Polls - Some research results on accuracy

[Stalker0]

So a key question around the polling has been "how accurate are they?" I did some research to try and found out.

So the first thing we have to take in mind is that polls on the forum is not representative of all people who play VP. Its more accurate to say its "people that are involved with VP's forum".

So then the next big question is "how big is the population of people involved on the VP forum?"

That number is probably pretty small, maybe 100? (could use @Gazebo thoughts on this). This effectively means we are making mod changes based on the feedback of 100 people.


So if we assume 100 people, how accurate are our polls? (with 95% confidence levels). Lets take a look.

For 20 responses, its +/- ~20% (70/30)
For 60 responses, its +/- 8% (58/42)

This means that for one of our lighter polls (in which we get about 20 responses), if we had a yes/no poll, you would need the response to be more than 70/30 to have any actual meaning (which is showing in the parenthesis above). For one of our bigger polls (we do see around 60 responses).

The million dollar question remains: Is 100 the right population number? Based on the polls alone we know the community minimum of ~60 people, because we have gotten those responses. So at least 100 seems reasonable enough. But here are a few other population numbers for context.

Population: 80 people
20 responses, its +/- 19% (69/31)
60 responses, +/- ~7% (57/43)

Population: 150 people
20 responses, +/-21% (71/29)
60 responses, +/- 10% (60/40)

Population: 300 people
20 responses, +/- 22% (72/27)
60 responses, +/- 12% (62/38)

Population: 500 people
20 responses, +/- 22% (72/27)
60 responses, +/- 12% (62/38)


So you can see that from 150-500 the confidence level doesn't change a whole lot.

So what that suggests is, any yes/no poll with 60 responses and a result of 63% (either way) could be considered "definitive". And a 20 response poll with a result of 73% (either way) could be the same.

This of course assumes the poll is reasonable and unbiased, and again we can't say these results apply to everyone playing the mod, just really those who are active on the forum.

Now polls with multiple responses are much trickier and I'll have to dig harder to get an understanding of those.

 

[Gazebo]

Ultimately, IMO, activeness on the forums should be rewarded with greater impact on the project. Otherwise, why be active? I've waffled on this in the past but, given the age of the project and veterancy of many of our most active members, we'd be fools to ignore polls even if their turnout is on the edge of 'questionable.'

That said, change for the sake of change is often tempting in an iterative project like this. So we need to temper a desire for 'novelty' with a cold, heavy-hand of 'practicality and time-impact.'

G

 

[azum4roll]

Ideally the voters should be familiar with the game mechanics too (misunderstandings rig the poll), but we can't be sure of that.

 

[Stalker0]

So in doing more research, for single choice polls, the number of responses just means the margin of errors have to be looked at more closely, but the same general numbers apply. For example, a poll with 3 answers, using our original 100 population numbers.

Again for reference: For 60 responses, its +/- 8%

We have to apply that margin to every answer, so in order to get a truly definitive answer, we would need at minimum 44/28/28 response to ensure that one of the answers is definitively favored over the others.


Now polls with actually multiple answers allowed (aka a multi-choice vs single-choice poll) gets much more complicated. So its much cleaner to rely on single choice polls statistically speaking.

 

[LarryAR]

Thanks for the analysis.
How can we enable the results of the Poll of Polls?

That is to say, some control of poll frequency helps prevent “poll fatigue.”
Poll fatigue results in reduced participation.
Reduced participation=reduced accuracy