I haven't seen this kind of abuse of statistics since the last major political debate.
The solution isn't to abuse statistics in the opposite direction.
If you flip a coin, the odds of getting heads are 50%. If you flip it again, the odds are still 50%. The coin does not remember what your last flip was.
If you flip eight times and get eight heads in a row, it's not because it's a trick coin; it's because the odds of getting heads are 50% each time, and that's what happened.
Specifically, you're rejecting the very notion of statistical inference, and the empirical method itself.
The problem, here, is that you've started with the premise that "the coin is fair" is absolute, unquestionable fact. So, obviously, seeing 8 heads in a row has no affect on our confidence that the coin is fair.
But in reality, we don't have an absolute guarantee that any particular coin is fair. If I pull a coin out of my pocket, I'm merely
virtually certain that it's fair. If I flip it 8 times and see 8 heads, that counts as evidence against the hypothesis it's fair -- but the evidence is relatively weak, and so it doesn't shake my confidence.
If I got 50 heads in a row, though, I would be a fool to remain certain that the coin flip was fair.
(aside: there are reasons why the coin flip could be unfair other than the coin itself being unfair)
However, there is a huge selection bias effect which, while I have no idea how to quantify, I imagine to be immense, rendering any one-off test like this one to be fairly unreliable.
Mylene had winning odds in nearly all of those battles. She won most of them. She only won against bad odds, what, three times? That's not abnormal, I don't think.
You see, this is what probability and statistics is
meant to do -- so that we don't have to "think" or "guess", but instead get cold hard facts.
This is something that is easily quantified -- assuming a random sample according to the given the sequence of percentages, the odds that the number Serial computed would have magnitude of 800 or more is strictly worse than 1 in 7.8. A little unlikely, but only a little. I strongly expect a full analysis could raise that to a double digit number, or maybe even low triple digits.
(which, incidentally, I do not find significant due to selection bias)
The problem with statistics is not that they're unreliable or anything like that -- the problem is that people just don't know how to reason with them. The test Serial came up with is a good one...
To look at all that and imply that something was off--and then to throw out a number that means nothing from a statistics standpoint (800% gained!)--is nuts.
... but as you rightly point out, the result of the test was originally reasoned with badly.
