UK court rules against use of Bayes' Theorem

Leoreth

Bofurin
Retired Moderator
Joined
Aug 23, 2009
Messages
37,545
Location
風鈴高等学校
The Guardian said:
It's not often that the quiet world of mathematics is rocked by a murder case. But last summer saw a trial that sent academics into a tailspin, and has since swollen into a fevered clash between science and the law.

At its heart, this is a story about chance. And it begins with a convicted killer, "T", who took his case to the court of appeal in 2010. Among the evidence against him was a shoeprint from a pair of Nike trainers, which seemed to match a pair found at his home. While appeals often unmask shaky evidence, this was different. This time, a mathematical formula was thrown out of court. The footwear expert made what the judge believed were poor calculations about the likelihood of the match, compounded by a bad explanation of how he reached his opinion. The conviction was quashed.

But more importantly, as far as mathematicians are concerned, the judge also ruled against using similar statistical analysis in the courts in future. It's not the first time that judges have shown hostility to using formulae. But the real worry, say forensic experts, is that the ruling could lead to miscarriages of justice.

[...]

Specifically, he means a statistical tool called Bayes' theorem. Invented by an 18th-century English mathematician, Thomas Bayes, this calculates the odds of one event happening given the odds of other related events. Some mathematicians refer to it simply as logical thinking, because Bayesian reasoning is something we do naturally. If a husband tells his wife he didn't eat the leftover cake in the fridge, but she spots chocolate on his face, her estimate of his guilt goes up. But when lots of factors are involved, a Bayesian calculation is a more precise way for forensic scientists to measure the shift in guilt or innocence.

[...]

The data needed to run these kinds of calculations, though, isn't always available. And this is where the expert in this case came under fire. The judge complained that he couldn't say exactly how many of one particular type of Nike trainer there are in the country. National sales figures for sports shoes are just rough estimates.

And so he decided that Bayes' theorem shouldn't again be used unless the underlying statistics are "firm".
The decision could affect drug traces and fibre-matching from clothes, as well as footwear evidence, although not DNA.
Source, emphasis mine.

I know that the title simplifies a little, because it's technically not Bayes' Theorem as a whole that's rejected, but the data used to calculate probabilities with it. Given that statisticians should be trusted to use reliable estimates, though, it's a typical case of the judiciary system not understanding science and especially math.

(By the way, if you're not acquainted to Bayes' Theorem, why it's important and how it helps overcome wrong intuitive conclusions, take a look at Simpson's Paradox)

Thoughts?
 
Guilt or innocence are not determined by the statistical probability of guilt. They are based on whether or not the evidence supports a verdict of guilty or not. To do otherwise opens the door to all sorts of miscarriages of justice.
 
LOL Bayes Theorem (rule) is undergraduate statistics!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Failing that is like failing rules of algebra.



If it was about an individual Bayesian model that was developed for the case and was poorly explained during the proceedings, then it really is the statistician's fault and not the concept of using statistics in court. Developing Bayesian models is specific practice. They are custom made for specific problems.


I don't see a problem in using statistics regarding specific evidence to make an argument about guilt, as long as it is only used to support the evidence. I don't think statistics was used irresponsibly here, i.e. they weren't trying to say that men of certain of race wearing certain sneakers are typically found guilty of crimes, therefore the man in this case is statistically guilty.


@Cutlass, statistical probability does play a role in validating court evidence. Statistics certifies that finger-prints are viable court evidence. I.e. that finger-prints are uniquely identifying of persons.
 
I'm a math major with a good understanding of Bayesian statistics and I agree wholeheartedly with this verdict. Just because the theorem works with perfect inputs doesn't mean it's much use if our data is bad. And people using questionable data in statistics is so common it is really a miscarriage of science/math.
 
And what NC said: if the data was bad, then it will corrupt the application of its use. If the statistician did an incomplete job of validating the data before using it, then that is where the problem lies.
 
@Cutlass, statistical probability does play a role in validating court evidence. Statistics certifies that finger-prints are viable court evidence. I.e. that finger-prints are uniquely identifying of persons.


So it can in some ways be a tool. But to what extent? What is your level of proof that a "statistical certainty" is an actual certainty? You may be able to use it in a number of cases. But unless it is certified in some specific instances, there is a lot that can go wrong. As nc-1701 said, your accuracy of the statistics is only as good as the accuracy of the models and variables used in making it.

Recall that Wall St used the best statistics money could buy to predict that what happened in 2007 could not happen. But their models were wrong.

I can see models used for some things in a court of law successfully and accurately. But I can also see a lot of room for both incompetence and fraud.
 
ummm good?
 
Well as long as people don't think Bayes' Theorem is not correct ;)

Applying Bayes' Theorem when you don't know real probabilities is suspect though (see: Economics).
 
I think this is wrong. Simply obscuring the fact that we are often dealing with uncertain events and not allowing to explicitily name assumptions as such (and trying to quantify them) cannot be a step in the right direction.
 
The article doesn't explain where these 'rough estimates' for national sports shoe sales come from or how they are estimated. If they're just back-of-the-envelope calculations the statistician did, then fair enough, if he can't explain his assumption clearly then the judge is correct to call foul. But it seems strange to have a completely worthless estimate knocking around. Bayes' Rule is quite robust regarding dodgy priors IIRC.

In any case, you can't ban Bayes' Rule, that's meaningless, and that doesn't appear to be what is being done (although it is what the article wants you to believe).

Recall that Wall St used the best statistics money could buy to predict that what happened in 2007 could not happen. But their models were wrong.

Not really sure what you mean here TBH, or why it's relevant :dunno:
 
Bayes' Theorem is a proven foundation of probability theory. Numerous proofs on the web (it's easy to prove).

Of course it assumes the probabilities are a priori known... not estimated via statistics (which is applied probability).
 
I agree wholeheartedly with this verdict. Just because the theorem works with perfect inputs doesn't mean it's much use if our data is bad.

Applying Bayes' Theorem when you don't know real probabilities is suspect though (see: Economics).

This, and this, I agree.

This reminds story me an awful, awful lot of all the stories we're hearing about overturned arson verdicts in the US (including death penalty level cases) where the original cases were real travesties of justice. When the underlying evidence and data is wrong (scientifically incorrect assumptions about how things burn, glass cracks, etc...) it doesn't matter if you then did math correctly to come out with a probability of arson. GIGO

Given the details we have available here, I would be very suspect of a the expert claiming it's the statistical model that proves it was this guy and his shoe. Without more details I couldn't disagree with the verdict, though of course there shouldn't be such statements like statistical evidence can't be used in the future.
 
Not really sure what you mean here TBH, or why it's relevant :dunno:



The point is that you can build any statistical model that you want. But if you make mistakes in building them, then you "prove" things that turn out not to be true.
 
Bayes' Theorem's legitimacy as a theorem has no relevance on its use in a court of law. A person must never be convicted based on statistical probability. That's just a horrible road to go down. I cannot believe a modern country like the UK would allow such a thing in a criminal trial where a man's freedom hangs in the balance.

P.S. - Fingerprints are not statistical probabilities. They are flat out unique. Until they are shown to not be unique, it is fine to use them.
 
This isn't much of a surprise to me. American courts are no different from British in that they are not required to use science or math to arrive at verdicts. They merely operate according to their own set of principles and do not make any effort to conceal this. For example, the use of the insanity defense has no bearing on actual scientific understanding of mental illness, nor on psychiatric knowledge.
 
We already imprison people based on probabilities, we just don't usually quantify said probability. As a judge you cannot never really rule out that a fingerprint match was erroneously found, it happened before. You can also never really rule out that the police officer in charge of the investigation tampered a little with the evidence because he's personally convinced the suspect is guilty. Very unlikely? Yes. But not impossible and it has already happened.
 
Another instance is that a certain percentage of eye-witness reports are flat out wrong. Wouldn't it be usefull to use the information of how often eye-witnesses err in general to acess a specific case?
 
Bayes' Theorem's legitimacy as a theorem has no relevance on its use in a court of law. A person must never be convicted based on statistical probability. That's just a horrible road to go down. I cannot believe a modern country like the UK would allow such a thing in a criminal trial where a man's freedom hangs in the balance.

Almost every trial can be summed up as "is the probability that person A did it great enough that we can say they did it". It's almost impossible to say with complete certainty that a specific person committed a crime.
 
Top Bottom