http://www.law.yale.edu/intruders/5493.htm
A pretty long article describing the rising success of relatively statistical models compared to expert opinions. It mostly deals with legal matters, with a bit at the end about wine-tasting.
Some excerpts:
The article goes on to discuss the release of sex offenders as an example of increasing reliance on computer models. The issues are obvious: On the one hand, it does seem to be generally more effective to put the machine in charge; on the other hand, the machine's decision-making process isn't always politically correct (e.g. the recidivism calculus penalises any sex offender with at least one male victim); and with no human discretion at all, it's possible that somebody could be penalised for innocuous behaviour that for some reason flags one of the machine's factors.
I think the first case cited, the Supreme Court competition, is the most striking thing about the article, but it's all interesting. How accurate should a machine be before we make it an ultimate arbiter over human freedom? Should human discretion be removed from decision-making, if it were shown that trust in a statistical model yielded superior overall results?
A pretty long article describing the rising success of relatively statistical models compared to expert opinions. It mostly deals with legal matters, with a bit at the end about wine-tasting.
Some excerpts:
Six years ago, Ted Ruger, a law professor at the University of Pennsylvania, attended a seminar at which two political scientists, Andrew Martin and Kevin Quinn, made a bold claim. They said that by using just a few variables concerning the politics of the case, they could predict how the US Supreme Court justices would vote.
Analysing historical data from 628 cases previously decided by the nine Supreme Court justices at the time, and taking into account six factors, including the circuit court of origin and the ideological direction of that lower courts ruling, Martin and Quinn developed simple flowcharts that best predicted the votes of the individual justices. For example, they predicted that if a lower court decision was considered liberal, Justice Sandra Day OConnor would vote to reverse it. If the decision was deemed conservative, on the other hand, and came from the 2nd, 3rd or Washington DC circuit courts or the Federal circuit, she would vote to affirm.
Ruger wasnt buying it. As he sat in that seminar room, he didnt like the way these political scientists were describing their results. They actually used the nomenclature of prediction, he told me. [But] like a lot of legal or political science research, it was retrospective in nature.
After the seminar he went up to them with a suggestion: why didnt they run the test forward? As the men talked, they decided to run a horse race, to create a friendly interdisciplinary competition to compare the accuracy of two different ways to predict the outcome of Supreme Court cases. In one corner stood the predictions of the political scientists and their flow charts, and in the other, the opinions of 83 legal experts esteemed law professors, practitioners and pundits who would be called upon to predict the justices votes for cases in their areas of expertise. The assignment was to predict in advance the votes of the individual justices for every case that was argued in the Supreme Courts 2002 term.
And, interestingly:The experts lost. For every argued case during the 2002 term, the model predicted 75 per cent of the courts affirm/reverse results correctly, while the legal experts collectively got only 59.1 per cent right. The computer was particularly effective at predicting the crucial swing votes of Justices OConnor and Anthony Kennedy. The model predicted OConnors vote correctly 70 per cent of the time while the experts success rate was only 61 per cent.
Several studies have shown that the most accurate way to exploit traditional expertise is merely to add the expert evaluation as an additional factor in the statistical algorithm. Rugers Supreme Court study, for example, suggested that a computer that had access to human predictions would rely on the experts to determine the votes of the more liberal members of the court (Stephen Breyer, Ruth Bader Ginsburg, David Souter and John Paul Stevens, in this case) because the unaided experts outperformed the super-crunching algorithm in predicting the votes of these justices.
The article goes on to discuss the release of sex offenders as an example of increasing reliance on computer models. The issues are obvious: On the one hand, it does seem to be generally more effective to put the machine in charge; on the other hand, the machine's decision-making process isn't always politically correct (e.g. the recidivism calculus penalises any sex offender with at least one male victim); and with no human discretion at all, it's possible that somebody could be penalised for innocuous behaviour that for some reason flags one of the machine's factors.
I think the first case cited, the Supreme Court competition, is the most striking thing about the article, but it's all interesting. How accurate should a machine be before we make it an ultimate arbiter over human freedom? Should human discretion be removed from decision-making, if it were shown that trust in a statistical model yielded superior overall results?