CSI on Trial

by on May 15, 2008 at 9:45 am in Law | Permalink

…to judge by the most
comprehensive study on the reliability of forensic evidence to date,
the error rate is more than 10% in five categories of analysis,
including fiber, paint and body fluids. …DNA
and fingerprints are more reliable but still not foolproof….a 2005 study in the  Journal of Criminal Law & Criminology
suggests a fingerprint false-positive rate a bit below 1%, a widely
read 2006 experiment shows an alarming 4% false-positive rate.

How can we preserve the
usefulness of forensic evidence while protecting the public when it
breaks down? The core problem with the forensic system is monopoly.
Once evidence goes to one lab, it is rarely examined by any other. That
needs to change.
Each jurisdiction should
include several competing labs. …

This procedure may seem like a waste. But such checks would save
taxpayer money. Extra tests are inexpensive compared to the cost of
error, including the cost of incarcerating the wrongfully convicted….

Other reforms should include
making labs independent of law enforcement and a requirement for blind
testing. When crime labs are part of the police department, some
forensic experts make mistakes out of an unconscious desire to help
their "clients," the police and prosecution. Independence and blind
testing prevent that.

That’s forensics expert Roger Koppl writing in Forbes.  If anything I think Koppl is being kind to CSI.  Take bullet lead analysis a procedure used by the FBI for decades that turns out to have no scientific validity whatsoever.

Full Disclosure: Koppl’s op-ed is based on a paper in a book called Law Without Romance edited by Ed Lopez to be published by Independent Institute where I am director of research.

Craig May 15, 2008 at 9:59 am

Competing labs would help, but that may solve only a small part of the problem. Any test, even if correctly done, will always have a non-zero false positive rate. This is unavoidable. The other source of error is, well, error itself. Competition can help solve the unforced error problem, but not the underlying false positive rate.

Mercutio.Mont May 15, 2008 at 11:54 am

It makes me sad to say this, but I really don’t believe that Law enforcement, including prosecutors, really gives a darn about convicting innocent people. Stories like the FBI’s bullet lead “evidence” and the “bite marks” expert below are so disappointing.

http://www.reason.com/news/show/125151.html

andres May 15, 2008 at 2:16 pm

@ andrew

that is how it works. the prosecution needs to introduce evidence about the procedure, but the defense can always question the prosecution’s witness and nail him on the procedure’s unreliability or even bring its own witness in. the problem is that 99% reliability, though not perfect, is pretty damn close in a jury’s eyes. if i told you the democrats have a 99% chance of winning the election this fall, do you have a reasonable doubt that it will happen?? the prosecutors ought to be required to confirm the tests on their own before bringing the case at all. and no, defendants can’t afford to run these tests themselves.

Andrew May 15, 2008 at 3:19 pm

“if i told you the democrats have a 99% chance of winning the election this fall, do you have a reasonable doubt that it will happen?”

Yes, because I wouldn’t believe you. I’d also be allowed to do my own research. Ask for other expert opinions, and not just take your word as you presented it, regardless of how good my defense attorney is.

andres May 15, 2008 at 5:03 pm

@andrew

you missed my point. the article says, “a 2005 study in the Journal of Criminal Law & Criminology suggests a fingerprint false-positive rate a bit below 1%.” my question is whether a 99% chance of something being true leaves you with a reasonable doubt that it is, in fact, true. we can go down the list to the other methods and other percentages, and somewhere along the line, it’ll break down–there will be enough of a likelihood that the test gave a false-positive to sustain a reasonable doubt. i’m thinking a false positive rate of over 10% is definitely there. under 5% is probably not. in between, i’m not sure.

my point is only that jurors are frequently presented with these error rates, which you think is some sort of novel idea–“How ’bout you present the jury with all the facts.” the problem is not that they aren’t given the facts, but that the facts themselves support the legal conclusion of guilt, even if the moral implications of a 1% false positive rate are alarming, to say the least.

as i understand it, you don’t think you should believe it when you are told that something has a 99% accuracy rate. your skepticism is something of which you are obviously proud and i’m sure it serves you well. but are you seriously suggesting that jurors go off on their own, in the midst of trials, to determine the accuracy of the tests being used by a party?? that’s what the adversarial system is for. the party’s opponent will challenge it, and if they fail to persuade the jury, they lose. you don’t honestly believe for a second that having jurors conduct independent evaluations of evidence or evidentiary methods is a good idea, whatever the merits of skepticism and independent evaluation in our daily lives. don’t pretend otherwise.

Ned May 15, 2008 at 9:07 pm

With regard to bullet lead analysis (actually trace element analysis), the definitive experiment seem easy to design. Buy a couple of hundred boxes of ammunition. Take some bullets from each and analyze them for whatever trace elements you desire. And then see if a blinded observer can match them up as to box of origin. My guess is that it won’t be possible, but it seems that the FBI (or anyone else) has never done the experiment. That said, you have to understand that someone “from the FBI Laboratory” has a tremendous impact on a jury, which will be composed mostly of poorly educated, low IQ, scientifically illiterate individuals, and that such an expert’s testimony is likely to be entirely believed without question. For an enjoyable experience as to how a pompous expert witness from a government laboratory spewing bogus testimony can be blown out of the water by a really sharp defense expert, rent a DVD of “My Cousin Vinnie.”

Mike Fladlien May 15, 2008 at 10:30 pm

The jury instructions tell the jury that “all evidence is evidence including what you believe.” This evidence includes circumstantial, real, and documentary evidence. The forensics evidence has probative value and should be allowed in court. I agree competition would add to the accuracy of the evidence.

Bob Murphy May 16, 2008 at 10:08 am

I don’t get it – if you believe “well this test has an error rate of 1% then it’s worthless and is inadmissible as evidence” – how is any one going to get a conviction?

Well, part of the answer is that you focused on the best stat of all the ones Koppl mentioned. To repeat:

…to judge by the most comprehensive study on the reliability of forensic evidence to date, the error rate is more than 10% in five categories of analysis, including fiber, paint and body fluids. …DNA and fingerprints are more reliable but still not foolproof….a 2005 study in the Journal of Criminal Law & Criminology suggests a fingerprint false-positive rate a bit below 1%, a widely read 2006 experiment shows an alarming 4% false-positive rate.

And so I think Koppl’s point here is that when someone might go to jail for the rest of his life, steps to increase the reliability ought to be taken. If airplanes had a 1% failure rate, that would be pretty serious.

Andrew May 16, 2008 at 10:34 am

“I don’t get it – if you believe “well this test has an error rate of 1% then it’s worthless and is inadmissible as evidence” – how is any one going to get a conviction?”

There used to be this little thing called witnesses. Tests can support real evidence.

It’s also important how statistics are worded. I’ll try to find the example I read one time about DNA evidence, but it goes something like this. The probability that this DNA is someone else’s is X, but the probability that this DNA is the defendant’s is Y. X and Y are not the same.

And, I don’t think anyone here is arguing for throwing stuff away that has value. I basically said that crappy tests should be used for investigation, and less crappy tests should be laid bare for the jury to decide, and I’ll add maybe the whole system needs an overhaul because it doesn’t seem like our country’s problem is NOT convicting enough people. The lawyers seem to disagree, which goes along with one of my tenets in life, never complain to someone for whom the system is working.

We also spend an @$$load on trials. We could probably spend a bit less on trials, improve tests, convict a few less innocent people, and come out way ahead.

Why would competing labs work? Oh, I don’t know, maybe it’s just a theory based on the idea that competition helps in almost everything else. Start with the theory that decoupling the interests in investigation and analysis and prosecution will improve objectivity. Add some adversarial competition between labs to keep them on their toes. Then you go from there and do some experiments, then do some trials with a few. Send tests to both labs and see how often they agree for a while. Have them cross-check. Or, maybe you are right, there’s nothing to this idea.

Person May 16, 2008 at 11:10 am

“I don’t get it – if you believe “well this test has an error rate of 1% then it’s worthless and is inadmissible as evidence” – how is any one going to get a conviction?”

There used to be this little thing called witnesses.

*epic facepalm*

Witnesses are even *less* reliable than the physical evidence just criticized, genius!

***

I want to look at the criminal justice system as a “guilt-finding device”. Based on all the handwringing, it has a “false positive” (wrongful conviction) rate of 1%, or 10%, or something like that, and is characterized as “too high”.

Someone forgot to ask: what’s the false *negative* rate? What’s the fraction of crimes committed (and let’s confine it to real crimes, not paper crimes) where the perp isn’t convicted. I’m not just talking about not-guilty verdicts, but also, cases where:

-the crime isn’t reported or discovered
-the police don’t pursue the case, or the at least the right perp
-the prosecutor declines to go through
-the charges are dismissed by a judge

I’d put the false negative rate at over 99% across real crimes, ~10% for homicide.

While I think the alternatives proposed can decrease both the false negative and false positive rates, and improve the welfare of victims, I don’t think it’s meaningful, within a system, to complain about the false positive rate without also comparing it to the false negative rate. (Remember, “It’s better for 1000 guilty to go free than one innocent get convicted” is a slogan, not an actual logical justification.)

Imposing ever-stricter conviction standards and safeguards will decrease false positives, but increase false negatives.

IIRC, the Middle Ages had obscenely high standards for guilt in homicide (one witness to the act or a confession were required), and, stuck between a rock and a hard place (can’t convict, can’t let murderers roam), law enforcement felt no inhibitions against underhanded tactics to get convictions, like beating confessions out of people.

Another metric more important than false positive rate, would be: how difficult it is for any given person to reduce his own chances of being falsely convicted.

Andrew May 16, 2008 at 2:40 pm

Oh, okay, then we throw out witnesses too. You guys are a trip.

Suggest ways to improve a system, and to you I must be saying toss it all out. When it is stated that convictions would be impossible if tests were improved or scuttled (e.g. bullet analysis) I suggest that there actually were, believe it or not, convictions prior to all these new fangled scientific-like, and not-so-scientific forensics, and the experts to schill them at trial. One of these technologies was witnesses. Well, the ridicule spews forth. I suggest we improve one link in the chain, and the criticism is levied that you haven’t improved the entire chain, so it is all for nought. Fine guys. We all get the justice system you deserve.

No, erring on the side of non-conviction over false-conviction isn’t a slogan. It’s a tradeoff. And, it isn’t even a constraint. We could in fact improve both sides of the equation if people didn’t automatically jump to either fur ‘em or agin ‘em. But, given only those two choices, I err on the side of non-conviction.

Andrew May 16, 2008 at 3:36 pm

So, expert witnesses aren’t…witnesses? They have to testify to the results of the tests. We can also have tests for other witnesses. I’m not opposed to polygraphs. You’d still have to convince the jury they aren’t fool proof.

“Imposing ever-stricter conviction standards and safeguards will decrease false positives, but increase false negatives.”

Not necessarily true. It just increases the cost of positives. All we have to do is decide to pay the cost. We can reduce false positives and false negatives. Off the top of my head I can think of:
1) wait until you have sufficient evidence by expanding investigation
2) spend more on the process by freeing up resources from hard/expensive to convict victimless crimes to those with victims/witnesses
3) streamline trial process and obtain higher quality jurors, maybe assigning a devil’s advocate role
4) redirect resources currently wasted on negative return investments (bullet analysis)
5) give me half a day and I could probably come up with 100 ways the miracle of the web could improve the investigation and trial process. How much of the current process is based on information transaction costs? Send the fingerprint electronically to 100 labs in India.
6) finally, beat confessions out of people as you suggest

If you consider the conviction rate in terms of statistical process control and total quality management, then yes, all other things being equal, tightening the control limits at the end of the process results in more falling outside tolerance. But, all other things aren’t equal. You can tighten the variability by improving the process.

People say, “they already do this” and “they already do that” as if the legal system fell out the sky fully formed without need for improvement or rethinking as technology advances. Who owns the overall process? It seems like the falsely accused are the only ones with a solid and direct incentive to improve it. Not a good customer base.

Bob Murphy May 16, 2008 at 4:17 pm

I’d say at 50 innocent people, $1 million dollars each per year of false incarceration wouldn’t be an unreasonable restitution. You could improve a lot of legal processes for a billion bucks or so.

I personally don’t mind your estimate here, but I just want to point out that one of the advantages of a market economy is that individuals don’t need to decide on rules in this way; the price of things emerges spontaneously.

So that’s what I was getting at in my earlier post about a truly market-based system of law enforcement. (Tabarrok has written/edited a lot in this area; why doesn’t he jump in here?? :)) We’re all arguing here about how much falsely convicted people should be paid, how many independent tests should be conducted, what the right tradeoff is between false positives and false negatives, etc. But to me this is like Cubans arguing about how many loaves of bread a kidney transplant should cost, etc.

Again, it’s a bit tricky to even imagine how you could meaningfully have a “free market” in law enforcement, evidentiary procedures, etc., but if you could it would solve all of these problems. The ratio of false positives to false negatives would be a market outcome, just as the number of broken eggs to non-broken eggs in cartons in your grocery store.

Mike Huben May 17, 2008 at 5:07 am

Bob Murphy:

First, if you want to use the dichotomy monopolistic versus competitive, then you have to say that jury trial is monopolistic the same way that elections are monopolistic or markets are monopolistic. After all, markets are imposed by governments instead of competing with alternative arrangements such as socialism or mixed models.

And of course there is competition in a trial: that is the nature of an adversarial system. Good competitors are rewarded on both sides: prosecution and defense.

Nor is there a history of “better” methods of conviction for crimes in the sense of accuracy, though there has been an enormous amount of experimentation in the innumerable governments, national and local. Largely because there’s no good way to measure the accuracy rate: no better “truth” to compare to.

We know that juries can be biased. We know that witnesses have a far higher error rate than forensics labs. We know that defense and prosecution attempt innumerable invalid arguments attempting to sway or mislead juries. We know that there are “hanging judges”. We know that laws can be unjust and biased.

But there’s no evidence that “let the market sort them out” will work any better than “kill them all and let god sort them out”. Perhaps markets could fix this one source of evidence: but of course, there is the same moral hazard of corruption because of the large stakes. The difference is that hiding behind private rights, we’re less likely to observe the corruption than in a public process.

Micha Ghertner May 18, 2008 at 10:41 am

No, Gil, Those are not all entities of a central government system. Courts, judges, juries, and laws are all entities of private arbitration systems as well. And police and prisons would be potential entitles of such non-central government legal systems were they not forcibly banned from competing with the government monopoly.

Gil May 20, 2008 at 2:21 am

You got to be kidding M Ghertner. If two parties have a dispute they want to non-violently resolve then they find an arbitrator they can both abide by. They would then both state their case and how each is right and the other is wrong. The arbitrator presumedly is able to sort out the problem and show where the rightness and wrongness is and shows how the two parties can come to a truce and resolve the problem. On the other hand how is the rigmarole of the justice system not part of a centralised system? Where would private operators get their right to a law or legal code that they could use physical force to restrain or punish somebody? Why would a known violent offender submit to a private justice system? History has shown protecton from crime in a village has been the realm of the individual within the comman law agreement between other villagers as to what constitutes a crime and what a home owner can use as defence.

aion kina March 18, 2009 at 10:02 pm

Comments on this entry are closed.

Previous post:

Next post: