How Statistics Can Foul the Meaning of DNA Evidence 215
azoblue writes with a piece in New Scientist that might make you rethink the concept of "statistical certainty." As the article puts it, "even when analysts agree that someone could be a match for a piece of DNA evidence, the statistical weight assigned to that match can vary enormously, even by orders of magnitude." Azoblue writes: "For instance, in one man's trial the DNA evidence statistic ranged from 1/95,000 to 1/13, depending on the different weighing methods used by the defense and the prosecution."
DNA is politically charged (Score:5, Informative)
Genetics means "out of your control" and touches on some raw nerve issues, so there's a lot of throwing around of "statistical" information and unrealistic mental models.
For example of statistical confusion:
New research shows that at least 10 percent of genes in the human population can vary in the number of copies of DNA sequences they contain--a finding that alters current thinking that the DNA of any two humans is 99.9 percent similar in content and identity.
http://www.hhmi.org/news/scherer20061123.html [hhmi.org]
And broken mental models:
http://en.wikipedia.org/wiki/Lewontin's_Fallacy [wikipedia.org]
Until our knowledge improves, you're going to see more "politicization" of DNA-related science.
It's fine for saying "it's somebody else". (Score:5, Informative)
This sounds like a good reason to stop releasing all of those convicted murderers and rapists who were freed on DNA evidence.
Not at all.
There is no problem determining that the DNA is from somebody else than the accused. All it takes is a single marker that's different. That's easy.
The problem is going from some bunch of markers that match to saying "This IS the bum! (Well, except for a one-in-[some number] chance it really isn't.) That requires a lot of information about prevalence of genetic markers, whether there is a correlation between their distribution. That information isn't well researched and the different estimates are based on different wild guesses by different experts. Further, the whole independent-probability thing gets knocked into a cocked hat with FAR lower numbers if the police found the accused by searching a DNA database for matches. And what if he had an evil identical twin? Or somebody with access to PCR gene-amplification materials, a DNA sample, and an atomizer decided to frame him?
IMHO DNA evidence is decisive for the defense. But pending a lot more research it's still voodoo for the prosecution.
Re:1/13 (Score:4, Informative)
No, we did that with our copies of sales receipts.
You used them already? Hmm...well this is embarrassing.
Re:Here’s a tip: Go fuck yourself. (Score:3, Informative)
That procedure is called charge inflation and plea bargaining. It's done all the time.
Re:Damn Lies and Statistics! (Score:5, Informative)
Sadly true, but there's so much about DNA analysis that you don't get on an episode of CSI. On TV DNA analysis only takes a few minutes and matches are proudly announced by flashing messages on the DNA machine.
In real life good DNA matching takes days, cost a lot of money and, as the article points out, matching can be in the eye of the beholder. DNA samples are incredibly easy to contaminate, whole labs can become contaminated over time if they don't have and follow strict contamination protocols. And there has been more than one reported case of harried techs gun-decking DNA analysis when police and prosecutors were certain they had the right guy.
Well done DNA analysis can be an amazing crime fighting tool but the science is not perfect and it's okay to be skeptical. There is no magic identification test that's completely fool proof. And DNA tests are only as good as the fool running the test.
This happened to me ... (Score:5, Informative)
... in a case I was on the jury for. (Sorry for the bait-and-switch title, couldn't resist.)
This was a case of armed home invasion. The victim was a big bruiser of a man, a multiple convicted drug addict. The defendant was a scrawny young Cape Verdean guy. (Cape Verdean drug gangs are common in the area: this is important later.) The victim testified that, after buying drugs from the defendant, he got a series of enraged voicemails demanding the return of the defendant's cell phone. A few hours later, the defendant allegedly shows up at the victim's house with a gun and barges in yelling. A struggle ensued, a shot was fired into the floor, and the guy with the gun fled.
Evidence against the defendant included eyewitness testimony from the defendant, matching ammunition found at the defendant's house, and crucially a do-rag found at the scene of the scuffle. DNA tests matched the do-rag to a mixture of at least 3 people, including the defendant. The DNA mixing was probably due to really awful police work: a paper bag borrowed from the defendant's cupboard is not a proper evidence collection container.
As in TFA, mixed DNA dramatically affected the "probability of exclusion" statistics: the state's expert testified there was a 1 in 50 chance that a random man on the street would match the DNA on the do-rag. The odds that a random *black* man on the street would match were much higher, like 1 in 20; the defense pointed out that the odds that a random *Cape Verdean* would match would be much higher.
We've grown used DNA evidence saying things like, "not one other person on the planet could match this DNA", but in this case, the odds were good that the DNA evidence would match at least one other person sitting in the *courtroom*. The defense also took the unusual tactic of introducing the defendant's sister, who testified that her *other* brother looks very much like the defendant, and she said it was *his* voice on the enraged voicemails. What are the odds that the DNA matches the *brother* instead? Damned good.
Between the fact that the eye witness seemed shifty and unreliable and was probably on crack at the time of the incident, and the fact that all the physical evidence could just as well implicate the brother as the defendant, we couldn't rule out the possibility that the cops got the wrong guy, so we found him not guilty. If I had to take a bet, I'd say he did it, but I wouldn't bet his life on it.
Anyway. Moral of the story is: on cop shows and in the public awareness, DNA evidence is rock solid and incontrovertible. But in the real world, the statistics of DNA mixtures make things a whole lot less cut-and-dried.
Re:But each time you measure it you reduce the (Score:3, Informative)
Temperature effectsm, the same properties of the nucleotides (even if from distinct DNA), the same crime scene contamination effects, the same statistical laws (if it turns out not to be as unique as we think, all tests necessarily fail) and at least some of the chemical reactions are in common.
In one documented case, a surprise common factor was a quality control person who didn't know the non-sterile swabs still couldn't be touched before packaging. (The CSI:NY episode borrowed that from real life).
Wall street fits in because they too believed that a series of high risk propositions (this test is accurate or this sub-prime loan won't default) could be somehow bundled together to make a low risk proposition (AAA bonds in the Wall street case, beyond reasonable doubt in the DNA case). In both cases, the unexamined dependencies are the downfall.
Re:People don't understand statistics (Score:2, Informative)
Your first example is known as the prosecutor's fallacy, and your second example is known as the defense attorney's fallacy. It's distressing that people get convicted on statistical evidence when the evidence really doesn't support it. For example, in Britain, someone named Sally Clark was convicted of killing her children after two apparent SIDS cases. The prosecution had someone testify that the chance of two children in the same family dying of SIDS was 1 in 73 million (erroneously assuming that the death of one child from SIDS is uncorrelated with siblings dying of SIDS), but even granting that the figure is accurate, you would expect there to occasionally be families with two SIDS deaths. She was convicted purely on the statistics. (Her conviction was overturned a few years later, though.)
Re:Damn Lies and Statistics! (Score:5, Informative)
Well, then, if you expect your opponent to pull something like that, bring in a statistician, qualify him as an expert witness and let him rip the assertion to shreds.
That doesn't always work so well. Read about John Puckett sometime...
Rather than try to sort out the disparities between its numbers and database findings, the FBI has fought to keep this information under wraps. After Barlow subpoenaed the Arizona database searches, the agency sent the state's Department of Public Safety a cease-and-desist letter. Eventually, the Arizona attorney general obtained a court order to block Barlow's distribution of the findings. In other instances, the FBI has threatened to revoke access to the bureau's master DNA database if states make the contents of their systems available to defense teams or academics. Agency officials argue they have done so because granting access would violate the privacy of the offenders (although researchers generally request anonymous DNA profiles with no names attached) and tie up the FBI's computers, impeding investigations. These justifications baffle researchers.
Source: DNA's Dirty Little Secret [washingtonmonthly.com]
Re:Lies, Damn Lies, and Statistics! (Score:5, Informative)
And most people don't understand statistics, however good they are, and draw wrong conclusions. Case in point: the author of TFA doesn't seem to have a clue what a likelihood ratio (LR) is. In the article it comes across as a type of comparison - contrasted with RMP (random match probability) and RMNE (random man not excluded), which are different tests to apply to the data. But actually LR is a way of presenting a probability which is used by forensic scientists because it's supposed to be easier for juries to understand - so you could present an RMP result as an LR, or an RMNE result as an LR.
FWIW, I'm not defending any of the statistics in TFA as good. I notice a complete absence of any error estimates. And I distrust forensic match probabilities in general because I've seen forensic fingerprint analysis software which uses pseudorandom numbers in the computation of the match probability and can vary the LRs presented (to 16s.f., believe it or not) by an order of magnitude if you recalculate.
Re:It's fine for saying "it's somebody else". (Score:3, Informative)
Your example is not singular:
In Colorado we have a prosecutor explaining away a DNA exclusion [reason.com] in the case of molestation of an 8 year old because "Depending on how long she had been wearing those panties and where, they could have rubbed up against the back of her chair at school, a restaurant, the couch at home that someone else had been sitting on, a bus seat, someone's toilet seat if she did not pull them down far enough — there are many ways to get unknown DNA on clothing. "
Still thinking this is an isolated incident? Don't believe there could be more than one prosecutor out there who would believe that an 8 year old got semen on her privates accidentally? Here's an Illinois prosecutor [reason.com] who refuses to believe a DNA exoneration. He actually claims that the semen that was found in the mouth, vagina and rectum of the 8 year old murder victim "must have found its way into the girl’s body while she was playing in a patch of woods where teenagers were known to have sex."
Prosecutors are immune from the real consequences [reason.com] of their fuckups, so you can't really expect them to work to overcome their natural resistance to being found wrong.