FBI Fights Testing For False DNA Matches

Statesman writes "The Los Angeles Times reports that an Arizona crime lab technician found two felons with remarkably similar genetic profiles, so similar that they would ordinarily be accepted in court as a match, but one felon was black and the other white. The FBI estimated the odds of unrelated people sharing those genetic markers to be as remote as 1 in 113 billion. Dozens of similar matches have been found, and these findings raise questions about the accuracy of the FBI's DNA statistics. Scientists and legal experts want to test the accuracy of official statistics using the nearly 6 million profiles in CODIS, the national system that includes most state and local databases. The FBI has tried to block distribution of the Arizona results and is blocking people from performing similar searches using CODIS. A legal fight is brewing over whether the nation's genetic databases ought to be opened to wider scrutiny. At stake is the credibility of the odds often cited in DNA cases, which can suggest an all but certain link between a suspect and a crime scene."

well, well...

What we're seeing here is a crack in the government's facade of fake-goodness. The ideas we're fed are that justice is blind, which means (we're taught), ultimately fair; that prosecutors and judges and the legal system in general have our best interests at heart, and so on, platitude after platitude...

But the truth peers 'round the corner here. They're not interested in accuracy, else they'd be all for determining how well this works, or not. The process and the results would both be open. What they're interested in are convictions, because that's how they keep score. That's how the public is keeping score.

This is unfair and irresponsible on two fronts. First, if you get the wrong person for any reason (including using DNA evidence that is supposed to be basically infallible, but is, in fact, fallible); then you've done a wrong to that person, of course. But secondly, for every false conviction the prosecutor and their accomplices notch into their pistols, the real criminal is now completely free -- the case is closed. They're not even looking.

As a society, we need to stop trying to raise up any part of the system based upon count of arrests, convictions, tickets, etc. The temptation to go for easy answers is too high; obviously, if the FBI itself is victim to this, an organization that prides itself on its organizational integrity, groups that have less tradition of trying to do right -- like the local cops who broke down your neighbors door last week -- are going to fall even more prey to such pressures.

As we see that the FBI tries to prevent the truth from coming out about a tool that is less effective than they claim, as they try to prevent exonerating information from reaching the defense, we see true colors.

These people are not our defenders; they claim to be, but they have their own agenda, and it is not about fairness. They're simply counting scalps.

Re:well, well...

but one felon was black and the other white

Well according to statistics it was clearly the black one
(note to mods, this is a joke)

Re:well, well...

Unless they were looking for a serial killer, who tend to be white males.

We're seeing no such thing.

What we're seeing is a consequence of basic math.

1/113 billion chance a particular person has the same DNA profile as me. 6 million records. So I have a 6 million / 113 billion chance of matching someone else in the database. Drop some zeros and thats 6/113,000. Of course, each of the 6 million people in the database has a the same chance of matching someone - so that's 6/113,000*6,000,000 - which means there should be 318 people who match someone else in CODIS, or 159 'matches'.

# of people matching = size of group * size of group * chance of match

Anytime you have something that has a small chance of matching, but a fairly large sampling group, your chances of matching are high, because your chances of a match existing within the group is the SQUARE of the group size.

So it would be surprising if there were NOT people who matched in CODIS. The question is, are there more or less that 318 of them, and how much more or less?

Re:We're seeing no such thing.

The other question is: since the database was started, our ability to study DNA, examine genetic markers and recognize which areas are better for identifying individuals has drastically improved. Has the database improved to match this knowledge, or are we relying on outmoded methods? If the former, then it's as good as it is going to get - for now. If the latter, the probability of a false positive could be massively slashed. That would surely be desirable for a crime-fighting system... wouldn't it?

Re:We're seeing no such thing.

So what you are saying is that a new sample has a 1 in 19,000 chance (6e6/1.13e11) of finding a match in CODIS at random, even though this is a new sample not present in the database. With thousands of police departments nationwide running samples against CODIS every week, false matches like this may occur frequently. If the consequence is that an innocent person is charged or even convicted of rape or murder, this is frightening.

There is a big difference between telling a lay jury "this match had a one in a 113 billion chance of occurring at random" versus "this is an event that occurs randomly on a routine basis." Non-statisticians have a hard time getting their head around the concept of correction for multiple hypothesis testing.

The real problem is that the FBI match criteria were developed years ago when the CODIS database was small compared to its current size, and these criteria have not been updated in light of growth in the database and new technology. Using state-of-the-art genotyping technology, it should be possible to design a test with a small chance of a false positive match even if the database contained the entire US population.

Re:We're seeing no such thing.

>>There is a big difference between telling a lay jury "this match had a one in a 113 billion chance of occurring at random" versus "this is an event that occurs randomly on a routine basis." Non-statisticians have a hard time getting their head around the concept of correction for multiple hypothesis testing.

To give an apocryphal quote by Mark Twain: "People use statistics the same way drunks use lampposts - for support, not illumination."

The lack of ability to reason statistically is extremely common in America. I mean extremely common - even in grad students publishing papers on stats, or in the technologically literate crowd. I'd used to write examples of egregiously bad stats in my livejournal in papers and news reports, but gave up because it was so common.

The DNA testing example is actually an example we studied in the Bayseian/conditional chapter of my stats textbook. It described an actual court case in LA where I got was convicted solely by DNA evidence (there was no other evidence to convict him, and he wasn't lucky enough to have an alibi) because the prosecutor confused the odds that (in this case) the odds of the match randomly matching being only one-in-a-million, and those are some pretty powerful odds. Of course, that would mean that in LA alone, there would be 6 people (on average) matching the DNA, and so the chance of the guy being guilty is actually only 1/6 or so.

The problem I have with the DNA "this has a one in 113 billion chance of matching" is that this is an extrapolated number based on certain premises of independence between the different loci. Whereas the more we learn about DNA, the more we learn that there is a high degree of covariability, certainly enough that (as the article shows), the odds of a match are actually much much higher.

Re:We're seeing no such thing.

It's probably even worse than the naive math indicates, because odds are not everybody involved will actually be unrelated in the mathematical sense anymore.

On the other hand... as others point out, in the birthday paradox there's a huge difference between two people have the same birthday and someone has the same birthday as me. However, in a prosecution situation, it is the latter that matters, not the former. Simply showing that the birthday paradox applies doesn't really prove that for a given criminal, that the odds are high that the match is a false positive.

This is also why government attempts (by well meaning people) to simple run dragnets over large datasets looking for "bad people" need to be resisted; while the birthday paradox is not always in play, similar statistical effects can be found in many other situations that cause enormous amounts of false positives. "Due process", where we investigate someone with all available tools only after we already have good reason to suspect them (to simplify, of course), turns out to be important not just for our rights in the abstract, but statistically sound, as well.

At the risk of potentially defending the government (pause for the shocked intake of breath), I find it quite plausible that the government knows all of this, and is resisting this investigation because they do not look forward to explaining this to every jury unto the end of time. While I support open disclosure and letting the chips fall where they may as a matter of principle, if you take a moment to look at this from their point of view, even if you dare take the step of assuming they're not actually out to get you personally (pause for another shocked breath), you might find it hard to avoid having a little sympathy for their position. (If you're still having a hard time, engage the "Most jury members sure are dumb, ha-ha!" cynicism circuit and consider the implications.)

Re:well, well...

I agree 100%. We see all the time stories about new evidence in death row cases. The first thing the prosecutor always argues is "he doesn't deserve a new trial, I stand by the conviction". If they truly cared about justice they'd say "hey we need to take another look at this". I don't know how they can live with that on their conscience anymore than I understand how defense lawyers can live with setting rapists and killers free on technicalities. I think the answer is that most lawyers in general lack a conscience, you'd have to to be successful.

I have seen first hand how the justice system truly works. Back in my youth I was a bit of a delinquent, and was convicted of many fairly petty crimes, mostly misdemeanor but nothing worse than a gross misdemeanor. I freely admit that I was usually guilty as charged, and I took my licks, but there were two cases where I was absolutely innocent.

Part of their strategy is to charge you with everything in the book, that way when plea bargaining comes around they can act like they are doing you a favor by dropping the extraneous charges if you plead to the main charge. The prosecutor had absolutely no interest in justice and wasn't interested in explanations or evidence, it was made very clear that if I refused to plead guilty it was going to cost me. In retrospect i'm confident I would have won both cases had I gone to trial, but the scare tactics can be quite effective, especially on a 20 year old kid.

I accepted the plea bargain in both cases to stay out of jail and avoid a costly trial and missing more work. Being innocent is just as costly as being guilty. Ironically I had to tell the judge under oath that I was really guilty and wasn't just saying so for the sake of the plea bargain. The convictions went on my permanent record and I got some hefty fines and probation, and the prosecutor got 2 more notches on her holster. 8 years later these convictions in particular, which were gross misdemeanors, cost me a very nice job.

Like I said, it was fairly petty and doesn't compare with the innocent people who are rotting in prison right now, but I can definitely sympathize. We see it all the time, people pleading guilty to avoid the death penalty or long prison sentences. When it is more of a punishment to prove your innocence than it is to plead guilty, something is wrong with the system.

Re:well, well...

"If they truly cared about justice they'd say "hey we need to take another look at this". I don't know how they can live with that on their conscience anymore than I understand how defense lawyers can live with setting rapists and killers free on technicalities."

There's a difference. A prosecutor in the situation you give is placing his own career interests above the interests of justice and the law.

A defense lawyer getting someone off "on a technicality" is at the very least demonstrating an allegiance to the letter of the law and a commitment to see that all people are treated under the law equally... and often may be the last line of defense against a government bent on violating civil rights.

There's really no such thing as "getting off on a technicality." Whenever you hear that phrase it's coming from a prosecutor or police officer who at the least didn't do their job properly and at worst violated the law themselves, and got caught.

When the police searched you illegally, you "got off on a technicality."
When they came into your home without a warrant, you "got off on a technicality."
When they didn't properly document and control the chain of possession of evidence used to convict you, thus throwing into doubt whether it's even legitimate evidence, you "got off on a technicality."
When they interrogated you improperly or otherwise throw into question the accuracy of what they claim is your testimony to police, you "got off on a technicality."
When they selectively show your photo to or otherwise lead the victim into presupposing you are the guy who did it, you "got off on a technicality."
When the prosecutor has withheld information that might have exonerated you or changed a jury's verdict, you "got off on a technicality."

"Got off on a technicality" = not guilty.

Re:well, well...

Just wait until they start outsourcing prisons to China. Won't that be all kinds of fun?

God, I wish I were joking.

Re:well, well...

I've noticed something: government conspiracies always get modded up here and are usually based on "Think about it. You know it's true, man."

The reason overly pessimistic posts about the government get modded up and not overly optimistic ones seems to be that your average slashdotter is insecure and doesn't want to get lauged at for being naive.

Well call me naive, but I don't think things like this are driven by greed so much as incompetence, hubris, and an "us vs them" mindset.

Re:well, well...

Possibly significant in terms of sample size.
If person A has a DNA profile that matches one other person in the country, it is still very strong evidence.
If upon checking the other states there was found to be an average of one matching person per state, 50 matches, still strong evidence, but not nearly so conclusive. Would now require stronger supporting evidence to be "beyond reasonable doubt".
If (prison population being approx 1%) there are found to be 100 matches per state, 5000 matches, then DNA becomes more useful as evidence for aquittal than for conviction, ie: non-matching still proves it wasn't you but matching doesn't prove it was you.

Re:This is why the death penalty is a bad idea.

It's screw ups like this that make the death penalty a bad idea. While life in prison is a terrible punishment, perhaps more cruel than death, it gives the state a chance to fix its mistakes.

Falsely sending a person to prison cannot be 'fixed', only perhaps ameliorated. The frequent unfunny jokes about homosexual rape in prison show that not only the conviction system is out of control, but the punishment system is as well.

DNA can disprove only

Unless the crime labs start encoding the full DNA sequence, even then identical twins will duplicate, the best DNA, for that matter finger prints, can do is prove it is *NOT* that person.

Re:DNA can disprove only

DNA and fingerprints are useful in conjunction with other evidence, just as any other type of forensic or circumstantial evidence is. If someone passed out and died during a movie from a blow dart, for example, it would not be prudent to arrest a random person if they had tickets to the movie; similarly, if there is a particular DNA profile on the dart that 10,000 people match, those 10K should not be brought in. However, if someone had tickets to the movie && matched the DNA it would probably be a good idea to bring them in. CSI type shows are partly to blame - the average citizen on the jury trusts scientific evidence on its own far too much instead of the old detective story trio of means, motive and opportunity (forensics cannot help at all with the second).

Transparent government

Please support radical transparency [wikipedia.org] and open source the government [wikipedia.org].

If everything is out in the open, there can be no hiding.

Re:Transparent government

If everything is out in the open, there can be no hiding.

Everything, I suppose, but your nic.

An example of the birthday problem

From the description, this seems like an example of the birthday problem [wikipedia.org]. Briefly, in a group of 23 people the odds are 50-50 that two of them will have the same birthday, while in a group of 57 the odds are better than 99%. However, the odds that someone in the first group will share *your* birthday are far less, roughly 6.1%. Quoting the Wikipedia article, "For a greater than 50% chance that one person in a roomful of n people has the same birthday as you, n would need to be at least 253. Note that this number is significantly higher than 365/2 = 182.5: the reason is that it is likely that there are some birthday matches among the other people in the room."

Likewise, the odds of there being two people with matching DNA in a database are far higher than the odds of someone else matching *your* DNA. So it seems possible that the FBI could be quoting accurate odds, while at the same time there being lots of matches within the database.

very worrysome

As an American once arrested by the SS/FBI for computer related crime a while ago, DNA testing always worried me. I can understand it for violent offenders (which is how it was started and then carried over to all felons).

I can also tell you, if you refuse to submit to testing, they give you what they call "diesel therapy," taking you away from the cushy club fed camp you were in and busing you around the system until you relent. If you were given a half way house sentence or probation, they can revoke either for not submitting a sample.

Dispite turning my life around, finishing my degree and now working as a developer for a medium-size firm, I worry at times that one night I'll be hauled away because some flunky at the FBI mixed up DNA samples or didn't compare them correctly. I can imagine this being a horror scenario for anyone who's never broken the law, but can anyone imagine there being even a slight chance a bunch of narrow minded, non-technical cops are going to believe me, even if the crime is something totally unrelated to my history?

Birthday paradox

The FBI says that the chance of any given person matching another unrelated person is 1 in 113 billion. They claim that the reason the Arizona lab tech found as many matches as she did ("dozens") is because she was checking the whole database (6 million entries) against itself. This is a straightforward birthday paradox issue, then. According to the Wikipedia birthday problem page, the number of collisions expected given d= 113 billion different "birthdays" and n = 6 million "people in the room" is n - d + d((d-1/d)^n). This is about 160 matches! So in fact the FBI may be right. Note that the chance of a given person matching _anyone_ in the database is about 0.0053%, which is much greater than 1 in 113 billion.

Practical truths...

Not to troll, but law enforcement agencies are really more interested in convictions than the truth. For instance, Virginia has a law that places a 21-day time limit on new evidence that can be used to exonerate someone wrongly convicted. I'm sure the FBI doesn't want it's coveted CODIS database subject to any doubt.

Re:So, the 1:113 Billion estimate is wrong

I believe the problem with this is outlined in this article: http://en.wikipedia.org/wiki/Prosecutor's_fallacy [wikipedia.org]

Have a read. It shows that you can't trust statistics when you only have half of the picture, and why it can be so dangerous to do so.

Re:False matches my ass.

There has never yet been a 13-locus match seen between unrelated people in the national database- despite the 5 million or so profiles currently in it. I'm sure the average Slashdot reader can manage to work out how many pairwise comparisons that is. (Hint- it's a pretty fucking big number.)

Well it seems that your statement of 'never' is simply because that comparison has not been run, since the FBI is doing everything possible to prevent testing of that theory. If its so unimaginably rare, it would seem that the FBI would be all for doing these searches to bolster the value of their evidence. However the results in MD would seem to challenge that dogma.

A 9-locus match between unrelated people is not surprising. That's why we don't sue only 9 loci, idiots!

Well it seems that California prosecutors are idiots and were using a 9 locus match to prosecute a man for a 2 decade old murder. From TFA: "Its implications became clear as she prepared to defend a client accused of a 20-year-old rape and murder. A database search had found a nine-locus match between his DNA profile and semen found in the victim's body. Based on FBI estimates, the prosecutor said the odds of a coincidental match were as remote as 1 in 108 trillion." So just to explain why we 'idiots' would use a 9 locus match: DNA collected at a crime scene is not a complete genome. Often it is only fragments. You may not have a full 13 loci to check because the fragment you have would not cover all 13 loci. There is, however a statistical nicety here that you've completely side-stepped in your haste to call us morons (which TFA mentions). The likelihood of finding matches of 9 of the 9 you have in your genome fragment is far less than the likelihood of finding any two people in a database with 9 of any 9 of 13 loci that match.

Re:False matches my ass.

I match 12 out of 12 Y-DNA markers with 9 other people who have had their DNA tested. Based on our last names, none of us are related.

I don't think you should be able to use blind searches of a DNA database as evidence, because it's too easy to get false positives. It's only useful evidence against someone you've already found by other means, or as a way to generate leads.