Jun 01 2020

Junk Science in the Courtroom

In the last 20 years I have been called to jury duty several times. Every time I was dismissed almost instantly, once I made it known that I am a professional skeptic. Apparently lawyers fear that kind of skepticism on their juries (at least one side always did). The same is true of many of my skeptical colleagues, so I am not an isolated case. Once my brother said during the process that he wrote an article on the fallibility of human memory and eyewitness testimony. His but barely hit the seat when he was dismissed.

It is unclear how best to interpret these anecdotes, but what is clear is that justice requires facts and needs to align optimally with reality. Falsehoods and pseudoscience do not generally lead to justice. It is for this reason that courtrooms have elaborate rules of evidence, and generally they work well. Even in our adversarial system, you need to use generally valid arguments, you need to back up your statements with evidence, and there are rules of admissibility. Each side provides a check on the other, as a neutral arbiter presides over the process. It is imperfect (because imperfect people are involved) but at least it has a process.

One area where this process has historically had significant problems, however, is in forensic science, and the admissibility of science itself. The main problem, as I see it, is that it is based largely on authority, in both a good and bad way. Each side is allowed to find their own experts, and they can cherry pick experts whose opinion aligns with their needs. Often a non-expert jury is then tasked with sorting it out. There are standards for which expert testimony is admissible, and this has been a controversy unto itself. Here is a good summary:

Prior to 1993, the Frye standard for admitting expert testimony was the prevailing standard for guiding federal and state courts in their consideration as to whether scientific expert testimony should be admitted at trial. Frye v. United States[1]. The Frye standard requires that the proponent of the evidence establish the general acceptance of the underlying scientific principle and the testing procedures. Notably, Frye only applies to new or novel scientific evidence. However, in 1993, following a revision to the Federal Evidence Code by Congress, the Supreme Court of the United States annunciated the new standard in Daubert v. Merrell Dow Pharmaceuticals, Inc.[2] The Daubert inquiry was meant to be flexible and focused on scientific principles and methodology, not conclusions. The Daubert opinion emphasized that the Federal Rules of Evidence governed admissibility and suggested a series of factors a court could consider, but did not establish a test per se. Under Daubert, the admissibility of expert evidence rests squarely within the discretion of the trial court judge. In contrast to FryeDaubert applies to all expert witness testimony.

This article is about the fact that Florida has reverted to the Frye standard recently. This highlights the fact that legal precedent is largely how this is sorted out, and may differ for every state and at the federal level.

But let me broaden the question a bit – what should the standard for scientific evidence in the courtroom be? I think the “first do no harm” standard is a good one. The law shares with medicine not only that they are professions with ethical standards, but that they often use applied science. They must take the science of the day and not just make abstract conclusions about what might be true, they must apply those conclusions in a way that can dramatically affect people’s lives. In medicine we acknowledge that the current science is always incomplete and imperfect, and so we should err on the side of not harming patients through our ministrations. In criminal law the standards is – innocent until proven guilty – which implies the same thing, erring on the side of not convicting someone with questionable science.

Unfortunately, that is not the history of forensic science, which has relied upon new and questionable techniques later shown to be pseudoscience. A recent article in the Boston Globe reviews a lot of this history. Bite mark analysis was often used to tie the accused with their alleged victims – matching a mold of the teeth of those accused with bite marks left on a body. It seems like impressive science, and can be compelling to a jury if you have blown up pictures to show. It also has the air of plausibility, directly matching pieces of physical evidence.

Another layer is added by television shows that glorify forensic science. That much – making a hero out of science and showing its power – is a good thing, but not if the science being glorified is pseudoscience. But now jurors bring with them the expectation that every case will solved with high-tech forensic science, and may predispose them to giving more weight to such evidence than it deserves.

Bite mark analysis is a dramatic example. Many people are sitting in prison convicted based on bite marks. According to The Innocence Project at least 30 overturned wrongful convictions were based on such evidence, and there are perhaps more out there. The problem with bite mark analysis is that there is a subjective component, a pattern matching done by a flawed human. This is perhaps the cause of the deeper problem – the technique was never adequately validated. What does that mean? A validated test must be checked in a blinded and objective way against reality. This does not mean matching what is already known, but predicting what is not known (to the person making the call).

For example, rather than having one bite mark expert examine the mold of the teeth of the accused and the bite marks on the victim, they should be given 10 molds of different people. They then need to see if they can confidently match one of the molds over the other nine. Then repeat with an independent expert to see if they come to the same conclusion. That kind of blinded, controlled, and replicated evidence is convincing, and courtrooms should demand no less before convicting someone of a serious crime like murder. Otherwise you are doing astrology. Astrologers can explain lots of things, once they know the answer, but utterly fail to make blinded matches.

The same is true for partial fingerprints, microscopic fiber analysis, and shoe print analysis. Experts can make a match when they know the target, but not when blinded. There are also problems with line-ups for eyewitness identification. These are supposed to be blinded, but can easily be tainted if anyone who knows the target is in the room. They need to be completely blinded.

The principles of skeptical science need to apply strongly in the courtroom, with full knowledge of how pseudoscience can deceive and all the fallacies of evaluating evidence. There is a well-developed body of knowledge that informs us how to tell science from pseudoscience, and how to weed out flaws and deception. Why aren’t these simply codified into law? Instead the courts grope their way case by case, and often state by state. The problem with this approach is that decades of wrongful convictions and injustice can happen before the pseudoscience is weeded out. And there are also new techniques being introduced, so the process is never-ending. We can do better.

The Boston Globe article points out that there was a Federal commission to review forensic science and develop rules of admissibility. But this was eliminated by Sessions, and now the states are all left to their own devices. Many don’t have the resources to do the job.

One final side point, which I think sheds light on this topic. In 1993 DNA evidence of deemed to be admissible in the courtroom. DNA is powerful and legitimate scientific evidence (not that it can’t be abused if not done properly). Since then there have been numerous convictions (including those based upon bite marks, hair analysis, or other bogus forensics) overturned with DNA evidence. In a way, the DNA revolution revealed the flaws and biases in the system. It was a wake up call, but not as much as it should have been. The result should be a renewed humility, and dedication to the principle of innocent until proven guilty. Proof (in this context – beyond reasonable doubt) means that you don’t rely on questionable science.

No responses yet