This post is a follow up to one from last week about reproducibility in science. An e-mailer had a problem with the following statement:

“I tend to accept claims based upon published rigorous evidence that shows a consistent robust result with reasonable effect sizes with evidence in proportion to the plausibility. “

In response they wrote:

“I think this might open up for arbitrary amounts of discrimination based on gut feelings. As an example, if it really was so that aliens regularly conducted semi-stealthy visits to planet earth (as the proponents seems to suggest), we might actually never realise so, because there could be a double standard demanding arbitrarily high levels of evidence based on gut feelings about prior probability.

Allowing prior conceptions this power could even open up for psychological effects such as post hoc adapting the prior judgement such that it is just low enough to allow to discard the presented evidence.”

This is a common reaction, especially when prior probability is used as part of an argument for rejecting a scientific claim (to be clear, I don’t think the e-mailer is doing this, they just seem to have an honest question). I am a strong proponent of prior probability, used as part of a Bayesian analysis. That is, in fact, at the heart of the difference between science-based medicine (a term I coined to reflect my approach) and evidence-based medicine.

Prior probability or plausibility is simply a reflection of our current overall understanding of a scientific question from existing evidence. Let me first lay out what I think are the arguments against it, and then I will explain why I disagree.

The e-mailer is essentially expressing a concern that taking plausibility into account will tend to lead to false negatives, that we may reject a scientific hypothesis which actually has reasonable evidence simply because it does not meet an arbitrarily high bar of evidence. His second concern is that judgments about plausibility are just that, judgments, and therefore are inherently arbitrary and will simply reflect the biases of the person making the judgment. In the extreme, invoking prior plausibility can result in denialism.

These are the most common criticisms of prior plausibility (and by extension science-based medicine), as reflected in this editorial by David Katz (a proponent of integrative medicine) in which he soundly rejects plausibility arguments in medicine.

The idea of considering prior plausibility, however, to me seems like ineluctable common sense. It is, in fact, how we operate day to day. If, for example, your spouse came home and claimed they hit a deer with the car, and showed you as evidence a large dent in the fender with a bit of blood and brown hair, you would probably accept that as sufficient evidence for their claims (and so would the insurance company). If, on the other hand, they claimed to have hit a bigfoot with the car and showed you the exact same evidence, you might want to see some further evidence before accepting that it was a bigfoot they hit.

This is the Bayesian approach. You have a belief according to existing evidence and theories. If a new bit of evidence comes in you don’t discard all prior knowledge, or pretend that we currently know nothing. You simply update your belief, adding the new information to existing information. In this way our beliefs slowly evolve, tracking with new evidence and ideas (unless you have a large emotional investment in one belief, but that’s another post).

Science operates the same way. Each new hypothesis or experiment is always put into the context of existing evidence. The amount of evidence necessary to add one small bit of incremental understanding about a phenomenon is much less (and should be less) than the amount of evidence necessary to entirely overturn a well-established theory. Science could not possibly function any other way.

Another way to look at it is this: the amount of evidence necessary to overturn an established conclusion is proportional to the amount of evidence that established the conclusion in the place.

Consider the alternative – looking at each new claim or piece of evidence with blinders on, as if we currently have no accumulated scientific knowledge. Failing to consider prior probability would also mean overturning a mountain of prior evidence with one tiny new bit of evidence. This amounts to favoring the new evidence simply because it is the most recent, but there is no justification for this approach.

Now let me address the two common criticisms of prior plausibility I outlined above. The first is that it will tend to favor false negatives. While this is true, it needs to be put into perspective. All approaches to evidence occur along a spectrum favoring either false positives at one end and false negatives at the other. As you more along the spectrum you change the balance of false positives to false negatives – this is a zero sum game, as you decrease one you have to increase the other.

You can reduce both false positives and false negatives by increasing the amount and quality of the evidence, but that is irrelevant to the point. I am talking about the approach to the evidence. Given any set of data, how we interpret that data is a zero-sum game between false positives and false negatives.

For example, for any medical test there needs to be a cutoff between normal and abnormal. Where you set that cutoff affects the false positive and false negative rates of how the outcome is interpreted, and as you increase one you decrease the other.

Failing to consider prior probability massively favors false positives. The downside of false positives in science is that we then waste tremendous resources chasing a hypothesis that is unlikely to be true. Therefore, utilizing prior probability does push the final assessment toward the false negative, but only because the alternative is starting way out on the false positive end of the spectrum. Obviously what we want is an optimal balance of false positive and false negative, which I would argue requires the proper consideration of prior probability.

This brings us to the second objective, that such considerations are ultimately a judgment call. This is correct, sort of, but ignores the fact that judgment is inherent to the process of science. You cannot completely eliminate judgment from science. Ultimately scientists have to consider how much evidence is enough, how to balance conflicting evidence, how to assess the degree of rigor of a study, which of two competing theories better accounts for the evidence, and if the flaws in a study invalidate the results. Science is messy, and prior probability is just one more judgment among many unavoidable judgments.

Further, judgments about prior probability are often crystal clear at the extremes, even if they get progressively fuzzy toward the middle. For example, if a new hypothesis requires overturning well established laws of physics, it does not take much judgment to consider the plausibility to be low.

For example, Orbo is once again claiming they have a free energy device. Their latest device is a small cube that can be used to recharge USB devices. Their claims require overturning the laws of thermodynamics. Should we accept the evidence they have produced (essentially recharging devices off the cube but not allowing anyone to open the cube and see what’s inside)? What if they were claiming they had a small incremental improvement in the lithium ion battery? Would you require the same level of evidence?

At the extremes there isn’t much judgment required, just a basic understanding of science. If entirely new phenomena are being claimed, like undetectable human energy fields, it only makes sense to set the bar higher.

How do you avoid bias in such judgments? That is where the community of science comes in. The hope is that the more people you have weighing in on a question, the more individual biases will average out. We definitely need transparency and openness in science, a health disrespect for authority while respecting the process and the consensus of the community (while still being open to dissent). It is a delicate balancing act – and it requires judgment.

But in the end, high quality evidence trumps everything else. That is not a zero-sum game, but a win-win. If you think your pet theory is unfairly being rejected by considerations of prior probability, you can whine about prior probability, or you can create high quality evidence that convinces the skeptics.