Jan 08 2016

What Is Bayes Theorem?

I have written a little about Bayes Theorem, mainly on Science-Based Medicine, which is a statistical method for analyzing data. A recent Scientific American column has some interesting things to say about it as well. I thought a brief overview would be helpful for those who are not sure what it is.

This statistical method is named for Thomas Bayes who first formulated the basic process, which is this: begin with an estimate of the probability that any claim, belief, hypothesis is true, then look at any new data and update the probability given the new data.

If this sounds simple and intuitive, it’s because it is. Psychologists have found that people innately take a Bayesian approach to knowledge. We tend to increment our beliefs, updating them as new information comes in.

Of course, this is only true when we do not have an emotional investment in one conclusion or narrative, in which case we jealously defend our beliefs even in the face of overwhelming new evidence. Absent a significant bias, we are natural Bayesians.

That is really the basic concept of Bayes Theorem. However, there are some statistical nuances when applying Bayes to specific scientific situations. There are two additional wrinkles to a Bayesian analysis that I think are worth pointing out.

The first is that Bayes begins with a prior probability. This is one of the things I really like about Bayes – it expressly considers the probability that a claim is true given everything we know about the universe, and then puts new evidence into the context of that prior probability.

This approach is inherently skeptical. It means I would require more evidence before believing someone probably saw Bigfoot than that they saw a deer. When you get into the math even a little, it helps put evidence into even better perspective.

Here is Bayes formula: P(B|E) = P(B) X P(E|B) / P(E). In this formula “P” = probability, “B” = belief, and “E” = evidence. Translated into English the formula means the probability of the belief given the new evidence = the probability of the belief absent the new evidence time the probability of the evidence given that the belief is true divided by the probability of the evidence absent the belief.

Don’t worry too much about the math, I know formulas can tend to make people’s eyes glaze over. Here is what this means in practice – the implication of new evidence depends heavily on the prior plausibility. This is mathematically saying what skeptics have been saying for years, that a weak study with slightly positive evidence for ESP is not convincing evidence that ESP is real because it changes the very low prior probability by only a little.

This is an important realization because it counters what we often refer to as the frequentist fallacy – the notion that because there is statistically significant evidence for a hypothesis the hypothesis must be true, no matter how slight the effect and improbable the hypothesis given everything else we know about reality.

Stated another way, we can ask, what are the odds that the hypothesis is true vs that the new evidence is wrong? That is exactly what Bayes seeks to calculate.

As the Scientific American article points out, physicians are very familiar with this question, because we face it on a regular basis in our jobs. The example the author give is very revealing: if you have a test that is 99% accurate (by which he means 99% sensitive and specific), and you test for a condition that is present in 1% of the population, what does a positive test mean? You may be surprised to learn that a positive outcome on a 99% sensitive and specific test for this condition only carries a 50% probability that the patient actually has the disease.

This is because of false positives. If even 1 in 100 tests is false positive, but only 1 in 100 people have the disease, then a false positive result is as likely as a true positive, hence a 50% predictive value of a positive test.

Doctors have to be specifically trained to think in this new way, not how accurate a test is but what is the predictive value is of a positive or negative test, given everything we know about the disease and the patient.

In the same way we can ask – what is the predictive value of a positive outcome in a research study for ESP? Given what we know about the high incidence of false positive outcomes in science, and the extremely low prior probability of rewriting the laws of physics, the answer should now seem obvious.

Bayes also shows mathematically why confirmatory tests are so powerful. In the medical example, a second test of the same accuracy if it is positive now has a 99% chance of being a true positive, because the prior probability has increased from 1% to 50%.

In science, replication is the key. When a result can be consistently replicated the Bayesian probability that it is a real effect becomes high.

The second aspect of probability that Bayes helps us understand is the importance of considering alternative hypotheses. The conspiracy theorist, for example, is impressed when they find information that supports their conspiracy narrative. What they are failing to consider is two thing: what is the predictive value of that fact, and closely related to that, is that fact also consistent with any alternative explanations?

If the fact in question is consistent with a hundred different interpretations, then it does not much affect the probability of any one of those hundred explanations.

This is where confirmation bias comes in – if you only consider your own hypothesis, then positive tests (correlations, coincidences, etc) can seem very compelling. If you are unconsciously seeking out positive correlations then the illusion of confirmation can be powerful because you won’t be aware of either the negative correlations or all the other possible explanations for the apparent correlations.

Bayes slices through all of this by organizing information into a fairly simple formula and giving us specific (and often counter-intuitive) predictive values.

The primary criticism of a Bayesian approach, and one I hear often and in many contexts, is that we don’t always know the prior probability, and in fact estimates of prior probability may simply reflect our current bias. There is some truth to this. It may take scientific judgement to decide how likely it is that something is true.

In some contexts, like disease frequency, we have a specific answer. We can know with high reliability what the prevalence of a disease is in a specific population – we can put a solid number on prior probability. In other contexts, however, it’s hard to put a number on it. What is the probability that ESP is real?

However, even in these situations Bayes is still very useful. First, we can plug representative or likely numbers into a Bayes calculation and see what happens. For example we could ask, if we think that there is a 1% chance that ESP is real, then what will be the post probability given this new evidence?

In other words, Bayes can still tell us how much a new study changes the probability of a phenomenon being real, whatever we think the prior probability is. In fact we can calculate how much the prior probability would change, without having to commit to any specific prior probability.

What we find is that probability changes much less than what we may think given, for example, high levels of statistical significance. Bayes shows that statistical significance is deceptive and tends to overestimate the sense of how likely it is that a hypothesis is true.

Conclusion

Bayesian analysis is an important concept for any scientist and skeptic to understand. It is extremely practical, and is already used (whether or not it is explicitly named) in professions that need to deal with probability in a practical way, such as in medicine.

Bayes Theorem makes explicitly clear several skeptical principles, including the need to consider predictive value, the impact of false positives, the need to consider alternative hypothesis, and the need to put statistical significance into its proper context.

In many ways a Bayesian approach to knowledge is a skeptical approach.