Dec 04 2012

The Power of Confirmation Bias

It is my contention that scientific skepticism is an intellectual discipline and a cognitive skill set more than anything else. It is also a philosophy, a value system, and an approach to knowledge – but these are hollow without the knowledge and skills to apply that philosophy.

This is especially true in our complex world, with sophisticated pseudoscience alongside mature and highly technical real science, ideologies of every stripe pushing their agenda, governments with power to protect, and markets and corporations with a profit motive to deceive. The internet is also drowning us in information, much of it dodgy.

It is therefore not enough to have a generally skeptical outlook, or even to call oneself a skeptic. Skepticism is a journey of self-knowledge, exploration, and mastering the various skills that comprise so-called metacognition – the ability to think about thinking. <shameless plug> For a thorough discussion of metacogntion, you can check out my Teaching Company course: Your Deceptive Mind: A Scientific Guide to Critical Thinking Skills. I also understand it makes a wonderful gift.</shameless plug>

As an example of the need for metacognitive skills in navigating this complex world there is confirmation bias. This is definitely on my top 5 list of core skeptical concepts, and is a major contributor to faulty thinking. Confirmation bias is the tendency to perceive and accept information that seems to confirm our existing beliefs, while ignoring, forgetting, or explaining away information that contradicts our existing beliefs. It is a systematic bias that works relentlessly and often subtly to push us in the direction of a desired or preexisting conclusion or bias. Worse – it gives us a false sense of confidence in that conclusion. We think we are following the evidence, when in fact we are leading the evidence.

Part of the illusion of evidence created by confirmation bias is the fact that there is so much information out there in the world. We encounter numerous events, people, and bits of data every day. Our brains are great at sifting this data for meaningful patterns, and when we see the pattern we think, “What are the odds? That cannot be a coincidence, and so it confirms my belief.” Rather, the odds that you would have encountered something that could confirm your belief was almost certain, given the number of opportunities.

Another factor that plays into confirmation bias is using open-ended criteria, or ad-hoc or post-hoc analysis. This means that we decide after we encounter a bit of information that this information confirms our belief. We retrofit the new data into our belief as confirmation.

Confirmation bias is further supported by a network of cognitive flaws – logical fallacies, heuristics, and other cognitive biases – that conspire together to reinforce our existing beliefs. In the end you have people who, based on the same underlying reality, arrive at confidently and firmly held conclusions that are directly opposing and mutually exclusive.

I encounter examples of confirmation bias every day. (My now favorite quote about this is from Jon Ronson, who said, “After I learned about confirmation bias I started seeing it everywhere.”) Of course, at first it is easy to see confirmation bias in others, and only later do we learn to detect it in ourselves, which forever remains challenging. A recent e-mail provides an excellent example, and was the prompt for this post. The following comment is from a forum dedicated to body asymmetry disorders as an explanation for chronic conditions such as multiple sclerosis and chronic fatigue syndrome:

“I personally have serious doubts if there is anything like “MS” or MS being an “autoimmune disease”.

Literally everyone I have seen and treated with “MS” was actually a TMJ dysfunction patient. I am anxiously looking for my first “Real MS Patient”.

I am currently treating at least 20 “MS” patients. Many of them have had short lived benefits from CCSVI. They all appear to be substantially improving with TMJ correction and most symptoms previously labelled “MS related” have disappeared.

I believe that most “MS lesions” are a consequence of CSF leaking into the brain stroma – they are not lesions in the neural tissue as such otherwise I could not so rapidly correct the symptoms.

The immune system tries to get rid of this fluid which is in the wrong place but is unable to do so and the deposits calcify leading to the typical opaque lesions seen on MRI and CAT scans. At autopsy the increased immune function is wrongly diagnosed as an auto immune disorder.

Imagine taking a slice of pathological tissue around any infective lesion one is bound to see an increased immune function. To go onto interpret it as an autoimmune disease would be pure folly. The immune system needs help – not an assault on its function which the drugs do.

As per your question I have treated hundreds of patients with Myalgic Encephalomyelitis (ME), Chronic Fatigue Syndrome (CFS) and others who present just with jaw problems for very many years. I have never found a real ME or CFS patient either. The common denominator is tooth/Jaw dysfunction and some who also have their Atlas out of line.

I remain to be corrected on my hypothesis”

Sounds pretty confident. It is also all utter nonsense. The commenter, a practitioner of dental therapy, may be just promoting their practice, but let’s assume they are sincere in the beliefs they state above. For background, we have over five decades of research telling us that MS is an auto-immune disease. There are literally thousands of published studies supporting this conclusion, with multiple independent lines of evidence. The commenter, who clearly knows very little about immunology or autoimmune disease, tries to dismiss some of this evidence, such as from autopsy examination. They argue that seeing inflammation around MS lesions in the brain does not prove the lesion is autoimmune. It could just be the immune system doing its job.

Whenever you hear a claim like this about a mature science you have to ask yourself – so all the researchers dedicated to this discipline over decades missed this simple idea? If something apparently obvious occurs to you as a non-expert, it’s a good bet that it has occurred to people who have spent their careers thinking about the issue and researching it. The difference between reactive immune activity and auto-immune activity is a basic concept – it is something that is thought of every single time a pathologist looks at immune activity on a slide – is this reactive immunity, or a primary inflammatory lesion?

MS is an autoimmune disease. There is evidence of chronic central nervous system immune activity, lesions are inflammatory, and immunosuppressant treatment works.

The commenter claims they never met a real MS patient. What are their criteria? This sounds like confirmation bias through definition. They imply that if someone who is diagnosed with MS has improvement in any of their symptoms with TMJ correction, then they don’t have MS. They do not seem to be aware of the illusory power of placebo effects and confirmation bias. Any treatment, if looked at in an open-ended way, will seem to work for almost any condition. That is the power of confirmation bias. Only by carefully studying a treatment is a blinded fashion, to eliminate the effect of all biases, can we get reliable information about the real effects of the treatment.

What published research is there to support the claims of this commenter? Very little – a couple of pilot studies (meaning preliminary type studies that are not blinded, controlled, or otherwise rigorous). Preliminary studies themselves are little more than confirmation bias – they have a huge positive bias and are likely to support the researcher’s hypothesis. They are not confirmatory – meaning they really are not evidence at all, but just an exploration to guide later research.

The commenter is further impressed by the fact that they can speculate wildly about an “explanation” for apparent MS lesions. This is just another form a confirmation bias – we think that because we can think of an explanation, this in itself is a sort of confirmation (the entire field of astrology, for example, is based upon this process). We underestimate our ability to invent explanations for things post-hoc. The ability to do so says absolutely nothing about the viability of our beliefs, because we can invent post-hoc explanations for anything. The real question is – is there any objective evidence for the hypothesis. In this case, regarding CSF leakage, reactive immunity, and MS lesions, the answer is no.

Conclusion

One primary difference between science-based medicine and practices on the fringe, like TMJ treatment for diseases like MS, is that SBM is based as much as possible on objective rigorous evidence. It recognizes that failings of anecdotal or uncontrolled observations. Fringe practice, by contrast, is little more than a massive exercise in confirmation bias. Disconnected from the reality check of science, there is no apparent limit to the nonsense in which humans will confidently believe.

22 responses so far