Mar 08 2007

No Love for Anecdotal Evidence

Scientists and skeptics have come to use the word “anecdotal” as a derogatory dismissal of certain kinds of weak evidence – and with good reason. Believers in the paranormal and unscientific healing modalities chafe at this and have rushed to the defense of anecdotal evidence, as it is often the only substrate out of which they construct their fantasies and attempt to pass them off as science.

Noted “alternative medicine” guru, Andrew Weil, for example, has advocated the increased use of “uncontrolled clinical observations” (a euphemism for anecdotal evidence) in evaluating non-traditional medical methods. Our favorite paranormal apologist, Winston Wu, also defends the anecdotal, or at least criticizes skeptics for not giving it the proper respect.

This difference in attitude toward anecdotal evidence is a persistent and critical difference between skeptics and believers, so it is important to understand why skeptics (by which I mean most scientists) do not trust it. Science, as I have written before, is a cumulative process – it builds upon itself (E.O Wilson’s book, Consilience, is a great discussion of this). Scientists not only build upon the facts and theories that have been developed by their predecessors, they also learn from prior experience with scientific methodology and discovery itself. In other words, scientists are getting better at doing science.

One of the hard won lessons of the process of scientific discovery is that anecdotal evidence is very unreliable. Psychologist Barry Beyerstein summarized it well when he wrote, “anecdotal evidence leads us to conclusions that we wish to be true, not conclusions that actually are true.” So anecdotes can be worse than worthless, they can be misleading. This conclusion was arrived at after centuries of being misled by anecdotes. Treatments seem to work and then later are shown to be useless or harmful. Common experience leads us to believe in many things that more carefully collected evidence later disproved. After being hit on the head so many times by anecdotal evidence throughout history, modern scientists are now appropriately wary.

What we mean, exactly, by anecdotal evidence is the report of an experience by one or more persons that is not objectively documented or an experience or outcome that occurred outside of a controlled environment. Such evidence is unreliable because it depends upon the accurate perception of the witness(es), often in a situation where the event was unexpected or unusual; it is dependent upon subjective memory, which has been overwhelmingly demonstrated to be extremely error prone and subject to a host of flaws; and it cannot account for the random vagaries of a chaotic world.

At this point in the history of science, with all that has gone before, it is hopelessly naïve to put any faith in anecdotal evidence. It is utter folly to use such weak and flawed evidence as a basis for concluding that a new and unusual phenomenon is real and that our science textbooks need to be rewritten. At best, anecdotes can be used as an indication of a possible (not even probable) phenomenon that is deserving of further research or exploration. But it should not be used as a basis for firm conclusions.

What I just described is generally accepted within the scientific community, and (as I outlined) with good reason. Yet proponents of the paranormal and unconventional would have us believe that this attitude toward anecdotal evidence is an invention of the closed-minded skeptic, nothing more than a debating tool to dismiss inconvenient evidence. Winston Wu uses the typical arguments of the paranormal crowd in an attempt to rescue anecdotal evidence (for where would they be without it?). Let’s take a look.

Anecdotal evidence is mostly reliable in regard to everyday things.

Wu writes: “The main problem with the “anecdotal evidence is invalid” argument is that anecdotal evidence IS in fact mostly reliable with regard to everyday mundane things.” I think this is both a false premise and a false analogy.

There are numerous psychological experiments, mostly designed to look at memory, that have documented the fact that in everyday encounters our anecdotal recollection of events are extremely flawed. Further, our interpretation of events is even more flawed, subject to a host of psychological factors (an excellent summary of this is the book How We Know What Isn’t So by Thomas Gilovich – I defy anyone to read and comprehend this book and still defend anecdotal evidence).

All of us have likely had the experience of being engaged in a disagreement with one or more people. Afterwards (usually after emotions have calmed down), when all sides compare notes it is apparent that each person involved in the discussion has a significantly different recollection of who said what when. I can’t tell you how many times I wish I had a camera rolling during a discussion so I could prove later what actually happened. So, if anything, everyday experience should have taught us that we cannot rely upon our anecdotal experience.

Also, comparing scientific evidence to everyday experience is a false analogy because science requires a level of detail that is not important in everyday mundane occurrences. I do trust that the anecdotal reports of other people are accurate enough with regard to broad brushstrokes – if my wife tells me she drove home after work and encountered some traffic, it is reasonable to trust this general account. However, I would not trust her memory to details such as – exactly what time did you leave, what was your average speed, or perhaps even what was the exact cause of the traffic.

I think it is accurate to say that trusting in everyday experience is what led to our pre-scientific view of the world. Scientific methodology is largely about controlling for all those factors that make anecdotal experience misleading.

Anecdotal evidence is dependent upon perspective.

Wu writes: “My firsthand direct experiences are anecdotal evidence to others, while their direct experiences are anecdotal to me too. Therefore, whether something is anecdotal or not depends on whether or not you are the experiencer, rather than on it being true or false.”

Wrong. My own experiences are anecdotal too – they just seem more compelling because I experienced them. One of the hardest things for a scientist to do is to have a proper suspicion of their own experience – and this is where many pseudoscientists fall down.

I teach medical students and residents, and one “clinical pearl” I always stress is that our own experience with patients is anecdotal and is therefore not necessarily representative. In fact, there is published evidence that following treatment guidelines based upon scientific data leads to better outcomes than relying upon personal clinical experience. That’s because personal clinical experience, no matter how compelling, is anecdotal.

The Credible Witness

Wu argues that we should judge anecdotal evidence based upon the following criteria: “a) The number of eyewitnesses; b) The consistency of the observations and claims; c) The credibility of the witnesses; d) The clarity of and proximity of the observation; e) The state of mind of the witnesses.”

Certainly it is reasonable to consider these factors in judging eyewitness testimony, but even under ideal conditions anecdotal evidence is unreliable as scientific evidence – for reasons that I have stated above and that are not mitigated by Wu’s factors.

Also, it has been adequately demonstrated that each of these factors is no guarantee of accuracy. I mentioned yesterday the writing of Robert Bartholomew, who has published excellent reviews of mass hysteria, the UFO phenomenon, and the Airship Hysteria of the late 19th and early 20th centuries. He documents numerous episodes, for example, of multiple eyewitnesses recounting the same story (seeing an airship, for example), and afterwards it was proven that all of the accounts were mistaken. Large amounts of low quality data is still low quality data.

Further, the number and consistency of eyewitnesses could be accounted for by memory contamination. It is common for eyewitnesses, especially if they were together at the time, to talk during and after the experience about what they were seeing. The perceptions of one person can thereby easily contaminate the memory of the other witnesses (and again, this ability to easily contaminate memory has been demonstrated in the psychological literature).

It is reasonable to consider the viewing conditions and the clarity they would afford, but even under apparently ideal conditions we are still subject to visual illusions. For example, it is difficult to impossible to judge the distance, size, and velocity of objects viewed in the sky, without any background for perspective. I wrote about this in more detail in this blog entry.

The “credible witness” fallacy is a fallacy because there is no such thing. People are people, we all have the same brains (more or less), are subject to the same optical illusions, psychological factors, and distortions of memory. Being trained as a pilot, police officer, astronaut, or brain surgeon does not free us from the foibles of human memory and perception. In fact, it is not even a guarantee of being free from psychosis – as the recent incident with the female lovelorn astronaut reminds us.

Likewise, being in an uncompromised “state of mind” is a necessary but insufficient criteria for reliability. Even someone who is wide awake and uncompromised can make profound errors in perception and interpretation. Also, people often underestimate the effects of certain factors on their mental state, such as moderate sleep deprivation. Gullible ghost hunters often find that if they stay up all night in an allegedly haunted house, eventually weird things start happening. I bet.

But the bottom line is this – we know for certain from countless historical examples that even people who have all the traits of a reliable witness can be profoundly mistaken.

Why would they lie?

I am often asked if I think a purveyor of obvious nonsense is a con-artist or true believer. Of course, I often cannot know for sure (unless they are caught red-handed in fraud), and I suspect that many are a combination of the two to some degree. But it is often argued by believers that if you cannot either prove that someone is lying or clearly identify an ulterior motive, they must be telling the truth, or at least are believable.

Of course, whether or not someone believes what they are reporting says nothing about the accuracy of the information – just that they are not willfully lying. But believers, like Wu, miss some important possible motivations for prevarication. First, people sometimes stretch the truth simply because it makes life more interesting. It breaks the dull routine of our mundane lives for others to believe that we saw a ghost or were abducted by aliens. And that’s motivation enough.

Others may commit “pious fraud.” They are a true believer, but they may embellish their experience in order to convince those nasty skeptics. This is like a cop planting evidence on a suspect they “know” to be guilty, just to make sure justice is done.

And finally, people do weird things for weird reasons. The inability to figure out why someone might lie is no reassurance that they are in fact telling the truth.

Skeptics are unfair.

Wu writes: “Ordinarily, anecdotal evidence this strong is accepted as valid evidence in most circumstances, so why not in regard to paranormal or psychic phenomena, especially when it’s so common? The reason is because skeptics and certain scientists don’t think these things are possible, therefore they assume that the fallibility of anecdotes must be the cause.”

Again, wrong. First, this is another false premise. As I stated above, this kind of evidence is NOT accepted in mainstream science for non-paranormal claims. Wu and others are just naïve about how rigorous mainstream science really is. Wu also ignores all the reasons I outlined for distrusting anecdotes. It is worth noting that all of the reasons I discuss here have been written about extensively in the skeptical literature, seriously calling into question Wu’s scholarship.

The notion that skeptics assume anecdotes are unreliable to fit their a-priori assumptions is an extremely common claim among critics of skeptics, but it is without factual basis. Essentially, Wu and others just make this up because it sounds good and dismisses the skeptics. While it is true that scientific plausibility is a significant and legitimate consideration (see my blog entry yesterday on extraordinary claims), it is not the deciding factor. Evidence (if it is sufficient in BOTH quality and quantity) trumps plausibility.

Further, some of the phenomena that Wu is talking about are not paranormal or impossible. There is no reason why space faring aliens cannot be visiting the earth. I hope to live to see this happen one day. It would be totally cool. If there were credible evidence to support the conclusion that aliens were in fact visiting the earth I would be dedicating a large portion of my time to investigating this, and trying to prove it to the world.

If ESP were real it would tell us something important about neurology and physics that we do not currently know. Scientists love true anomalies and mysteries – that’s where the money is. That is how we discover new things and add to our growing model of reality. It is a cartoonish self-serving notion that scientists and skeptics fear change, support the status quo, and abhor genuine mysteries. Believers who claim this are not even part of the real conversation – they’re just talking to themselves, lulling each other into pseudoscientific complacency.

So there is no love for anecdotes in skeptic-town. And no matter how much the paranormal apologists whine, anecdotes will never be granted a place of respect within the halls of rigorous science.

One response so far