Dec 07 2010

The Context of Anecdotes and Anomalies

The most succinct criticism of postmodernist philosophy as applied to science that I have heard is this – that proponents confuse the context of discovery with the context of  later justification. It occurred to me that the same is true of the role of both anecdotes and anomalies in science. Often when I criticize reliance on anecdotes or so-called anomaly hunting, I get feedback that makes the exact same confusion of context.

The context of discovery refers to how new ideas are generated in science. Playing off of Thomas Kuhn’s work on paradigms (and without getting into a side discussion of Kuhn’s own position), some post-modernists argued that science is a humanist-type of endeavor because scientists come up with their ideas in quirky and culturally contingent ways, rather than rigorous or methodical ways.

However, what makes science methodologically rigorous is not how new ideas are generated (the context of discovery) but how they are tested (the context of later justification).

Anecdotes

Anecdotes are uncontrolled subjective observations. I have often criticized reliance on anecdotes, which is especially problematic in medicine. The problem with anecdotes is that they are subject to a host of biases, such as confirmation bias. They are easily cherry picked, even unintentionally, and therefore can be used to support just about any position. For every anecdote, there is an equal and opposite anecdote.

Because anecdotes can be used to support any position, in reality they support no position – they have no predictive value. As they are often used, they are worse than worthless because they are misleading – they have a tendency to support our biases and give the false impression that our biases have merit.

I have often repeated the above criticism of reliance on anecdotes to form conclusions. And just as often it is pointed out that anecdotes are useful, in medicine and elsewhere, in pointing the way to new possible treatments or phenomena. This is true, but confuses the context of my criticism in the same exact way that post-modernists confuse the process of science.

Anecdotes are useless and misleading in the context of scientific justification – as a way of testing hypotheses. But that is often how they are used, especially in the marketplace and the arena of public opinion where I focus much of my criticism.

In the context of discovery, however, they can be useful. Many medical discoveries started as anecdotal observations. But then those observations have to be tested with controlled observations or experiments – and most anecdotal observations will turn out to be wrong or misleading, because they are quirky and uncontrolled.

In short – anecdotes are useful for generating hypotheses, but not for testing them. Problems arise when anecdotes are used to support a hypothesis or claim, rather than just to raise a possibility to be tested.

Anomaly Hunting

Similarly, anomalies are useful in the process of science to point the way to deeper understanding. An anomaly is some fact or observation that cannot be explained (to use Kuhn’s term) within the current paradigm. Anomalies are vital to scientific progress – they are a giant sign with an arrow pointing and reading “scientific discovery to be made here.”

There are countless examples in the history of science – the orbit of Mercury showed anomalies within Newtonian mechanics and pointed the way toward General Relativity, for example. Today we have anomalies in the speed of distant probes that may indicate that further revisions to gravitational theory are needed.

But just as with anecdotes, anomalies are the beginning of discovery and of the generation of new hypotheses. But by themselves they are not evidence for any particular conclusion.

When an anomaly is encountered in science the first step is to make sure it is an actual anomaly. The simplest answer is always that there is an error in measurement and observation, so that has to be ruled out first. When the measurements are all triple-checked and scientists can be as certain as possible that the anomaly is real, then all possible explanations need to be sought within our current scientific theories or paradigms. Only when an exhaustive attempt to explain the apparent anomaly within conventional theories fails, should scientists conclude that they have a genuine anomaly on their hands.

Then scientists can start to generate new hypotheses to explain the anomaly, usually requiring a revision to existing theories. Kuhn characterized this phase as “revolutionary science” in which one paradigm will shift to another in order to resolve the anomalies. He thought such shifts were large and rare. But later philosophers criticized this position as being a false dichotomy – revisions to theories in order to resolve anomalies are actually quite common and range the spectrum from minor tweaks to wholesale replacement with no sharp demarcation at any point.

Further, often revisions to theories involve not replacing old ideas but deepening them – adding layers of complexity that are compatible with the simpler theories but increase their precision and predictive power. Again – a perfect example is the progress from Newtonian mechanics to relativity. Newtonian mechanics still work in most situations, but at high velocities and gravities relativistic effects become important and measurable.

The role of anomalies, therefore, is entirely within the context of discovery and generating new hypotheses. Anomaly hunting refers to the misapplication of anomalies to the testing of hypotheses. The process involves generating a hypothesis then pointing to apparent anomalies as if their existence supports the specific hypothesis.

At best the mere presence of anomalies would indicate weaknesses in the current theory, but would not support any particular new hypothesis. That would require demonstrating that the new hypothesis resolves the anomalies. Even then this would only indicate that the new hypothesis is plausible and interesting – but it would need to make predictions that are later validated in order to be truly scientific.

Often anomaly hunting is far worse than just misusing anomalies because it often entails pointing to apparent anomalies and prematurely concluding that they are genuine. This comes up most often in conspiracy thinking. Conspiracy theorists often take a complex historical event, then hunt for apparent coincidences or quirky events that are not easily explained and treat them as if they are genuine anomalies – and therefore evidence for a conspiracy.

There are a number of logical fallacies inherent in this process, such as confusing currently unexplained with unexplainable. There is also a false major unstated premise – that if there were no conspiracy at work, then all aspects of the historical event would be easily explained to arbitrary detail and precision. Therefore, the inability to explain every aspect of the event in great detail is treated as an anomaly, rather than just the natural and expected result of the chaos of complex and contingent historical events and human behavior.

For example, 9/11 truthers will point to apparent anomalies in the debris of the jet crash into the Pentagon as evidence for a conspiracy. The unstated premise is that a casual observer would have a good instinctive idea of what such a debris field should look like.

Moon Landing hoax conspiracy theorists look for apparent anomalies in photos and film from Apollo and declare anything they cannot immediately explain as evidence for a hoax. JFK conspiracy theorists look for quirky action on the part of Oswald (as if human behavior is predictable and logical) and point to anything that seems funny as evidence of a conspiracy.

These are all examples of anomaly hunting – prematurely assuming that an apparent anomaly is genuine and then concluding that the anomaly supports a specific hypothesis (an argument from ignorance logical fallacy).

Conclusion

Context is king. With regard to the history of science, the use of anecdote, and the use of anomalies in science play an important role in the context of discovery and the generation of new ideas. But they are problematic and misleading in the context of scientific validation, which relies upon rigorous methodology.

I also find that the very concept of context itself is often overlooked. I have found this to be one of those generic and invaluable intellectual skills, or metacognition (as psychologists call it) – thinking about thinking. It is a common error to fail to consider the context in which one is making an argument – are the proper rules being used in the proper context?

Failure to consider context leads to hopeless confusion – confusion that cannot be resolved until the concept of context is addressed.

Share

17 responses so far

17 Responses to “The Context of Anecdotes and Anomalies”

  1. Jim Shaveron 07 Dec 2010 at 11:43 am

    Nicely done, Dr. Novella. With luck, maybe some of this scientific philosophy can make its way to a public television series.

  2. Rayon 07 Dec 2010 at 2:15 pm

    Very nice explanation of a common problem. Sadly, even people with doctorates do not always appreciate the differences you explain.

  3. SARAon 07 Dec 2010 at 3:43 pm

    Excellent Post!

  4. BillyJoe7on 07 Dec 2010 at 4:19 pm

    JFK conspiracy theorists look for quirky action on the part of Oswald (as if human behavior is predictable and logical)

    This reminds me of the case of Azaria Chamberlain in Australia some decades ago. The family were camping at Uluru (Ayre’s Rock) and one evening her mother, on returning to their tent, saw a dingo running off with her daughter in its mouth.

    Azaria’s body was never found and nearly the entire population became convinced that the mother killed her daughter based mainly on her idiosyncratic reaction to her daughter’s death.

    Although she undoubtedly would have grieved in private, in public she was always calm and methodical in discussing the incident. She was eventually charged with the murder of her daughter, convicted, and sentenced to life imprisonment.

    Many years later the remains of her daughter’s matineee dress were found by a tourist at the rock. From the evidence gleaned from the jacket, it appeared that a dingo had indeed taken her baby as Mrs. Chamberlain had said all along.

  5. sonicon 07 Dec 2010 at 4:51 pm

    Excellent points well explained, could be a section in a basic science text. (Maybe should be…)

  6. VRAlbanyon 07 Dec 2010 at 10:44 pm

    Great post Steve. Maybe I’m a little behind but I just have a few more questions about anecdotes that maybe you (or anyone here, really) could answer just to help me cement my understanding of the concept.

    Your basic definition was that they are uncontrolled, subjective observations. I think the common understanding is that they are personal accounts of very singular or uncommon events. But what happens when the same observations start to occur in large numbers? When the same anecdote shows up again and again, and a seemingly great amount of people swear by the same story (e.g. “My child began to display signs of autism immediately after they were vaccinated.” or “Acupuncture is the only thing that relives my pain.”), then does that lend any evidentiary strength to the claims that are being made? Is there a point where the prevalence of an anecdote garners more attention and time and research effort from the scientific community?

    In discussions I’ve seen about homeopathic treatments or the supposed vaccine/autism link, when charged with relying on anecdotes, proponents will often point to all the other people with similar observations as them to prove that their experiences aren’t singular. Their implication is that the more these stories pile up, the more evidence there is. And to be honest, I would often find this argument difficult to dispute.

    But after thinking about it, it’s kind of easy to see how our susceptibility to biases can explain the spread of such anecdotes rather than their prevalence lending them authority. (Parents are “biased” by drive to find an answer about their child’s condition, and people in pain are “biased” by a drive to find relief. They would be more likely to latch onto a solution that makes sense on the surface before examining it further.)

    And if I properly learned anything from this blog post, in the context of scientific discovery, the subjectivity and … um… uncontrolled-ness of anecdotes are what make them unreliable as evidence and the prevalence doesn’t matter so much at all.
    It looks like the common understanding of a word and it’s meaning in relation to science clash yet again.

    Am I getting this? Please steer me in the right direction if I am mistaken.

    Thank you.

  7. TheRedQueenon 08 Dec 2010 at 2:41 am

    Do actual human researchers conducting the craft of science really cleanly cleave between the ‘context of discovery’ and the ‘context of justification’ ?

    Bruno Latour and Paul Feyerabend (“Farewell to Reason”, “Against Method: towards an Anarchy of Knowledge”) have argued no.

    I understand the ideal of clean separation but “Motivated reasoning”
    sneaks up quite frequently.

  8. BillyJoe7on 08 Dec 2010 at 4:53 am

    VRAlbany,

    ..when the same observations start to occur in large numbers…does that lend any evidentiary strength to the claims that are being made?

    The plural of anecdotes is not data.
    However, a large number of independent subjective reports do strengthen the case for subjecting it to scientific study.

  9. Steven Novellaon 08 Dec 2010 at 8:15 am

    As BillyJoe said – the plural of anecdote is anecdotes, not data. And you already hit upon the reason – systematic and reinforcing biases. Anecdotes generate beliefs which generate more anecdotes in a process of confirmation bias that can seem compelling but is not predictive.

    In other words – millions and even billions of people can be systematically wrong.

    RedQueen – I would say no, but the separation of discovery and confirmation is simply just another demarcation problem. Most such dichotomies are, in fact, continua – but that does not mean they are not real. There is a difference between science and pseudoscience, even though there is not bright line between the two but rather a continuum.

    So while many scientific activities combine or blur the lines between discovery and validation, that does not mean these are not two distinct concepts that need to be understood.

  10. VRAlbanyon 08 Dec 2010 at 10:34 am

    “As BillyJoe said – the plural of anecdote is anecdotes, not data.”

    I like that. It’s an easy, concise concept to remember. And I now feel better equipped to recognize confirmation bias.
    Thank you both for the clarification.

  11. stompsfrogson 08 Dec 2010 at 7:19 pm

    @ VRAlbany

    The witnesses to Joseph Smith’s miraculous golden tablets were three people, because if it was just one guy nobody would’ve believed him.

  12. elmer mccurdyon 09 Dec 2010 at 2:07 am

    VRAlbany:

    Seems to me that acupuncture and homeopathy are not the most relevant examples, since, beyond having mere anecdotal evidence to back them up, there is good evidence showing that they don’t work (I still think some kinds of dry needling that haven’t been studied adequately could possibly have non-placebo effects, but presumably we’ll see, eventually). Of course, many treatmnets commonly practiced by doctors have only anecdotal evidence to back them up.

  13. VRAlbanyon 09 Dec 2010 at 8:53 am

    Haha, great example, frogs.

    And Elmer,
    Perhaps those weren’t the best examples, but they were the first things that came to mind. I was just using them to try and gain some tools for rational discourse in the future with people who advocate for whatever unconfirmed remedy/theory/what have you with a stockpile of anecdotes.

  14. petrucioon 09 Dec 2010 at 4:15 pm

    Man, this post is priceless.

    And so are these two quotes:
    “For every anecdote, there is an equal and opposite anecdote”
    “The plural of anectode is not data”

    Steve, do you mind if I translate it to portuguese and post it with linkback?

    I think you don’t read older posts comments, so I’ll email you later with that request.

  15. Darrickon 11 Dec 2010 at 11:03 am

    Excellent article. I’m sure many of us know someone who swears by [insert kooky idea] because he/she has *personally* experienced its veracity.

    Now that the good doctor has supplied us with the intellectual ammo, it’s time we start shooting down anecdotal ‘evidence’ wherever it rears its fallacious head.

    Post duly linked to and promoted: http://the-attempts.blogspot.com/2010/12/plural-of-anecdote-is-anecdotes-not.html

  16. BillyJoe7on 11 Dec 2010 at 3:33 pm

    “For every anecdote, there is an equal and opposite anecdote”
    “The plural of anectode is not data”

    Anecdotes are useful for generating hypotheses but useless for testing hypotheses.

  17. zoe237on 12 Dec 2010 at 11:54 pm

    I agree with most of what you’ve written, that anecdotes and anomalies are considered in generating hypotheses. I don’t think, however, that you’ve proven that they aren’t used at all in the later justification. Many many research studies are nothing but a collection of anecdotes, particularly non rcts. We *attempt* to collect these observations methodically, but science is biased, there is no doubt in my mind about that. Confirmation bias is common every step of the way, not just the first, and subject to file drawer effect. I do agree that it is the most unbiased way of thinking that we currently have, but to ignore the human aspect of science is folly. Scientists must always be aware that they are human and subject to fallacy. Anomalies are usually found and measured via standard deviation and we attempt to explain them away. Sometimes anomalies accumulate until we DO need a new paradigm. So I’m unconvinced that anomalies and anecdotes are never a part of the later steps of the scientific method. Or maybe I’m missing the point?

Trackback URI | Comments RSS

Leave a Reply

You must be logged in to post a comment.