The most succinct criticism of postmodernist philosophy as applied to science that I have heard is this – that proponents confuse the context of discovery with the context of  later justification. It occurred to me that the same is true of the role of both anecdotes and anomalies in science. Often when I criticize reliance on anecdotes or so-called anomaly hunting, I get feedback that makes the exact same confusion of context.

The context of discovery refers to how new ideas are generated in science. Playing off of Thomas Kuhn’s work on paradigms (and without getting into a side discussion of Kuhn’s own position), some post-modernists argued that science is a humanist-type of endeavor because scientists come up with their ideas in quirky and culturally contingent ways, rather than rigorous or methodical ways.

However, what makes science methodologically rigorous is not how new ideas are generated (the context of discovery) but how they are tested (the context of later justification).

Anecdotes

Anecdotes are uncontrolled subjective observations. I have often criticized reliance on anecdotes, which is especially problematic in medicine. The problem with anecdotes is that they are subject to a host of biases, such as confirmation bias. They are easily cherry picked, even unintentionally, and therefore can be used to support just about any position. For every anecdote, there is an equal and opposite anecdote.

Because anecdotes can be used to support any position, in reality they support no position – they have no predictive value. As they are often used, they are worse than worthless because they are misleading – they have a tendency to support our biases and give the false impression that our biases have merit.

I have often repeated the above criticism of reliance on anecdotes to form conclusions. And just as often it is pointed out that anecdotes are useful, in medicine and elsewhere, in pointing the way to new possible treatments or phenomena. This is true, but confuses the context of my criticism in the same exact way that post-modernists confuse the process of science.

Anecdotes are useless and misleading in the context of scientific justification – as a way of testing hypotheses. But that is often how they are used, especially in the marketplace and the arena of public opinion where I focus much of my criticism.

In the context of discovery, however, they can be useful. Many medical discoveries started as anecdotal observations. But then those observations have to be tested with controlled observations or experiments – and most anecdotal observations will turn out to be wrong or misleading, because they are quirky and uncontrolled.

In short – anecdotes are useful for generating hypotheses, but not for testing them. Problems arise when anecdotes are used to support a hypothesis or claim, rather than just to raise a possibility to be tested.

Anomaly Hunting

Similarly, anomalies are useful in the process of science to point the way to deeper understanding. An anomaly is some fact or observation that cannot be explained (to use Kuhn’s term) within the current paradigm. Anomalies are vital to scientific progress – they are a giant sign with an arrow pointing and reading “scientific discovery to be made here.”

There are countless examples in the history of science – the orbit of Mercury showed anomalies within Newtonian mechanics and pointed the way toward General Relativity, for example. Today we have anomalies in the speed of distant probes that may indicate that further revisions to gravitational theory are needed.

But just as with anecdotes, anomalies are the beginning of discovery and of the generation of new hypotheses. But by themselves they are not evidence for any particular conclusion.

When an anomaly is encountered in science the first step is to make sure it is an actual anomaly. The simplest answer is always that there is an error in measurement and observation, so that has to be ruled out first. When the measurements are all triple-checked and scientists can be as certain as possible that the anomaly is real, then all possible explanations need to be sought within our current scientific theories or paradigms. Only when an exhaustive attempt to explain the apparent anomaly within conventional theories fails, should scientists conclude that they have a genuine anomaly on their hands.

Then scientists can start to generate new hypotheses to explain the anomaly, usually requiring a revision to existing theories. Kuhn characterized this phase as “revolutionary science” in which one paradigm will shift to another in order to resolve the anomalies. He thought such shifts were large and rare. But later philosophers criticized this position as being a false dichotomy – revisions to theories in order to resolve anomalies are actually quite common and range the spectrum from minor tweaks to wholesale replacement with no sharp demarcation at any point.

Further, often revisions to theories involve not replacing old ideas but deepening them – adding layers of complexity that are compatible with the simpler theories but increase their precision and predictive power. Again – a perfect example is the progress from Newtonian mechanics to relativity. Newtonian mechanics still work in most situations, but at high velocities and gravities relativistic effects become important and measurable.

The role of anomalies, therefore, is entirely within the context of discovery and generating new hypotheses. Anomaly hunting refers to the misapplication of anomalies to the testing of hypotheses. The process involves generating a hypothesis then pointing to apparent anomalies as if their existence supports the specific hypothesis.

At best the mere presence of anomalies would indicate weaknesses in the current theory, but would not support any particular new hypothesis. That would require demonstrating that the new hypothesis resolves the anomalies. Even then this would only indicate that the new hypothesis is plausible and interesting – but it would need to make predictions that are later validated in order to be truly scientific.

Often anomaly hunting is far worse than just misusing anomalies because it often entails pointing to apparent anomalies and prematurely concluding that they are genuine. This comes up most often in conspiracy thinking. Conspiracy theorists often take a complex historical event, then hunt for apparent coincidences or quirky events that are not easily explained and treat them as if they are genuine anomalies – and therefore evidence for a conspiracy.

There are a number of logical fallacies inherent in this process, such as confusing currently unexplained with unexplainable. There is also a false major unstated premise – that if there were no conspiracy at work, then all aspects of the historical event would be easily explained to arbitrary detail and precision. Therefore, the inability to explain every aspect of the event in great detail is treated as an anomaly, rather than just the natural and expected result of the chaos of complex and contingent historical events and human behavior.

For example, 9/11 truthers will point to apparent anomalies in the debris of the jet crash into the Pentagon as evidence for a conspiracy. The unstated premise is that a casual observer would have a good instinctive idea of what such a debris field should look like.

Moon Landing hoax conspiracy theorists look for apparent anomalies in photos and film from Apollo and declare anything they cannot immediately explain as evidence for a hoax. JFK conspiracy theorists look for quirky action on the part of Oswald (as if human behavior is predictable and logical) and point to anything that seems funny as evidence of a conspiracy.

These are all examples of anomaly hunting – prematurely assuming that an apparent anomaly is genuine and then concluding that the anomaly supports a specific hypothesis (an argument from ignorance logical fallacy).

Conclusion

Context is king. With regard to the history of science, the use of anecdote, and the use of anomalies in science play an important role in the context of discovery and the generation of new ideas. But they are problematic and misleading in the context of scientific validation, which relies upon rigorous methodology.

I also find that the very concept of context itself is often overlooked. I have found this to be one of those generic and invaluable intellectual skills, or metacognition (as psychologists call it) – thinking about thinking. It is a common error to fail to consider the context in which one is making an argument – are the proper rules being used in the proper context?

Failure to consider context leads to hopeless confusion – confusion that cannot be resolved until the concept of context is addressed.