Science only works when it works.

In other words – science itself does not lead to an understanding of the universe unless that science is done correctly, rigorously, and honestly. This is a lot harder than I think is generally appreciated. In order to really reach firm scientific conclusions about any complex question we need to follow the arch of the research as it matures. We need to see what overall patterns emerge in the evidence. Eventually a tentative but reliable scientific consensus can be achieved.

There are many ways in which this process can go off the rails, however. With ESP we see researchers chasing the noise – trying to find tiny signals but only chasing their tails. With acupuncture we see proponents choosing to ignore, misinterpret, and then abandon well-controlled clinical trials in favor of “pragmatic” studies that will show them what they want. There is “cargo cult” science that goes through the superficial motions but lacks true scientific methodology. There is “Tooth Fairy” science that nibbles around the edges but never addresses the core premise – is the phenomenon actually real?

There is a huge positive bias in science – researchers have a tendency to tweak their methods to get the results they want, publishers have a tendency to publish positive exciting research, and other scientists have a bias toward citing positive interesting research. Funding sources affect research outcome. When pharmaceutical companies fund research the results are much more likely to be favorable to their drug than independent research. Scientists make mistakes, take shortcuts, and often have blinders on. And then there is outright fraud, which is uncommon but still crops up on a regular basis.

Somehow, through all of this we manage to grind slowly forward. Because of the nature of science, reality has an effect. It is a slow wind gently blowing us in the right direction, despite our efforts to find the answers we want rather than the actual truth.

Anti-vaccine Pseudoscience

The latest example of how bias can distort the process of science comes from the anti-vaccine movement. Another study from researchers Christopher Shaw and Lucija Tomljenovic has now been retracted. These researchers, darlings of the anti-vaccine movement, seem to have a dedicated anti-vaccine mission. They apparently are not doing studies to find out if vaccines cause any problems, but to prove that they do. Real science, of course, doesn’t work that way.

As usual Orac has all the details. Last month their paper was published allegedly showing that injecting aluminum adjuvant, sometimes used in vaccines, activated immune regulating genes that they claim are “homologous with biomarkers of autism.”

The paper was quickly criticized on several grounds. First, their line of reasoning is paper thin, making leaps of inference that are not justified by the data. There were also many methodological complaints, some small (like injecting subcutaneously rather than intramuscular) and others devastating. They were accused of using outdated genetic data and genetic tests, and inappropriate statistical analysis.

Further, as Orac points out, their study was a “fishing expedition.” This is yet another type of problematic science in which researchers hunt around for any correlation. This is a reasonable type of study to do in preliminary exploratory research, but any findings are dubious because you are basically data mining. Results of exploratory research should only be used to generate hypotheses, not to test them. Findings need to be confirmed by later studies designed to specifically test the alleged correlation.

But a lot of exploratory research, especially in service to an agenda, is not even legitimate as far as it goes. The biggest reason is that the statistical analysis does not account for multiple comparisons, and so what is presented as statistically significant really isn’t.

So the Shaw and Tomljenovic paper was fatally flawed, and in line with their previously established anti-vaccine bias. But there is an update – when an online journal club, PubPeer, looked at the paper they found convincing evidence of fraud. Images used to represent data in the paper appeared to have been duplicated and manipulated.

This prompted the journal editor to do their own review, and they concluded:

Our own analysis showed some figures had been altered. We requested a retraction because we could not understand how that had happened. We felt the data had been compromised.

Really – they could’t understand how it happened? I guess they need to be circumspect, but that struck me as an odd way to say it. I can understand how it happened (using the passive voice). It’s called fraud. That at least should be the working hypotheses until another viable explanation is offered with evidence.

At least the paper has now been retracted (another retraction for these authors). But, as I discussed yesterday on Science-Based Medicine, even retracted papers can live on in an undead “zombie” state. This is what happens within dedicated anti-science echochambers. The Shaw-Tomljenovic papers are evidence for anti-vaxxers, and criticism and even retraction of the studies are just evidence of a cover-up. We see the same thing happen in the anti-GMO community with Seralini’s studies.

That is the real lesson here, and it goes way beyond this one crappy study. Science has to be transparent and communal. Different groups can’t have their own science that says what they want it to say.

Science only works when it works.