Mar 26 2013

Debating Homeopathy Part II

Yesterday I discussed a recent debate in which I participated at UCONN, focusing on the plausibility of homeopathy. Today I will discuss the clinical evidence, and address some of the strategies employed by my opponent in the debate, Andre Saine.

Does Homeopathy Work?

Yesterday I made the case that homeopathy is highly implausible in many ways, and after two hundred years of scientific advance this extreme implausibility has only become greater. Two centuries has apparently not been enough time for homeopaths to make their case and convince the mainstream scientific community. The only reasonable explanation for this is that homeopathy is simply not valid.

I also took the position that overall scientific plausibility must be considered when looking at any new claim – how well does it comport with existing scientific evidence? In medicine this means, when considering clinical evidence for a treatment, that evidence needs to be put into the context of the scientific plausibility of the treatment.

Given homeopathy’s extreme implausibility, it would take a great deal of iron-clad scientific evidence to convince most scientists that there was a real effect.

However, the clinical evidence for homeopathy is so clearly negative, that even if we set aside plausibility arguments entirely we could still confidently conclude that homeopathy does not work. In other words, the clinical evidence regarding homeopathy is negative using any reasonable threshold, and not adjusting for the extreme implausibility.

Systematic Reviews

Individual studies are not a good way to understand the evidence base for any medical claim. Differences in methodology, rigor, and statistical fluctuations will ensure a random scatter of results. Meanwhile, researcher bias and publication bias will tend to bias that random scatter toward the positive end. For this reason, if you cherry pick individual studies on any topic that has been widely studied you can support any position you choose to take. Only by systematically looking at all the evidence for the overall pattern can you see what the science is really saying.

There are several ways to do this: systematic reviews are just that – they look at all published studies on a given question or topic and then come to a conclusion about what that evidence says.  One pattern to look for is the relationship between the quality of the study and the outcome. You can assess heterogeneity – do high quality studies tend to agree, or are the results all over the place? A systematic review may include a funnel-plot analysis – a measurement of publication bias.

Another way to look at multiple studies is a meta-analysis, which combines data from multiple studies into one new statistical analysis. This can be combined with a systematic review.

Finally there is a best-evidence synthesis, which is similar to a systematic review but emphasizes a detailed analysis of the best studies, including effect sizes, and the relationship between rigor and outcome. (It sounds very similar to the four criteria of compelling evidence I outlined in yesterday’s post.)

So – what do systematic reviews tell us about homeopathy? The most comprehensive such review was conducted by Edzard Ernst in 2010, a systematic review of systematic reviews, and he concluded:

The findings of currently available Cochrane reviews of studies of homeopathy do not show that homeopathic medicines have effects beyond placebo.

A 2000 review came to a similar conclusion:

There is some evidence that homeopathic treatments are more effective than placebo; however, the strength of this evidence is low because of the low methodological quality of the trials. Studies of high methodological quality were more likely to be negative than the lower quality studies. Further high quality studies are needed to confirm these results.

While the author tried to put a positive spin on the result, the penultimate line is where the money is – high quality studies were more likely to be negative than lower quality studies.

The evidence is clear and overwhelming, after years and hundred of trials, studies of homeopathy have been unable to generate convincing evidence that homeopathy works for anything. Different reviewers can spin the results in different ways depending on their biases, but the evidence is the same – no consistent, replicable, clinically significant effect seen in the highest quality studies.

Saine Responds

Saine echoed the standard response from the homeopathic community. First he played the “no true Scotsman” card, claiming that most studies of homeopathy are not really studies of homeopathy, but of “homeotherapeutics.” Only individualized treatment by a homeopath is true homeopathy, not the standardized treatment in most clinical trials.

The results of individualized homeopathy, however, are similar to that of standardized homeopathy. A recent review concludes:

The results of the available randomized trials suggest that individualized homeopathy has an effect over placebo. The evidence, however, is not convincing because of methodological shortcomings and inconsistencies. Future research should focus on replication of existing promising studies. New randomized studies should be preceded by pilot studies.

By now you can recognize the positive spin of a proponent and look past it – the bottom line is that the evidence is “not convincing.”

Saine also tried to dismiss the Ernst review by making an ad hominem attack. First he stated the straw man premise that I rely heavily on Ernst for my opinions (not true – I am capable of assessing the literature on my own, thank you), and then he character assassinated Ernst, saying he is not a “real” homeopath and therefore doesn’t know what he is doing. The fact that Ernst was the first professor of complementary and alternative medicine and has over 1000 published studies to his credit apparently did not impress Saine.

In any case – the evidence is what it is. Ernst is simply reviewing the evidence. His findings are also no different than anyone else, only the language used in the conclusion is different based on author bias.

At one point in the debate I was able to ask Saine, putting aside all his special pleading excuses,  if he thinks there is any indication for which homeopathy has been adequately proven to be effective. He cited the Jacobs 2003 review of three clinical studies of homeopathy for diarrhea. Jacobs concluded:

The results from these studies confirm that individualized homeopathic treatment decreases the duration of acute childhood diarrhea and suggest that larger sample sizes be used in future homeopathic research to ensure adequate statistical power. Homeopathy should be considered for use as an adjunct to oral rehydration for this illness.

However – other reviewers (Altunç U, Pittler MH, Ernst E.) looked at the same studies and concluded:

The evidence for attention-deficit/hyperactivity disorder and acute childhood diarrhea is mixed, showing both positive and negative results for their respective main outcome measures.

Further, a 2006 study by Jacobs of homeopathic combination therapy was entirely negative.

So – Saine referred to the 2003 review that was positive, despite the fact that other reviews of the same data were negative, and a high quality follow up study by Jacobs was negative. This was his best evidence for homeopathy, which supports my point – homeopathy has not been adequately demonstrated to work for anything.

Saine also spent a great deal of time quoting from epidemiological evidence, historical cases of homeopathy used to treat epidemics (mostly from the 19th century) and similar data. This is all, of course, uncontrolled and unblinded data.

The pattern is clear – when the rigorous, blinded, controlled data is negative, turn to the unblinded and uncontrolled data. That is the pattern of pseudoscience, used to support a claim that is not true.

Conclusion

There is a great deal more detail that I could go into, and Saine and I might engage in a written exchange to follow up our live debate. This would be a great opportunity to delve deeply into the research of homeopathy.

On every point proponents fail to make the case for homeopathy. It remains extremely implausible, and even setting aside that implausibility, the clinical evidence is negative. The pattern of results that we see are consistent with the null hypothesis – a scatter of results, a positive bias in the preliminary studies, but as methodological rigor increases studies tend to become negative. The evidence is a giant arrow pointing at the null hypothesis – homeopathy does not work.

The basic science and clinical evidence for homeopathy has not come anywhere close to the four criteria I outlined for compelling scientific evidence – statistically significant outcomes with adequate signal to noise ratios in high quality studies that survive replication.

What we have instead are excuses, special pleading, appeals to low grade evidence, some conspiracy mongering and bashing of mainstream medicine, ad hominem attacks, and other logical fallacies.

And yet Saine marvels at how skeptics can remain skeptical.

 

19 responses so far