I was recently pointed to an excellent article by an Irish journalist, Jonathan McCrea, about homeopathy. He discussed the topic on his radio show, and was highly critical of the practice. In response a homeopath wrote in a letter complaining that the shows was too negative, and McCrea wrote a nice reply pointing out the scientific evidence. He quoted science-based medicine extensively, as well as systematic reviews.

It’s wonderful to see a science journalist who is not a scientist themselves do such a great job. As he explained himself in the reply, a good journalist will reflect the consensus of expert opinion. He also pointed out the dangers in medical nonsense and clearly understands his duty as a journalist when reporting on such issues – so good for him.

Here is the money quote from the homeopath:

What I would like to point out to your presenters is that never, in all their dismissal of homeopathy, do they seem to take into consideration the homeopaths who practice this particular brand of ‘heresy’ (heresy according to their own particular view of the world that is). If it is nonsense as they claim, then we are all either stupid, deluded or charlatans who take people’s money for something which we know does nothing. I may be many things but I am neither stupid, deluded nor a charlatan as anyone who knows me will verify. Neither are those thousands of doctors and vets who have changed from using conventional medicine to homeopathy. No homeopath or user of homeopathy comes to it from ‘belief’. We come to it from trying it, sometimes from curiosity, but often in desperation when other things haven’t worked. Then when we see it works we try it again. Then it works again and we try it more often, and it keeps working. Not every time, any more than antibiotics, or any other medicine works every time. But enough, more than enough to make us trust it.

This is a typical response – the appeal to anecdotal evidence. Clearly this homeopath does not understand the scientific position. She sets up an interesting forced choice straw man, that critics think homeopaths are either stupid, deluded, or charlatans. First, these are not mutually exclusive states. But then she equates these possibilities to using treatments which “we know does nothing.” That is a non-sequitur, however. If a homeopath is stupid or deluded then they don’t know that what they are using does nothing.

What they leave out is the actual scientific skeptical position – that homeopaths are people. People are subject to a host of biases, such as confirmation bias. They can be fooled by placebo effects, without being stupid or especially prone to delusions. There is a strong tendency for uncontrolled observations to lead to incredibly biased conclusions, creating the false impression that a worthless treatment works. The only way to really know if a treatment works (aside from obvious cases, like using parachutes when jumping out of planes) is to perform careful efficacy trials. When homeopathy is studied in such trials, the overall result is that homeopathy does not work for any indication.

This is now where depression enters the story – in response to the claim that the scientific literature shows that homeopathy is indistinguishable from placebo, it is common for proponents to reference one or a few studies, or to cherry pick studies, in order to refute the claim. It is as if they feel that if they can point to a single study showing that homeopathy works, then homeopathy works and the skeptics are wrong. The latest study being put forward as single proof of homeopathy is this study of Individualized Homeopathic Treatment and Fluoxetine for Moderate to Severe Depression in Peri- and Postmenopausal Women.

Before I get into the study itself, it is important to review what we mean when we say the the scientific evidence demonstrates efficacy. There are two ways of answering this question, one is the evidence-based medicine (EBM) approach and another is the science-based medicine (SBM) approach. EBM essentially looks only at the clinical evidence, while SBM considers the clinical evidence in the context of scientific plausibility.

A reasonable EBM threshold for concluding that a treatment works is that multiple high quality and independent clinical trials show a statistically significant and clinically significant effect. A key component there is independent replication. SBM also requires the same level of evidence, but would take a Bayesian approach – given the overall scientific plausibility of the treatment, is the clinical evidence sufficiently robust to conclude that the treatment probably works?

To be clear – homeopathy fails the lower EBM threshold as well as the SBM threshold.

What about the new study? On paper, the methods seem very solid. It is actually a well-designed study and they did not fall prey to many of the methodological flaws that plague the homeopathy literature. There are limitations, however. The study is relatively small, randomizing 133 subjects to three treatment arms. The duration of the study is also relatively short, only six weeks. The outcomes are also all subjective reporting. While the described blinding seem adequate, there was no independent assessment of blinding efficacy (for example, asking subjects if they thought they were receiving treatment or placebo).

The results showed that homeopathy and fluoxetine were better than placebo in one depression scale, the Hamilton scale, but there were no differences in the Beck Depression Inventory. The results also show improvement in the homeopathy group, but not fluoxetine, in the Greene Climacteric Scale of peri-menopausal symptoms.

The disconnect between the Hamilton scale and Beck depression scale is interesting, and does weaken the results significantly. The Hamilton scale focuses more on somatic symptoms, while the Beck Depression Inventory is considered as having a better emphasis on symptoms of depression itself.

The study, while showing some positive results and generally being well-designed, is certainly not a home run. There are also few other studies of homeopathy in depression. A 2011 systematic review found no clinical trials. A 2013 study by Adler et al had to be terminated early and showed weak and mixed results.

Conclusion

While it is likely that this study will be put forward by homeopaths as evidence that homeopathy works, it is actually a small study with mixed results. Even by the looser EBM standard, this is not sufficient to conclude efficacy. To put this into context, the FDA would never approve a drug based upon this level of evidence.

The SBM standard would go a bit further, stating that this amount of clinical evidence has an insignificant effect on the near-zero prior plausibility of homeopathy. In this study they used either 30C or 200C dilutions of individualized ingredients – meaning that the homeopathic pills given to patients were indistinguishable from the placebo pills. When testing placebo vs placebo if you find a difference, that does not indicate that one placebo was actually more effective than the other. Occam’s razor would favor the simpler interpretation of random results or perhaps there were methodological issues.

Further, single studies are rarely compelling. Even when a study looks pristine on paper, we cannot know (unless the study was directly monitored) if the blinding was adequate, or if any researcher degrees of freedom were exploited but not reported. Simmons et al demonstrated what scientists already know – that exploiting a little wiggle-room in methodology can easily manufacture positive P-values (referred to as p-hacking). The most effective control for p-hacking is independent replication.

Therefore, when advocates of SBM state that the evidence does not support the conclusion that homeopathy works for anything, we mean that there is no single indication for which there are multiple independent studies replicating clear positive results. We have either negative studies, fatally flawed studies, studies with mixed results, or one-off studies without replication. Given the extreme scientific implausibility of homeopathy, it still remains the best conclusion we can make that homeopathy almost certainly doesn’t work.  This conclusion can never be refuted by a single study.