Feb 22 2016

The Evidence Says – Homeopathy Does Not Work

airguitarIn a recent blog post for the BMJ, Paul Glasziou wrote about the recent Australian review of homeopathic remedies of which he was head:

…I lost interest after looking at the 57 systematic reviews (on 68 conditions) which contained 176 individual studies and finding no discernible convincing effects beyond placebo.

He is not the first person to look at the totality of clinical evidence for homeopathy and find it wanting. Glasziou was chair of the working party that produced the 2015 NHMRC report on homeopathy, which concluded:

Based on the assessment of the evidence of effectiveness of homeopathy, NHMRC concludes that there are no health conditions for which there is reliable evidence that homeopathy is effective.

So, after more than two centuries, and thousands of studies in total, no homeopathic treatment has crossed over the line of what would generally be considered sufficient evidence to prove that it works. That is very telling. I liken the evidence to other dubious claims, such as ESP. After a century of research and thousands of studies there is no clear evidence that ESP is real.

For both homeopathy and ESP there is a great deal of noise, but no clear signal. There are many flawed or small studies, but no repeatable high quality studies.

This lack of convincing evidence also has to be looked at in the context of the scientific plausibility of homeopathy (or ESP). Homeopathy’s scientific plausibility approaches zero.

To quickly summarize: homeopathic potions are diluted to the point where there is little to no “active ingredient” remaining. Homeopaths have to argue that the original substances leave behind their “essence,” which is just covering one highly implausible claim with another.

Further, the “active ingredients” themselves are based upon pure magical thinking, including the false notion that like cures like, which is little more than sympathetic magic, and even more fanciful notions about the relationship between personality and illness. I like to say that homeopathic treatments are essentially fairy dust diluted into non-existence. Others have called homeopathy the air-guitar of medicine, which is also an apt analogy.

When we combine maximal scientific implausibility with weak evidence, in fact convincing evidence for lack of efficacy, there is only one reasonable conclusion to reach – homeopathy does not work, and in fact, as far as we can tell, it cannot work.

Despite this, Glasziou, who is a professor of evidence based medicine (EBM), went into his review with an open mind. I find that interesting, as that is a premise of EBM, to not consider prior plausibility and just look at the evidence. So he did a real EBM review – and found that there isn’t a single indication for which there is sufficient evidence to claim that homeopathy works.

Unsurprisingly, Dana Ullman made an appearance in the comments to Glasziou’s blog post in order to represent the “science denial” position of the die-hard homeopath (as he often does). Ullman had two points to make: 1) the threshold for evidence in the NHMRC review was too high, 2) and that mainstream medicine does not fare well under the same strict standards. Both of these claims are dubious.

When someone is arguing for lowering the standards of evidence, that is a huge red flag. That is rarely a strong position. Ullman complains that the NHMRC report set as their threshold three independent studies, each with at least 150 subjects total. To me, that is a perfectly reasonable standard.

Independent replication is the key to science. There are numerous biases which can affect the outcome of studies – researcher bias, publication bias, citation bias, researcher degrees of freedom, and even occasional fraud. We only know if something is really real if the effect can be independently replicated. This point cannot be emphasized enough.

I have written before, in fact, that before a new phenomenon should be deemed probably real we should expect evidence that simultaneously meets the following thresholds:

1 – Rigorous trial design
2 – Statistically significant positive results
3 – Adequate signal to noise ratio
4 – Independent replication

Three out of four are not enough. You can get three out of four with a fake treatment or a phenomenon that is not real. It is difficult to get all four unless the phenomenon is real.

Further, these are general principles. Where exactly we draw the line is a judgment call. It is partly based on an understanding of the history of science – in the past, where is the line that has reliably indicated that a phenomenon is real? It is also partly based on scientific plausibility. The bar is higher for claims that represent a greater dissonance with existing evidence. Extraordinary claims require extraordinary evidence.

These rules reflect a genuine desire to know what is real, rather than a genuine desire to prove that one’s pet theory is true.

The NHMRC review and other reviews have concluded that no homeopathic remedy has crossed this line for any indication. You can quibble about exactly where to draw the line, but honestly homeopathy is not even close. You would have to abandon all reasonable standards of evidence in order to argue that the evidence supports homeopathy – which is exactly what Ullman does.

Ullman’s second claim is that if you apply the same standards to mainstream medicine it also does not fare well. He cites as his primary source for this claim a BMJ clinical evidence review, which Ullman summarizes:

In fact, when the BMJ’s “Clinical Evidence” analyzed common medical treatments to evaluate which are supported by sufficient reliable evidence, they reviewed approximately 3,000 treatments and found only 11% were found to be beneficial.

Let’s take a closer look at this review. They found that 11% met the highest category of “beneficial” but 24% were in the “likely beneficial” category. Further, 7% were in the category – trade off between benefits and harm. This does not mean they are unproven, just that they have side effects that represent a significant trade off.

Only 8% were unlikely to be beneficial or likely to be ineffective. That is the category, by the way, in which I would place homeopathy.

But here is the biggest omission of Ullman in summarizing this review:

‘Unknown effectiveness’ is perhaps a hard categorisation to explain. Included within it are many treatments that come under the description of complementary medicine (e.g., acupuncture for low back pain and echinacea for the common cold), but also many psychological, surgical, and medical interventions, such as CBT for depression in children, thermal balloon ablation for fibroids, and corticosteroids for wheezing in infants.

In the biggest category of “unknown effectiveness” (50%) the review included CAM treatments like herbs and acupuncture. It is therefore incredibly dishonest to portray these numbers are representing mainstream medicine.

Even further, the authors report that these percentages represent just the number of treatments they looked at, not the frequency with which they are used. In other words, it is possible that most practitioners largely stick within the 42% (higher if you exclude CAM therapies) of treatments for which there is reasonable plausibility and evidence of efficacy.

In fact, that is standard practice – to follow a hierarchy of evidence, starting with the safest and most evidence-based treatments first, and then working your way down only when such treatments have failed.

Ullman is also cherry picking. One review found that on average such reviews conclude that 76% of treatments are reasonably evidence-based. This is admittedly a complex question and you can address it in many ways. How often do practitioners follow strict published practice guidelines, for example?

Keep in mind, those of us at Science-Based Medicine and others in the medical community are arguing that we need to be raising our baseline threshold for evidence. We need to more thoroughly consider plausibility, and if anything raise the bar for accepting new treatments. There is increasing recognition of the systemic biases in published medical research, and more than ever we need to dedicate our profession to rigorous standards of evidence.

In the midst of this Ullman and others want to lower the standards of evidence, all so that they can admit their preferred unscientific treatment.

Conclusion

The evidence still shows that homeopathy does not work. Even if you take a strict EBM approach and do not consider scientific plausibility, the evidence does not support the conclusion that homeopathy works for anything. There is not one indication for which any homeopathic treatment has been reliably shown to work, if you use a fair and reasonable threshold of evidence.

And of course, homeopathy is as scientifically implausible as you can get. It is essentially magic. Homeopathic products are magic potions that have far more in common with a witches brew than a scientific remedy.

23 responses so far

23 Responses to “The Evidence Says – Homeopathy Does Not Work”

  1. mlegoweron 22 Feb 2016 at 8:28 am

    Find/Replace “judgement” w/ “judgment”.
    http://www.merriam-webster.com/dictionary/judgement

  2. mlegoweron 22 Feb 2016 at 8:33 am

    For something more substantive, I imagine medical interventions follow something close to the Pareto principle where 20% of the interventions account for roughly 80% of the case volume. If this is the case, then the fact that 35% of the interventions studied were likely to be beneficial means that probably something like 90% of patient-treatment pairs are supported by good evidence. That’s pretty good.

  3. carbonUniton 22 Feb 2016 at 9:37 am

    Nice summation of the homeopathy evidence gap. In other news, water is still wet. 😉

    I can’t make ‘head’s or tails of the opening sentence. Is ‘of which he was head:’ some sort of slang?
    2) and that mainstream medicine does not fair well UNDER the same strict standards. ?

    Steve, hope you don’t mind proof-reading along with the comments. I stand in awe of the amount of data you consume and analyze to produce these fantastic blog entries for us!

  4. mumadaddon 22 Feb 2016 at 9:47 am

    Also it’s *fare* well, not fair well. 🙂

  5. mlegoweron 22 Feb 2016 at 10:03 am

    I actually didn’t think it was confusing at all. But in the interest of making it as clear as possible, “…of which was head” is avoiding the use of a preposition at the end of a sentence. It may be more clear (but arguably less correct) if he had written:
    “In a recent blog post for the BMJ, Paul Glasziou wrote about the recent Australian review of homeopathic remedies which he was the head of:”

    Or just avoid it altogether:
    “The head of the recent Australian review of homeopathic remedies, Paul Glasziou, wrote the following in a recent blog post:”

    This is the first post of my new blog “Plagiarizing Neurologica” where I just slightly rephrase Steve’s posts and then claim credit 🙂

  6. mlegoweron 22 Feb 2016 at 10:06 am

    (Of course, my pedantic post would also contain a typo. “…of which was head” should obviously read “…of which he was head”)

  7. hardnoseon 22 Feb 2016 at 10:06 am

    “Ullman complains that the NHMRC report set as their threshold three independent studies with at least 150 subjects total. To me, that is a perfectly reasonable standard.”

    No, this is what Ullman said:

    “unless there were at least three studies conducted by three separate groups of researchers, with each study having over 150 subjects, the results were deemed to be “unreliable.””

    There is a big difference between “150 subjects total” and “each study having over 150 subjects.”

    Experimental studies do not normally have such a large number of subjects. Ullman’s complaint is legitimate if the reviewers arbitrarily decided to exclude good studies that have been replicated, just because they have a normal number of subjects.

  8. Steven Novellaon 22 Feb 2016 at 10:08 am

    I don’t mind the copy editing. That is the one thing I usually don’t have a lot of time to do, although sometimes I get to it later in the day. I have, at times, had volunteers doing copy editing post publishing, but that has variable availability.

  9. Steven Novellaon 22 Feb 2016 at 10:09 am

    HN – that is what I meant, but I see now that it was ambiguous. Studies often have multiple arms, so we often refer to total subjects in the study, as opposed to the number of subjects in each arm. I will clarify it.

    And his criticism is not fair. Essentially the review was looking at the best evidence. If the best evidence is negative, who cares what less rigorous evidence showed (unless you have an agenda).

  10. Steven Novellaon 22 Feb 2016 at 10:36 am

    Also, I could have gone into far more detail here, please read the methods section of the review: http://www.nhmrc.gov.au/_files_nhmrc/publications/attachments/cam02a_information_paper.pdf

    Ullman is not being even correct in his characterization. If a study had <50 subjects it earned a -2 on its precision score, 51-149 was -1 point, and 150 or greater, no penalty. There was also a rigor score. The total score was then considered in grading the quality of each study, using an internationally accepted and standardized system.

    So - Ullman is simply wrong when he says that an absolute cut off of 150 subjects was required, or that this was somehow unfairly rigging the game against homeopathy.

  11. civilon 22 Feb 2016 at 10:50 am

    ‘ It may be more clear (but arguably less correct) if he had written:
    “In a recent blog post for the BMJ, Paul Glasziou wrote about the recent Australian review of homeopathic remedies which he was the head of: … ‘

    The Chicago Manual of Style (on which many publications’ style sheets are based) has said it’s really not necessary to avoid ending a sentence or clause on a preposition. Unfortunately, the news hasn’t filtered down to the average reader, and the writer risks looked uneducated in doing so.

  12. mlegoweron 22 Feb 2016 at 10:52 am

    Hence the “arguably” 😉

  13. steve12on 22 Feb 2016 at 12:01 pm

    It is fascinating that people still believe in this despite the state of the evidence, but then will fight tooth and nail against probably the greatest (in terms of of loves saves) medical treatment of all time – vaccines.

  14. carbonUniton 22 Feb 2016 at 12:24 pm

    mlegower – thanks, I just wasn’t reading it correctly. ‘of which he was head’ does make sense.

  15. hardnoseon 22 Feb 2016 at 12:39 pm

    It makes no sense to consider studies with N < 150 as being of inferior quality. If a result is statistically significant and there was a reasonable number of subjects (and much less than 150 would be considered reasonable by most researchers), then there is no reason to discount or minimize their importance.

    The fact that they did makes me wonder how objective they were being.

    Maybe they tried the meta-analysis with the smaller studies included and didn't like the results they got. There are many ways to make a meta-analysis come out the way you want it to.

  16. Steven Novellaon 22 Feb 2016 at 12:59 pm

    HN – this was not a metaanalysis. This was a systematic review of systematic reviews, combined with a review of studies that were referred to them.

    Taking one point off for having fewer than 150 subjects is perfectly reasonable. Statistical significance is not everything, as any serious medical researcher can tell you. Also, 150 total could be 50 per arm for three arms, or 75 per arm for two arm studies. Fifty subjects per arm is a reasonable rule of thumb to be considered an adequately sized clinical study.

    They weren’t discounting the smaller studies entirely, they were just weighting them less. That is the standard for systematic reviews. If you don’t weight the studies then you end up with the large number of smaller studies overwhelming the few larger studies, when the opposite is what you want.

    This is all basic and completely standard stuff. Nothing new. Ullman is trying to weave this into a conspiracy and you appear to be buying it.

  17. jsterritton 22 Feb 2016 at 1:27 pm

    Hmmmm, weighting studies for statistical power and rigor — sounds like a trick to me!

    🙂

  18. mumadaddon 22 Feb 2016 at 1:31 pm

    “Ending a sentence with a preposition is something up with which I will not put.” (Attributed to Winston Churchill.)

  19. Willyon 22 Feb 2016 at 1:39 pm

    Ya gotta love Winston Churchill!

    mumadadd-You mentioned a Sam Harris podcast a few weeks ago. Thanks for that; I am enjoying him very much.

  20. mumadaddon 22 Feb 2016 at 1:55 pm

    Willy — excellent, glad to have introduced you to something you like!

  21. hammyrexon 22 Feb 2016 at 4:12 pm

    I always find the special pleading argument highly amusing for homeopathy, because it would only be consistent that their same reduced standards be applied across the board for all treatments. Ullman’s speech about “look, these are also really great studies, I promise!!” could just as easily be imposed on a faceless pharmaceutical drug representative trying to pimp a new me-too drug.

    It does get to the heart of why a lot of us are so frustrated with alternative medicine – the standards for what passes for evidence already isn’t as great as it could be. I’m terrified of lowering them even more. Of course, alternative medicine advocates like Ullman don’t see this massive issue because they don’t have the intellectual honesty to recognize it’s special pleading to begin with – they want lower standards, but just for pet project. People can very easily rationalize why their belief is special, but they don’t respect the consequences of arbitrarily changing standards when it comes to medical science as a whole or a meaningful philosophy of science we share as a culture.

  22. jsterritton 22 Feb 2016 at 4:37 pm

    @hammyrex

    “I always find the special pleading argument highly amusing for homeopathy, because it would only be consistent that their same reduced standards be applied across the board for all treatments.”

    This is an excellent point, as Ullman is essentially pleading for special (lowering of the bar) consideration for one thing (homeopathy) while arguing (spuriously) that conventional/mainstream interventions be held to the highest standards of evidential rigor. Dr Novella correctly zings Ullman for rounding cherry-picks and special pleading for homeopathy up to positive results and rounding EBM and SBM down to negative results (11%!).

    Sure, everyone wants special consideration for their thing, but that is literally an impossible accommodation for EBM or SBM to make.

    You nailed it, Hammyrex!

  23. fudiostudioon 24 Feb 2016 at 10:13 am

    “There is not one indication for which any homeopathic treatment has been reliably shown to work”

    I disagree. You can go “off label” with any homeopathic treatment and reliably treat dehydration.

Trackback URI | Comments RSS

Leave a Reply

You must be logged in to post a comment.