Mar 11 2013

Revenge of the Woo

Sometimes the targets of our skeptical analysis notice, and they usually are not pleased with the attention.

Last year the Acupuncture Trialists Collaboration published a meta-analysis of acupuncture trials in which they claim, “The results favoured acupuncture.” The report was widely criticized among those of use who pay attention to such things. In my analysis I focused on the conclusions that the authors drew, rather than their methods, while others also had concerns about the methods used.

The authors did not appreciate the criticism and went as far as to publish a response, in which they grossly mischaracterize their critics and manage to completely avoid the substance of our criticism.

To review, the original meta-analysis concluded:

Acupuncture is effective for the treatment of chronic pain and is therefore a reasonable referral option. Significant differences between true and sham acupuncture indicate that acupuncture is more than a placebo. However, these differences are relatively modest, suggesting that factors in addition to the specific effects of needling are important contributors to the therapeutic effects of acupuncture.

In my critique I pointed out that the results do not show that acupuncture is effective, nor that it is a reasonable referral option. What they characterize as “modest” differences were, rather, not clinically significant. Further, such tiny differences are most parsimoniously explained as the result of researcher and publication bias, two phenomena that are well established in general and specifically within the acupuncture literature. Unblinding alone would be sufficient to explain these results.

What they call “factors in addition to the specific effects of needling” the rest of the scientific community would call “placebo effects,” which are not an indication that a treatment works, but rather the result of bias, noise, and statistical illusions. These results are due to unblinded comparisons with untreated groups in clinical trials – they are not evidence of any kind of efficacy.

Their conclusions are part of a pattern visible within the acupuncture community – attempting to parlay placebo effects into the mirage of a real effect from acupuncture. I commented in my original article that such a conclusion was evidence of pro-acupuncture bias in the authors.

In their response, the authors write:

Although there was little argument about the findings in the scientific press, a controversy played out in blog posts and the lay press.

Only one substantive critique of the paper has appeared in a scientific forum.

We find that there is little argument in the scientific press because most scientists pay little attention to what they consider fringe practices. That is precisely why it is left to those of us who do care and pay attention to fringe medicine to provide a detailed analysis and point out the flaws in reasoning used by proponents.

In fact we did submit a letter in critique of the study, in a traditional scientific forum, but it was not published. Only the brief letter by David Colquhoun was.

This represents a typical strategy by proponents of dubious fringe medicine – interpret lack of resistance by mainstream scientists as acceptance. Whey they do encounter resistance, they try to minimize it as irrational – as Vickers et. al. have done here.

They continue:

This controversy was characterised by ad hominem remarks, anonymous criticism, phony expertise and the use of opinion to contradict data, predominantly by self-proclaimed sceptics.

This is a remarkable exercise in cherry picking and distortion. Their example of “ad hominem” remarks was my article in Science-Based Medicine (linked above) in which I said their conclusions were not justified and were therefore evidence of pro-acupuncture bias. This was followed by a substantive critique of their analysis, demonstrating the bias.

The majority of the criticism was not anonymous. All the usual players (myself, David Colquhoun, Edzard Ernst, Mark Crislip, Andy Lewis) posted articles or comments under our names. There are a few medical bloggers (like Orac) who prefer to remain anonymous (although they also blog under their real name) so as to preserve their rhetorical freedom and minimize professional harassment. To characterize the criticism as “anonymous criticism” is extremely unfair.

Under “phony expertise” they explain:

Many blog posters threw around methodological concepts such as I2 or funnel plots, or made claims about the nature of chronic pain or acupuncture placebo techniques. At the same time, many admitted to not having read the paper,4 and none appear to have published scientific research on pain, acupuncture or meta-analysis.

The reference is to Orac’s criticism. I reread it, and nowhere do I see a statement that Orac did not read the study. In fact he read not only the study but analyzed the studies in the meta-analysis themselves. Perhaps they meant to reference another article.

They also deride the concept of “Science-Based Medicine” as if that is a strange concept. What they fail to realize is that our collective expertise is in the distinction between science and pseudoscience, and the various mechanisms of self-deception. Most of us are also physicians, and we share our respective specialty expertise when collectively analyzing such studies. The original article, and this response, are excellent evidence of why such expertise is desperately needed in medicine, especially when dealing with unusual claims, such as acupuncture.

In response to my article they wrote:

One blogger asserted that acupuncture ‘has an effect size that is very small and, in my opinion, overlaps with no effect at all’.3 It is simply bizarre to dismiss years of careful statistical analysis on the grounds that results ‘might’ change; similarly, it should go without saying that whether an effect size overlaps with no effect is not a matter of opinion but of CIs.

Wrong and wrong. I did not simply substitute my opinion. My criticism was also not based on confidence intervals. We are not talking about statistical analysis, but systematic bias. I specifically cited the paper on “researcher degrees of freedom” to document this point. I further cited the authors in admitting that unblinding is a source of bias.

The point is that you can generate a small statistically significant result even when a treatment has zero effect. The authors falsely and naively assume that statistical significance equals a real effect, and they retreat to this position as if that counters the meat of our criticism. But it is simply not true. This is the point that those who are not aware of the principles of science-based medicine often miss. This is precisely why we advocate using a Bayesian analysis rather than p-values to assess clinical data.

Researcher and publication bias tend to produce a small (but statistically significant) positive result in clinical trials, and a meta-analysis will show that. We reject the results because:

- The effect size is not clinically significant

- There is a-priori and empirical evidence of bias in the acupuncture research

- Acupuncture is inherently implausible

- There is a clear pattern in the research that the best controlled and designed trials have no effect (no difference between true, sham, and placebo acupuncture.

There are two ways to interpret the acupuncture literature. One is that there is a real effect but it is too small to be clinically relevant. The second (the one I advocate) is that the small effects that tend to emerge from the research are likely not real and due to well-established sources of bias in clinical research. This is the most parsimonious interpretation. It is partly justified by the fact that the effect sizes and patterns in the research are similar to other phenomena, such as homeopathy and ESP, that are almost certainly not true.

We also criticized this statement from the authors:

With respect to the debate about clinical implications, the Collaboration argued that, while a treatment should ideally be shown to be superior to placebo, evaluation of clinical significance should be based on overall benefit, including any non-specific effects.

Yes – this should be debated. We maintain that clinical significance should absolutely not include “non-specific effects,” because such effects do not support a specific benefit from the procedure in question and are largely the product of illusion and bias. Further, useful non-specific effects, such as a therapeutic relationship between doctor and patient, can be had with legitimate treatments that are not based upon dubious principles.

Conclusion

Criticism of the Vickers et. al. article have been substantive and perfectly legitimate. Some were indeed intended for a lay audience, and meant to counter gullible treatments in the lay press. The authors, however, are unfair to dismiss all of this as “political muckraking” as they do in their response.

They did try to address some of the substantive criticism, but failed to do so, in my opinion. They entirely missed the main point of the effect of systematic bias in clinical research. They do so, it appears, because they lack expertise in pseudoscience – the very expertise they derided.

The pattern is very clear. Acupuncture is an implausible treatment with a pattern of clinical evidence that mirrors other highly implausible treatments. Researcher degrees of freedom alone is enough to explain the small residue of positive results, and it should not be ignored that the best designed studies tend to be entirely negative.

The unblinded comparisons to a no-treatment group do not justify acupuncture. Imagine if a pharmaceutical company tried to get away with such an argument. What Vickers and his coauthors have demonstrated is the dire need for science-based medicine.

 

Share

28 responses so far

28 Responses to “Revenge of the Woo”

  1. Eric Thomsonon 11 Mar 2013 at 10:01 am

    Did they cite the studies in rodents suggesting pain reduction with certain types of needle insertion? That seems the best way to respond to the placebo objection (while it is possible to get placebo effects in rodents, it requires explicit conditioning paradigms that are not done in these studies).

  2. Steven Novellaon 11 Mar 2013 at 10:16 am

    Eric – You are thinking of “placebo effect” too narrowly. These (as measured in studies) includes researcher bias and other things that are still relevant, even in animal research.

  3. Skepticoon 11 Mar 2013 at 10:48 am

    To characterize the criticism as “anonymous criticism” is extremely unfair.

    Ironically, it is also ad hominem – attacking the person (they are “anonymous”) rather than attacking the arguments.

  4. daedalus2uon 11 Mar 2013 at 10:59 am

    As they say in the schoolyard, “them’s fighting words”.

    If a flagship journal of Big Acupuncture publishes such a diatribe, misstating so much about the criticism (misstatements are not simply “unfair”, they are lies and so represent scientific misconduct (if they were published in an actual scientific journal, which arguably ‘Acupuncture in Medicine’ is not)) they are inviting a lengthy response, and can’t hide behind journal policy to stifle it. Particularly since some of the authors of the diatribe are editors of the journal in question.

    I would say go for it. Put together a lengthy response. You and the SBM crowd have done most of the work already. If (when?) they refuse to publish it, you will demonstrate that Big Acupuncture has no clothes science to cover their practices.

  5. Murmuron 11 Mar 2013 at 11:09 am

    Did anyone else hear “The Imperial March” in their head when they read the title?

    I love seeing this happen:

    CAM People: “See! Science says our stuff works!”
    Skeptics: “Cool, let’s see the evidence.”
    CAM: “Oh… *cough* here it is *cough* don’t look at it too closely and don’t be too “sciencey” about interpreting it.”
    Skeptic: “Ah, no, sorry, it appears it doesn’t work.”
    CAM: “What!!! STOP MAKING FUN OF ME YOU NASTY SCIENCIES!!!”

  6. PharmD28on 11 Mar 2013 at 11:58 am

    I think that when such matters of disagreement arise, a public dialogue or debate should ensue…not just publishing responses….

    Vickers dropped into orac’s post, made his points, and made no further replies.

    Seemed lame to me….if they have the evidence and the logic/reason on their side, then let them debate things openly.

    After reading orac’s post, then vickers, I felt like many of the points were not acknowledged.

    In pharmacy school, we almost ALWAYS discussed “clinical significance”. If this were a particular pain drug, there is no doubt the conclusion would be “not clinically significant”….and yet….

  7. tmac57on 11 Mar 2013 at 12:11 pm

    Imagine if a pharmaceutical company tried to get away with such an argument.

    This to me is the ‘slam dunk’ in the case against all CAM that tread on that razor’s edge of low plausibility and small effect size.

  8. mmon 11 Mar 2013 at 1:31 pm

    Can you link to your original critique?

    Also what do you say to someone who says that after years of looking, acupuncture was the cure to their problems with X ailment?

  9. Ori Vandewalleon 11 Mar 2013 at 2:58 pm

    mm:

    Because acupuncture doesn’t appear to work when studied on a large, thorough scale, it is safe to assume that someone who claims acupuncture cured X is (unintentionally) failing to see what really cured their ailment. It’s possible that whatever they were sick with simply went away on its own, it’s possible that they moved to a new location that had less detrimental environmental factors, it’s possible they got a new job with reduced stress, etc. There are any number of possibilities, but we tend to only believe/see what we expect with this sort of thing.

    That doesn’t really answer your question, though, because you shouldn’t really say that to an acupuncture believer. It sounds like you’re belittling their intelligence and/or beliefs.

  10. PharmD28on 11 Mar 2013 at 4:46 pm

    “Also what do you say to someone who says that after years of looking, acupuncture was the cure to their problems with X ailment?”

    One example that I loved when reading simon singh and Ezdard Ernst’s book (just finished it last week) – was “internal mammary gland ligation”….

    Here was a procedure in which patients subjectively reported significant benefits in reducing frequency and quality of chest pain…

    Problem was, was that simply having the excision alone as a control produced similar results. Why is that? Why did just an incision work all the same?

    In the case of acupuncture, we have a number of higher quality studies that use a control group (a “sham” acupuncture group) in an effort to show this effect that some people report.

    Interestingly, these “sham” treatments were not “acupuncture” and yet, they too were equally effective in the research with patients reporting benefit.

    So the question is in that case…why after years of looking for something to help a given patient’s pain, why a “sham” form of acupuncture “cured” their ailment all the same as “real acupuncture”?

    Conclusion? pseudorandopuncture (sham acupuncture without any penetration or regard for location) may be worth looking into as a useful treatment modality folks…

    One argument with a fellow local skeptic I had led to the conclusion that now we need a new study: blinded pseudorandopuncture vs. unblinded pseudorandopuncture (in this case we tell them we are giving them the sham, not penetrating or having regard for location)….

  11. tmac57on 11 Mar 2013 at 7:23 pm

    PharmD28- Yeah, I like your example as well,and what do you want to bet that there are still people out there claiming the internal mammary (I think you meant artery) ligation is a valid procedure,even if it’s done in the sham mode.
    I often have the discussion with people who claim that ‘such and such’ dubious treatment did wonders for them or someone else,and when I raise the question of plausibility,they almost invariably (and irritably) invoke the “Well if it works for them,then how can you argue with that!?” defense.

  12. daedalus2uon 11 Mar 2013 at 9:58 pm

    One of the placebo procedures that I like is using a laser to blast holes in someone’s heart from the inside (but only part way through). The idea was that if you did that, the holes would be the sites for new vascular growth and the heart would have an improved blood supply. They tried it and it worked! People who had the procedure done reported improvement and objective measures showed improvement too.

    The problem was that when some of the subjects eventually died, their hearts did not show increased vascular growth. The holes filled up with scar tissue and there was no increased perfusion. So they did placebo procedures, and some of them experienced the same types of subjective and objective improvements too.

  13. Davdoodleson 11 Mar 2013 at 11:24 pm

    Even if we accept the woo-afficionados’ claim that there is a therapeutic effect above zero, the question still remains: Why get so excited about it?

    Aspirin, paracetamol, hot and/or cold packs, exercise and all manner of other stuff safely provide significant positive effects on pain.

    So what is so bloody thrilling about accupuncture, which even its proponents agree provides, at best, almost no therapeutic effect?
    .

  14. PharmD28on 11 Mar 2013 at 11:38 pm

    TMC –

    YES – ARTERY – FAIL :D

    Oh, and on the note on “well it worked for me” – Dr. Novella dedicated a blog post on that very subject in my honor :D – it seems quite relevant in this discussion to dig up again…

    http://theness.com/neurologicablog/index.php/well-it-worked-for-me/

    And one of my favorite studies so far to best distinguish “it works for me” vs. a more useful objective reality is the NEJM published study with asthma ..treatment groups were albuterol inhaler, placebo inhaler, no treatment, and sham acupuncture…..all the placebos has similar subjective improvements reported, but only albuterol produced reduced FEV1….

    http://www.nejm.org/doi/full/10.1056/NEJMoa1103319#t=abstract

    I hope that our comments help answer MM’s questions better….the above study should really muddy the waters for the quick assumption that such anecdotes are worth as much importance as our quick intuitions would make it to be.

    Daedalus – I had never heard of that example….fascinating!

  15. sonicon 12 Mar 2013 at 1:18 am

    A blog post responded to at ncbi.nih
    That’s amazing.
    When I used the sites search and tried ‘blog posts’, the data that came up indicated that this might be a first.

    Is that true?

    I’m guessing this fits into the ‘beyond my wildest dreams’ category for when Dr. N started blogging.

  16. ConspicuousCarlon 12 Mar 2013 at 5:03 am

    I admire the boldness with which they brush off criticism from real doctors as anonymous “lay press” as if you were just a bunch of ghost writers for Perez Hilton.

  17. tmac57on 12 Mar 2013 at 11:06 am

    D2U- I vaguely remember the laser treatment that you cited,but I had never heard of the follow up that apparently showed that it was a placebo effect. Anybody know of a comprehensive list of these kind of interventions that turned out to be placebo? That would be a great tool for refuting the ‘worked for me’ argument.

  18. daedalus2uon 12 Mar 2013 at 11:33 am

    here is the link

    https://www.ncbi.nlm.nih.gov/pubmed/15950570

  19. Eric Thomsonon 12 Mar 2013 at 12:30 pm

    Daedalus that is a great example thanks for the link.

    Also I think DavDoodles has hit the key point. It seems clear that inserting needles (especially with twisting or electricity attached) can reduce pain locally. But so does an ice pack, and the latter costs a wee bit less. This is why I am skeptical of its clinical relevance, but also a bit underwhelmed the the skeptical community’s attempts to dismiss all the effects as placebo. That just seems unlikely given the work in rats using needle insertion etc..

    I think the auxilliary weirdness associated with acupuncture needle use (e.g., the theory of chi, and all that), that is, the larger theoretical edifice of Acupuncture (capital A), rightly puts off skeptics, but it sometimes dulls their critical faculties with claims that are largely independent of the weirdness.

  20. Murmuron 12 Mar 2013 at 12:47 pm

    @Eric, if it acts like a duck and sounds like a duck and looks like a duck, Occam’s Razor says it is most likely a duck. If it acts like a placebo, then why would we assume it is anything else?

    This is the second reference to the study on rats, I don’t recall any links to it previously. The only one I could find referred to elecro-acupuncture reducing neuropeptide Y, which is a compound associated with stress done in 2011 (Georgetown University). There have yet to be any follow up studies, and I could not find the specifics of the study size or any of the actual results.

    If I have the wrong study please feel free to correct and point me in the right direction.

  21. Eric Thomsonon 12 Mar 2013 at 12:50 pm

    Murmur I was thinking of this paper in Nature Neuroscience. It is a very good paper (despite the first paragraph, which seems to have stopped many people from reading the science).

  22. Eric Thomsonon 12 Mar 2013 at 12:56 pm

    Note it was actually mice, not rats.

    We discussed that paper quite a bit here, which is when I first learned to not trust interweb skeptics for intelligent critical discussions of acupuncture.

    That thread ended up with people quibbling about whether it should be called ‘acupuncture’ because while the researchers indeed used acupuncture needles, it didn’t subscribe to or promote the specifics of the weird theory of Acupuncture (which, because of that interaction, I am now sure to capitalize to make the distinction for those who get caught up in such wordplay).

  23. Eric Thomsonon 12 Mar 2013 at 12:57 pm

    http://theness.com/neurologicablog/index.php/potential-new-mechanism-of-pain-relief-discovered/

  24. BillyJoe7on 12 Mar 2013 at 4:59 pm

    Eric,

    Acupuncture works only if you define it out of existence.

    There is no evidence of qi, meridians, or anything anatomically special about acupuncture points.
    Clinical trials have demonstrated that it doesn’t matter where you stick the needles in, how far you stick them in, or even if you don’t stick them in.
    What is left of acupuncture then?

    Traumatising a mouse’s leg releases endorphins, yes, but what has that to do with acupuncture aside from the fact that you used a acupuncture needle to traumatise that poor mouse.

  25. Eric Thomsonon 13 Mar 2013 at 8:45 am

    BillyJoe you should read the Nature Neuroscience paper, as it seems you have not. It is important for people to distinguish Acupuncture (the ancient and discredited theory of Chi and such) from acupuncture (the use of needle insertion for treatment). People get all caught up on the former, and it dulls their ability to think well, and dispassionately, about the latter.

  26. Oracon 13 Mar 2013 at 12:43 pm

    I presume you mean this paper:

    http://www.sciencebasedmedicine.org/index.php/another-overhyped-acupuncture-study-misinterpreted/

    Hint: It doesn’t show what the authors try to make you believe it shows.

  27. BillyJoe7on 13 Mar 2013 at 4:29 pm

    Eric,

    I read the paper at the time it was covered at SBM (link provided by Orac above).
    Have a read and see what you think.

  28. madmidgitzon 17 Mar 2013 at 12:51 am

    repost from: Another Acupuncture Meta-Analysis – Low Back Pain

    @everyone and specifically Eric
    there is a huge issue here no-one noticed
    if stabbing rats with needles is what we are basing things on shouldn’t we scale up and stab people
    with nails
    just a fun thought
    a needle to a rat is a fencing sword to us
    trauma causes effects, nothing new here
    r’Amen

Trackback URI | Comments RSS

Leave a Reply

You must be logged in to post a comment.