May 06 2016

Acupuncture for Tension-Type Headache

acupuncture2A newly updated Cochrane systematic review of 12 studies looking at acupuncture for the treatment of tension-type headaches (TTH) concluded:

The available evidence suggests that a course of acupuncture consisting of at least six treatment sessions can be a valuable option for people with frequent tension-type headache.

This has led to another round of headlines that, “Acupuncture works but no one knows how.”

A closer look at the data, however, does not back up that conclusion, in my opinion. Cochrane is generally considered to be the gold standard for evidence-based systematic reviews, but their history is dodgy when it comes to unconventional treatments. For example, they famously had to withdraw their review of homeopathic occillococcinum for the flu because they concluded, although the evidence was insufficient to recommend, it was “promising” and deserved further research.

Their updated review is not much better, however:

There is insufficient good evidence to enable robust conclusions to be made about Oscillococcinum® in the prevention or treatment of influenza and influenza-like illness. Our findings do not rule out the possibility that Oscillococcinum® could have a clinically useful treatment effect but, given the low quality of the eligible studies, the evidence is not compelling.

Oscillococcinum is fairy dust – there is no scientific reason to suspect that it even exists. This is sufficient to justify a very robust conclusion that it cannot possibly work for anything. That the clinical evidence is also negative (even though poor quality) is an extra and unneeded nail in the coffin. At least they didn’t call for more research.

Acupuncture for TTH

That Cochrane has a blind spot for so-called alternative medicine treatments is often evident. Many of their reviews, like the one above, also highlight the limitations of a strictly evidence-based approach, and the exact reason why my colleagues and I found it necessary to advocate for a science-based medicine approach.

I think you will see the same blind-spot evident in the current review. They looked at 12 studies. Two of the trials were unblinded. It is a mystery why they would even included these trials.

It is known from countless studies that introducing any new treatment will have a placebo effect, and this is maximal for subjective symptoms like pain. Further, the greater the cost, invasiveness, and the more elaborate the treatment ritual, the greater the placebo response. Acupuncture is therefore almost the perfect placebo treatment. It is, in fact, nothing but an elaborate placebo ritual.

I’m not going to even discuss these two studies further. They are, in my opinion, worthless as confirmatory evidence, have no place in such a review, and can comfortably be ignored.

Four studies compared acupuncture to other “alternative” treatments like massage. These studies did not show that acupuncture was superior to any other treatment, which could simply mean that the studies were not powerful or rigorous enough to detect any differences.

That leaves only six studies in which there was some blinding. I say “some” blinding because this is a very sticky issue when it comes to acupuncture studies (pun intended). Inadequate blinding is perhaps the single largest weakness of the acupuncture literature.

I also have to make a comment about the use of meta-analysis. This involves combining data from multiple trials to increase the overall power, and then treat it like one large study. Doing this well is very tricky; it involves picking the right trials to include, adjusting the data so that they can be combined, and then examining patterns in the data that suggest bias. In an NEJM review meta-analyses they looked at agreed with later definitive large clinical trials only 65% of the time.

In addition, clinical trials that are conducted at one treatment center are often contradicted by later multi-center trials. There is also a well-known positive bias in the literature, partly stemming from researcher bias and partly from publication bias.

This reality is often summarized as, “Garbage in, garbage out.” In other words – you can’t really combine a bunch of small studies that are not individually rigorous and come out with one large rigorous study just by doing a meta-analysis.

Having said that, a well done meta-analysis can be useful, if the studies examined are sufficiently rigorous, and the data analysis is thorough and unbiased. The results an be reliable if they are consistent and robust. That is not what we have with the acupuncture and TTH review, however.

Six studies is not a large number, and most were very small. The authors note:

The results were dominated by one large, good quality trial (with about 400 participants), which showed that the effect of true acupuncture was still present after six months.

So the author’s conclusions are largely based on one study (conducted in Germany). Let’s take a look at that study.

The first thing to note is that this is not a double-blind study. It is a, “sham-controlled, patient and observer-blinded” study. This may seem like a subtle difference, but it isn’t. In the study the acupuncturists were not blinded. They knew if they were giving “real” (called “verum”) acupuncture or sham acupuncture. The patients were not told, and the person on the phone recording their results did not know their allocation, but the treating acupuncturists did.

This is, in my opinion, a fatal flaw. In previous studies, when the interaction between the acupuncturist and the subject was controlled for, a more positive interaction was associated with a better outcome. In fact, it was the only variable that predicted outcome.

In the German study, subjects estimated that they spent 8.8 minutes with the acupuncturist for verum treatment, and 7.8 minutes for sham. There were also fewer needle insertions per session (16.5 vs 14.6).

In addition – another fatal flaw – the verum group received needles on their head, while the sham group did not. Needling the location of the pain is likely to contribute to a placebo response.

In short, the two groups were not properly blinded and were not sufficiently similar, in a manner that is likely to result in increased placebo response in the treatment group. Despite all this – the results for the primary outcome were negative. That’s right, this was a negative study.

The treatment group met the primary endpoint of a 50% reduction in headaches 33% of the time, while the sham group did 27% – this was a small difference and not statistically significant.

Some, but not all, of the secondary measures were improved in the verum group, but again the effect size was small, and the researchers did not correct for multiple comparisons.

These small and unimpressive results are easily due to the lack of rigor in this study, specifically the lack of blinding for the acupuncturists and the differences in treatment between the two groups. This low quality and essentially negative study “dominated” the evidence in the systematic review. Essentially what they did was add a few smaller trials to eke over the line for statistical significance, but this does not make the results compelling.


The pattern of clinical evidence for acupuncture remains consistent. Overall the research quality is moderate to low, as with this review. The results generally show small clinical effect sizes with marginal statistical significance, resulting from trials that have one or more fatal flaws, such as improper blinding and asymmetries between the treatment groups.

When you do get high quality properly blinded studies they generally show the same thing – that it does not matter where or even if you place the needles, meaning that acupuncture does not work. The ritual surrounding acupuncture does have a significant placebo effect, which is enhanced by the duration and quality of interaction with the acupuncturist and how elaborate the ritual is. Actually performing acupuncture, however, seems to be irrelevant.

The only logically consistent and evidence-based way to interpret the totality of the acupuncture literature is to conclude that acupuncture does not work, for any indication. Added to this is that lack of any plausible mechanism. Chi is not a scientific concept and there is no reason to conclude that it exists. Neither do acupuncture points have any basis in reality – so it should not matter if you stick needles in these nonexistent points to manipulate a nonexistent energy force.

It is therefore no surprise that when properly studied there is no effect from acupuncture.

What is surprising is that researchers continue to waste time and resources on this clear dead end, and that academics can look at this data and come to their wishy-washy conclusions, when a clear science-based conclusion is evident.


26 responses so far