As Carl Sagan observed, “randomness is clumpy,” which means that sometimes, for no specific reason, I write two or more blog posts in a row about the same topic. Perhaps it’s not entirely random, meaning that when a topic is being discussed related news items are more likely to come to my attention.

In any case, there was recently published yet another meta-analysis of acupuncture, this time specifically for low back pain. The findings and interpretation add to the pile of evidence for two important conclusions:

1 – Acupuncture does not work.

2- Acupuncturists refuse to admit that acupuncture does not work.

I would further infer from these two unavoidable conclusions the dire need for a greater understanding of the core principles of science-based medicine.

From the abstract of the new study:

Compared with no treatment, acupuncture achieved better outcomes in terms of pain relief, disability recovery and better quality of life, but these effects were not observed when compared to sham acupuncture. Acupuncture achieved better outcomes when compared with other treatments. No publication bias was detected. Acupuncture is an effective treatment for chronic low back pain, but this effect is likely to be produced by the nonspecific effects of manipulation.

The authors expand upon this in their discussion:

When acupuncture has been compared with no additional treatment or blank treatments, the results have consistently shown that acupuncture is significantly more effective. However, the results are relatively inconsistent when acupuncture is compared with sham acupuncture. Some studies found there is moderate evidence that acupuncture is more effective than sham acupuncture, while other studies have published strong evidence of no significant difference between acupuncture and sham acupuncture.

The meta-analysis itself seems well done and fairly rigorous. That is something that I have generally noted about the acupuncture literature – while it is littered with poor studies, preliminary studies, and unblinded studies (as is most medical science), it also contains many very well designed and executed studies, and solid systematic reviews and meta-analysis. This is typical of clinical science in general. Preliminary studies eventually lead to definitively designed studies that give us the most reliable answers to our clinical questions, in this case, does acupuncture work for specific indications.

It takes time to sort out the answer to these clinical questions, and we have to look at the overall pattern in the literature to see that answer. Several patterns may emerge. With treatments that actually work we see initial conflicting results, with a positive bias, and then as better and better studies are executed we zero in on the actual safety and efficacy of the treatment. The ultimate magnitude of the treatment effect tends to be lower than initial studies indicated, but it is still effective.

With treatments that do not work we see a similar pattern, but the effect size declines to the point that it overlaps with zero. For such treatments the consensus conclusion is that the treatment does not work. Sometimes proponents claim we need higher doses, subgroup analysis, or longer studies to see the effect. Sometimes they are right, but most often this is just special pleading that does not rescue the therapy from the discard bin.

Related to the above pattern there is a broader pattern within the world of science in general (not specific to clinical studies). It often happens that effects appear to exist when they are observed in an unblinded fashion, but then they vanish when properly blinded controls are put into effect. The poster child for this phenomenon is the infamous N-Rays. Dozens of scientists thought they saw the ephemeral rays, until blinded experiments showed that they do not exist.

When a phenomenon disappears under proper blinding, scientists conclude that the phenomenon is not real. That is the essence of science – controlling for bias and other variables to separate what’s really real from what only seems to be real.

Let’s apply this basic scientific principle to the evidence for acupuncture. What the researchers above found in this case, and admit is what reviewers have found in general, is that when you look for an acupuncture effect in an unblinded way it seems to be there, but when you look for it with blinded controls the effect vanishes.

Acupuncture = N-rays = not real

Studies find either no effect at all, or a tiny effect that is indistinguishable from no effect. This latter point deserves some clarification, however.

In my prior post on the response of the authors of an acupuncture meta-analysis to skeptical criticism, I briefly discussed the difference between statistical significance and clinical significance, and this concept is important here. Clinical researchers generally understand the concept of statistical significance – that the probability of the outcome of a study being what it is, or greater, is less than 1 in 20 (p-value of 0.05 or lower). There is also something called a confidence interval which is an attempt to statistically measure the range of the real effect size given the measured effect size. If this range overlaps with zero, then we should not reject the null hypothesis.

This is all fine, as far as it goes, but it is all premised on a massive assumption – the lack of any systematic bias in the research. The statistics are based on the probability of random results, not biased results. There is therefore another way entirely that the results of an experiment can be indistinguishable from no effect – the first is statistical, the second is bias.

One of the core bits of wisdom of SBM, well documented in the literature and clear from any proper historical understanding of science in general and medical science specifically, is that it is difficult to impossible to completely eliminate bias and error from research. We have to therefore assume a minimal level of such bias and error, which we can think of as noise in the data.  When the signal is very small, the signal to noise ratio is therefore very low, and the results are not reliable – meaning that they are indistinguishable from no effect (even if they are statistically significant).

We see this with ESP research, with homeopathy research, and with countless practices that are based on fantasies and not reality. The acupuncture research looks like ESP research because neither are real phenomena.

How do we overcome these limitations, and how do we detect a small effect if any such signal will likely be lost in the noise? That is where replication comes in, doing large rigorously designed studies, and carefully controlling for every variable. Meta-analysis really does not solve this problem – that only addresses the issue of statistical significance, but not bias and error. This is why meta-analysis is a poor predictor of later definitive research. It is also why some researchers recommend “best-evidence synthesis” over meta-analysis.

In recent years researchers have developed tools for detecting and correcting for publication bias, although this is not always done or done properly. There is no way, however, to detect or measure researcher bias. This can only be minimized with the most rigorous methodology, and corrected through replication.

We also need to take prior probability into account. If a treatment defies well-established science, we should be more skeptical of the small signals that are lost in the noise of bias and error. The threshold for acceptance should be higher. This is also why we favor a Bayesian approach to analyzing data, over the more common p-value approach.

This new meta-analysis adds to the consensus of research that shows that acupuncture, as defined by sticking acupuncture needles into acupuncture points, is not a real phenomenon. It does not matter where you stick the needles. It does not matter how deep you stick them, or even if you stick them. There is no evidence to support the reality of acupuncture points or any proposed underlying mechanism. We have low prior probability, with evidence of lack of efficacy for the variables specific to acupuncture.

We also have the completely unsurprising finding that when unblinded comparisons are made a typical magnitude of placebo effects is seen. This does not mean that acupuncture works, by any sound reasoning. Proponents keep desperately trying to make this argument, however, in defiance of all scientific principles.

Acupuncture does not work. The therapeutic interaction that surrounds the administration of acupuncture has the predictable non-specific effects, bias, and illusion that comprise observed placebo effects. The same would be true of any intervention, no matter how worthless or absurd. You can make up an unlimited number of fanciful treatments and substitute them for acupuncture and you would get the same non-specific effects. That does not mean that any of them work either.

The authors conclude:

The current data shows that acupuncture is effective in providing long-term relief of chronic low back pain, but this effect is produced by non-specific effects that arise from skin manipulation. Further research is needed to evaluate which the effects of acupuncture compared with other treatments. Moreover, acupuncture should be used in combination with other treatments in the treatment of chronic low back pain. In view of the often refractory nature of chronic low back pain, comprehensive treatment plans, which may involve acupuncture, are urgently needed.

This conclusion is absurd and not justified. It should read:

The current data show that acupuncture is ineffective for low back pain, as it is indistinguishable from sham treatments in blinded comparisons. Acupuncture is therefore not recommended as a treatment and has no role to play in a science-based treatment plan for low back pain. Given the amount and quality of existing data, in addition to the low prior probability, no further research is warranted and in fact would be ethically dubious at this point. In view of the refractory nature of chronic low back pain it would be extremely counterproductive to waste any more time or resources on this dead end. 

There, fixed.