Alternative Medicine’s best friend, and in my opinion largely responsible for what popularity it has, is a gullible media. I had thought we were turning a corner, and the press were over the gushing maximally clueless approach to CAM, and were starting to at least ask some probing questions (like, you know, does it actually work), but a 2006 BBC documentary inspires a more pessimistic view.

The documentary is part of a BBC series hosted by Kathy Sykes: Alternative Medicine, The Evidence. This episode is on acupuncture. The episode is from 2006, but was just posted on YouTube as a “2014 documentary.” Unfortunately, old news frequently has a second life on social media.

First, let me point out that Sykes is a scientist (a fact she quickly points out). She is a physicist, which means that she has the credibility of being able to say she is a scientist but has absolutely no medical training. It’s the worst case scenario – she brings the credibility of being a scientist, and probably thinks that her background prepares her to make her own judgments about the evidence, and yet clearly should have relied more on real experts.

She does interview Edzard Ernst in the documentary, but he mainly just says generic statements about science, rather than a thorough analysis of specific claims. I wonder what gems from him were left on the cutting room floor.

The documentary does get better in the second half, as she starts to mention things like placebo effects, and the problems with the evidence-base for acupuncture. But she follows a disappointing format – setting up a scientific premise, then focusing on the positive evidence. There is a clear narrative throughout, that acupuncture is amazing and surprising.

A few examples illustrate my point. She showcases a patient in China having open heart surgery without general anesthesia, but with acupuncture “instead.” The framing of the case is massively biased to exaggerate the role of acupuncture. Then, tucked into the reporting, she mentions that the patient had sedation and local anesthesia (her chest was numbed), as if this is a tiny detail. There is no mention of whether or not you could have the same procedure with conscious sedation and local anesthesia but without the acupuncture.

In the end she perpetuated the myth of acupuncture anesthesia without putting the case into any perspective.

The worst part of the documentary, however, was when she came to evaluating the clinical evidence for acupuncture. After gushing over all the usual nonsense about chi, life force, holistic Eastern medicine, and setting up the audience for how magically wonderful acupuncture is, she then gives us the “but I’m a scientist” bit. Here is where her failure is greatest.

She frames her approach to the evidence as – well, most studies are small, disappointing, and negative, finally a large well-controlled trial was done on migraine, and this study showed that acupuncture works. She then repeats that process with acupuncture for knee osteoarthritis, relying on the Berman trial as if one trial can be definitive.

In other words, the approach she takes is to rely on a single trial, presenting the trial as if it is definitive (finally answering the question), and then concludes that we can “safely say” that acupuncture works for these indications.

This is profoundly wrong. We always need multiple trials with a consistent effect, as revealed by systematic reviews. For example, the The American Academy of Orthopaedic Surgeons reviewed the evidence, and made a strong recommendation against acupuncture:

There were five high- and five moderate- strength studies that compared acupuncture to comparison groups receiving non-intervention sham, usual care, or education. The five moderate-strength studies were included because they reported outcomes that were different than the high-strength evidence. High-strength studies included: Berman et al, 61 Suarez-Almazor et al.,62 Weiner et al.,63 Williamson et al.64 and Taechaarpornkul et al.65 Moderate-strength studies included: Sandgee et al.,66 Vas et al.,67 Witt et al.68 and Berman et al. 69
The majority of studies were not statistically significant and an even larger proportion of the evidence was not clinically significant. Some outcomes were associated with clinical- but not statistical- significance. The strength of this recommendation was based on lack of efficacy, not on potential harm.

This is in line with other reviews – overall the effects are not statistically significant, and in those studies that find an effect it is generally not clinically significant (meaning that it is likely background noise and placebo effects).

The same is true for migraine. Reviews of the evidence are consistent with placebo effects only.

After mangling the clinical evidence, Sykes then gives a credulous review of the fMRI evidence. Gee – when you stick needles into the skin, stuff happens in the brain. There is no mention of how tricky it is to perform and to interpret such studies. There is no mention of anomaly hunting, or the need to confirm the results, and find out what they actually mean.

An article in The Guardian by Simon Singh nicely attacks this stunt.

To the average viewer, they will see – wow, science shows that acupuncture has a real effect on the brain, and something to do with pain.

Conclusion

Sykes BBC review of acupuncture was an unmitigated fail. Throughout she follows a clear narrative – as a scientist, she was initially skeptical, but then was surprised to find that there really is something to acupuncture. She fails to put any of the evidence she presents into perspective, she fails to give a real skeptical view, or to even mention systematic reviews.

In the end she comes to a conclusion that, in my opinion, is the opposite of what science and the evidence say. She failed as both a scientist and a journalist, and did a disservice to anyone watching her documentary.

The series did receive significant criticism at the time – like this article from Ben Goldacre – and now that it has been posted on YouTube as if it were new, I guess we need a new round of criticism as well.