Jul 15 2014

BBC Fail on Acupuncture Documentary

Alternative Medicine’s best friend, and in my opinion largely responsible for what popularity it has, is a gullible media. I had thought we were turning a corner, and the press were over the gushing maximally clueless approach to CAM, and were starting to at least ask some probing questions (like, you know, does it actually work), but a 2006 BBC documentary inspires a more pessimistic view.

The documentary is part of a BBC series hosted by Kathy Sykes: Alternative Medicine, The Evidence. This episode is on acupuncture. The episode is from 2006, but was just posted on YouTube as a “2014 documentary.” Unfortunately, old news frequently has a second life on social media.

First, let me point out that Sykes is a scientist (a fact she quickly points out). She is a physicist, which means that she has the credibility of being able to say she is a scientist but has absolutely no medical training. It’s the worst case scenario – she brings the credibility of being a scientist, and probably thinks that her background prepares her to make her own judgments about the evidence, and yet clearly should have relied more on real experts.

She does interview Edzard Ernst in the documentary, but he mainly just says generic statements about science, rather than a thorough analysis of specific claims. I wonder what gems from him were left on the cutting room floor.

The documentary does get better in the second half, as she starts to mention things like placebo effects, and the problems with the evidence-base for acupuncture. But she follows a disappointing format – setting up a scientific premise, then focusing on the positive evidence. There is a clear narrative throughout, that acupuncture is amazing and surprising.

A few examples illustrate my point. She showcases a patient in China having open heart surgery without general anesthesia, but with acupuncture “instead.” The framing of the case is massively biased to exaggerate the role of acupuncture. Then, tucked into the reporting, she mentions that the patient had sedation and local anesthesia (her chest was numbed), as if this is a tiny detail. There is no mention of whether or not you could have the same procedure with conscious sedation and local anesthesia but without the acupuncture.

In the end she perpetuated the myth of acupuncture anesthesia without putting the case into any perspective.

The worst part of the documentary, however, was when she came to evaluating the clinical evidence for acupuncture. After gushing over all the usual nonsense about chi, life force, holistic Eastern medicine, and setting up the audience for how magically wonderful acupuncture is, she then gives us the “but I’m a scientist” bit. Here is where her failure is greatest.

She frames her approach to the evidence as – well, most studies are small, disappointing, and negative, finally a large well-controlled trial was done on migraine, and this study showed that acupuncture works. She then repeats that process with acupuncture for knee osteoarthritis, relying on the Berman trial as if one trial can be definitive.

In other words, the approach she takes is to rely on a single trial, presenting the trial as if it is definitive (finally answering the question), and then concludes that we can “safely say” that acupuncture works for these indications.

This is profoundly wrong. We always need multiple trials with a consistent effect, as revealed by systematic reviews. For example, the The American Academy of Orthopaedic Surgeons reviewed the evidence, and made a strong recommendation against acupuncture:

There were five high- and five moderate- strength studies that compared acupuncture to comparison groups receiving non-intervention sham, usual care, or education. The five moderate-strength studies were included because they reported outcomes that were different than the high-strength evidence. High-strength studies included: Berman et al, 61 Suarez-Almazor et al.,62 Weiner et al.,63 Williamson et al.64 and Taechaarpornkul et al.65 Moderate-strength studies included: Sandgee et al.,66 Vas et al.,67 Witt et al.68 and Berman et al. 69
The majority of studies were not statistically significant and an even larger proportion of the evidence was not clinically significant. Some outcomes were associated with clinical- but not statistical- significance. The strength of this recommendation was based on lack of efficacy, not on potential harm.

This is in line with other reviews – overall the effects are not statistically significant, and in those studies that find an effect it is generally not clinically significant (meaning that it is likely background noise and placebo effects).

The same is true for migraine. Reviews of the evidence are consistent with placebo effects only.

After mangling the clinical evidence, Sykes then gives a credulous review of the fMRI evidence. Gee – when you stick needles into the skin, stuff happens in the brain. There is no mention of how tricky it is to perform and to interpret such studies. There is no mention of anomaly hunting, or the need to confirm the results, and find out what they actually mean.

An article in The Guardian by Simon Singh nicely attacks this stunt.

To the average viewer, they will see – wow, science shows that acupuncture has a real effect on the brain, and something to do with pain.

Conclusion

Sykes BBC review of acupuncture was an unmitigated fail. Throughout she follows a clear narrative – as a scientist, she was initially skeptical, but then was surprised to find that there really is something to acupuncture. She fails to put any of the evidence she presents into perspective, she fails to give a real skeptical view, or to even mention systematic reviews.

In the end she comes to a conclusion that, in my opinion, is the opposite of what science and the evidence say. She failed as both a scientist and a journalist, and did a disservice to anyone watching her documentary.

The series did receive significant criticism at the time – like this article from Ben Goldacre – and now that it has been posted on YouTube as if it were new, I guess we need a new round of criticism as well.

Share

24 responses so far

24 Responses to “BBC Fail on Acupuncture Documentary”

  1. mumadaddon 15 Jul 2014 at 8:50 am

    There is no mention of whether or not you could have the same procedure with conscious sedation and local anesthesia but without the acupuncture.

    Or with acupuncture but without conscious sedation and local anaesthesia. Ouch…

  2. Neurobonkerson 15 Jul 2014 at 10:31 am

    It looks like you’ve been caught out by a cunning, blatant attempt at misinformation. This isn’t a 2014 documentary – whoever uploaded it on Youtube mislabeled the date. It is in fact a six year old documentary that was widely slated and has had complaints upheld against it by the BBC: http://www.badscience.net/2007/07/complaints-against-bbc-over-alternative-medicine-the-evidence-upheld/

  3. Steven Novellaon 15 Jul 2014 at 10:45 am

    Thanks. I wasn’t familiar with the series as it was out prior to my blogging career. I will update the article.

  4. Skepticoon 15 Jul 2014 at 10:46 am

    Media reports (especially the BBC in my experience) are notoriously bad what it comes to acupuncture. I’ve virtually given up on commenting on it. Yesterday a website calling itself Science 2.0 followed the same format. With the headline “Acupuncture Works To Reduce Menopause Hot Flashes – Meta-analysis” the article reports (in an almost off the cuff manner 3/4 the way through) that yes well those on sham acupuncture also benefited. Article here:

    http://www.science20.com/news_articles/acupuncture_works_to_reduce_menopause_hot_flashes_metaanalysis-140499

    I commented briefly. See the usual misinformed comments following. Depressing for a supposedly “science” website.

  5. Steven Novellaon 15 Jul 2014 at 11:01 am

    Nice commenting, Skeptico. You nailed it.

  6. pdeboeron 15 Jul 2014 at 12:47 pm

    Skeptico, your bravery is the face of madness is admiral.

    I’m going to hide here in my escape to reality.

  7. WeWeeon 15 Jul 2014 at 3:03 pm

    This topic has caused many controversies in Italy.
    Acupuncturists are very sensitive about what is said about them, they say in all the ways that they use a scientific method that is already proved, they reported also this BBC video and the WHO report about acupuncture. I wrote a book in which I pointed out that acupuncture is a little more than a placebo, and that the studies show that its effects are not yet proven. This has caused weeks of controversy and has moved even the italian society of acupuncture and many acupuncturists with threats and insults against me. The president of the italian society of acupuncture claims that words such as “flow”, “Chi” and “meridians” “are only “metaphorical “and which today have been identified with modern concepts (hormonal “flows” and nervous reactions i.e.) and fully recognized by the medical science.
    :)

  8. Teaseron 15 Jul 2014 at 6:31 pm

    @Novella

    Your continued assault on acupuncture as a threat to patient safety is illogical. Death or serious injury by acupuncture is nominal when compared to prescription drug deaths, deaths from care in US Hospitals or death/injury from distracted driving. (To name but a few)

    Are skeptics are only skeptical of fringe topics that receive gratuitous media attention? Are these other, more pervasive, causes of death simply the collateral damage of science based medicine/technology and therefore “off the hook” from equivalent skeptical analysis?

    Prescription drugs:
    “Every day in the United States, 113 people die as a result of drug overdose, and another 6,748 are treated in emergency departments (ED) for the misuse or abuse of drugs.2 Nearly 9 out of 10 poisoning deaths are caused by drugs.3″

    ww.cdc.gov/homeandrecreationalsafety/overdose/facts.html

    Medical Mistakes in U.S. Hospitals:
    Now comes a study in the current issue of the Journal of Patient Safety that says the numbers may be much higher 2014 between 210,000 and 440,000 patients each year who go to the hospital for care suffer some type of preventable harm that contributes to their death, the study says.

    That would make medical errors the third-leading cause of death in America, behind heart disease, which is the first, and cancer, which is second.

    http://www.scientificamerican.com/article/how-many-die-from-medical-mistakes-in-us-hospitals/

    Distracted Driving:
    The number of people killed in distraction-affected crashes decreased slightly from 3,360 in 2011 to 3,328 in 2012. An estimated 421,000 people were injured in motor vehicle crashes involving a distracted driver, this was a nine percent increase from the estimated 387,000 people injured in 2011.

    http://www.distraction.gov/content/get-the-facts/facts-and-statistics.html

    Acupuncture Death rate:
    “These studies have shown an incidence of mild, transient acupuncture-related adverse events that ranges from 6.71% to 15%. The most common adverse events of this type were local pain from needling (range: 1.1–2.9%) and slight bleeding or haematoma (range: 2.1–6.1%). In a prospective observational study of 190 924 patients, the incidence of serious adverse events (death, organ trauma or hospital admission) was about 0.024%.5 Another large-scale observational study showed a rate of adverse events requiring specific treatment of 2.2% (4963 incidents among 229 230 subjects).6 Studies such as these have shown that in extremely rare cases acupuncture can lead to serious, sometimes life-threatening complications, in addition to mild and transient adverse events.”

    http://www.who.int/bulletin/volumes/88/12/10-076737/en/

  9. AmateurSkepticon 15 Jul 2014 at 7:02 pm

    @Teaser

    You have completely missed the point. Any evaluation of a proposed course of action must consider both the cost and the benefits of that course of action.

    Unless you are some sort of crank you would not propose that someone avoid going to a hospital for medical treatment because of the existence of medical mistakes which may occur there. Frankly the same logic would rule out any form of medical treatment in which case we might as well leave accident victims lying on the side of the road.

    By contrast, acupuncture provides no benefit other than a possible placebo effect. (At least that conclusion is strongly implied by the studies to date.) So while the harm is no where near as great as the other examples which you have raised, the net result of a cost benefit calculation is clearly negative.

    Perhaps you can suggest some other topic for discussion in this forum with an even greater net negative result. The ones which you have so far suggested do not qualify.

    By the way, please do not suggest that by inference I believe that there is a benefit associated with “distracted” driving — only driving in general. Now if someone were to argue in favor of distracted driving I suspect that topic would get a thorough airing here.

  10. jesse.huebschon 15 Jul 2014 at 7:03 pm

    Teaser – Even a single adverse event is too many when the denominator in cost / benefit is zero.

  11. RickKon 15 Jul 2014 at 7:11 pm

    Teaser is right again.

    Look at all the money we waste pointing out the dangers of certain items or behaviors when there are much more dangerous issues to be dealt with. Vastly more people die of drowning than of brain cancers, yet we pay brain surgeons much more than we pay lifeguards. What a waste!

    And Teaser is right to point out the comparison of acupuncture to pharmaceuticals. It is stupid of us to fund the FDA and other agencies to ensure drugs do something other than just provide placebo theatrics. By demanding proof of efficacy, we force drug companies to put chemicals in their pills that actually DO stuff to our bodies and brains. Think of the risk! We all would be so much safer and better off if we removed the efficacy requirements for drugs and just let pharmaceutical companies sell us sugar pills wrapped in convincing advertising. Then we’d have a safe and happy public not living under the constant fear that their drugs or procedures might actually affect them physically.

    When I had pneumonia and when my wife had Lyme disease, we were constantly afraid that the drugs we took might actually do something in our bodies. How much better we would have slept if we’d followed Teaser’s philosophy and just gone to a homeopath or a reiki healer or an adcupuncturist with an unblemished safety record.

  12. Fitzon 15 Jul 2014 at 7:46 pm

    It seems some lessons may have been learned.

    http://www.telegraph.co.uk/culture/tvandradio/bbc/10944629/BBC-staff-told-to-stop-inviting-cranks-on-to-science-programmes.html

  13. roadfoodon 15 Jul 2014 at 8:21 pm

    Am I the only one wondering why the BBC put this up eight years after it was originally broadcast? And so soon after they made the very admirable move of dropping false balance? Was someone at the BBC just blindly posting old shows without paying attention to the content? Just seems odd (and before anyone even thinks to ask: no, I am not suggesting any conspiracy, just dumbness).

  14. Malfeitoron 15 Jul 2014 at 9:01 pm

    So because more people die from things other than acupuncture we should just ignore the misinformation?

    How I loathe the “don’t you have better things to worry about?” Gambit!

  15. Sylakon 16 Jul 2014 at 12:40 am

    Nice review Dr Novella.

    It’s Disappointing to see a scientist science training go off the rails like that. She’s a physicist, No medical training, but she suppose to be trained in understanding how studys, and stats work. She must had a previous belief in it. it seem that again even critical thinking is not embedded enough evn in scientist minds.

    I wonder if SSR will end up here, Each time the word acupuncture is written in a article on SBM, you can be sure that our master troll SSR will come with his backpack full of fallacies. Well Teaser seems to have replaced him here.

    @skeptico : Nice work over there, really spot on comments.

  16. SteveAon 16 Jul 2014 at 7:16 am

    Unfortunately a lot of BBC science programming now has to include some sort of narrative to make it palatable to a general viewing audience (perceived as drooling knuckle-draggers for the most part).

    In this case the story was: ‘Sceptical scientist changes her views on intriguing ancient medical practice.’

    They could have gone the other way: ‘Professional acupuncturist reviews evidence and discovers that he has wasted his life pedalling worthless sham medicine.’

    Not so upbeat though, and they like to keep it cheerful.

  17. Pete Aon 16 Jul 2014 at 9:23 am

    A guaranteed recipe for concocting a lucrative media story is to include two basic ingredients: scientist and fMRI.

    Of course acupuncture will show a changed fMRI response in the patient. Far less obviously, but equally convincingly, the fMRI of a dead salmon could demonstrate a neurological response:
    http://www.newscientist.com/article/mg22029391.500-neuroscience-wrongs-will-make-a-right.html

  18. BillyJoe7on 16 Jul 2014 at 10:20 am

    The placebo effect induces changes in fMRI:

    http://www.ncbi.nlm.nih.gov/pubmed/14976306

    How could it not?
    So, to say that acupuncture induces changes in fMRI is consistent with it being a placebo effect.

  19. Skepticoon 16 Jul 2014 at 10:52 am

    Steven/pdeboer

    Thanks. To be honest though, it wasn’t that hard. No arguments I haven’t heard before (numerous times).

    Also, nice article today on Science Based Medicine, Steven. You summed it up very well, as usual.

  20. grabulaon 16 Jul 2014 at 11:12 pm

    @Teaser

    “Your continued assault on acupuncture as a threat to patient safety is illogical. Death or serious injury by acupuncture is nominal when compared to prescription drug deaths, deaths from care in US Hospitals or death/injury from distracted driving. (To name but a few)”

    Your first sentence is nonsensical – opposition to anything that costs someone money, and misleads the public on science is never illogical.

    Prescription drug deaths…and how many lives are these drugs having an actual positive effect on? You woo believers seem to forget this aspect of modern medical science. You’d rather provide something that does absolutely nothing – letting people die or remain sick regardless, than apply modern science to the problem and treat the vast majority of issues something like acupuncture can’t. It’s misleading and disengenuous. As for people dying in hospitals, yeah – when people are ill or injured they sometimes die, and where do those people congregate primarily?

    Finally distracted driving? strawman? non-sequitor?

    “Medical Mistakes in U.S. Hospitals:”

    I notice you don’t provide any links to the number of successes in US Hospitals. Too coneveniently works against your ridiculous argument?

    “Acupuncture Death rate:”

    lol, so it doesn’t TREAT anything but apparently does harm….hmm see where you’ve gone wrong here?

  21. Teaseron 17 Jul 2014 at 10:55 am

    @Grabula

    You are the King of Redundancy, the Prince of Rehash and The Duke of A Day Late and A Dollar Short.

    (I apologize in advance if my gender assumption is incorrect.)

  22. grabulaon 17 Jul 2014 at 10:28 pm

    @teaser

    So no rational response from you then? Oh wait, you can’t have one, sorry my bad. I’d retreat to ad hominem too if I had nowhere else to go.

  23. mumadaddon 18 Jul 2014 at 7:14 am

    Actually I’ve started a change.org petition to replace all potentially risky medical procedures with giving the patient a signed picture of The Queen. I’ve done the risk/reward calculations myself:

    Risk = 0
    Reward = 1 (who doesn’t want a signed photo of the Queen?)

    This will also save the NHS an absolute ton of money, which we can invest in research to find more completely risk free procedures.

  24. grabulaon 18 Jul 2014 at 9:24 pm

    I guess you get kicked in the pants so hard on the way out the door it’s hard to turn around and come back in.

Trackback URI | Comments RSS

Leave a Reply

You must be logged in to post a comment.