Mar 12 2013

Another Acupuncture Meta-Analysis – Low Back Pain

As Carl Sagan observed, “randomness is clumpy,” which means that sometimes, for no specific reason, I write two or more blog posts in a row about the same topic. Perhaps it’s not entirely random, meaning that when a topic is being discussed related news items are more likely to come to my attention.

In any case, there was recently published yet another meta-analysis of acupuncture, this time specifically for low back pain. The findings and interpretation add to the pile of evidence for two important conclusions:

1 – Acupuncture does not work.

2- Acupuncturists refuse to admit that acupuncture does not work.

I would further infer from these two unavoidable conclusions the dire need for a greater understanding of the core principles of science-based medicine.

From the abstract of the new study:

Compared with no treatment, acupuncture achieved better outcomes in terms of pain relief, disability recovery and better quality of life, but these effects were not observed when compared to sham acupuncture. Acupuncture achieved better outcomes when compared with other treatments. No publication bias was detected. Acupuncture is an effective treatment for chronic low back pain, but this effect is likely to be produced by the nonspecific effects of manipulation.

The authors expand upon this in their discussion:

When acupuncture has been compared with no additional treatment or blank treatments, the results have consistently shown that acupuncture is significantly more effective. However, the results are relatively inconsistent when acupuncture is compared with sham acupuncture. Some studies found there is moderate evidence that acupuncture is more effective than sham acupuncture, while other studies have published strong evidence of no significant difference between acupuncture and sham acupuncture.

The meta-analysis itself seems well done and fairly rigorous. That is something that I have generally noted about the acupuncture literature – while it is littered with poor studies, preliminary studies, and unblinded studies (as is most medical science), it also contains many very well designed and executed studies, and solid systematic reviews and meta-analysis. This is typical of clinical science in general. Preliminary studies eventually lead to definitively designed studies that give us the most reliable answers to our clinical questions, in this case, does acupuncture work for specific indications.

It takes time to sort out the answer to these clinical questions, and we have to look at the overall pattern in the literature to see that answer. Several patterns may emerge. With treatments that actually work we see initial conflicting results, with a positive bias, and then as better and better studies are executed we zero in on the actual safety and efficacy of the treatment. The ultimate magnitude of the treatment effect tends to be lower than initial studies indicated, but it is still effective.

With treatments that do not work we see a similar pattern, but the effect size declines to the point that it overlaps with zero. For such treatments the consensus conclusion is that the treatment does not work. Sometimes proponents claim we need higher doses, subgroup analysis, or longer studies to see the effect. Sometimes they are right, but most often this is just special pleading that does not rescue the therapy from the discard bin.

Related to the above pattern there is a broader pattern within the world of science in general (not specific to clinical studies). It often happens that effects appear to exist when they are observed in an unblinded fashion, but then they vanish when properly blinded controls are put into effect. The poster child for this phenomenon is the infamous N-Rays. Dozens of scientists thought they saw the ephemeral rays, until blinded experiments showed that they do not exist.

When a phenomenon disappears under proper blinding, scientists conclude that the phenomenon is not real. That is the essence of science – controlling for bias and other variables to separate what’s really real from what only seems to be real.

Let’s apply this basic scientific principle to the evidence for acupuncture. What the researchers above found in this case, and admit is what reviewers have found in general, is that when you look for an acupuncture effect in an unblinded way it seems to be there, but when you look for it with blinded controls the effect vanishes.

Acupuncture = N-rays = not real

Studies find either no effect at all, or a tiny effect that is indistinguishable from no effect. This latter point deserves some clarification, however.

In my prior post on the response of the authors of an acupuncture meta-analysis to skeptical criticism, I briefly discussed the difference between statistical significance and clinical significance, and this concept is important here. Clinical researchers generally understand the concept of statistical significance – that the probability of the outcome of a study being what it is, or greater, is less than 1 in 20 (p-value of 0.05 or lower). There is also something called a confidence interval which is an attempt to statistically measure the range of the real effect size given the measured effect size. If this range overlaps with zero, then we should not reject the null hypothesis.

This is all fine, as far as it goes, but it is all premised on a massive assumption – the lack of any systematic bias in the research. The statistics are based on the probability of random results, not biased results. There is therefore another way entirely that the results of an experiment can be indistinguishable from no effect – the first is statistical, the second is bias.

One of the core bits of wisdom of SBM, well documented in the literature and clear from any proper historical understanding of science in general and medical science specifically, is that it is difficult to impossible to completely eliminate bias and error from research. We have to therefore assume a minimal level of such bias and error, which we can think of as noise in the data.  When the signal is very small, the signal to noise ratio is therefore very low, and the results are not reliable – meaning that they are indistinguishable from no effect (even if they are statistically significant).

We see this with ESP research, with homeopathy research, and with countless practices that are based on fantasies and not reality. The acupuncture research looks like ESP research because neither are real phenomena.

How do we overcome these limitations, and how do we detect a small effect if any such signal will likely be lost in the noise? That is where replication comes in, doing large rigorously designed studies, and carefully controlling for every variable. Meta-analysis really does not solve this problem – that only addresses the issue of statistical significance, but not bias and error. This is why meta-analysis is a poor predictor of later definitive research. It is also why some researchers recommend “best-evidence synthesis” over meta-analysis.

In recent years researchers have developed tools for detecting and correcting for publication bias, although this is not always done or done properly. There is no way, however, to detect or measure researcher bias. This can only be minimized with the most rigorous methodology, and corrected through replication.

We also need to take prior probability into account. If a treatment defies well-established science, we should be more skeptical of the small signals that are lost in the noise of bias and error. The threshold for acceptance should be higher. This is also why we favor a Bayesian approach to analyzing data, over the more common p-value approach.

This new meta-analysis adds to the consensus of research that shows that acupuncture, as defined by sticking acupuncture needles into acupuncture points, is not a real phenomenon. It does not matter where you stick the needles. It does not matter how deep you stick them, or even if you stick them. There is no evidence to support the reality of acupuncture points or any proposed underlying mechanism. We have low prior probability, with evidence of lack of efficacy for the variables specific to acupuncture.

We also have the completely unsurprising finding that when unblinded comparisons are made a typical magnitude of placebo effects is seen. This does not mean that acupuncture works, by any sound reasoning. Proponents keep desperately trying to make this argument, however, in defiance of all scientific principles.

Acupuncture does not work. The therapeutic interaction that surrounds the administration of acupuncture has the predictable non-specific effects, bias, and illusion that comprise observed placebo effects. The same would be true of any intervention, no matter how worthless or absurd. You can make up an unlimited number of fanciful treatments and substitute them for acupuncture and you would get the same non-specific effects. That does not mean that any of them work either.

The authors conclude:

The current data shows that acupuncture is effective in providing long-term relief of chronic low back pain, but this effect is produced by non-specific effects that arise from skin manipulation. Further research is needed to evaluate which the effects of acupuncture compared with other treatments. Moreover, acupuncture should be used in combination with other treatments in the treatment of chronic low back pain. In view of the often refractory nature of chronic low back pain, comprehensive treatment plans, which may involve acupuncture, are urgently needed.

This conclusion is absurd and not justified. It should read:

The current data show that acupuncture is ineffective for low back pain, as it is indistinguishable from sham treatments in blinded comparisons. Acupuncture is therefore not recommended as a treatment and has no role to play in a science-based treatment plan for low back pain. Given the amount and quality of existing data, in addition to the low prior probability, no further research is warranted and in fact would be ethically dubious at this point. In view of the refractory nature of chronic low back pain it would be extremely counterproductive to waste any more time or resources on this dead end. 

There, fixed.

60 responses so far

60 thoughts on “Another Acupuncture Meta-Analysis – Low Back Pain”

  1. Meta-analysis is a poor predictor of later definitive research? This surprises me since there are ways to “statistically correct” for some biases in meta-analyses, like the file-drawer effect (although this requires some rather strong assumptions in itself, so the solution may be just as bad as the original problem).

    Would you occasionally get dizzy and confused if you were stuck doing a meta-analysis on the effectiveness of meta-analyses?

  2. I added a reference for the lack of predictive power of meta-analysis, and I also added a few more relevant references.

  3. rmaxwell710 says:

    One of my favorites is this from Arch Internal Med 2007:
    The researchers found that sham acupuncture worked as well as verum, but instead of concluding that that placebo effect is some effect, they decided that apparently acupuncture is so effective that it doesn’t matter where you stick the needles or how deep they go or whatever. Nice save.

  4. egiro25 says:

    I saw an interesting meme on facebook this morning that said…

    “The placebo effect is scientific proof that we have the ability to heal ourselves. Our thoughts are powerful enough to bring things into existence – when will we begin to absorb this?”

    I find this to be an intersting spin on the placebo effect and a possible way for practitionors of psuedoscience to trick people into believing the efficacy of their shams. They are just triggering the inner ability of our minds to heal ourselves.

  5. Thanks much, I’d never heard of Best Evidence Synthesis before. Interesting that there are so many different formal ways to interpret a collection of studies. Doesn’t make a reviewer’s job very easy.

  6. Murmur says:

    As a statistician, I would have to agree with being very suspicious of meta-analysis. If I do not have a very good view of the data going into a system, I am very suspicious of data coming out of it (Garbage in/Garbage out is not just a computer programming mantra). Meta-analysis takes data from studies I have had no sight of and then lumps it all together into some big mish mash in order to try and tease out some kind of “interesting” number. You can then include or exclude any number of the studies until you get the result you want and either make up some excuse to exclude that study (cohort too small… not done in America… or even if the study is too big, you laugh, but I have known people who have justified excluding data from populations that are too big as they are too “homogenous”… or even simply pretending they did not see that particular study).

    You really have to be a bit intelligent when looking at figures and try and understand what they are telling you. If you start off with a premise you have to be darn sure that the effect you are getting is what you were looking for and not just an artifact of how you have cut the data (I am talking about bias in doing the stats, nevermind the study itself) and how you present it. I once worked for a manager who insisted that a very mediocre 2% increase from 10% to 12% in performance was in fact a wonderful 20% increase in productivity that he could report to his directors.

    I could rant on for ages on this…

  7. locutusbrg says:

    I am continually enraged by the Shruggies in spinal treatment. I have attended national spine surgeon conferences. I am constantly at the mic in the question and answers sections. Complaining because board certified spine surgeon treatment options include acupuncture. It has moved forward to the front of Accepted/Safe useless treatment. It used to be marginalized but the drumbeat of useless data keeps propping it back up.

    Steve in your opinion what are a few of the best examples of well structured studies? When I do searches I have to wade through crap. Is it in your new books that you have published over at SBM. I haven’t finished homeopathy yet will move onto cam next?

  8. Enzo says:

    And another speedbump added to the forward momentum of acupuncture apologists. I used to think that CAM like this only persisted because it already had a foothold in culture, but with conclusions like that taken from the data, I’m pretty sure anything at all could gain momentum as CAM – recent or historical.

  9. Eric Thomson says:

    The study at least suggests that localized trauma (such as needle insertion) to the skin helps relieve pain, and that the whole theoretical apparatus of Acupuncture (chi with specific locations) is not important for this at all. They say as much in the discussion in a nice description that is itself skeptical of Acupuncture:
    According to the theory of traditional Chinese medicine, acupuncture outside Meridian
    lines cannot disturb or harmonize the Yin and Yang (Guan and Liu, 2009). In other
    words, theoretically sham acupuncture should be of no analgesic or therapeutic benefit. In
    our meta-analysis, acupuncture does not achieve better outcomes in terms of pain relief,
    disability rehabilitation or spinal flexion improvement compared with sham acupuncture.
    This indicates that the partial analgesic effect of acupuncture does not derive from the
    stimulation of the meridians, namely “De Qi”, but from the skin manipulation.

    So the title of the OP I would take issue with. Yes, the paper goes against Acupuncture, but not against the insertion of needles to relieve localized pain (which people can call ‘acupuncture-inspired approaches’ if they are worried about semantics).

    Just because it is not better than “sham” acupuncture (which also locally perturbs the skin) doesn’t mean that needle insertion doesn’t work (any more than saying because aspirin is not different from Tylenol shows that aspirin doesn’t work to relieve pain). This is the kind of sliding logic that Novella has consistently used wrt acupuncture studies.

    Note I have no horse in this race, I really don’t care, but if you look at studies like this there is an interpretation that doesn’t involve assuming the authors are stupid, subject to perceptual illusions, and which is antecedently plausible (especially given results in mice with adenosine receptors and skin irritation) and consonant with all the data out there. I tend to go with that one. I am frankly nonplussed by the knee-jerk name-calling and presumption of stupidity going on when analyzing acupuncture: it is not skepticism it is dickishness.

  10. eric – I have to disagree with your interpretation of the data.

    The evidence does not show that needle insertion has an effect.

    First – unblinded comparisons are not useful for drawing conclusions.

    Second – when needle insertion is controlled for in studies (toothpicks poking the skin compared to actual needle insertion) again, there is no difference. So the variable of needle insertion also does not work. There is no evidence to suggest that poking the skin with toothpicks has any physiological effect, and again no blinded comparison showing an effect.

    With acupuncture – all blinded comparisons are negative.
    Only unblinded comparisons show a difference.

    That is a pattern we see with phenomena that are not real.

  11. PharmD28 says:

    also worth mentioning as an analogy is the initial homeopathy study from what was it nature magazine as I recall from the 90’s (1997 as I recall) – where researchers thought their homeopathic solution was inducing an immune response…and did so repeatedly….then once Nature sent in a team to evaluate (which included James Randi) and they blinded them for analysis – no effect….taaahhhdahhhhh 😀

  12. ConspicuousCarl says:

    Eric cares so little that the entire concept of acupuncture as a healing system is completely obliterated and reduced to a [possible] comforting side effect. What if we found that there was no difference in outcomes between melanoma patients who had their cancerous spots removed and melanoma patients who had some other random patch of skin removed? Would we say “but it makes them feel a little better for a short time, so we’ll still suggest surgery”, or would we say “looks like this treatment is crap, we should tell them to seek the same minor comfort from any number of cheap and existing options instead of promoting and paying for a ruse.”

  13. Eric Thomson says:

    Steven: I agree that it is very important to have double-blind studies, and if it were not for rodent studies I would be much more skeptical about all this research. However, that doesn’t let them off the hook: they should come up with some way to double blind future studies.

    In terms of toothpicks poking the skin versus needles actually going in, I would hypothesize that the former is enough to trigger localized adenosine release (if it is enough to cause any pain, then the pain-dependent adenosine release should happen). However, if not, I would happily admit I was wrong for those particular cases.

    Like I said, I really don’t care much either way, if anything I came into this thinking acupuncture was bunk, but the research is pretty clear that it is not total bunk, but is a very expensive technique that produces some effect in a relatively special subset of cases.

    Because there is a small and real effect, I put acupuncture is in a qualitatively different class from such unequivocal crap as homeopathy, Reiki healing, etc..

    Conspicuous carl: you think it is placebo effect, which is what a lot of people here are saying. You should read the study I linked to above from Nature Neuroscience. It is a good paper.

    I doubt that mice are comforted by having needles stuck in their legs, that placebo effects caused adenosine release localized to the location of needle insertion in mice, and that placebo effects can explain why the pain ameliorating effects to go down in mice that are lacking adenosine receptors.

  14. eiskrystal says:

    Since sham acupuncture has been shown to be as effective as “normal” acupuncture, surely this means that sham acupuncture should now be investigated at great time and cost to see if it works.

    After all if they are willing to spend all this time and money proving one failing modality, surely a modality with similar effects should also be thoroughly investigated to the same degree.

    More research needed!!!

  15. Eric – the adenosine study has nothing to do with acupuncture. These studies show that there is a local physiological response to trauma – no surprise there. There is a temporary increase in pain and inflammation followed by a temporary normalization. This is all transient and non-specific.

    This is so far away from the claims made for acupuncture that it really does not rescue it to any significant degree.

    Also – the studies involving mice need to be put into perspective. The relative amount of trauma is fairly large, given the small size of the animals compared to the needle insertion. It is a huge stretch to compare this to poking the skin with toothpicks or dull non-penetrating needles. You absolutely cannot assume that a similar physiological response occurs.

    I do agree, however, that acupuncture is in a different category than homeopathy, because something physical is happening. It just turns out that the something physical is not specific at all to anything that can be labeled “acupuncture,” and is also clinically insignificant, making acupuncture a useless therapeutic modality.

    “Something physical happening” is not a sufficient threshold for legitimacy.

  16. Eric Thomson says:

    Steven: I have pointed out numerous times that just because something else would have the same effect doesn’t mean acupuncture (lower-case a, refering to needle insertion, not Acupuncture, the whole weird theoretical apparatus including Chi and such) doesn’t have the effect. I can’t discount the efficacy of aspirin for pain relief by pointing out that Alleve has the same effect.

    What is very odd is that we have a paper with mice where is an effect, and you have to say it isn’t actually acupuncture? That’s just strange. They inserted needles for pain relief, and it worked, but that isn’t acupuncture? We are really degenerating back to this semantic argument? As I’ve said, if you truly think such semantic points are that important then call it ‘acupuncture-inspired’ or whatever. I have been very consistent in my use of capitalization to refer to the kooky ancient theory, just to accommodate concerns about semantics.

    Finally, this is a largely academic point, as I would agree that on the clinical side there seems to be little of use here. I am also surprised it seems so easy for these people to get grant money; it is disproportionate to the results seen thus far, which are underwhelming to say the least (when compared with the older claims of Acupuncture).

  17. PharmD28 says:

    Has there been any discussion – like concepts for how the heck you could double blind acupuncture?

  18. tmac57 says:

    Too bad that it wouldn’t be safe or ethical to ‘blind’ patients with Propofol for the procedure/non-procedure. I have had it in 3 colonoscopies and two esophageal endoscopy exams,and never remembered anything. Of course they do have to stick you with a needle so…. 😉

    Maybe they could just use people with profound short term memory loss.

  19. Eric Thomson says:

    PramD28 that is the tricky part. I think it wouldn’t be too hard to use a servo mechanism to push pins within an external sleeve that hide the needles, but stop once a certain pressure is attained. You could even have it deliver electrical stimulation or twist the needles. The researcher would simply lower the external sleeve to the skin of the subject, and would not know the state of the servo/other mechanisms.

    To me that seems the easy part (though frankly the practitioners, I worry, would say they need to be able to see and “feel” the needles and that there is something important about actually being able to manipulate the needles with their hands–but there should be some scientifically-minded folk who will work hard with researchers to create a mechanism that will reproduce what they do with their hands).

    To me that seems the easy part (i.e., blinding the experimenter). This is even easier to do in rodent studies: one person can do the acupuncture, another person, who doesn’t know what treatment each animal received, doing everything else, and the treatments aren’t revealed until after the data are collected.

    For human studies, the tougher parts have to do with controls in the subjects. One, they should control for the pain response that is created with “sham” acupuncture such as needles in the “wrong” location or sharp objects that do not penetrate the skin but still generate pain. All of these likely produce localized adenosine release (a known pain reduction pathway).

    If you have completely naive subjects, you could give instructions like “Sometimes the needles feel like a little prick, but sometimes they are so thin that you cannot even feel them at all, or it feels like a light touch.” And then, using your servo-controlled mechanism, you can push with a relatively thick rod that will cause zero pain, but simply a light touch on the surface of the skin. This will not cause (pain-mediated) adenosine release, but will recreate any apparent placebo effect.

    You could have the servo make the exact same noise each time, so that is controlled. You could also just have cases where literally nothing actually is happening inside the sleeve.

    Something like this should be standard. You could then compare up to four cases: acupuncture needle insertion, painful poke with sharp object without skin break, painless skin touch, and no touch at all.

    This is all assuming it is worth it to study this, and spend a lot of money to do so. That is where I am not so sure.

  20. sonic says:

    eric- PharmD-
    I think the difficulty in blinding is great–

    If one defines acupuncture as ‘exact needle technique in exact locations’, then it would be possible to have a robot preform the actions and this could then be blinded through some method as eric suggests.
    (You could tell people they might not feel the needle at all- sometimes you won’t).
    But my experience (I went to an acupuncturist who was the ‘teacher’ in my area) is that acupuncture is not done that way. The person I saw considered the ‘ancient points’ as ‘guidelines’. She would ask me about what I was experiencing as she did the needle work and would sometime move a needle or put in another close to one all ready in– depending if I was getting the ‘response’ she was looking for.
    I’m not sure how others do it, but this lady was the ‘master’ around here, so I’m assuming she wasn’t doing anything wrong.

    I don’t know how it would be possible to double blind in that situation.

    Note– the authors of this paper are saying that ‘exact needle technique in exact locations’ doesn’t work. Of course the person I saw would tell you that too.

    My comments should not be taken as a recommendation or not of any treatment or non-treatment as they are not intended to be. 🙂

  21. Eric Thomson says:

    sonic those are good concerns, and interesting to hear (I have never been to one), but ultimately the practitioners should be able to work with researchers to get something with which they can all agree is good enough, even if it is not perfect it should be an improvement.

    They could likely need to work with some that are not blinded, with the servo mechanism with transparent guide tubes, to get it to a point where they were happy with its performance in the optimal case. And then do it blinded.

  22. Orac says:

    Perhaps Eric would like to see this take on the rat acupuncture/adenosine study:

  23. “When a phenomenon disappears under proper blinding, scientists conclude that the phenomenon is not real. That is the essence of science – controlling for bias and other variables to separate what’s really real from what only seems to be real.”

    Brilliant! Had to steal and post that on my FB … thanks for this article, excellent analysis as always.

  24. Eric Thomson says:

    Orac: my responses should be obvious based on everything I’ve written here. That post seems like an exercise in extreme confirmation bias. The final three paragraphs are truly astounding in their lack of application of the principle of charity.

    It tries to downplay the paper while actually finding little scientifically to criticize (and for good reason, as it is a good paper that everyone in this discussion should have already read). The one concern, that mice are small, and humans are big, so the results won’t generalize, is the only idea in there that I saw that is new, but it is a bit of a stretch (no pun intended), and something that can easily be addressed experimentally.

    Funny thing is, the post pretty much agrees with what I’ve written here, but spinning everything negatively just like Novella did, really just going through contortions trying to find anything bad, even scouring the press release associated with the paper to find fault. He pulls out what seems to be the standard chorus of saying the paper has little to do with acupuncture because it only shows that acupuncture needles can relieve pain, but does little to support the broader theory of Acupuncture. I’ve dealt with this concern ad nauseum.

    The third to last paragraph is worth reading.

  25. Eric Thomson says:

    Note I didn’t mean to hijack the thread with a new discussion of that paper, which we went through when it first came out. I’ll not comment anew on that in this thread.

  26. Enzo says:


    If we are inappropriately having this discussion in this thread, I’m sorry for contributing. But it seems relevant, so I’m quickly responding.

    I get what you mean about any usage of needles being sufficient to be called acupuncture, so let’s disregard meridians and such. If you are trying to argue that the adenosine paper presents a possible mechanism for acupuncture having a physiological response, then there are still some things that don’t match up.

    It isn’t simply that the needles are large relative to the mouse size. Remember, the paper only describes a LOCALIZED pain response. This doesn’t fit with acupuncture studies that demonstrate the acupuncture effect is just as strong regardless of the location of needle insertion — the paper would have to demonstrate systemic effects. Also, other studies show just twisting toothpicks over the skin generates the acupuncture effect, while this study shows the pain relief only occurs when the needle is inserted and twisted repeatedly during the data collection. The duration of the effect (1-3.5 hours in the study) also calls the feasibility of acupuncture into question (time and money for brief relief).

    So the study is very limited in how it can be used to explain the acupuncture effect. Maybe a launching pad for further study to control for the above concerns at best.

  27. daedalus2u says:

    If “sham” acupuncture with toothpicks and “real” acupuncture with needles have clinically equivalent results, then “sham” acupuncture with toothpicks should be done in preference to “real” acupuncture with needles because if you don’t break the skin, there is less potential for infection and other adverse side effects.

    It is unethical and should be considered malpractice for an acupuncturist to use needles instead of toothpicks.

  28. Eric – I think your characterization of David’s post and my criticism of the interpretation of the adenosine paper is unfair and misses our points.

    The phenomenon they found does not have anything to do specifically with acupuncture. It found a physiological mechanism that partly explains the local response to tissue trauma, which is an initial phase of increased pain and inflammation, followed by a normalization phase that suppresses these two things. These effects are local and transient. They are therefore only relevant to local and transient effects from any manipulative intervention (acupuncture, massage, heat, etc.)

    The effects are not relevant to the vast majority of claims made for acupuncture.

    We never denied that physical manipulation can have (and probably does) a local transient counter-irritation effect. There are other proposed mechanisms for such effects also.

    This is not acupuncture, however. Further, such effects are not clinically relevant to most of the uses of acupuncture, but probably does cause a short term non-specific effect that contributes to the overall placebo effect of such interventions.

    Further, the magnitude of the effect probably is proportional to the magnitude of the trauma – and that is where the size of the needles, manner of insertion, and size of the animals is relevant. To equate this to poking the skin with toothpicks is frankly absurd and ignores the high probability that the effect is proportional to the magnitude of the trauma.

    The fact that poking the skin with toothpicks has the same effect as needle insertion to depth suggests that this adenosine response is likely not relevant to what is being measured in acupuncture trials, which is (again) placebo and non-specific effects, not a response to acupuncture itself.

    Our objection to the paper was not the basic science showing the role of adenosine receptors, but to the interpretation that this “shows how acupuncture works.” That is grossly misleading, and actually has the potential of distracting from otherwise useful science. David hit the nail on the head when he observed that this is one way in which woo poisons real science.

  29. The difficulty is that practitioners see these papers, they assume they are relevant. When I have ongoing discussions with a naturopath on twitter, her response is to cite the paper- and feels comfortable sending people off to get needled.
    The idea that a treatment, even if it has no efficacy, is ok as long as it is not harmful neglects that we have a limited amount of resources to apply to medicine – so to pay for treatments that have no value diminishes that revenue for treatments that work.
    One might argue as to “how it works” or what effect a fine needle in a skin is- if it is gating- but the overall question is this: does it work.
    It is a treatment without a disease, and a group of people who practice it seeking for relevance.
    Great post, as always

  30. Nice post. Thank you for fighting pseudoscience.
    You pointed out a big problem that affects the thinking not only of the laypeople: statistics should be based on the probability of random results, not biased results.
    Unfortunately many people are blinded by statistics without stopping to think whether experiments were carried out rigorously or the results really make sense. My experience is that some scientists will be more willing to accept without questioning a graph if you just put a few stars over the bars; never mind whether the differences are relevant or just 1%.
    Regarding acupuncture itself: I use to be a die hard rationalist that would ban any pseudoscience; now that I am older I became more softy and philosophical I have this ethical problem: if you can give someone a gelatine pill and this person will feel better (and we know it has nothing to do with what is in the pill but with what happens in his brain) will you give it to him or rather tell him the truth It is a tough decision.

  31. PharmD28 says:

    Eric, I want to hear all of what you are saying….I think all of the “interweb skeptics” here want to give your arguments full attention and consideration…but I think still, I find your arguments to be soooo far reaching still. I do not need to entirely close the door and say that acupuncture but the point that it may have some local effects…or even IF it had some form of systemic effects that were potentially useful….you would still need to translate that into measurable differences in human subjects through good research….

    so your point seems to be irritation with what you see is a confirmation bias driven too quick to dismiss stance by some folks….I find this charge in light of the clinical research on this subject to be a huge stretch IMHO. Especially since they seem to have appropriately responded and do not deny apparent local tissue effects that you are pointing to no?

    Another question for you Eric regarding daedalus’s last point…are you at this point comfortable with the assertion that various forms of “sham acupuncture” are clinically in humans equally effective? If so, then we need only indeed study non-puncture types of “acupuncture”. The assumption there is that these local skin effects could causes more systemic pain relief…adenosine and other perhaps?

    I dont know Eric…I want to give your points full consideration….but it just seems so darn far of a stretch friend….I am at this point comfortable with the idea that we leave acupuncture as a thing of the past….inneffective….

    It seems also a basic matter of practicality for us to give up and at the very minimum call it “functionally a placebo”….what do you think of that concept? So perhaps you could maintain that perhaps there are effects…and perhaps in some highly specific subgroups they could concievably have a meaningful and robust (more than a control group) effect…but it seems at this point very impractical to take this research further…

    I for one think that using government money at this point for an exotic robot driven double blinded acupuncture treatment is perhaps foolish? In principle, I can entertain the notion that it could make a difference, but I am very very very highly skeptical of this based on the clinical evidence to date…I think it is highly reasonable and highly practicle to indeed call acupuncture at a very minimum a “functional placebo” at this point.

    If the acupuncture community pools their money to pay for 2-3 quality studies in coming years showing finally clear benefit, then I will change my mind about the treatment….but at this point failing to see this as needed, I feel is useless….lets move on and use that money to study something new folks….my 2 cents.

  32. locutusbrg says:

    @ Eric Thomson
    I simultaneously agree with you on one point while finding the underlying practicality of your argument questionable.
    Meaning that I agree with you about saying that criticizing effects from the procedure because they are not “acupuncture” is actually a little weak. To criticize acupuncture on the basis of the definition is a little bit of(to use a novella expression) hand waiving excuse. There are situations where the reason for a medical treatment is not properly known but it is effective. I do not think that it justifies the treatment just that this particular argument lacks vigor.

    “In terms of toothpicks poking the skin versus needles actually going in, I would hypothesize that the former is enough to trigger localized adenosine release (if it is enough to cause any pain, then the pain-dependent adenosine release should happen). However, if not, I would happily admit I was wrong for those particular cases.
    Like I said, I really don’t care much either way, if anything I came into this thinking acupuncture was bunk, but the research is pretty clear that it is not total bunk, but is a very expensive technique that produces some effect in a relatively special subset of cases.
    Because there is a small and real effect, I put acupuncture is in a qualitatively different class from such unequivocal crap as homeopathy, Reiki healing, etc..”

    If you assume that the same mechanism related to skin/adenosine appears in toothpick controls why use the needles? The only reason to stick needles into people is because people would not pay $800.00us for getting poked with toothpicks. At many levels it is sham to impress upon the patient/victim, that they are in fact getting a miraculous treatment. If it is a physical effect not a psychological one why haven’t practitioners moved en-mass to toothpicks. I mean it is safer and it could be done much more frequently. Why break the skin if you don’t have too?

    If David Blaine came out and before every stunt and told everyone exactly how it was done no one would stay for the show. If Acupuncturist took out a box of toothpicks and said here we go. No one would stay. It is a magic show and if you lifted the curtain so would the benefit.

    As soon as they are willing to drop the unnecessary puncture part I am on board.

  33. Piebald Skeptic says:

    re studies on acupuncture.

    In the 1990’s some German researchers showed identical FMRI responses to sham acupuncture and ‘real’ acupuncture. They used the concealed needle technique without skin penetration.
    The problem with researching placebo remedies is one of belief. People do not subject themselves to such remedies unless they believe, hence there have been no studies that compared efficacy between believers, and non believers. Also no studies have compared suggestibility crossover in acupuncture subjects that would allow comparison with an unbiased sample.
    If the objective is to look for a biological response it is necessary to randomly allocate believers and non believers to the sham and ‘real’ treatments, and to obtain an objective measure of response and not simply a reported response which is subject intervention bias.
    Despite clear evidence of lack of efficacy proven over several years with biased samples( i.e. biased in favour of proving efficacy) it is to my great shame, that both Australian Colleges of General practice recognise acupuncture as a real and legitimate treatment and offer points for CME in acupuncture.
    The Commonwealth of Australia also uses my tax money to pay for such ‘treatments’, and divert scarce public health money away from proven interventions in indigenous communities.
    Hopefully the Obama health system will avoid reimbursing useless flim flam and pay only for scientifically proven interventions.

  34. BillyJoe7 says:

    I’m sorry but I’m calling Dunning-Kruger on Eric.

    By his own admission he is “not bothered either way” which could only mean that he has not studied acupuncture to any great extent. Yet he feels competent to criticise those who have studied and written about acupuncture for many years. He accuses them of failing to give credit where credit is due because of what he characterises as their “extreme confirmation bias”. Yet he doesn’t realise that his deficient knowledge of the subject is was caused him to make that accusation in the first place.

    It seems to me he should be asking for explanations, not making accusations.

  35. madmidgitz says:

    @everyone and specifically Eric

    there is a huge issue here no-one noticed
    if stabbing rats with needles is what we are basing things on shouldn’t we scale up and stab people
    with nails

    just a fun thought
    a needle to a rat is a fencing sword to us
    trauma causes effects, nothing new here


  36. elmer mccurdy says:

    One thing that I think is missing here is that the type of acupuncture for which toothpicks are a valid placebo is a fairly recent development, originating as I understand it in Japan (I don’t have any citations for this offhand, sorry). Traditionally acupuncture has used larger needles and been more painful (and therefore would be disliked by most sufferers). I know that some MDs, rightly or wrongly, use dry needling, which is itself excruciatingly painful, on patients with with very severe pain, and a lot of such patients claim it works. From my own observation of my own sensations over the years, my feeling is that painful stimuli to the skin (or really almost any strong, novel stimuli, at least sometimes) can distract from muscle pain and help overcome muscle guarding (though I’ve no experience with needling). I have no idea how you could test this rigorously, but presumably there will be some sort of technology for this eventually, if there isn’t already (unfortunately, nobody even thought of the toothpick idea until three decades after acupuncture was first popularized in the West).

    This has nothing to do with acupuncture points, but even there, I know there is something called an “a-shi (‘Oh, yes!’) point,” based on the patients painful response to pressure, which, if nothing else, contradicts Dr. Novella’s claim that this is purely based superstitions based on life force or whatever. An it seems to me that, if we were somehow able to do controlled experiments with painful needling, we might also want to take a look at those.

    All that said, I’ve never seen fit to try acupuncture myself, largely because of Dr. Novella’s analyses of those toothpick studies, which were what drew me to this site in the first place, and I’m not sure I would cput any more resources into researching it were my choice. But it seems to me that Dr. Novella’s absolute confidence that there’s nothing there worth further study is unwarranted.

  37. elmer mccurdy says:

    One other thing: from what I’ve read or been told, the reasons for studying multiple forms of pain therapy is that, as I understand it are 1) different treatments work for different people, and finding the right match is often a matter of trial and error 2) different treatments can be used in combination, often with a total effect that is greater than the sum.

  38. madmidgitz says:

    @ elmer

    that is the most self deluding rationalization i have ever heard on this blog
    even the apologists pretend to be rational

    no offence but you have been told wrong or your listening to the wrong people

  39. elmer mccurdy says:

    What am I wrong about?

  40. elmer mccurdy says:

    If you’re referring to the comment starting “One other thing,” I don’t even think those 2 statements are particularly controversial among pain doctors.

  41. elmer mccurdy says:

    Oh, well, it doesn’t really matter. I’m going to go look at my book.

  42. BillyJoe7 says:


    “if stabbing rats with needles is what we are basing things on shouldn’t we scale up and stab people
    with nails”

    Nail on the head.
    But you haven’t read the commentary if you think that has gone unnoticed.

    And, just a word of advice, elmer mccurdy is definitely someone not worth responding to other than with laughter or derision. His contributions are either intentionally or unintentionally comical. Since it is never clear which, I think silence is the appropriate response.

  43. elmer mccurdy says:

    But not from me!!!

    The only citation I could find through quick googling for my comment about Japanese acupuncture is the following from wikipedia: “Japanese acupuncturists use extremely thin needles that are used superficially, sometimes without penetrating the skin, and surrounded by a guide tube (a technique adopted in China and the West).”

    I do believe that this is standard now, and that it’s a fairly recent development, but it’s possible I’m wrong on this one point.

    I’m confident about the other statements of fact I made, though (i.e. chronic pain treatment is largely trial-and-error on an individual basis; pain specialists combine multiple treatments with the effect that, in the words of a pain specialist I spoke to, “2+2=5”; some MDs treat chronic pain with excruciatingly painful dry-needling). These statements are true!

  44. elmer mccurdy says:

    … and there happen to be a bunch of citations in that wikipedia article on the painfulness of various kinds of acupuncture, in case anyone feels inspired to take a look-see whilst silently deriding me.

  45. elmer mccurdy says:

    I’m on a roll now, baby!

    One of the more mystifying moments in my life was when a doctor first told me that I could treat my pain with heat. Or cold. Or heat. Or cold. It seemed absurd! It was absurd! It is absurd! But it works! Sort of.

    But here’s a tip: it works even better if you turn up the heat high enough to cause pain. Leastwise for me. Just be careful not to burn yourself.

  46. elmer mccurdy says:

    Acupuncture today! Acupuncture tomorrow! Acupuncture forever!

  47. BillyJoe7 says:

    BJ: “And, just a word of advice, elmer mccurdy is definitely someone not worth responding to other than with laughter or derision. His contributions are either intentionally or unintentionally comical. Since it is never clear which, I think silence is the appropriate response”.

    As you can see, I was not being unkind.

  48. elmer mccurdy says:

    Oh, poop, BillyJoe7, normally when I stop by here you’re in the midst some all-day debate about something utterly trivial, and you seem to have a habit of lobbing insults at whomever you disagree with.

    Obviously I’m being facetious with some of the things I’ve said here (and, yeah, I do like to be silly), but I don’t apologize, because, what the hell, you and whatisname decided to dismiss me out of hand.

    My claims are not “self-deluded”; they’re correct (as I say, I’m a bit less confident on my claims about the history of acupuncture; though I’ve done some reading on this history, I’m not an expert on it, but clearly no-one here is, including our host, and he’s seen fit to post on it).

    I’m not exactly arguing for acupuncture, nor disagreeing with Dr. Novella’s analyses of individual studies, but I think that, as he occasionally does on some subjects, he went the extra step into demagoguery in dismissing the possibility of any value from studying it further. That’s all.

  49. Harbo says:

    Its a pity, that Dr Novella goes to great pains, to explain that the signal to noise ratio, makes the further expensive pursuit of this modality futile and wasteful, and is accused of dickishness and demagoguery.

    Yes we have many ways to treat pain… but as the old physician said “if there are many ways to treat a problem, it means that most of them are no good”.. so when we establish that one way is as close to useless as can be proved we should spend our limited resources elsewhere.

  50. PharmD28 says:

    “But it seems to me that Dr. Novella’s absolute confidence that there’s nothing there worth further study is unwarranted.”

    I disagree…and we have gone around and around as to why. Also use of the term “absolute” is the problem here….science does not deal in absolutes…and I think many would agree….it is rather instead that the reserach to date is highly consistent with placebo effects…”highly consistent” does not equal “absolute”….but practically at this point, it is getting as close to it as we need to be in my view.

    I mean, at what point to we move on from feeling any need to research this treatment?

    My thoughts are that no longer should the funding for further such research come from the government or tax payers…the acupuncturists can tax themselves within some form of organization or something to do all the research that others can think of to tease out what little plausible clinically useful treatment may be there….

    And it has no place within the VA, DoD, medicare, or coverage by private insurance companies because it is simply not evidence based…it should be relegated entirely to out of pocket expense by customers, not from my insurance premiums or tax dollars. period.

  51. BillyJoe7 says:


    Actually I was trying to do MMG a favour.
    You were just collateral damage.
    Sorry about that.

  52. Eric Thomson says:

    Revisiting this, I don’t have much new to add. It is clear that my disagreement with Steven comes down largely to emphasis and semantics, as I don’t find anything substantive to critique in his previous post. I agree that the papers in question do not promote the practice acupuncture as it is presently used by practitioners.

    PharmD Your earlier post directed at me seems mostly reasonable: I have never advocated for clinical use of acupuncture (I lean against it, fairly strongly, because of the concerns Steven brings up).

    I have simply pointed out one fact: the pain alleviation that comes with needle insertion (in some cases) is unlikely to be explained away as placebo. It seems almost Creationist levels of denial to say otherwise at this point, given the rodent studies. Yet people here keep parroting the “placebo” charge as a universal truth about acupuncture. (obviously this isn’t to say that many instances are not placebo).

    BillyJoe stepped in with some non sequitors, evaluating my posts with:
    By his own admission he is “not bothered either way” which could only mean that he has not studied acupuncture to any great extent.

    This doesn’t follow. I have studied it quite a bit since two years ago, even reviewed a paper for a journal on the topic. By saying I wasn’t bothered either way, I meant I don’t have a vested interest in the truth or falsity of any particular claims. It doesn’t mean I am not interested in the topic. It seems many people here know the truth already, but clearly haven’t looked over the relevant studies I am citing (I do not include Steven in this). This is how Creationists work.

    Yet he feels competent to criticise those who have studied and written about acupuncture for many years….It seems to me he should be asking for explanations, not making accusations.

    Instead of ad hominem, perhaps try to address the actual ideas. If it is frocks and genuflection you hunger for, perhaps you should join a church.

  53. PharmD28 says:

    “I have simply pointed out one fact: the pain alleviation that comes with needle insertion (in some cases) is unlikely to be explained away as placebo. It seems almost Creationist levels of denial to say otherwise at this point, given the rodent studies. Yet people here keep parroting the “placebo” charge as a universal truth about acupuncture. (obviously this isn’t to say that many instances are not placebo). ”

    I think it is unwise to compare this to “creationist levels of denial” Eric. This in of itself is comitting an obvious logical fallacy in your rebuttal – the false equivalency. Even if I grant your point as legitimate, then this comparison and denial by Novella is by a long shot nothing close to denial of the theory of evolution. I am glad that you are willing to argue your point, I KINDA see where you are trying to go with it, but do not poison the well of your argument with such an obvious logical fallacy. I mean, you said it yourself: “It is clear that my disagreement with Steven comes down largely to emphasis and semantics” – when was the last time disagreement with a creationist was merely “emphasis and semantics”?

    If you are correct about his “denial”…it is clear that it would not qualify as “denial” but rather a “misinterpretation” at the very least. Evidence of this fact is that he does seem to reply and acknowledge the study and what he (and many others) see as the limited implications of it. His last post seems very reasonable and addresses all of your points just fine from where I am sitting.

    “I don’t find anything substantive to critique in his previous post”

    If that is true, then how can you continue to trump up the concern that he is overlooking the obvious effects demonstrated in the rat study that supposedly negate the conclusion that it is entirely a placebo?

    I think perhaps your hang up is that you are seeing the comparison between acupuncture and sham different than he sees it….you used the analogy of comparing aspirin and alleve: “any more than saying because aspirin is not different from Tylenol shows that aspirin doesn’t work to relieve pain”

    Problem with your analogy there is that aspirin and tylenol both do in fact work….this prior to such a hypothetical study here is established….when studying acupuncture and sham acupuncture – neither are assumed to be effective….it is mostly assumed that sham acupuncture should have no clinically significant measurable repeatable effects, and thus may be a good control group…and when both groups show equal efficacy, it better supports the conclusion that both are in fact placebos.

    The rat study does not prove that acupuncture inspired prodding of people has “an effect”…ok, yeah, here is the semantics, yes there was “an effect”…probably poking a toothpick at the skin could perhaps produce some form of measurable physiological local change…but that is so far besides the point…this fact that there are possible measurable local effects are a far cry from making folks give much pause at all that there there are “effects” there that are anything more than local transient changes that have little or no bearing on acupuncture inspired treatments.

    now if the rat study and perhaps a few more like it existed AND the clinical experience in humans was positive for acupuncture….then we could say one of two possibilities:

    1) acupuncture works, but we are not really sure why and/or
    2) acupuncture work, see x, y, and z studies to show the likely mechanism

    But with x, y, and z local tissue effects established with local trauma + clinical studies perfectly consistent with no clinically significant effect….the conclusion is in my view “functional placebo effect” ONLY – and because we do not deal in absolutes, this distintion of “functional” is entirely unnecessary and really we can call it just plain ol’ “placebo”.

  54. BillyJoe7 says:


    You seem to be trying hard to be the sceptic’s sceptic, but you just come off as being a sceptic in need of a sceptic’s sceptic yourself. If, after two years of studying acupuncture and after the analysis provided here, you still cannot see how the frog study tells us nothing about acupuncture, perhaps even a sceptic’s sceptic is lost on you. Or perhaps you are willing to define anything at all as acupuncture in which case that wprd means nothing at all. And also you’re falling for CAM bait and switch.

    As for ad hominem, how would you classify this in response to SN:

    “I am frankly nonplussed by the knee-jerk name-calling and presumption of stupidity going on when analyzing acupuncture: it is not skepticism it is dickishness”

    And to Orac:

    “That post seems like an exercise in extreme confirmation bias. The final three paragraphs are truly astounding in their lack of application of the principle of charity”

    But apologies if I’ve misinterpreted you again. |:

  55. Eric Thomson says:

    PharmD28: you’ll notice in that post Steven didn’t dispute what has been my main claim this whole time, so that is why I had nothing to criticize. People have consistently been reading a lot more into my posts than I have actually said or implied. I’m not disputing everything, for goodness’ sake.

    As for my comparison to creationists, I think you are overreacting, as it was an analogy, and there are always disanalagies. The point is there is clear strong evidence with some studies and people ignore it when they make overgeneralizations because they think they already know what is true even if they are not familiar with all the evidence.

    BillyJoe you need to learn what ‘ad hominem’ means. While your at it, look up ‘tu quoque.’

  56. PharmD28 says:

    “The point is there is clear strong evidence with some studies and people ignore it when they make overgeneralizations because they think they already know what is true even if they are not familiar with all the evidence.”

    I think plenty of folks are perfectly aware of the evidence you speak of…but somehow, some way, they are coming to a different conclusion: that there is not “clear strong evidence” that supports what limited point you have attempted to make and points made in referencing such evidence are by far not “overgeneralizations”. I think you are just restating your premise that seems to have been, frankly, torn to shreads on this thread.

    As far as the creationist analogy, IMO, I am not overreacting….I think I was calmly stating the obvious. Keep the “creationist level denial” threat for when it is in that ballpark…it ain’t here…”just sayin”…take that advice or leave it…your choice.

  57. PharmD28 says:

    I am not entirely sure those previous examples are ad hominem, but “interweb skeptics” most definately was ad hominem….not that it is at all important in the deconstruction of your argument here.

  58. BillyJoe7 says:

    Put it this way, Eric….

    Before the rat study:
    Plausibility = zero; no qi, meridians, acupuncture points. It seems you agree.
    Evidence: real = sham = placebo = acupuncture does not work.

    After the rat study?
    Well, we already knew that trauma releases a chemical that acts on nerve endings to relieve pain, so why would traumatic acupuncture needle insertion be any different; and rats are considerably smaller than people who are actually not traumatised by acupuncture needles.

    I don’t know about you, but my prior probability is unaffected by this result.

  59. shanrocks666 says:

    in my personal opinion it is at least less of a scam than homeopathy. After all you are sticking needles into people.

    However, the chinese meridian theory is probably wrong. At least no one has ever found meridians – anatomically.

    There have been studies done where they stuck exactly on the chinese-defined points or a little “off”, and it did not matter.

    I can see where it could work by nerve or muscle stimulation.

    however, studies have been rare and are difficult to perform well – after all, how do you stick a “placebo needle” into someone as a control?

  60. PharmD28 says:

    “however, studies have been rare and are difficult to perform well – after all, how do you stick a “placebo needle” into someone as a control?”

    well….there have been at least a good number of pretty well designed studies looking at “real acupuncture” vs. some form of “sham” type of acupuncture as a control – toothpicks poking the skin, needles poking without penetration, or needles barely penetrating the skin. These “sham” treatments – were meant to be a control arm. Results from these studies show clearly that “real acupuncture” seems to be overall equally effective as “sham” forms of acupuncture.

    These finding are consistent with the conclusion that effects measured here are placebo effects. Or at the very very most absurdly generous interpretation, one could argue that perhaps there is something to randomly poking people with toothpicks….

Leave a Reply