Aug 29 2008
Another Worthless Acupuncture Study
Comming (by coincidence) on the heels of my recent blog entry on why I am skeptical of acupuncture, another major acupuncture study has just been published. This study looked at acupuncture for headaches, and was published in a major headache journal – Cephalalgia. The media, in typical fashion, gets the bottom line wrong, declaring: “Acupuncture may ease chronic headache pain.”
It took me about 10 seconds (literally) to realise that this study was utterly worthless. That is because it is unblinded – meaning that both subjects and physicians knew which patients were getting acupuncture and which were not, which further means that there is no way to know if the measured effect was all placebo.
Still I read the study to analyze it more carefully. My fellow skeptical blogger, Orac, has already beaten me to the punch, however, in deconstructing this study. I will not, therefore, repeat what he has written. He correctly points out the major flaws in the study – that it is not blinded, there is no sham-acupuncture group, and it is not fully randomized. He also points out many of the minor flaws, such as not distinguishing among the various headache types, some self-selection in the non-randomized group, and the reliance on subject memory (rather than real-time diaries) for data collection. Read his entry for a more detailed discussion of these fatal flaws.
What I want to discuss in more detail here is the very fact that this study was done in the first place. What purpose did it fill?
The purpose of medical research is to change the practice of medicine by giving practitioners better data from which to make treatment decisions. The typical progression of studies over time is such that they often begin with pilot or preliminary studies. These are often small, may or may not be blinded or randomized, and are usually straightforward in their design and the outcome being measured. Such studies are done to test the waters – to see if a treatment is generally safe and has any potential to work.
If preliminary studies are promising, then larger and more tightly controlled studies are done. If a treatment is controversial, such studies will often go through many permutations, as both sides argue about the need for certain controls or comparisons, variables to consider, and how to interpret the study outcomes.
Eventually this process leads to large, multicenter, tightly controlled consensus trials. These trials are designed to be as reliable as possible so that all sides will feel comfortable abiding by the results.
Once the clinical research on a specific treatment has reached these later stages, it makes little sense to go backwards and do poorly controlled preliminary trials. What would be the point of this? Sometimes there may be sub-questions, dealing with cost-effectiveness, combination therapy, sub-populations, etc., and these smaller questions can be dealt with by smaller or less well-controlled trials once the big question has been answered to satisfaction. But this is entirely different than doing a poorly designed study to answer the big question – does this treatment work at all; are its underlying assumptions sound?
Acupuncture remains scientifically controversial, despite the fact that proponents have managed to acquire some of the trappings of legitimacy, such as institutes and licensure. The proposed mechanisms (chi and meridians) are pre-scientific, and the post-hoc speculations for how it might work (counter-irritation) remain unproven at best. The clinical data, as I discussed in my previous post, shows a clear pattern that the better controlled trials tend to be negative – showing no difference between sham and “genuine” acupuncture.
So what, then, was the purpose of this trial? It is completely worthless scientifically. We are already way past the preliminary phase of doing uncontrolled trials. The only studies on acupuncture that are of any value at this point are rigidly controlled ones, with true sham-needles and tight double-blinding. That is – if you are interested in resolving the main question, does acupuncture work?
Of course, any peer-reviewed published study serves an academic purpose for those who publish it – it helps their career (the old saying of publish or perish still holds). Poor studies that are virtually guaranteed to generate a positive result (like this one) are also useful for marketing propaganda. They create great headlines – and most of the public are not going to read much beyond the headlines and so will be left with the sense that there is more and building evidence that acupuncture works. As propaganda this study is very effective.
As science – it is worthless.
Orac speculates, and I agree, that acupuncture proponents may be moving beyond double-blind sham-acupuncture studies out of frustration that they tend to be negative. They use a great phrase – calling this study a “pragmatic” study, meaning that acupuncture was studied how it would be used in practice. This is clever. Such studies have their use. One such study type is called an “intention to treat” study, where subjects are prescribed a certain treatment and then followed whether or not they are compliant with the treatment. This type of study is meant to answer specific pragmatic questions – if primary care doctors prescribe a daily aspirin to all patients who should be getting it what is the ultimate effect on public health?
But again – these types of trials are used to assess treatments that have already been shown to actually work. They are not used to answer questions about the effectiveness of a certain treatment or mechanism of action.
Doing a “pragmatic” study on a treatment that does not work, or has not been shown to work, is worthless.
But it sounds good. It is, in the end, and elaborate and deceptive strategy for bypassing good scientific evidence. This is what non-science-based practitioners do when good scientific evidence does not give them the answer they want. They stop doing well-designed trials and make excuses for poorly-designed trials. They make excuses for why their claims cannot be tested by accepted scientific practices. They make excuses for the negative results from rigorous studies while promoting the positive results from worthless studies.
A funny and blatant example of this is reported by Ray Hyman who was investigating applied kinesiology (the use of bogus muscle strength testing to assess for medical conditions, like allergies). After doing a blinded study of AK that was completely negative, he reports:
When these results were announced, the head chiropractor turned to me and said, “You see, that is why we never do double-blind testing anymore. It never works!” At first I thought he was joking. It turned it out he was quite serious. Since he “knew” that applied kinesiology works, and the best scientific method shows that it does not work, then — in his mind — there must be something wrong with the scientific method.
I predict we have not seen the end of the worthless acupuncture trials. I expect to see more of them, gullibly or lazily reported by the media as more evidence for acupuncture. What we cannot allow is for the proponents of dubious modalities and claims to alter the very definition of scientific medicine. They cannot simply change the rules to get the results they want.