Feb 17 2010

Choosing Your Evidence

The current process of districting in this country has many critics. Essentially their point is that congressmen carve up districts in odd ways so as to create an artificial majority for their party in their district. Of course, this leaves a majority for the other party in other districts, so everyone’s happy. This leads to the excessively high re-election rate in the House of Representatives of about 98%.

The best summary of this distorted practice in my opinion is by George Will, who says (and I paraphrase) – this is the practice of having politicians choose their voters rather than voters choosing their representatives.

There is a very similar phenomenon happening in the bizarro world of complementary and alternative medicine (CAM). CAM advocates seem intent on being able to choose their evidence, rather than have the scientific evidence choose which treatments are safe and effective.

This strategy is possible because there are many kinds of evidence in medicine. There is basic science dealing with biochemistry and physiology. There are observational studies that deal with observing what is happening in the world. And there are experimental studies where variables are carefully controlled and outcomes tracked. And within each category of scientific evidence there is a wide range of specific techniques and degrees of rigor.

If you take any topic in medicine that has been researched for years, you will likely find studies pointing in many different directions, with different inferences. Eventually we like to see that the majority of studies are pointing in a common direction, and it is this consensus of evidence that we base our opinions on. But there is almost always some disconfirming evidence, or studies that suggest a different interpretation.

As a result, if you choose which evidence to focus your attention, you can build a case for almost any conclusion you wish. This process is often called cherry-picking, and it is a common problem. It is also difficult to detect, unless you have a thorough knowledge of the relevant scientific evidence. Even then, it is a subtle mechanism by which bias asserts itself – for even if you look at all the evidence, knowing what relative weight to give to each kind of evidence requires expertise and judgment.

For example, advocates of science-based medicine (SBM) often refer to the plausibility of a claim or treatment. Plausibility generally refers to basic science evidence – does the alleged mechanism of a treatment make sense in light of what we know about physics, chemistry, and biology.

CAM advocates, however, often dismiss plausibility arguments as being “closed-minded” (an accusation which serves the same functional role as an accusation of lacking faith). To dismiss plausibility, however, is to dismiss a large segment of scientific evidence.

Anti-vaccinationists dismiss the extensive observational data showing no connection between vaccines and autism, and insist that the only relevant clinical evidence would be randomized experimental studies – which essentially cannot be done for ethical reasons.

Acupuncture advocates, meanwhile, dismiss randomized and blinded experimental studies that show acupuncture does not work, and prefer instead open-label “practical” studies which are almost guaranteed to give positive results.

Sometimes CAM advocates want to change the rules of evidence entirely. The existing rules do not give them enough elbow room – in other words, choosing the type of evidence to focus on still does not give them the results they want, they need to include low grade evidence as well. So they advocate specifically for allowing forms of evidence that are generally considered not scientific (like anecdotes), or only preliminary, and treating them as if they can be used to form clinical conclusions.

Further, they want to create a double-standard, where only CAM modalities are allowed to be based upon these unreliable or unscientific forms of evidence. In fact, the very label of CAM (or any of its variants) is designed to create such a double-standard.

SBM, on the other hand, advocates a single science-based standard for all health claims. Further, this standard is based upon an appropriate assessment of all scientific evidence, putting the different kinds of evidence into proper context.

Unfortunately, there is no simple formula for how to translate scientific evidence into clinical practice – this requires a nuanced understanding of the strengths and weaknesses of different kinds of evidence and how they apply to different kinds of questions. But there are many rules of thumb that the scientific community have developed over the last century – and it is largely these rules that CAM advocates wish to turn on their head.

Here are some examples: Basic science evidence is used to suggest new potential treatments, but those treatments still need to be subjected to clinical testing. Basic science also is a good indicator of plausibility, but plausibility is not sufficient. This can also be stated as – you cannot reliably extrapolate basic biochemical, physiological, or pharmacological phenomena into net clinical outcomes.

Basic science may also suggest that a treatment is implausible. While there is a base level of clinical evidence required to support even a highly plausible treatment, the more implausible a claim becomes the higher the bar should be raised for the level of clinical evidence required to conclude that it works none-the-less. For treatments that are highly implausible (homeopathy, say) the bar for clinical evidence should be raised high enough to counterbalance the basic science evidence that tells us it is implausible.


The reasons for exploring the complex issues surrounding the application of science to the practice of medicine is simple – we want to use treatments that are actually safe and effective. We want to know that a treatment really works, and does not just appear to work because of bias and self-deception.

I don’t think anyone can reasonably disagree with these goals. For health care professionals this is part of due diligence that is required, in my opinion, by professional ethics.

My core problem with CAM is that, in practice, it places allegiance to specific modalities above the interests of the patient, which require scientific due diligence. They do this largely by changing the rules of evidence so that they can choose, not the kinds of evidence that give us the most reliable information, but the kinds that tend to validate their preferred treatments.

This practice, which has been overtly adopted as CAM philosophy of medicine, functionally eliminates science as the basis for choosing treatments, since these practices allow anything under the umbrella of CAM. If homeopathy can get through the door, then anything can. And as further evidence of this my open challenge to name a CAM modality that has been rejected following negative scientific evidence remains unanswered.

More and more, as CAM culture becomes more sophisticated, this does not mean rejecting science, but changing the rules of science so that advocates can choose their evidence, rather than evidence choosing the treatments.

In that way CAM is more like politics than science.

9 responses so far

9 thoughts on “Choosing Your Evidence”

  1. banyan says:

    The solution to gerrymandering is proportional representation. Unfortunately the solution to quack claims seems to be a bit more complicated.

  2. daedalus2u says:

    Nice article. However a slight quibble, I know what you mean and I think that most every other scientist knows what you mean, but it is clear that the pseudoscientists don’t understand what you mean and won’t understand what you mean.

    Medical researchers don’t use double-blind studies because those are the scientific “rules”, dictated from on high by the scientific elite. They use double-blind studies because that eliminates the possibility of bias. If the researcher doesn’t know which patients got which treatment, the researcher can’t use that information to bias their observations, consciously or subconsciously.

    You can do “science” without double blind studies, however what ever studies you do that are not double blind, you cannot eliminate the possibility of bias, either on the part of patients or researchers. Your conclusions will always be unable to eliminate the possibility that the effect is due to some sort of bias. You can reduce that possibility by using instrumental measures, blood tests, tests that produce results independent of human subjectivity, but the possibility of some sort of unknown bias, mind-body thing, or even supernatural non-physical healing powers on the part of the researcher, cannot be eliminated.

    I am not suggesting that any researchers actually do have supernatural non-physical healing powers, but the possibility that such powers are causing a differential healing effect can be eliminated in a double blind study because the researchers don’t know which individuals to apply those healing powers to.

    The purpose of medical research is to produce treatments that are generalizable, that is treatments that essentially any clinician can apply. Unless clinical trials are done under conditions where the interactions of the clinicians are known and controlled, the possibility of the treatment “working” because of supernatural healing powers on the part of the clinician can’t be eliminated. That makes any treatment developed non-generalizable.

    CAM promoters want to have it both ways, they want to put the trappings of science on their non-blinded studies and claim that the outcomes are non-magical, but then pretend that the non-blinded clinician played no role in the outcome other than as an inert automaton going through the (magical) motions.

  3. BKsea says:

    To me, one of the great things that distinguishes real medicine from CAM is the presence of internal dissent. When new claims are made based on scientific evidence, there are always other science-based medicine advocates questioning the evidence. Even established consensus opinions are frequently subjected to new analyses that may call the consensus into question (e.g. are current guidelines for mamography screening or colonoscopy optimal?). Science-based medicine survives and thrives in this environment.

    In contrast, I can think of few (one) example from CAM where dissent was expressed from within the field. To me that says that everyone in CAM is afraid to throw stones lest their glass houses all come crashing down. The one case that comes to mind is the takedown of subluxations in chiropractic by chiropractors (see http://www.sciencebasedmedicine.org/?p=3022). This study actual made me more inclined to see a chiropractor.

  4. Joe says:

    @BKsea on 17 Feb 2010 at 3:28 pm “In contrast, I can think of few (one) example from CAM where dissent was expressed from within the field. To me that says that everyone in CAM is afraid to throw stones lest their glass houses all come crashing down. The one case that comes to mind is the takedown of subluxations in chiropractic by chiropractors (see http://www.sciencebasedmedicine.org/?p=3022). This study actual made me more inclined to see a chiropractor.”

    Well, it shouldn’t. Chiropractic “education” still centers on subluxations http://www.chirocolleges.org/paradigm_scope_practice.html Look at item 4. Also, read their “definition” of subluxation and tell me how it compares to the original definition of a bone out of place. You see, when their original description was shown to be invalid- they simply re-defined it.

    Also, a survey by chiros shows that 88% believe in subluxations http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B75KC-4F1H9GS-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=754fe88415cd702aa52be6484f7005b8

    You might think this is encouraging that 12% do not; but you’d be wrong. Most of them simply use a different term such as dysfunction or lesion (someone has compiled a list of 300 alternatives). In the end, those who realize that the subluxation is a fairy-tale are very few, and much reviled by the others in the business.

    That is the chiro way- redefine and rename the 19th century claims and carry on as usual.

    So, if you find a chiro who really does not have a subluxation-based practice, what should you expect? Well, they claim to be able to substitute for a PT; but they lack the formal education. How many of them do imagine are competent as such? Sure, they learn “conversational medicine” so they can dance around you with terminology; why not simply go to a PT and take out the guess-work?

  5. sonic says:

    I think there is an aspect that is being overlooked.
    When I go to get a treatment one thing I want to know is, ‘will this treatment do more good than harm?’
    If I go to an MD, often the treatment that is prescribed will actively harm me. (Even aspirin fits this statement). Therefore I would want the treatment being prescribed to be better than a placebo (as placebo should be truly safe).
    If someone offers a treatment that doesn’t harm (massage for example) I would want to know- ‘will this do me more good than if I did nothing?’ (because it will not actively harm me in the way that most drugs do)
    A different standard is not irrational in this case.

  6. tmac57 says:

    sonic-“Therefore I would want the treatment being prescribed to be better than a placebo (as placebo should be truly safe).
    If someone offers a treatment that doesn’t harm (massage for example)”
    Are you really sure that massage is completely benign? I know that a person with symptoms of deep vein thrombosis, for example should avoid a massage, but what if they are unaware that that is what is causing their leg pain? Massage therapists use a variety of oils and lotions during the treatment.What if the client is allergic to those things? What if someone seeks a massage therapist who uses a particularly vigorous form of massage and the client has an unrealized tissue or bone injury?
    The bottom line is that all forms of intervention can run the risk of some harm.The question is, ” do the benefits outweigh the risks?”
    P.S. Has anyone in history ever choked to death trying to swallow a placebo? I bet you can find at least one case.

  7. sonic says:

    I would agree- “Do the benefits outweigh the risks?” is the question. That is more clearly stated than what I wrote. Thanks.
    I would also agree that any possible intervention could contain risk under some conceivable circumstance. (and I have a rich and vivid imagination…)

Leave a Reply