Nov 19 2007

How Much Modern Medicine is Evidence-Based

In response to my Homeopathy entry last week, reader Jasonb left the following comment:

I think debunking homeopathy is important and worthwhile. There is, however, a more pressing matter at hand.

Following on the heels of recent news that many drugs prescribed by doctors (and sold by pharmacies) lack an FDA approval, how many people are aware that LESS THAN FIFTY PERCENT (50%) of western medical practice is based on valid scientific evidence and some experts estimate that number is low as FIFTEEN PERCENT (15%).


At 15%, I have to imagine the whole of Alternative Medicine has more valid science behind it than that.

It would be a tremendous service if you could identify the top dozen or so western practices that would utterly astonish people to know have as much scientific validity as homeopathy. There has to be some real doozies at the top of the list.

Thanks for the comment, Jason, but I am afraid that the less than 50%, and as low as 15%, figures are myths, propagated by the critics of scientific medicine and advocates of “alternatives.” Like so many ideologically useful myths, the figures are now embedded in the culture, and it seems we will be hearing them over and over again.

Like most rumors and myths the “source” that people cite as support always turns out to be a secondary source – someone just spreading the rumor. It is nothing more than a “friend of a friend” story, where you can never get to the original friend that was the source of the information. Critics of mainstream medicine happily repeated the figure without ever verifying it. I will also note that the Washington Monthly article that was cited is a highly political piece, and while there are kernels of truth within it, it is also a highly distorted and agenda driven view of the current situation.

Fortunately, Bob Imrie (a skeptical veterinarian and great guy who sadly recently passed away) did the leg work of tracking down the original source. The 15% figure comes from a small survey of primary practice offices in the north of England conducted in 1961 – that’s right, almost half a century ago. Further, the survery was never intended to assess the degree to witch primary practices were evidence-based, but rather was looking at whether treatments were “specific” from the point of view of insurance reimbursement. So the 15% figure is misrepresented and half a century out of date.

What are the real figures? Well, there is some subjectivity to this because evidence-based is not black or white; there are degrees of evidence. Also, modern medicine has picked all the low hanging fruit, meaning that obvious treatments have been developed for straightforward situation. Bob Imrie gives the example of blood transfusions. If someone is bleeding to death, we stop the bleeding and transfuse blood to stabilize blood pressure. This is such a no-brainer that it would be unethical to conduct an experiment in which we let people continue to bleed and/or not transfuse them just to prove that doing so is a good idea.

But putting the obvious stuff aside, there are many institutions that expend a great deal of time and effort assessing the evidence-basis for what doctors do. Prominent among them are groups dedicated to evidence-based medicine – the purpose of which is to systematically review the evidence for specific practices. Estimates vary among the various fields of medicine – some specialties lend themselves to placebo-controlled trials more than others. But Imrie summarizes the evidence thusly:

Thus, published results show an average of 37.02% of interventions are supported by RCT (median = 38%). They show an average of 76% of interventions are supported by some form of compelling evidence (median = 78%).

Three quarters is a completely different world than “as low as 15%.” Also, we must consider that the other 25% is based on some evidence, even if not compelling, and typically will have a high degree of plausibility. You also have to take into consideration the context of treatment. If there is no evidence-based treatment available for a patient (or what is available has failed or cannot be used) then it is reasonable to try plausible if unproven therapies.

My personal experience is that nearly 100% of the clinical decisions I make are based upon the best available evidence combined with plausible and rational extension of what is known. I can’t think of any time when I use treatments that are based upon nothing, or even nothing but anecdote. At the very least there is a biologically plausible mechanism of action and adequate evidence for lack of harm. The only exception to this, of course, is experimental treatments – but they are highly regulated and follow their own stringent ethical guidelines.

Regarding “off-label” use of pharmaceuticals, this is also highly misleading. The FDA requires a high threshold of evidence for safety and effectiveness for a specific indication, and then only allow the company to market the drug for the specific indication. But the reality of medicine is very different. Diseases processes and diagnoses are not neat separate entities. The FDA process pretends that they are – it’s a fiction of convenience and everyone understands this.

For example, the FDA has approved the drug Cymbalta for painful diabetic neuropathy – a specific disease diagnosis. But diabetes can cause different kinds of nerve damage and different kinds of neuropathic pain. Neuropathic pain can also be caused by many different things. The neuropathic pain caused by diabetes may be similar or even identical to the neuropathic pain caused by trigeminal neuralgia or post-herpetic neuralgia. There is no reason to think that Cymbalta would not work for these other forms of neuropathic pain, but these would be off label uses.

The bottom line is that you should not equate off label with not based upon evidence. In many cases there is compelling evidence for off-label use, but it is not worth the company’s money to get an additional indication from the FDA. Company’s rarely do this – they only do it when they think it will increase the market for their drug. Scientists, however, will study approved drugs to see what the extent of their use truly is.

What about so-called alternative medicine? Despite Jason’s hunch, it is based upon almost no evidence at all. Modalities are alternative for a reason, because they are highly improbable and lack compelling evidence. Otherwise they would be mainstream, not alternative. But the CAM spin machine is very effective at spreading misinformation and confusion. They don’t let the facts get in their way.

But the kernel of truth in all of this (the best lies always have a kernel of truth) is that we can do better in modern medicine than we are doing. The problem is not cultural or systemic. I am proud of my profession in that it is conscientiously self-critical and science-based. There are many academics whose job is basically to evaluate and improve the state of evidence in medicine, and then spread the results to the profession at large.

But science-based medicine is hard. We are overwhelmed with evidence and it is impossible to keep up with it all. I think we would benefit from a system of continuing medical education that was more systematic and less haphazard, including methods for educating practitioners about the established evidence-based standards. This happens, I just think we can do better. At the highest levels of academia I think the system actually works impressively well, it’s more that the evidence-based standards need to trickle down more quickly and systematically to practitioners in the field.

As high as the general level of evidence is within mainstream medicine, the general consensus is that we need to continue to push it higher – hence the development of evidence-based medicine, and the constant revision of our standards of practice. CAM proponents, on the other hand, seek only to create a double standard for themselves – one with lower and lower standards of evidence and plausibility for themselves. To cover themselves they promote the fiction that mainstream medicine is not based upon science either – always trying to lower the standard.

That is the real urgent crisis in medicine.

18 responses so far

18 thoughts on “How Much Modern Medicine is Evidence-Based”

  1. Sqrrl101 says:

    Similar tactics are used by the “Animal Rights” lobby when they’re complaining about animal-based research. The problem is that it can take hours to thoroughly bust a well crafted piece of disinformation, but only a minute to write it in the first place. To compound this, the public seem woefully ill-informed as to the process of bringing new therapies to market, and the overall research process in general.

    Luckily for me, however, the AR lobby tend to be fairly lazy and just copy-and-paste the same 33 “facts” as a substitute for actually debating, making it possible to just copy-paste the same 33 rebuttals each time.

  2. delaneypa says:

    Here’s one of my favorite (albeit tongue-in-cheek) articles about the limitations of evidence-based medicine:

    BMJ. 2003 Dec 20;327(7429):1459-61.

    Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. Smith GC, Pell JP.

    Department of Obstetrics and Gynaecology, Cambridge University, Cambridge CB2 2QQ.

    OBJECTIVES: To determine whether parachutes are effective in preventing major trauma related to gravitational challenge. DESIGN: Systematic review of randomised controlled trials. DATA SOURCES: Medline, Web of Science, Embase, and the Cochrane Library databases; appropriate internet sites and citation lists. STUDY SELECTION: Studies showing the effects of using a parachute during free fall. MAIN OUTCOME MEASURE: Death or major trauma, defined as an injury severity score > 15. RESULTS: We were unable to identify any randomised controlled trials of parachute intervention. CONCLUSIONS: As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials. Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute.

    PMID: 14684649 [PubMed – indexed for MEDLINE]

  3. daijiyobu says:

    Re SN’s “science-based medicine:”

    Wait, medicine should be based on actual evidence aka actual scientific support?

    Huh, I’m confused (wink-wink).

    When I went to naturopathy school at UB here in CT the profession told me that they were “the modern science-based primary care doctor” and “not a belief system” (quite a lie).

    And UB currently states that “life force” (a belief) is “in fact” “scientific medicine” (though actually scientifically-ejected since, charitably, the 1930s).

    I thought ‘medicine’ of whatever kind just has to be postured as ‘scientific,’ and like Peter Pan — ‘if you wish hard enough, it comes true.’

    You mean to say that they ACTUALLY have to be what they say they are?!?!?!?

    No way (not so far, in my experience).

    [I’m being entirely facetious, of course…helps me deal with ‘UB and FNPLA naturopathy fraud’ and $200,000 worth of student loan debt.]

  4. Scepticon says:

    “the degree to *witch* primary practices were evidence-based”

    Now there’s yer problem, they were looking at Witch Doctors.

    Seriously though, another great post Steve. I had not heard of this particular piece of mis-information.

  5. rbstansfield says:

    I got a job in a medical education department, and shortly afterward heard the term “evidence-based medicine” (EBM) for the first time. My first reaction was basically, “what is the other kind of medicine based on?”

    But over time, I’ve come to recognize the pushback that some clinicians make against EBM is part of, not counter to the self-critical and self-improving culture of medicine. Medicine is academic, and so inherently conservative. That BMJ article that delaneypa cites above is a good example*. There’s also some Radiology article (I believe an April Fools issue) statistically showing that radiologists with at least one eye open outperform radiologists with both eyes closed. Thank goodness for scientific rigor.

    In general, science baldly exhibits a strength that non-scientists misperceive as a weakness: self-doubt. I’m always tickled to see a scientist unabashedly admit that all theories are tentative. The critics immediately assume they’ve won, then slowly realize they’ve lost. It’s like beautiful chess.

    *Though it’s stunning to me that editors of prominent journals waste space with crap jokes like that. I prefer a little more “shame-based editing” (SBE) in my medical journals.

  6. Fred Cunningham says:

    I’m curious as to how you feel about prescribing Prozac to patients with MS. It appears that it reduces the rate of new lesions and maybe will reduce symptoms of depression that other MS drugs have.

  7. twelvesamwicheatin says:

    Over the last 13 years of being a pediatrician I have seen first hand the changes in my practice all based on EBM. There have been alterations in the treatment RSV infection, croup and ear infection (to name a few), all because of persistant study and an inherent desire for ensuring we are doing the right thing for our little patients. While I understand that some of our use of medicine is officially “off label” it is primarily because of not being studied in a specific age range. There is a large group of pediatricians nationwide that are constantly involved in research based in the office setting.

    Basically, we are trying. And we are striving to improve the evidence all the time. We don’t stand behind the smokescreen of homeopathy and we openly encourage shooting holes in treatment plans if they indeed do not work.

    By the way, I loved in typo of “witch primary practices”. I am sure you weren’t refering to homeopathy being akin to witchery

  8. superdave says:

    It blows my mind that there are people that think not being based on evidence is a virtue of alternative medicine.

  9. nfpendleton says:

    I know this is a day late and a dollar short, and maybe not exactly on topic, but this blog post reminds me of a comment James Herriot made in his first book, All Creatures Great And Small. To paraphrase, he stated that much of the “medicines” they used in those days (late 30’s) were just useless cocktails no more potent than home remedies. In the days before antibiotics, vet med was lax on real science and reliant on what we now call CAM. And, if I recall the success rate of these treatments, they were just as effective then as now. He lost at least as many patients as he saved.

    Flash forward to today and all the progress made in the interum…

    These assertions you highlight are just wreckless and irresponsible. There’s just too little old fashioned shame where these truthers are concerned. Corny perhaps; but true most likely.

  10. Amy Alkon says:

    I am no fan of “alternative medicine” (I like the Angell/Kassirer quote: “…There is only medicine that has been adequately tested and medicine that has not, medicine that works and medicine that may or may not work.”)

    That said, there’s a great deal of fudging of statistics in what we consider evidence-based medicine…so while I’m absolutely for evidence-based medicine (as opposed to the recommendations of gray-skinned health food store employees)…exactly how evidence-based is some (or even much) of this evidence-based medicine?

    The problem is, it probably takes some of the finer stats hounds/epidemiologists to properly deconstruct the math behind the studies, leaving the rest of us to simply hope that what we consider “evidence-based” medicine actually is.

    Again, no fan of woo here…but I think there’s a tendency in opposiing woo to deify the other side. Skepticism all around is called for: Science, as opposed to “science.”

  11. jasonb says:

    Not so fast. I was actually able to share emails with Ms. Brownlee who had this to offer,

    “The 50 percent figure comes from none other than the Institute of Medicine. Not exactly a bunch of pro-alternative medicine nutjobs.”

    And she goes on.

    “Here’s a short list right off the top of my head of widespread medical practices that have only recently been shown to be ineffective or harmful for at least some patients on whom they were commonly used: cadiac stents vs medical management; neurontin for biopolar disorder; vena cava filter in patients on low molecular-weight heparin; implantable defibrilators; lung
    reduction surgery for emphysema; pulmonary artery catheters, carotid endarterectomy for most patients. There are many many more.”

    “Here’s an off the top of the head list of current, common medical practices that have poor, contradictory, or no evidence; spinal fusion for low back pain; minimally invasive hip replacement surgery; orthognathic surgery for TMJ; PET scanning for putative Alzheimer’s; TpA for stroke; statins for primary prevention; PSA testing for prostate cancer; CT scan screening for lung cancer; stents for the prevention of stroke. I could go on and on.”

    “There is widespread antipathy towards evidence within the medical community, I think at least in part because it is very hard for physicians to admit they are practicing medicine on the basis of hunches, tradition, and all the other non-science-based reasons that doctors do what
    they do. The problem is, most doctors are not trained to tell the difference between valid evidence and garbage, much less valid evidence and somewhat less reliable evidence — the difference, for instance, between a randomized controlled clinical trial and an observational trial. Many think observation is just as good.”

    Out of all the sciences, it is the medical profession that sports the highest percentage of belief in supernaturalism (i.e., god). It’s no wonder there’s so much bullshit being passed off as “good, solid medical practice.” And for you to not go after these things as passionately as you do CAM, is frankly a big disappointment. I love when you call bullshit on the CAMers. Now, step up and learn to call BS even when it hits close to home.

  12. healthman says:

    Actually, there are more current references to the 50% and 15% numbers that make them somewhat less “mythical.”

    “More than half of all medical treatments, and perhaps as many as 85 percent, have never been validated by clinical trials.” Michael L. Millenson, AB: Demanding Medical Evidence: Doctors and Accountability in the Information Age, The Robert Wood Johnson Foundation, Issue 2, September 1998.

    “Only 10-20% of all procedures currently used in medical practice have been shown to be efficacious by controlled trial.” Office of Technology Assessment, US Congress. Assessing the Efficacy and Safety of Medical Technologies. Washington DC : Office of Technology Assessment, US Congress; 1978.

Leave a Reply