Mar 18 2016

Ioannidis – Evidence-Based Medicine Has Been Hijacked

In a recent commentary, framed as an open letter to David Sackett (the father of evidence-based medicine), John Ioannidis argues that EBM has been hijacked by various interests. He also clarifies his position in an interview with Retraction Watch.

Ioannidis hits many interesting points: EBM has become a way to market products and services, clinical studies are largely in the hands of corporations with vested interests, academics are under their own pressures which emphasize getting grant money, practitioners are likewise struggling to survive in an era of managed care, and quacks and charlatans are exploiting the whole mess.

It is an eye-opening roller-coaster ride, including many personal stories, through the mind of perhaps the most famous current critic of the industry of medical science. I agree with much of what he says, and in fact they coincide with a great deal of commentary here and at science-based medicine. He takes a more cynical and pessimistic tone than I would, but that is subjective.

Perhaps I am a bit more optimistic (on my good days) because I am actively working on a solution – SBM. That is, in fact, precisely what SBM was designed to be, a fix for the limitations and vulnerabilities of EBM, the very vulnerabilities that Ioannidis now exposes.

Some of the more solid points that Ioannidis makes includes the overhyping of basic science or preliminary research. He writes:

Right after my talk, everybody rushed to hear the launch of a new campaign, where the leader of the institution singled out this unique historic moment: that institution would single-handedly eliminate most major types of cancer within a few years. Several years have passed, and none of these cancer types have disappeared. I recently tried to find the name of that campaign online but realized that this institution has launched many similar campaigns. Which among many was the unique historic moment that I happened to be at?

This pattern is all too familiar – some basic science finding is extrapolated to an ultimate and dramatic clinical outcome. It is not just journalists who overhype, but researchers and their press offices play a role as well. In this industry of hype, every discovery about cell metabolism will cure cancer, every viral study will lead to a cure for the common cold, and every incremental advance in our understanding of brain physiology will lead to a cure for Alzheimer’s disease.

He extends this to epidemiology:

Second, instead of dealing with these major public health risks, the production of spurious, false-positive, or confounded putative risk factors is more dangerous than ever. Jumping from correlation to causation, data dredging is called causal evidence and fuels guidelines.

I wrote about this also in an article called Everything Causes Cancer. There is a lot of noise in data, and if you try even a little bit you can extract a false signal out of that noise. Sometimes the signal is real, but clinically meaningless. The public is then exposed to an endless stream of warnings about tiny or spurious risks, which dilute out warnings about the large and proven risks. People become worried about the wrong things. (This is now typified in my mind by the person who smugly reassured me that she only smokes organic cigarettes.)

Regarding industry sponsored research he says:

It is just that they often ask the wrong questions with the wrong short-term surrogate outcomes, the wrong analyses, the wrong criteria for success (e.g., large margins for noninferiority), and the wrong inferences, but who cares about these minor glitches?

This highlights how difficult it can be to properly interpret a study or series of studies. People focus on the p-value – are the results statistically significant. If they are, then whatever the authors are claiming must be true. Reality is far more complex.

Ioannidis also refers in his commentary and later interview to the fact that, as a meta-science researcher, and therefore critic of the practice of science within biomedicine, he has not always been popular. In fact there are many cranks and science deniers that try to find common cause with him, and cite him as evidence that science is broken or hopelessly corrupt.

The cranks and science-deniers, however, are pointing at the splinter in the eye of medicine while they have a plank in their own. All of the problems with biomedicine are magnified by orders of magnitude when it comes to alternative medicine, the science-deniers, and the fringe.

Further, Ioannidis is criticizing the science of medicine in order to make it better. He is pointing the way to improving how biomedical science is funded, evaluated, executed, reported, and translated into practice.

Enter Science-Based Medicine

There is a reason I have been following the publications of Ioannidis over the years and writing about them frequently – he is pointing (often in a rigorous empirical way) to the same issues that we address at SBM. As I said, SBM is our best effort at a fix. We agree that EBM has been hijacked. This does not mean it is worthless, it is a valiant effort, but as Ioannidis says, an unfinished project.

SBM addresses all of the issues that Ioannidis points to. The point of SBM is to take a global look at the big picture of any clinical claim (it is scientific skepticism applied to health care). This means we have to consider scientific plausibility, not just clinical trials. We have to look at what basic science and clinical science studies are actually telling us. What variables are being controlled for, what is the real relationship between the outcome measure and the disease of interest, what are all the biases, how certain are we really that this phenomenon is real?

This is why it is often the case that an SBM evaluation of a claim is more skeptical or negative than an EBM evaluation, or claims that don’t even satisfy the criteria of EBM. We are not naysayers or cynics – we simply recognize that science is complex, flawed, and often wrong or misleading.

The ultimate goal of SBM is to have as accurate as possible an idea of where the threshold should be before we accept a claim. In medicine this has a very practical and important implication – when do we recommend an intervention?

Right now that threshold is almost certainly too low, for all the reasons that Ioannidis discusses. When we look back at the literature, as Ioannidis has, we see that we have adopted interventions prematurely, sometimes resulting in causing more harm than good.

The stakes, if anything, are getting higher. Health care is swallowing up 20% of our economy, and we simply cannot afford it. We need greater efficiency in every aspect of heath care and biomedical research. We need to be researching the right questions, in the right way.

Ioannidis also claims that 80% of biomedical research is wasted. I can see how this is true – I see so many studies that fail to address to proper question (does this treatment actually work). I would argue that 99% of acupuncture studies are worthless, 100% of homeopathy studies are worthless, but many studies of even mainstream questions are also wasted due to poor design, or asking the wrong question.

The other challenge of medicine, as an applied science, is that there often is a huge emotional investment (not just a financial investment) in the outcome. Placebo effects are powerful in convincing people that certain treatments work (both patients and practitioners). When scientific evidence fails to support this, anger and frustration is a common response. There is also the temptation to shoot the messenger. (Just look at the cryotherapy article from earlier this week.)

We need to make the case to the public that the stakes are high and we have to get this right. We need to move massively in the direction of science-based medicine, and weed out all the pseudoscience, bias, and perverse incentives from medicine.


10 responses so far

10 Responses to “Ioannidis – Evidence-Based Medicine Has Been Hijacked”

  1. PeterEliason 18 Mar 2016 at 9:11 am

    Nice post. I would add three thoughts, based on how I cope in the real world of primary care.

    (1) Humility. We have to stop talking and thinking as if our current information is Truth or The Answer. As scientists, we may understand at a deeper level that it is all tentative (until more information is acquired) but we do ourselves, our patients, and the public a great disservice by not making this more clear. Every day.

    (2) Patient focus, not evidence focus. It helps to reframe evidence-based medicine as evidence-informed medicine. Saying our decisions are ‘based’ on the evidence gives the impression of a solid and enduring foundational truth and leads us to think of patients as ruled by evidence-based algorithmic overlords. It replaces irrational physician-based paternalism with more rational guideline-based paternalism. I ‘base’ my decision on…sounds pretty dogmatic. Saying instead that our (the clinician and the patient) decision is informed by the evidence does several things. It makes it easier to include things like common sense and patient preference/values. It makes it easier to adjust our decision when the evidence changes the landscape.

    (3) Advocacy. We need to consistently and loudly call out the policy makers and industries when they misuse the evidence.

    Peter Elias, MD

  2. hammyrexon 18 Mar 2016 at 9:32 am

    The comment about industry-funded trials is spot on. A lot of people have the misconception that industry research is “low quality”, but if anything it’s the exact opposite – it tends to do a much better job than independent studies on a regular basis in terms of the things we deem as quality – meeting recruitment, lower overall dropout, etc.. They just tend to ask questions that are not always particularly meaningful to a clinician or go into the “not even wrong” category (biomarkers for cholesterol as a measure of efficacy being a good example).

    I feel EBM has definitely become a victim of its own success and more or less become a mere buzzword. Apparently, *everything* is now evidence-based, even when it isn’t, and if you want to try and snake something through a committee, you just have to say it’s evidence-based enough and emphasize how important that is (also just whisper the word “patient safety”). I believe Dana Ullman now has arbitrarily granted himself the title of “evidence-based homeopath” and every naturopathic student is told they have an “evidence-based” curriculum. And to be fair – I’m also told I’m being a good little clinician when I follow “evidence-based” guidelines that are often 60% or more expert opinion, often written by specialists who are clueless about what actually happens in primary care.

  3. 107197on 18 Mar 2016 at 9:40 am

    Perhaps here would be a good place to post a link:
    and point out the four-part discussion of EBM vs. SBM, links to which are at the bottom of the page. I’m only on part 2 now (thank FSM for multiple browser windows!) but it’s explaining a lot of the background that helps this post make more sense.

  4. hardnoseon 18 Mar 2016 at 1:06 pm

    “We need to move massively in the direction of science-based medicine, and weed out all the pseudoscience, bias, and perverse incentives from medicine.”

    Scientific research is inherently difficult and always will be, so that aspect of your crusade is destined to fail.

    And what is or is not pseudoscience depends on your perspective and experience, so this is something the world will never agree on.

    And money is here to stay and it will always influence everything. The same with social status. You can’t subtract these from science until robot scientists are invented — and that won’t happen.

    Science is always going to be messy, no amount of idealism is going to change that.

    Do we need more skepticism about science? Yes absolutely, but skepticism, like science, it is the result of fallible human efforts.

  5. BillyJoe7on 19 Mar 2016 at 10:15 am

    Don’t even think about it.

  6. The Other John Mcon 19 Mar 2016 at 10:43 am


  7. The Other John Mcon 19 Mar 2016 at 10:44 am

    You are too funny

  8. BillyJoe7on 19 Mar 2016 at 11:27 pm


  9. BillyJoe7on 21 Mar 2016 at 4:38 pm

    I finally got time to read this:

    Seems to me John Ioannidis should start up a conversation with Steven Novella. After all, dead men like David Sackett, great as they may have been while alive, cannot contribute much to the converstion anymore.

  10. BillyJoe7on 22 Mar 2016 at 6:41 am

    Here is David Gorski’s take:

    It’s a little different from Steven Novella’s

Trackback URI | Comments RSS

Leave a Reply

You must be logged in to post a comment.