May 05 2014

David Katz on Evidence in Medicine

David Katz is a fellow physician at Yale, and he is also a strong proponent of so-called “integrative medicine.” He has written a recent commentary at the Huff Po, defending the integrative approach. He writes:

Integrative Medicine — a fusion of conventional and “alternative” treatments — provided patients access to a wider array of options. So, for instance, if medication was ineffective for anxiety or produced intolerable side effects, options such as meditation, biofeedback, or yoga might be explored. If analgesics or anti-inflammatories failed to alleviate joint pain or produced side effects, such options as acupuncture or massage could be explored.

His basic argument is this – when we lack strongly evidence-based options, we need to explore not-so-evidence-based options, for the good of our patients. Mainstream medicine is not that evidence-based either. And – mainstream medicine relies on money-driven research, which is biased against integrative approaches.

I like this article because I can see some common ground from which to work. Katz does not appear to be a hopeless ideologue (at least not in this article); his is attempting to lay out a reasoned argument, but simply makes some critical flaws in reasoning that can be addressed. To his credit he explicitly rejects the naturalistic fallacy:

The belief that treatments are intrinsically better just because they are “natural” is fatuous and misguided. Smallpox, botulinum toxin and rattlesnake venom are natural. Nature is not benevolent.

He also endorses some good basic principles of evidence-based medicine (EBM). His main error (which is shared by EBM itself) is that his evaluation is incomplete. I also take exception with the exact manner in which he executes his philosophy. Let’s take his three main points outlined above.

Patient-Centered Care

Katz argues for a more patient-centered approach, meaning that we should not just abandon patients when we run out of strictly evidence-based options. First, I find this to be a false choice – we can continue to care for patients and hold their hands, as he suggests, without delving below the line of acceptable science-based treatments.

The term “patient-centered”, in fact (just like “alternative” or “integrative”) is a misleading term meant to misdirect from the only real issue here – what process do we use to determine which treatments are sufficiently established to be safe and effective? Using safe and effective treatments is all that matters – saying that your approach is “patient-centered” is just an attempt to grab the moral high ground and turn attention from this key issue.

Katz acknowledges that this is a key issue (hence the common ground) and lays out his approach, what he calls “evidence-mapping.”

To address this very scenario, colleagues and I have developed and published a construct that examines therapeutic options across five domains: safety; efficacy; quality of evidence; therapeutic alternatives; and patient preference.

As with EBM, this is reasonable but tragically incomplete. He is glaringly missing a sixth domain – scientific plausibility. The term plausibility, in fact, is entirely missing from his article. It is a fatal flaw in his approach, in my opinion.

He correctly points out that we live often in the gray zone of evidence. So what do we do? The science-based medicine approach is to combine the factors he lays out with insight from basic and pre-clinical science as well as what is well-established in clinical science and what preliminary clinical evidence exists. We should look at all the science, in other words, not just the clinical science, which is the trickiest and often least reliable form of evidence.

In this article Katz does not want to “get too deep into such weeds,” but he links to another article in which he submerges himself. This pretty much captures his sentiment:

I would not deny the implausibility of the therapeutic influence from not touching, nor contest the improbability of healing messages left behind by molecules diluted out of solution. But once we acknowledge that little could be more unlikely or wondrously implausible than a handshake, the topic of plausibility must be broached with greater purpose. I don’t understand homeopathy, or believe in it per se. I find it inexplicable and farfetched, but perhaps slightly less so than the experience of a caress, and incalculably less so than our existence.

What Katz is desperately trying to do here is to eliminate plausibility from the equation (because that would doom his approach) without sounding too silly. So he says, sure homeopathy is implausible, but gee whiz science is awfully complex and mysterious, so I guess it’s OK, because everything is implausible. This is the “quantum dynamics” gambit – using the incomprehensibility of the cutting edge of theoretical physics to make a false-equivalency argument and essentially neutralize plausibility.

Katz also has a problem in the way he is applying his narrow EBM approach (minus plausibility). I was at a CAM conference with him at Yale a few years ago. While we were chatting he said to me (paraphrasing) that if a treatment did not work, then the clinical evidence should be a Bell curve around a net effect of zero. This is profoundly wrong.

There are many reasons to conclude that, even with a completely null effect, research is skewed toward the false positive. I write about this issue extensively here, and so won’t repeat the entire argument again. Suffice to say, there are multiple biases at work favoring false positives. Katz simply has his evidence-meter miscalibrated.

It is the combination of failure to properly consider plausibility, and a miscalibrated sense of clinical evidence, that leads Katz and others to embrace treatments as demonstrably absurd as homeopathy.  Homeopathy is not “patient-centered” care, it’s just pure nonsense that, when studied clinically despite its lack of plausibility, does not work.

Mainstream Medicine is not all that great either

This tu quoque logical fallacy is all-too-common with defenders of unscientific medicine. Let me state up front, that I am not arguing that the current scientific standards of mainstream medicine are optimal, nor are they executed sufficiently. One of our missions at Science-Based Medicine, in fact, is to promote higher standards of evidence and better implementation of evidence.

The imperfections of current medicine is not a reason for lowering the standards of evidence and rejecting scientific plausibility, any more than deficiencies in our current government should favor resorting to anarchy.

Katz and others, in addition to the poor logic, often overstate those deficiencies. They are again desperately trying to make a false-equivalency argument – integrative approaches are not that different than mainstream approaches when it comes to evidence. This is pure nonsense.

Katz quotes a figure of >50% of modern treatments are not truly evidence-based, and references a single commentary.  (At least he didn’t quote the 15% figure.) It is, of course, difficult to put a number on something so nebulous – there are many decisions that need to be made to operationalize such an evaluation. Those who have done so, however, come up with a figure closer to 78% as the number of treatments that are reasonably evidence-based.

Whatever the exact figure is, it needs to be higher, not lower, as Katz’s approach would do. Also, this is precisely why we need to consider plausibility, which helps us navigate in the 22% or so. Lowering the bar for evidence and abandoning plausibility would be disastrous (which it is within the world of CAM).

Katz also pulls another common ploy in defenders of CAM or IM – labeling treatments that are evidence based as “alternative.” He writes:

So, for instance, if medication was ineffective for anxiety or produced intolerable side effects, options such as meditation, biofeedback, or yoga might be explored.

Biofeedback is relaxation, and yoga is exercise – both evidence-based treatments with plausible mechanisms. Evidence-based nutrition is also part of science-based medicine, not alternative. This is simply just another deception.

Money-Based Research

Katz final big point is that the evidence-base for mainstream medicine is biased against the treatments he likes because they cannot be patented. There is a kernel of truth here in that the system for funding research and determining what should get researched is not perfect. It’s not entirely broken either, as Katz implies.

Pharmaceutical research is this country is profit-driven. This in uncontroversial. This biases the research, a problem that needs to be managed (and progress is being made – such as registering clinical trials, and closer scrutiny on research methods). But the profit-driven system is not all bad either. Katz argues that it costs nearly a billion dollars to bring a drug to market, and so it has to be patentable and profitable. True – but this also means that pharmaceutical companies will research potential drugs that are plausible, based on solid evidence and rationale, and are likely to have a big market (which can translate into – help the most people). There are strengths and weaknesses in the current system.

Katz, however, goes way to far, suggesting that the system is so flawed, his precious alternative treatments are simply not being studied. He gives the example of CoQ10 for heart disease. His own example, however, weakens his point, because eventually the research was done showing that CoQ10 is helpful.

The point is also completely moot where we have lots of clinical research into CAM modalities. There are hundreds of clinical studies in homeopathy (which show it doesn’t work), and thousands of studies with acupuncture (which show it doesn’t work). Yet Katz still defends these treatments.

We write daily at SBM about alternative modalities and there is usually some evidence to evaluate – these things are being studied. To the extent that they have been studied, the evidence ranges from unimpressive to clearly negative (that’s why they’re alternative). Combine poor evidence with low plausibility, and you have a treatment that should be clearly below the line and not considered appropriate for use.


While there is some important overlap in our stated approach to evidence-based medicine, there are also very clear differences between Dr. Katz and me. We both agree that evidence-based medicine is a good idea (as far as it goes), that evidence is tricky, and that clinicians need to make decisions based on imperfect and incomplete evidence.

Where we differ is in the inclusion of plausibility in evaluating treatments, and in how we calibrate our assessment of clinical evidence. Katz fails to consider plausibility and accepts far too low a threshold for evidence, in my opinion.

This is the real and only true issue – how to evaluate the evidence. The other points about how much modern medicine is evidence-based, not abandoning patients, and how research is funded, are diversions and excuses.

The proof of the pudding is in the tasting – Katz endorses homeopathy and acupuncture, and I reject them. These are well-studied treatments with non-existent plausibility for homeopathy and low for acupuncture. In both cases the extensive clinical evidence is clearly negative – evidence for lack of efficacy. They have been studied more than enough to reject them as fruitless therapies.

Katz clings to them with hand-waving arguments that have nothing to do with the core question – what does the science and evidence say?

Note: David Gorski also wrote a response at Science-Based Medicine.

102 responses so far