Feb 24 2017

Practicing Evidence-Based Medicine

SBM-coverAn excellent article in ProPublica by David Epstein discusses the problem of doctors not adhering to the best evidence-based standards. The full article is worth a read, and I won’t just repeat it here, but I do want to highlight a few points which align well with what I have been writing here and at SBM for years.

The essential problem is that there is a disconnect between the best evidence-based standards and what is actually practiced out in the world. There are actually two problems here. The first is the scientific evidence itself. The second is the alignment of practice to this evidence.

Scientific evidence in medicine has a few challenges. There is publication bias, researcher bias, p-hacking, the decline effect, and problems with replication. What all of this adds up to is that there is a lot of published preliminary evidence, most of which is wrong in the false positive direction. There is a tendency, in my opinion, of adopting treatments prematurely.

One study which supports this conclusion was cited by Epstein – a 2013 study that shows that it is not uncommon for research to contradict current treatments:

Dr. Prasad’s major conclusion concerns the 363 articles that test current medical practice — things doctors are doing today. His group determined that 146 (40.2%) found these practices to be ineffective, or medical reversals. Another 138 (38%) reaffirmed the value of current practice, and 79 (21.8%) were inconclusive — unable to render a firm verdict regarding the practice.

This does not mean that 40% of what doctors do is not backed by evidence. Treatments that are questionable are more likely to be studied. Also, a big journal like the NEJM is more likely to publish an interesting result, such as a reversal of current practice, than a boring result, that what we thought all along is really true.

But it does mean that there are many current practices which may not hold up to further research. It is possible this means we are adopting treatments too early. This is not necessarily the case, however – what we really need to know is the risk vs benefit of early adoption of treatment. Are more people helped by adopting a treatment that actually works before it is fully proven, or are more people harmed by adopting a treatment that does not work or is harmful before the evidence is more clear? That would be a difficult question to answer definitively. We are left to infer the best answer from existing evidence.

Essentially I recommend three things to fix this first part of the problem, with the research itself. The first is rather standard – greater dedication to evidence-based medicine. This means basing practice on actual evidence, not experience alone and not just on what makes sense.

Second, we need to significantly alter the medical research infrastructure to put more resources into higher quality studies and replications, and publish fewer preliminary studies. Researchers need to stop cranking out lower quality papers, and put more work into each publication (for example, do a couple of replications before you publish your data). We probably need more education for researchers to increase the typical rigor in research and avoid statistical errors. Institutions need to reward quality of quantity more, and boring replications over surprising high-impact results. And journals need to change their publication policies to optimize them to a positive impact on clinical practice, not maximizing their impact factor.

The third fix is what I would call a shift to science-based medicine. SBM goes a bit beyond EBM in a couple of ways. The first is to directly consider plausibility, or prior probability. EBM essentially eliminated such considerations, looking only at clinical evidence. We think you should look at all scientific evidence, including non-biomedical science, to illuminate the plausibility of an intervention. This will help put the clinical evidence into perspective.

Further, SBM explicitly considers all the factors I listed above, such a p-hacking and publication bias. Finally we also recommend moving beyond over-reliance on p-values and frequentist analysis. This means additionally using Bayesian analysis, and explicitly considering effect sizes and other considerations.

In a way, SBM takes a more holistic (yeah, I know) and critical approach to the scientific evidence.

The second problem discussed in the article is that, even when we do have good quality evidence, many practitioners do not follow best practice. This is also a complex problem, without a single cause.

There are many potential biases at work here. New treatments seem sexy and interesting, and carry with them the hope that they will be better. A lot of medicine is high tech, and this creates a technology bias. Sometimes a lower tech intervention may be better.

Doctors would rather do something than not do something. When you are confronted with a patient having a problem (a bothersome symptom, or worried about a bad outcome) they want you to do something. Sometimes the best thing to do is either nothing, or very minimalistic. Patients often feel uncomfortable with this, and in turn make their doctors feel uncomfortable.

There may (depending on the practice context) be a financial incentive to do something, or the more high tech option, than to do nothing or a low tech option.

Finally, the evidence moves quickly, and it is difficult to keep up with it. There are mechanisms in place for doing so, but we could argue that these mechanisms are not robust enough. Perhaps there needs to be more oversight and feedback, not to punish or stigmatize practitioners, but to keep them in a feedback loop that moves them continuously toward the standard of care.

Part of the problem here also relates to the first problem of the quality of evidence. It is easy to be overwhelmed with tons of low to moderate grade evidence. When the evidence itself is uncertain, it is easier to substitute your own clinical judgement, which is subject to multiple biases. If there was less but higher quality evidence it would be easier to keep up and feel confident in the current standard.

It is easy to become a bit disillusioned when all the problems with scientific medicine are summarized at once. To put this in perspective, however, most of what doctors do are solidly evidence-based, and most of the rest is at least reasonably evidence-based. The discussion, however, is extremely useful in order to raise those standards even further. There is a lot of room for improvement, and we actually know (at least in broad brushstrokes) what we need to do. We just need to do it.

14 responses so far

14 thoughts on “Practicing Evidence-Based Medicine”

  1. PeterElias says:

    Both Epstein’s article and yours are excellent, but I am bothered that they both fail to even mention the role of the patient in identifying the goal(s) of treatment and assigning values to both the possible benefits and harms.

    A hypothetical example illustrative of this point is the option of treatment A versus treatment B, where A makes the patient miserable and largely non-functional for 12 months but has an excellent change of long-term benefits and B is well tolerated short term but is likely to be helpful for only 18 months after which the disease will recur. A 22 year old is more likely to pick treatment A and a 72 year old is more likely to pick treatment B. The difference is driven by patient context and patient goals.

    Just as a specific benefit may be important to one patient and not another, side effects like impotence or muscle pain with exercise or changed sleep patterns may be devastating for one patient and irrelevant to another.

    I’m not quibbling with the content of either article, just unhappy at framing that treats the patient as an anonymous third party with no agency.

  2. Peter,
    Your point is well taken, but I think misplaced. We were talking about offering options that are the most evidence-based. You are talking about which options an individual patient will choose based on their context and preferences.

    So your point is valid, just unrelated to the topic of these articles. We were not framing the treatment as you say – using the best evidence-based options and individualizing to the patient are not mutually exclusive, nor does advocating the former imply you are ignoring the latter.

  3. techczech says:

    I wonder if your statement that most of what doctors do is ‘solidly evidence-based’ is in itself more than a little light on evidence. What doctors in what fields, what evidence? What is the unit of ‘do’. What is the quality of the evidence, what is the source of the doctors’ knowledge of the evidence, what are the foundations of the knowledge, what is the doctors’ ability to critically evaluate the evidence?

    It seems to me that this statement is massively subject to all kinds of confirmation bias you can think of. Kind of like the clinical judgment you are sceptical of. My impression of doctors is that they are often singularly unable to assess the evidence they’re using. And the main reason I tend to defer to them in my own treatment is their ‘holistic’ clinical judgment – even though I feel I could often do a better job of evaluating any single piece of evidence that goes into that treatment.

    I wonder what a comparison of knowledge use of, for example, general practitioners, surgeons, dentists and ob-gyn doctors would reveal. They all to me draw on different types of evidence and require different knowledge. While you might say that they are all based on the same foundations of biochemistry, the way they draw on the knowledge, generate new knowledge and the type of research that goes into the knowledge they rely on, would be quite different. Surgeons would be much more reliant on case studies, dentists might draw more on industrial innovation and apprenticeship, GPs more on basic science research. In fact, I’d think somebody like ob-gyn would be interacting the most with the sort of science-base research you mention. Of course, they all rely on pharmacological research but in all cases, they have to outsource most of the evaluation of the efficacy of the outputs to other evaluators. I doubt that many doctors have a good grasp of the epistemological foundations behind the medications they routinely prescribe.

    I don’t know what details of such a comparison would reveal but I suspect that they would not make science or evidence-based medicine look quite as straightforwardly idyllic. But I do know that knowledge and use of knowledge are complex (as in non-linear) phenomena and any single precept for dealing with them is doomed to failure. Which is not to say that many of the heuristics of both SBM and EBM are not really useful. But they will always operate in a complex epistemological environment which is saturated with individual judgment. (For instance, the sort of storytelling that happens around statistical results.)

  4. zorrobandito says:

    Is there any evidence that drug company advertising, drug reps coming by and talking, and so forth, influences doctors and other medical people? Obviously the drug companies think it does or they wouldn’t be doing it!! There is a certain bias in this culture for the Newest Thing, even if the Newest Thing is no better than, or maybe worse than, the Same Old Boring Cheap Remedy. And drug companies of course would much rather that you and your insurance company spend a lot of money, whether this is necessary or not.

    I’m thinking blood thinners immediately post-op, as for joint replacement surgery. Aspirin and warfarin are both old standbys, and according to what I’ve read they work reasonably well. There are several new and jazzy substitutes, and they are quite expensive. My top-of-the-line very prominent ankle surgeon recommended aspirin for this purpose.

    The rehab nurses were very uneasy with this. They felt that something newer and more costly (and more difficult to administer) would be better. (Also, perhaps they thought they were less likely to be blamed later if the Newest Thing had been used.) I allowed them to beat me up on this because I wasn’t exactly at my best after major surgery. I shouldn’t have done that though, and they shouldn’t have tried to countermand the surgeon’s orders.

  5. BillyJoe7 says:

    zorrobandito,

    Sorry to hear about your ankle surgery (I empathise a little – I’m sitting here with a blown up achilles bursa after yesterday completing the toughest marathon on Australian soil – the Roller Coaster Run through the Dandenong Ranges in Victoria – I was a bit underprepared!).

    Yes, there is evidence that drug company reps influence the precribing habits of doctors. The drug companies don’t think it does, they know it does. As you say, they would not spend the money they do on reps if it didn’t have a demonstrable effect. Also, the medical societies are well aware of this and have drawn up fairly strict guidelines for doctors accepting gifts from drug reps. On the other hand, many individual doctors are in complete denial of this fact. I have no idea why doctors see drug reps. They are not a source of reliable un-biased information and they take up their valuable time.

    There is also evidence that the newest and most expensive is not necessarily the best. The old anti-hypertensive drugs which are dirt cheap are about as effective as the more recent expensive drugs. Of course, as patents run out, these do become cheaper and there are some marginal benefits of these drugs above the older drugs.

    But, the evidence and guideline recommendations are against aspirin* as being an adequate preventative for DVT following surgery to the lower limb. It’s a choice between LMWH, heparin, warfarin, and the new expensive drugs (?Xarelto). The first two have to be injected and are very expensive, while the latter two are taken orally, and warfarin is the cheapest option ($2 per day versus $9 per day for xarelto). But it can take up to ten days for warfarin levels to stabilise and it requires regular monitoring with blood tests. When the patents run out, the price on those more costly drugs will likely drop signifcantly.

    As for nurses: Yes, they seem to be very personable, helpful, and practical in one sense but very regimented in another. And also prone to promoting pseudomedicine as a matter of course.(My two sister’s in law are nurses, and my mother had a lot of attention from nurses during her final few years). They also tend to have a pretty low threshold for referring off to either more senior nurse and to medical personal.

    *Here is a dissenting view regarding the merits of aspirin: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4054010/

    —————-

    techczech,

    I think your opinion of the various groups and their evidential knowledge is spot on (meaning agreing with mine 😉 ). My interactions with surgeons have been largely negative (ie my father’s uologist’s knowledge regarding prostate cancer was pretty poor – everything was based on his own personal experience. Even the College of Urologist’s statement about prostate cancer screening is at odds with the evidence!). But my inteactions with obstetricians has been very positive. They talked about options and evidence for and against before making recommendations* about what we should do. They are dealing with life and death at close quarters, which I guess is likely to concentrate the mind a little. General practitioners seem to be a mixed bag. Some put their own personal experience above evidence based guidelines and protocols, which some hate with a vengence, others are sticklers for the guidelines and protocols and see them as rules to be followed to the letter. Not many seem to have any idea of how to evaluate clinical trials and apply them to individual patient in front of them. Of course, I may be wrong about all this, based as it is on my personal and online experience of these medical groups. 🙂

    *I think it is important for doctors to make recommendations after supplying the evidence for the various options, provided they have the necessary knowledge and experience to do so. Most patients are at a considerable disadvantage in this regard – they have almost zero background knowledge, have no idea about clincal trial methodology, and therefore cannot evaluate the evidence to any degree of accuracy. I believe a sufficiently informed doctor is reneging on his responsibility if he does not also make recommedations – and give reasons for those recommendations – on what the patient should do taking their individual situations into account. Equally, they should continue to work positively with their patients when they choose another option. They should also be up front regarding pseudomedicine but accept that patients are going to use it anyway.

  6. zorrobandito says:

    @BillyJoe7 Sorry about your ankle! Ankles take a while to heal but they do heal eventually, like everything, so don’t lose hope.

    I’m a lawyer, not a doctor, so I have no informed opinion about blood thinners. That’s what I hired a doctor for. i did some shopping and got a good one, so I’m inclined to follow his directions. Anyway I didn’t get DVT, so that’s a win. Whether the cause was aspirin or not, no one can possibly know.

    Drug companies advertise to prospective patients too, as I learned by watching TV while I was immobilized. They spend a lot of money on those ads, and they surely feel that it increases sales. I cannot view this with anything but disapproval. For the same reason that I have no opinion on blood thinners, I am in every way disqualified to see a TV ad and then run in and demand Humera, a dangerous immunosuppressant, all because I have a rash that embarrasses me in my yoga class. This kind of thing cannot possibly have a good effect on the practice of medicine. Of course, improving the practice of medicine is probably the last thing on the minds of drug company executives.

  7. BillyJoe7 says:

    ZB,

    My achilles bursitis settled down within 24 hours with rest and an OTC topical anti-inflammatory gel, but thanks for your concern. It was not achilles tendonitis which, indeed, would have taken a while to heal. I had that a few years ago and had to resign myself to cycling for a year or so till it slowly resolved.

    In Australia we fortunately don’t have direct to consumer advertising by pharmaceutical companies. But they do get around it a little by “public service announcents”, by providing videos to current affairs media outlets, and of course by drug reps deluding doctors into thinking they don’t influence their prescribing habits.

    From my reading, the role of aspirin seems to be inferior compared to other blood thinners in the prevention of DVT following lower limb surgery. However, I did provide a link to the counterview to the present practice of using those other blood thinners. But here is a link that supports the current practice:

    https://www.ncbi.nlm.nih.gov/pubmed/24695091

    According to that article, the incidence of DVT after total knee replacement when one of those new oral drugs was given was 3%, LMWH 13%, and aspirin 16%. If the same is true for ankle surgery, your surgeon increased your risk of DVT five fold. The fact that you didn’t get a DVT is irrelevant as, even with aspirin, you had only a 1 in 6 chance of getting a DVT.

    Frankly, I wouldn’t trust a surgeon to give me reliable information about medical treatments such as DVT prophylaxis. For that, I’d speak to a medical specialist or Haematologist, who is much more likely to be up to speed on issues such as this.

    “I have no informed opinion about blood thinners. That’s what I hired a doctor for. i did some shopping and got a good one, so I’m inclined to follow his directions.”

    Well, the surgeon did not give you a “informed choice” as far as I can tell. He merely told you he uses aspirin and you agreed with him without question. He may be a good surgeon but a lousy haematologist – maybe your surgeon should talk to a haematologist.

  8. zorrobandito says:

    “maybe your surgeon should talk to a haematologist.”

    Maybe indeed. At the time I felt that I had more than enough doctors involved.

  9. BillyJoe7 says:

    ZB,

    “I had more than enough doctors involved”

    I said your surgeon should talk to a haematologist before deciding how to best prevent post operative DVT in his patients. If one of his patients died as a result of pulmonary embolism due to DVT, he may be held liable if he was not using best practice to prevent DVT.

  10. MaryM says:

    Related to evidence-based medicine, someone asked me about this site the other day: “Consumers United for Evidence-Based Health Care”. Intriguing–sure. But look over the member list a bit. NVIC? A couple of other dubious ones appeared there.

    http://consumersunited.org/about-cue/member-organizations

    Yet they also seem to be associated with US Cochrane, and Johns Hopkins somehow.

    Anyone know anything about this?

  11. Rogue Medic says:

    Anticoagulation is complicated.

    Normally we keep our blood circulating in our extremities by moving about, but with immobilizing injuries, we can end up growing blood clots, which may migrate to places that do not accommodate interruptions of blood flow, such as the heart, brain, and lungs. A clot obstructing any of those organs can be anything from asymptomatic to fatal.

    We expect that most hospitalized patients will have emboli (clots) lodging in the lungs, so there are a variety of methods of attempting to minimize the size of the clots. If moving around isn’t possible, compression stockings are one way to avoid having blood pool in the legs. There are also machines that vary the compression on the legs to try to stimulate blood flow.

    Drugs can be as familiar as aspirin or any of the new clotting factor Xa inhibitors, for which there was no reversal agent until late 2015. that reversal agent has not been well studied, but it meets a need, and people want to use something, even if the evidence comes from a very small number of patients.

    Why is a reversal agent for anticoagulants important? Gravity.

    People go home and fall down. Some fall down before they leave the hospital, even with the rails up on the side of the bed. It does not take a major impact to cause major bleeding in an anticoagulated patient. One study of over a thousand patients who fell while taking warfarin or clopidogrel, had any head injury that resulted in transport to the emergency department, and found that 6% of warfarin (Coumadin) patients and 12% of clopidogrel (Plavix) patients had some degree of intracranial bleeding.

    The reversal of the anticoagulant effects is not easy. The concern with falls is more of a problem with patients who have atrial fibrillation. The upper chambers of the heart quiver ineffectively and blood clots tend to develop. The major concern is that the emboli will end up in the brain, briefly depriving important parts of blood supply, but for long enough to cause irreversible harm.

    Don’t use an anticoagulant and risk a stroke.

    Use an anticoagulant and have a fragile cerebral blood vessel, or have a fall, or be in a motor vehicle collision, and risk a stroke.

    For an active person, the ability to reverse warfarin probably makes that the safest choice, although there is a need for repeated blood draws, changes to dosing, and dietary restrictions (spinach and other foods with a lot of vitamin K – one of the reversal agents for warfarin is vitamin K).

    For someone bed-ridden, there are plenty of new clotting factor inhibitors, which do not require blood draws, frequent dosing adjustments, or dietary restrictions. The problem is that there are no good (as far as valid evidence) reversal agents for these medications.

    I apologize for over-simplifying a very complicated topic.

    Warfarin is cheap, generic, and has a well known side effect profile, even if the side effects are very annoying.

    The others are great for people who are more worried about annoyances than the catastrophic problems that are not rare.

    This is one of the problems with evidence-based medicine (and science-based medicine).

    There are a variety of reasons for making decisions that will produce a bad outcome for one patient, which makes it seem as if the other choices were better, even though the other choices would more often produce bad outcomes.

    Then there are the lawyers, claiming that doctors should be able to avoid all bad outcomes, thereby discouraging good patient care.

    .

  12. BillyJoe7 says:

    RM,

    Am I correct in assuming that you would agree that an orthopaedic surgeon should have input from a haematologist when deciding how to anticoagulate his patients post-operatively. I am also interested to know if you know what that input is likely to be.

  13. Rogue Medic says:

    BillyJoe7,

    Am I correct in assuming that you would agree that an orthopaedic surgeon should have input from a haematologist when deciding how to anticoagulate his patients post-operatively.

    My anecdotal experience with orthopediasts is that they do not appreciate the nuances of pharmacology.

    I would hope that an orthopedist would know enough about the effects of coagulation to make decisions about dosing, drugs, and devices.

    I would hope that an orthopedist would know enough about the effects of coagulation to present enough accurate information to patients to be able to make informed consent/refusal decisions about their treatment.

    I am also interested to know if you know what that input is likely to be.

    I expect the advice to depend on the patients’ circumstances.

    I have told my mother that she needs to take warfarin, if she is ever in need of anticoagulant medication. She is in her 80s and has tried to fight with someone who stole her car, even being dragged along by the car, before letting go.

    She is more prone to trauma than the average octogenarian.

    My recommendation for my mother might be different, if she were a more sedate, less mobile, person.

    It is definitely not a one size fits all situation.

    On the other hand, treatment by an orthopedist strongly suggests a predisposition to enough trauma to break a bone (even if one weakened by osteoporosis) requiring reversal of the anticoagulant prior to surgery, since surgeon generally do not want to initiate uncontrollable bleeding. The surgery is followed by resumption of the anticoagulant.

    .

  14. DrGM says:

    Concerning “best medical practice” and with reference to the reply by Steven Novellaon (24 Feb 2017 at 3:30 pm) to Peter; medical treatment decisions are also directed by Government policy which is often based on whole population management and financial restraints; for example (in Australia) treatment of elevated cholesterol is based on cost versus return such that high risk individuals can be prescribed statin medications but moderate risk (for whom the research data suggest would benefit from statins) are not eligible under the Government prescription scheme.

Leave a Reply