Here is a curious article published recently in the New England Journal of Medicine (NEJM): Assessing the Gold Standard — Lessons from the History of RCTs. The article discusses the history and current role of randomized controlled trials (RCTs) in medical research and practice.

This is, of course, a very complex and important issue, worthy of serious discussion. The article, in my opinion, is a mixed bag. It correctly points out many of the issues with RCTs, but I feel draws the wrong conclusions from them.

Flawed or Broken

Flawed does not necessarily mean broken. I feel this concept applies frequently to such discussions, and if it is not explicitly explored then we end up just serving our own bias.

All complex human endeavors and institutions are flawed. It is extremely healthy and useful to explore those flaws in a constructive way in order to improve the institution. It is easy, however, or overhype those flaws into an argument that the institution is fatally flawed and needs to be abandoned, or at least undergo a major revolution.

This is where ideology and bias come in. Take any institution – if you are ideologically predisposed to that institution, then the flaws are relatively minor and can be fixed. If you are antagonistic toward the institution, then the flaws are fatal and evidence that the institution should be destroyed.

We have this debate frequently about science in general. Many of the articles here are about current flaws in the institution of science: publication bias, perverse incentives, P-hacking, conflicts of interest, poor reproducibility, etc. To me the institution of science is valid and solid, but humans are flawed, and so we need to be endlessly vigilant to keep making the system better.

But the exact same issues are often used by opponents to science to argue that the institution is hopeless and should not be trusted at all.

The same is true of democracy, capitalism, the skeptical movement, the health care industry, higher education – you name it.

In addition to serving an ideological agenda, I also find that the flawed vs broken debate has to do with personality and intellectual style. There are some people who always want a revolution, to burn down the institution in the hopes that something better will arise from the ashes. I often find this approach to be intellectually lazy.

There are certainly some institutions that are broken, because they are founded on invalid premises or principles, and therefore should probably be abandoned. Homeopathy, for example, is simply a failed system. There is no revision, adjustment, or improvement that could rescue homeopathy from its inherent fatal flaws. For most institutions, however, this is not the case.

I tend toward the reform end of the spectrum. My default approach to an institution’s flaws is to focus on improvements. The health care system, for example, needs a hundred thoughtful tweaks. I doubt a radical change in the entire system will be ultimately beneficial, and will likely just trade one set of problems for another.

RCTs

With all this in mind, my overall assessment of the NEJM article was that it was too far toward the revolution end of the spectrum. The challenges and limitations to RCTs that it discusses are all valid points, but the remedies offered were often worse than the problems they address.

RCTs are still the best method we have of determining the net efficacy of a medical intervention. Their strength is that they control for variables, control for bias, and can address the critical issue of the risks and benefits of a specific intervention. The RCT is simply too valuable to abandon, or to even significantly replace with other methods. No other research method has the power of the RCT.

At the same time, RCTs are limited because they are often contrived, and therefore may not be perfectly applicable to real world situations.

The authors write:

Even as RCTs have become standard in pharmaceutical research, clinical researchers have struggled to apply them to other areas of medicine. Although psychiatrists have conducted many RCTs of psychotherapy, critics have argued that it is inappropriate, and sometimes impossible, to evaluate such long-term, highly individualized interventions in that way. Some major psychotherapy trials have been undermined by methodologic concerns.

In one section they get dangerously close to post modernism, declaring that, “RCTs serve both objectives, as historical entities representing at once scientific, political, and economic developments.”

Their proposed solution:

The idea that RCTs would be the only authoritative arbiter to resolve medical disputes has given way to more pragmatic approaches. Experimentalists continue to seek new methods of knowledge production, from meta-analyses to controlled registry studies that can easily include large numbers of diverse patients. Observational methods are seen as complementary to RCTs, and new forms of surveillance can embed RCTs into the structure of data collection within electronic health records. RCTs are now just a part — though perhaps the most critical part — of a broad arsenal of investigative tools used to adjudicate efficacy and regulate the therapeutic marketplace.

The article is not a hit piece on RCTs, and they do acknowledge the central role of RCTs in medical research, but I think go too far in arguing that other types of research should serve a larger role. The problem is, observational research and pragmatic studies are not efficacy trials. They cannot properly control for variables or minimize bias. They have a very poor history of determining the bottom line – does a treatment work or not.

In the end the fix would be worse than the problem, they would trade in one set of problems for a bigger set of more serious problems.

Observational and pragmatic studies do serve a limited role – at the front end they indicate what interventions are worth further study, and at the back end they help us incorporate proven therapies optimally into practice. But they do not determine efficacy.

A better solution, in my opinion, is just to continue to improve the technology of RCTs. We have identified many issues, and there are fixes to all of them. We really are just facing a quality control issue. As my colleagues and I have argued at SBM, it is also important to include prior plausibility in the evaluation of RCT results (a Bayesian approach), and dethrone p-values as the final arbiter of significance.

As Vinay Prasad points out in his thoughtful criticism of the article, in the end it turned into an argument for replacing better evidence with weaker evidence.

I would add that this argument for relying on weaker forms of evidence in medicine comes at a very dangerous time, when proponents of dubious health claims are arguing for the exact same thing. In the last few decades, people making billions of dollars collectively on pseudoscientific, fraudulent, or just premature, unproven, and dubious treatments have been loudly arguing for a weakening of the standards of evidence in medicine, and they have their sites set on the RCT. Their motivation seems clear to me – they want to erode the standard of care and the scientific basis of practice so they can make more money selling nonsense. I predict they will look at the NEJM article as support for their position.

Rather, we should be doing the opposite. We should be increasing the standards of science and evidence in medicine. The evidence is overwhelming that we tend to adopt new treatments too easily, and they are very likely to be overturned when better evidence is produced. We should be more cautious, not less.

We rely too much on observational and pragmatic evidence. These types of studies are complementary only, they cannot replace the information we get from RCTs.

Ultimately I found the NEJM article to be regressive, although admittedly it was a bit confusing as they seemed to waffle quite a bit. I sensed they were dancing carefully around the issue, but were trying to not-so-subtly dethrone RCTs as the most reliable form of clinical evidence.

Instead the health care profession needs to continue its historic slow march toward higher standards of science in medicine.