January 1998
by Steven Novella, MD

The NIH Office of Alternative Medicine seeks to scientifically research alternative medicine modalities, such as acupuncture, but is the act of doing research enough to make alternative practices science?

The true nature and essence of science is of intense interest to skeptics, as it should be to any rational person. The question has become of particular importance to defenders of scientific medicine, as the promoters of alternative medicine practices have risen to seriously challenge the dominance of the scientific approach to HealthCare. Alternative practitioners have argued that their treatments are not amenable to scientific study, or that they are based upon anecdotal evidence which is compelling, and they have challenged that the art of medicine is more important than the science, and even that mainstream medicine is just as unscientific as any alternative philosophy.

Recently, the trend has been toward claiming that specific alternative practices are scientific, because they have research to support their claims. The new National Institutes of Health Office of Alternative Medicine (OAM) was established to fund scientific research into alternative practices. The stated purpose is to unite traditional, ancient, or unconventional modes of treatment with modern scientific methods. On the surface, this is a laudable mission, but the philosophical flaws of many alternative practices go far too deep for the salve of research to cure. Unless the full spirit of scientific inquiry and evidence based medicine is embraced, going through the motions of research is an empty exercise.

At first, such an assertion may seem contradictory to the spirit of skepticism and the skeptical movement, which overtly encourages scientific research into paranormal, fringe, and pseudoscientific claims. Scientific research, however, must be conducted in the proper scientific context. The purpose of such research must be a genuine search for the truth, whatever it may turn out to be. Researchers must be prepared to accept both positive and negative results of their research. Within the field of medicine, they must also be willing to change their practices based upon the results of such research. I will demonstrate that alternative practitioners are not operating within a scientific paradigm, are not engaged in a genuine search for truth and knowledge, are not prepared to accept the possibility that their treatments do not work, do not change their practices based upon new findings, and therefore are doing research without practicing science.

Mainstream Medicine

Modern Western medicine derives its power from its strong scientific tradition. Modern doctors are scientists, despite the fact that some may stray from this tradition. Medical training involves exposure to research, learning the statistical methods of epidemiology (the study of the frequency and location of disease within a population), and learning how to use and interpret the medical literature. Claims are defended by citing a reference in the research literature, and even the most senior physicians will not be respected if they defend their assertions only with authority, and not with evidence. Journal clubs, in which doctors in training analyze and criticize published research, are common in residency programs. In short, the culture of modern medicine is one in which doctors are trained to question everything, and to base diagnoses and treatments on a critical analysis of peer reviewed research.

Doctors also learn to become comfortable with change. Change is a sign of progress, which brings new tools and ideas to bear against disease and suffering and makes doctors more effective at what they do. Most physicians therefore maintain subscriptions to several or more medical journals, attend conferences and meeting, and do whatever they can to “keep up.” Medical boards now require recertification of specialists every ten years, reflecting the understanding that physicians who are out of training must prove that they have indeed kept up with the latest techniques and information.

Most importantly, scientific mainstream physicians change what they do based upon new information. This reflects the fact that physicians do not want to use treatments which merely seem to work, but ones which actually do work. In this way, good doctors are skeptical of their own knowledge and practices.

An excellent example of this principle is that of mammilary artery ligation. The mammilary artery comes off the aorta near the heart and feeds the chest wall. In patients with angina, chest pain due to insufficient blood flow to the heart muscle itself, mammilary artery ligation was used with the idea that if the mammilary artery were blocked off, blood would be shunted to the heart, increasing flow, and decreasing or eliminating angina. The procedure was tried in a few patients, with encouraging results, and therefore increased in practice. Many patients receiving this procedure reported to their doctors that their angina was decreased or eliminated, which was very satisfying to the doctors. The procedure was also very lucrative to the surgeons who performed it, for it was very quick and straightforward with few complications.

Scientific medicine, however, was not satisfied. A study was conducted in which one group of angina patients received mammilary artery ligation and a control group received sham surgery (admittedly, by today’s standards, a questionable practice). The results showed no difference in outcome between the two groups. Surgeons who made a practice of performing the procedure did not rationalize away the negative results of the study. Rather, they stopped performing the procedure altogether. Eventually, new effective methods were developed to manage angina.

There are countless similar examples within medical history. Many treatments are pursued based upon a reasonable scientific rationale, eventually the treatments are studied and restudied, perhaps debated for a time, and then either accepted into general practice, or discarded as a blind alley. This is the process of medical advancement which has brought us to the modern age of medical miracles and the doubling of the human life expectancy.

By contrast, there is not a single example of an alternative medicine practice which was completely abandoned as a result of negative research. Alternative practices do not advance, they merely justify themselves, and dismiss criticism. This is primarily due to the fact that they are philosophy based, not science based.

The Office of Alternative Medicine

In 1992 Congress passed an NIH Appropriations Bill which mandated the creation of a new division, to be called the Office of Alternative Medicine. The purpose of the OAM is to fund research into alternative medicine practices. Senator Tom Harkin, who is enamored of alternative medicine, was the prime architect behind the OAM, and remains its strongest supporter. The primary function of the OAM is to provide grant money for research projects which examine alternative medicine practices.

The budget in 1992 for the OAM was $2 million, but this has increased to $12 million in 1997, and the proposed budget for 1998 is $20 million. Tom Harkin is now talking about making the OAM its own national institute. At the same time, money for conventional research is on the wane, and many scientific researchers are finding it difficult to maintain their labs and their research. The long term agenda seems clear, to divert resources away from scientific institutions towards those with a different philosophy.

Dr. Wayne B. Jonas is the current director of the OAM. He is a strong supporter of alternative medicine, and in fact is himself a homeopath (see The Connecticut Skeptic, vol. 1, issue 3). Under his management, the agenda of the OAM has not lived up to its initial promise to scientifically study alternative practices in order to bring them into the mainstream. The primary function of the OAM, rather, seems to be to promote alternative practices.

Since its inception, and despite funding hundreds of studies, not a single alternative practice has fallen due to negative research. It seems unlikely that this is due to the fact that all of the alternative practices which have been studied so far just happen to work. This rate of success is unheard of in science, and is especially unlikely among methods which are selected for being the most speculative and unconventional. Rather, it is due to a philosophical flaw of the OAM.

The OAM is criticized for funding studies which have a loose, or poorly controlled, design. They lack the scientific rigor or clear rationale that ordinarily would be required of those applying for a grant. Performing poorly designed and conceived research in an anti-scientific philosophical context is worse than a complete waste of limited resources. In fact, regardless of the outcome of such research, the practice of medicine is harmed.

When a study is conducted, there are four possible theoretical outcomes. The study can accurately indicate that the treatment under study works (a true positive) or does not work (a true negative) or the study can falsely indicate that the treatment works (a false positive) or does not work (a false negative). Medical research techniques have been developed to maximize the chance that a study will show a true outcome and avoid a false outcome. When research is conducted, however, without fully embracing a scientific context, all of the four outcomes will be harmful to medical practice.

If, for example, a typical study funded by the OAM is negative, which many have been, the researchers and practitioners of the alternative treatment being studied are quick to devise a rationalization for why the study was negative. Their defense often entails the ironic argument that their treatment is not amenable to scientific study in the conventional sense. They also may introduce new principles or forces to explain the negative results, arguing that they now need to do another study. Because they will never accept negative results, however, this only produces an endless series of useless studies.

A dramatic example of this is homeopathic researcher, Jacques Benveniste. In one study investigating the effects of homeopathic doses of the hormone, thyroxine, on the climbing rate of frogs, no difference was found between the hormone administered in normal physiological amounts and homeopathic dilutions where none of the hormone remained. As a control, they placed the homeopathic remedy sealed inside a test tube, and again no difference in the climbing rate was found. This is a negative study. However, Benveniste concluded that the homeopathic remedy was transmitting its effect on the frogs through the test tube, and even postulated the existence of a new basic particle, a “biophoton” to explain this effect.(Benveniste, 1993) Use of such post-hoc special pleading to explain any outcome of an experiment as a positive result renders the experiment worthless. This behavior is a pathetic mockery of the scientific method; it is quintessential pseudoscience.

If a false positive study is conducted then a worthless treatment will gain false scientific support. Defenders of that treatment will now have ammunition to attack any criticism, and even to dismiss true negative studies of that same treatment. A treatment which does not work will then have a false scientific legitimacy, which is likely to increase its use and frustrate any attempts to control it. Most alternative practices which have been around long enough have accumulated one or a few false positive studies. The OAM is likely to dramatically increase this phenomenon.

An example of the selective use of positive studies is acupuncture, recently the topic of an NIH report. The report concluded “there is a paucity of high-quality research assessing efficacy of acupuncture,” and that there is some evidence of efficacy for pregnancy associated nausea, but insufficient evidence for efficacy in pain control.(NIH, 1997) Despite this very tentative and watered down positive report on acupuncture, the lay press inacurately represented the study as concluding, simply, “acupuncture works.”

The NIH report, as weak as it was, has already been highly criticized for failing to take into account the fact that, of all the acupuncture studies that have been done, the better designed the study the smaller the effect. Also, the best designed studies all showed negative results. The most positive studies, by contrast, were also the most poorly designed. Such subtleties were lost on the NIH committee, and certainly on the lay press and the public. This is a clear example of a few poorly designed positive studies overshadowing well designed negative studies. Also, despite all of this research and controversy, acupuncture continues to be performed without change.

A true positive study is also likely to be harmful. Research which is poorly designed or conceived is not likely to gain wide mainstream support. If a study is not well controlled, then there is no way to distinguish a true positive from a false positive, and the treatment will remain controversial. For this reason, only studies which are well designed and executed will be convincing and will therefore advance reliable medical knowledge.

If out of the many speculative treatments, one or a few turn out to work, and they gain a large enough body of evidence to convince even skeptical physicians, then there will be benefit in that a new and effective treatment method would have been added to the body of medical practice. This fact, however, will be used (as has been done in the past) to falsely support all speculative treatments. Quacks will then shout from the roof tops that “The skeptics were wrong,” and “alternative medicine works.” That treatment will become the poster child for all alternative medicine. The resulting increase in the practice of treatments which do not work is likely to far outweigh any benefit from the acceptance of the new treatment which does work.

Finally, a false negative study is also harmful, or at least not helpful. All false studies simply add to the confusion of any controversial treatment and do not advance knowledge. A false negative study is likely to keep a useful treatment out of the mainstream and on the fringe.

Simply doing studies, therefore, whatever the outcome, is not solution to the current problem of alternative medicine. Research should be done in the genuine spirit of distinguishing those treatments that work from those that do not work. In order to protect the public from treatments which are harmful or do not work, any new treatment should have to be reliably proven safe and effective before it is marketed to the public. Until that time it should be considered experimental. Use of experimental treatments should also be carefully regulated, and only used in two circumstances: as part of an approved clinical trial which is designed to determine if the treatment works, or on a compassionate basis for those with otherwise untreatable illnesses. In either case, patients should be fully informed about the experimental status of the treatment.

This philosophy is accepted and even institutionalized within mainstream medicine. The FDA exists to enforce these principles with regard to drugs and medical devices. The medical community also establishes standards of medical practice and ethical guidelines which are used by state licensing boards to enforce these principles. The system, of course, is not perfect, but it works and it is getting better.

Alternative medicine, on the other hand, operates outside of this system. Experimental methods are used with claims that imply that there is no doubt about the treatment’s safety or effectiveness. Most importantly, there is absolutely no incentive on the part of practitioners to conduct reliable research into their methods, because they are free to use them without the burden of proof that they work or are safe. They do not have to change their practices based upon new research, because what they were doing was not based on research in the first place. An almost universal response to research on the part of alternative practitioners is to cite those studies which are possible, and dismiss those which are negative, while they keep on doing what they were doing without change.

What must be done is to strengthen the laws which regulate health practices and broaden their application so that they will cover all practices, and not just those which are performed by physicians. If homeopathic remedies, for example, were forced through legislation to pass the same FDA standards as conventional medications, then and only then would homeopaths conduct well controlled research, for they would have to prove that their remedies work, or abandon their use.

Meanwhile, all resources spent on research into alternative practices is wasted. The OAM’s only impact upon HealthCare is to lend false scientific legitimacy to quackery while it diverts precious and limited research resources from more promising applications.

References:

1) Benveniste, J. 1993. Frontier Perspectives 3: 13.
2) NIH Consensus Statement Online 1997 November 3-5; 15(5): in press.

The following are Letters To The Editor concerning this article

Alternative Medicine

I received the first issue of the New England Journal of Skepticism, Winter 1998 issue (Volume 1, Issue 1). I enjoyed every article.

I was particularly moved by the essay on Scientific Research in a Mystical World. I have had some contacts with less than ethical doctors, both of mind and body, and I have formed some opinions based on this experience plus extensive reading.

I would like to add to the article by saying that perhaps one reason the general public has turned more toward non-scientific medicine is because our modern institutional doctors have omitted the most important ingredient in medical practice – the patient’s feelings. There are studies to show that the attitude of the patient can have a profound effect on the course of many ailments. Studies with the placebo effect and industrial efficiency studies have shown that when people get personal attention they work harder, recover faster and have fewer illnesses to boot. Witch doctors and others flourish because they provide a service that traditional medicine does not.

Your description of the attitudes of the practitioners of alternative medicine as being reluctant to change in the face of overwhelming evidence, could be summed up by a quote from an unknown source: “You cannot reason someone out of an idea that he was not reasoned into.”

– Robert Seay

Skeptical Philosophy

Godel’s Incompleteness Theorem applies to well-defined formal systems; it says that any formal system which meets certain criteria contains statements which cannot be proved or disproved within that formal system. Thus, every formal system is fundamentally limited; none can do the whole job one might ask of it.

It is not necessarily valid to say the same thing about Mathematics as a whole. Mathematics uses formal systems, and parts of mathematics are often formalized in formal systems; but notwithstanding projects such as the Principia Mathematica, there is no single agreed-upon formal system for doing all of mathematics. Even if all mathematicians adopted a particular formal system, there could not be just one, because you can always add additional axioms to the system when you want them.

As a practical matter, the formal systems used for mathematics are incomplete in other more important ways: several basic mathematical questions are undecidable based on the usual axioms of set theory, including the existence of choice functions, the Continuum Hypothesis, and the existence of certain kinds of infinite cardinal numbers. These questions are not just unprovable, but actually undecidable, in that either their truth or their falsehood could be consistent with the axioms.
Mathematicians take advantage of this situation by postulating additional axioms to resolve these questions: the paradigmatic example is the Axiom of Choice, which says that choice functions exist in all cases. Mathematicians most often use this assumption, but the contrary assumption is occasionally interesting.

But even if Godelian incompleteness does apply to mathematics as a whole, it makes little difference to what we can expect mathematics can do. It does not show that deduction is invalid, only that there are certain questions which it cannot directly settle. Excluding these questions leaves an infinity of statements that deduction could resolve; but mathematicians at any time have proved only a finite number of theorems. The provably unprovable Godelian statements are insignificant beside our actual mathematical ignorance. Their existence is an interesting aspect of mathematics, not a crippling injury to it.

– Richard Stallman

Filed under: Philosophy of Science, Science and Medicine, alternative medicine