Li Li, a kinesiology professor at the Louisiana State University (LSU), is conducting a study  of the effectiveness of tai chi in treating peripheral neuropathy. He reports very positive results from this ongoing study that started in 2004, uncritically passed along by the media. I thought it would be a good example of the difficulty in designing good clinical trials and interpreting their results. Below are some of the issues raised by this study.

What is the target disease and how is it defined?
Ideally a clinical trial will focus on a specific disease with both inclusion and exclusion criteria designed to make sure that everyone in the study has the same disease and does not have anything else. The tai chi study says it is studying “peripheral neuropathy.” But peripheral neuropathy is not a disease, it is a category of diseases with a long list of very different causes. There are different types of nerves, different types of nerve damage, and many different types of causes. Mixing them together makes for a very messy trial.

Is the population homogeneous?
In addition to having the same disease, it is typical to screen subjects so that they fall within some range of severity. This is to make sure that they are severe enough to ensure that they actually have the diagnosis, yet they are not so severe that there is little chance of them getting better.

Is the study double blind?
“Blind” means that it is not known if a subject is getting the treatment being studied or the placebo or comparison treatment. Double blind means that neither the subject nor the evaluator knows. In the tai chi study it is not possible for the subject to be blind. However, there is no indication that a blinded evaluator is scoring the progress of the subjects.

What is the outcome measure?
This is a critical question, often overlooked by those unfamiliar with clinical trials. What, exactly, is being measured, and how does this relate to the effect that is being looked for. Good outcome measures are objective, measurable and quantifiable, and should relate directly to the disease being studied. Often many outcome measures are looked at to see how they correlate. An overall subjective feeling of well-being is essentially worthless as an outcome measure.

With neuropathy it is common to use standardized nerve conduction study. This is an electrical test that looks directly, objectively, and quantifiably at nerve function. It does not require any subjective feedback from the patient. But sometimes sensation is important to measure, and this will always have a subjective component. However, this can be quantified to some degree using standardized exams. Other aspects of neurological exam can also be used.

The tai chi study reports improvement in balance, improved walking, quality of life, and pain. None of these are good outcome measures to use. The first two are closely related, as many people in the study may have had impaired balance as the primary problem with walking. This kind of outcome is called a “functional” outcome, because it measures some aspect of function, such as walking. Functional outcomes are helpful in assessing the overall impact of an intervention, but they are not good as primary outcome measures because they are difficult to interpret. For example, it is possible (even probable) that tai chi improves balance through simple practice and muscle strengthening. This would be expected even without any actual improvement in nerve function.

Pain is an especially bad outcome measure for neuropathy because it is common for pain (such as the burning pain reported in the article) to be maximal at the onset of neuropathy. As nerve damage worsens, burning pain may actually improve, and this is often misinterpreted as an improvement in the nerve damage itself.

Are there other factors that have not been controlled for?
The art of designing a good clinical trial is anticipating all potential factors influencing the outcome and controlling for them. This is not always possible, and typically study design for a specific question will evolve over years as the design and results are debated by experts.

The tai chi study involves may potential complicating factors that could skew the results. For example, subjects with diabetic neuropathy (the most common cause of neuropathy) may have benefited because the exercise from tai chi improved their diabetes, and this in turn improved their overall health and even their nerve function.

So what can we conclude from this study? Not much. It is certainly not possible to conclude that tai chi has any special benefit for health in general or neuropathy in particular (over other comparable exercise programs). Other studies have shown that tai chi is good exercise, which is enough to account for the perceived benefits in this study. Other studies have also found that gait training is more effective than tai chi in improving balance and walking.

The lesson is that many important and detailed questions should be asked about any clinical study before its outcome can be meaningfully interpreted. Also, it is rare for any single study to be definitive. Often, many studies have to be interpreted together to reach a reasonable conclusion. Unfortunately, the lay press is not well equipped to deal with this level of complexity, and the public are often treated to reports that are simplistic to the point of being wrong, or at least grossly misleading.