Oct 04 2011

A CCSVI Meta-Analysis

In April of 2009 an Italian vascular surgeon by the name of Zamboni published the first paper of chronic cerebrospinal venous insufficiency (CCSVI) in which he proposed that blockages in the veins that drain the brain are strongly associated with multiple sclerosis, and further “play(s) a key role in determining the clinical course of the disease.” The paper sparked a controversy that rages still, one I have been following fairly closely here.

Discussion has now been renewed by a just published meta-analysis of CCSVI trials. The authors of the meta-analysis conclude:

Our findings showed a positive association between chronic cerebrospinal venous insufficiency and multiple sclerosis. However, poor reporting of the success of blinding and marked heterogeneity among the studies included in our review precluded definitive conclusions.

In other words – the data are all over the place, making a meta-analysis all but worthless.

Reporting on this story has been variable, but overall not bad. There are two components to the conclusion, and reports can emphasize one, the other, or both equally. The first is that there is an association between CCSVI and MS in the studies that have been done to date. But the second conclusion negates the first – that there is considerable variability in the quality, success of blinding, and outcome of the studies, making it impossible to draw any definitive conclusion.

In my opinion, a meta-analysis was the wrong tool for reviewing this question. Essentially, a meta-analysis combines the results from various trials, treating them as if they were one large trial. The advantage of this approach is that the resulting combined data set has much more power than the individual studies. There is also the hope that by combining multiple trials error and bias will average out and the true signal will come through.

However, there are several weaknesses to meta-analysis. The first is the obvious fact that they are only as good as the trials they combines, following the old rule of garbage in – garbage out. Combining trials may compensate a bit for outliers, but does not correct for the general quality of the studies.

Second, a meta-analysis works best when the studies that are being combined are homogeneous – with similar methods and outcome measures. Otherwise it is difficult to know how to quantify different outcomes. And third, meta-analysis introduces another possible layer of bias into the analysis by which studies it chooses to include.

It is not surprising, therefore, that a 1997 study found that meta-analysis predicts the outcome of later large definitive clinical trials only 65% of the time, which is not much better than chance.

For messy preliminary data a systematic review is perhaps better than meta-analysis to get an idea about what the state of the research is. In this case, we have one researcher, Zamboni, who has generated the initial positive results. Most of the attempts at replicating this research have either had mixed or negative results. There is little consistency among the various studies.

There also appears to be a correlation between lack of blinding and positive outcomes. Zamboni’s original data was unblinded, and dramatically positive. In a follow up study he claims to have addressed the concerns, but critics point out that he gives no indication of how assessments were blinded and how successful the blinding was. Attempts to replicate the data with well-blinded protocols have tended to be negative.

It is a good rule of thumb that when a phenomenon tends to disappear when proper blinding protocols are put into place, then the phenomenon is likely not real.

Conclusion

The current meta-analysis can really only reach one conclusion – that the data is preliminary and mixed, with various degrees of blinding and overall quality, and therefore a meta-analysis cannot reach any conclusion as to the data. The researchers really could have stopped there. Actually doing the meta-analysis was pointless, and generated what the authors acknowledge is likely a spurious outcome. This outcome, however, is likely to add confusion to the reporting of the data.

A better approach to this set of studies is a science-based systematic review, taking into consideration the relationship between the quality of each study (especially the quality of the blinding) and the magnitude of the correlation between CCSVI and multiple sclerosis. The emerging consensus as more studies are being done is that there is no correlation.

Share

5 responses so far

5 Responses to “A CCSVI Meta-Analysis”

  1. Kawarthajonon 04 Oct 2011 at 1:18 pm

    This is a big issue in Canada, where MS patients have successfully lobbied the federal government to engage in a study to see if this approach to treating MS can be helpful.

    While I agree with Bob’s approach to studying the question of whether CCSVI is a scientifically viable treatment, it is a hugely emotional issue. Many MS patients, desperate for a reprieve from their illness, believe strongly that this treatment should be made available to them and are travelling all over the world to get it – India, Hungary, China, etc… There are many anecdotal reports that it helps – which often is enough to fuel widespread lobbying for the treatment to be made available. (There are also reports that it does not work and many patients have had their overall condition worsened by this procedure, although these reports do not sway the proponents of the treatment) I believe a clinical trial is underway in Manitoba (or Saskatchewan?).

    Overall, I think that this is a very emotional issue and is one where the science will not necessarily sway public opinion about the treatment. It has yet to be determined whether science or emotion will determine public policy in Canada (i.e. whether Provincial governments will cover the cost of the treatment).

  2. sonicon 04 Oct 2011 at 2:45 pm

    Meta-analysis uses mathematics that have very strict limits on applicability. It seems the data in this case are outside that limit.
    Doing a statistical analysis in this situation can give a false impression of certainty. The researchers make mention of that (lack of heterogeneity…) but do the analysis anyway.
    And some will take that and get a false sense of certainty.

    Once upon a time I thought it would be good to clean-up this area– the misuse of statistics.
    Perhaps this is a good example of where positive thinking is really wishful thinking.

  3. Kawarthajonon 04 Oct 2011 at 3:59 pm

    Sorry, I meant Steve, not Bob, in my comment above.

  4. JurijDon 04 Oct 2011 at 4:24 pm

    The exact same thing is happening here in Slovenia, where Zamboni has successfully found allies in certain private clinics near the border between Italy and Slovenia that have begun a systematic campaign propaganda aimed at the public and legislators to have balloon angioplasties to “relieve” the supposed CCSVI in MS paid for by the regular government health insurance.

    So far they have not succeeded but public pressure is mounting and the doctors who are pushing Zamboni’s agenda have often publicly said they are conducting a clinical study to measure the effectiveness of balloon angioplasty to relieve the symptoms of MS.

    When asked how the study is being conducted they specifically started to make up reasons why BLINDING is not at all necessary in their study.

    I can see them coming out with a bogus blinded, poorly executed study in a year or so that they will proceed to use as a shield against all criticism.

    If this indeed transpires, I don’t think the general public will be swayed by “academic” contra-arguments about the finer points of clinical study design and blinding and these guys will get exactly what they want. – MONEY

  5. LarryGon 13 Nov 2011 at 10:13 pm

    @Kawarthaion: “It has yet to be determined whether science or emotion will determine public policy in Canada (i.e. whether Provincial governments will cover the cost of the treatment).”

    The majority of Canadians are not in favor of politicians spending money on unapproved medical procedures when the wait lists for proven medical procedures are so long. The feds and most provincial governments are not generally moving in that direction.

    I may be wrong but the last time I read news about this, only one province was paying part of the cost for some MS patients. Even if some MS patients want to see it paid for, the majority of voters are against it.

    I agree that funding of a medical procedure shouldn’t be decided on the basis of anecdotal evidence. But it added to the cynicism to see MS patients being told that they shouldn’t depend on anecdotal evidence, and then the feds and their own “advocacy” group oppose funding clinical trials that would give those patients the hard evidence they need to make a more informed decision.

    People always have decisions to make. Unable to make a well informed decision, they’ll make a less informed decision. No doubt desperation plays a role for many people with MS, but so does risk analysis.

    Thanks to the lobbying (for the clinical trials), perhaps we’ll be able to get some more answers sooner rather than later. Then patients can make a more well informed decision, as can doctors, taxpayers, governments, insurance companies, etc.

Trackback URI | Comments RSS

Leave a Reply

You must be logged in to post a comment.