I wonder what you mean by common, because I find it concerning that such an error could get pass peer review. I could see this being very common in a covert sense – that the researchers cherry pick among many variables, but then do not disclose this information. But- to do this in a way that could be detected and for that error to be missed is disturbing, especially for something that is routinely taught in introductory statistics classes

For sure, if you think of moving a limb you’ll see which part of the brain gets more active, but the rather far fetched studies now being churned out by the dozens going as far as to ‘conclude’ that “While religious and nonreligious thinking differentially engage broad regions of the frontal, parietal, and medial temporal lobes, the difference between belief and disbelief appears to be content-independent” http://www.ncbi.nlm.nih.gov/pubmed/19794914 for example based on it is beyond belief. (pun intended)

With such a crude and errorprone device this kind of ‘research’ seriously undermines the status of the science. Imho.

I have read through the articles you referenced. First off, the largest trial is 37 people, the other two are 15 and 12. In any sort of trial, that is significantly underpowered to generate any sort of reliable results. In brain scan studies specifically this is even worse. John Ioannidis demonstrated that the corpus of brain scan studies are vastly under powered and – even when generous accessions were made – that positive results from brain scan studies were double what should be statistically possible (Arch Gen Psychiatry. 2011;68(8):773-780. doi:10.1001/archgenpsychiatry.2011.28). Granted this may make for a solid pilot study to then do more rigorous analysis, but that has been completely lacking. In fact, looking over a PubMed search on the topic of fMRI scans and acupoints, it is notable to see that there are only 50 studies, they are all small, and they all either come out of China, the authors are all affiliated with a Chinese institution, and/or are published in journals with a low impact factor and an obvious likelihood of bias such as “J Acupunct Meridian Stud” (including the ones that you referenced). It is also well known that studies coming out of China on TCM and acupuncture are extremely unreliable with a massive publication bias. It has been documented that 99% of acupuncture trials out of China are positive (Control Clin Trials. 1998 Apr;19(2):159-66.) which is simply an impossible statistic outside of obvious publication bias and/or poor study methodology. In fact at least two large review of reviews has demonstrated that while “Utilization of Chinese-language databases greatly increased the number of potentially relevant references for each search. Unfortunately, due to methodological flaws, this additional information did not generate any usable information.” (Journal of Evidence-Based Medicine Volume 5, Issue 2, pages 89–97, May 2012) and “The quality of trials of traditional Chinese medicine must be improved urgently. Large and well designed randomised controlled trials on long term major outcomes should be funded” (Review of randomised controlled trials of traditional Chinese medicine BMJ 1999; 319 doi: 10.1136/bmj.319.7203.160) As for the studies you cite specifically: As with pretty much all other studies that come out of China there are significant methodological flaws. In all 3 proper blinding is not observed. One in particular (the PNAS paper) is also striking in that they found differences in the fMRI signal which they then attributed to the participant being either of “yin” or “yang” disposition. They try to claim that the confirmation of “yin” and “yang” characteristics was independently and blindly verified, but they do not at all elucidate what those characteristics are, how the are determined, and whether or not the designation was given before or after the fMRI results were obtained. Additionally, they admit that 1 blind determination was incorrect, which is rather significant given that it is essentially a 50/50 chance of getting it right by pure luck and the sample size was so small. Additionally they discuss the methodology used for selecting whether the pixel (which should be called a voxel in MRI studies) was active by comparing a correlation coefficient to a threshold value (TH) which they say was set at 0.4 for “most of the study.” There is no discussion as to why that particular value was chosen, nor a discussion as to why only “most of the study” used that value, nor which parts used that value, how it was determined that a different value would be used, nor what different value was actually used. The remainder of the studies follow this similarly poor methodology which, as I described above, is known to be par for the course in Chinese studies. Additionally, large studies with adequate power have shown that in fact there is no difference between sham and verum acupuncture in outcomes. A study of 1162 patients with chronic low back pain (which we would a priori expect to have the largest placebo responses and thus the largest effect sizes with placebo intervention) demonstrated ” At 6 Months, Response Rate Was 47.6% In The Verum Acupuncture Group, 44.2% In The Sham Acupuncture Group…Verum Vs Sham, 3.4% (95% Confidence Interval, −3.7% To 10.3%; P = .39)” (Arch Intern Med. 2007;167(17):1892-1898) In other words, it really didn’t matter where you put the needles it had the same effect which clearly demonstrates that there is no intrinsic utility to acupuncture but a distinct placebo response amongst the cohort of subjects. A study in the journal “Pain” did a large review of 57 systematic reviews. Of note only 4 were considered “high quality.” The review of reviews concluded “numerous systematic reviews have generated little truly convincing evidence that acupuncture is effective in reducing pain. Serious adverse effects continue to be reported.” (PAIN Volume 152, Issue 4 , Pages 755-764, April 2011). For all of these reasons I am quite unimpressed with the studies you have provided and would most certainly NOT consider them to be more objective. To stress the point, any brain scan data – and especially fMRI data – is subject to a large amount of observer bias in interpretation. A lack of blinding the researchers can very easily lead to biased results (whether intentional or not). As John Ioannidis has pointed out on this topic, the only way to prevent this would be to both blind the researchers AND require that the trial declare beforehand exactly what type of measurements will be used and what will be measured. The resolution of brain scans and our understanding of the complex interconnectedness of neurophysiology is simply too poor to consider them immune to bias and indeed can be very easily biased unintentionally. So this is what I mean when I refer to prior plausibility of a so-called CAM modality. A rigorous look at the corpus of data surrounding the topic is required and must be implemented, which is much more than simply reading the abstract which will *always* paint the conclusions in the most favorable possible light. I still see no plausibility for a putative mechanism of acupuncture beyond placebo effect and large analysis demonstrate clearly that there is no putative effect. Even the most recent meta-analysis demonstrates a statistically significant but clinically insignificant effect on pain from acupuncture. An intellectually honest researcher would recognize that in subjective assessments there is invariably a large amount of noise in the signal and that the threshold for accepting something as significant must accordingly be higher. Even in that case though, clinical significance is also of importance and no study in acupuncture without serious methodological flaws has demonstrated clinical significance for any use of acupuncture.

The Igs are not just to make fun of bad science, but to recognize interesting science that is humorous in some way. So you end up with both ends of the spectrum.

Also, I guess I’m a bit confused about the purpose of the Ig Nobel awards. Often they are given to researchers who do sloppy science or to school boards that would prefer to teach kids superstition than science, for example. But in this case, the salmon fMRI study appears to be an example of good science, a straightforward way to demonstrate how easily statistical analyses can be misapplied or abused. These types of studies are, in my opinion, at opposite ends of the science quality spectrum.

We’ve seen this before. (See Comment 3: http://theness.com/neurologicablog/index.php/publishing-false-positives/comment-page-1/ )

I wonder if this is statistically significant?