Mar 30 2015

Peer-Review Scandal

Biomed Central, a UK company that publishes 277 peer-reviewed journals, announced that it is retracting 43 articles because of “fabricated” peer-review.

Peer-review is a process that many scientific journals use to vet submitted articles. Typically an editor will review the article but also send it out to two or three experts in the subject matter and have them take a close look at the article to make sure everything is high quality. Most submitted articles will come back with required changes before acceptance. Of course many articles are rejected outright.

The process is not perfect, but it is one critical layer of quality control. The “peer-reviewed literature” is therefore a body of evaluated knowledge that has met at least a minimal standard of quality.

Of course, your mileage may vary. Not all peer-reviewed journals are as rigorous. Also, whenever there is any system in place to separate the wheat from the chaff, someone will try to game the system for their own advantage. There also needs to be some monitoring or policing in place to ensure the integrity of the system.

Scientific fraud is a terrible thing. The institution of science requires complete transparency, and fraud violates transparency and reduces confidence in the whole system. Peer-review fraud is not as bad as fabricating data (the ultimate scientific sin) but it’s bad.

The concern is that the problem may be much deeper than these 43 retracted articles. This story, therefore, may not be over yet. I don’t think this will turn out to be a systemic problem but it may be widespread.

The Washington Post reports:

A partial list of the retracted articles suggests most of them were written by scholars at universities in China, including China Medical University, Sichuan University, Shandong University and Jiaotong University Medical School. But Jigisha Patel, associate editorial director for research integrity at BioMed Central, said it’s not “a China problem. We get a lot of robust research of China. We see this as a broader problem of how scientists are judged.”

That’s interesting – it seems that the problem, while not limited to China, may be concentrated there.  This brings up another highly sensitive issue. Science is both an institution and a culture. While there certainly is a universal culture of science, there are also subcultures – in different disciplines, and in different countries.

China’s scientific culture is embedded in China’s broader culture.

One potential problem arises when a particular scientific subculture includes an excessive deference to authority, without the proper balance of respect for freedom and openness. Junior researchers who feel they cannot challenge their superiors, or feel incredible pressure to get positive results, are more likely to bend or break the rules to please their boss. This can be poison in a scientific environment.

I will note, as I have in the past, that a review of the acupuncture literature, for example, showed that 100% of studies coming out of China are positive. This is not statistically possible, even if acupuncture worked, which I think, given the totality of evidence, it doesn’t. This is evidence of systematic bias – with 100% penetrance.

I am not trying to point the finger at one culprit, but rather just pointing out that ethical problems in science are not evenly distributed. This provides an opportunity to examine the institutional and cultural factors that may contribute to scientific misconduct. I agree with Patel that we should not overstate the significance of clumping in China or pretend the problem does not exist elsewhere, but neither should we ignore it. It’s data.

Conclusion

This latest scandal will hopefully provoke another round of self-reflection among scientific journals and improve quality control.  Quality control is an endless game, and there will always be problems like this. At least the peer-reviewed literature has the power of retraction, to purge itself of fraud or poor quality when it comes to light.

7 responses so far

7 Responses to “Peer-Review Scandal”

  1. evhantheinfidelon 30 Mar 2015 at 11:01 am

    This is a nice case that demonstrates, at least in part, that though there can be value in cultural variation, there are specific values that are integral to science which can be overpowered by contrary values in the larger culture. I think it further confirms some sort of objectivity in science, since these problems are rooted out (primarily through investigation and further research in the spirit of science) even though there are /some/ powerful cultural influences at play.

  2. Fair Persuasionon 30 Mar 2015 at 3:33 pm

    Retraction of papers in university biology labs is an issue between European research and the United States too. Some years ago at UIUC, student experimentation with lab fish species from American stock behaved exactly the opposite of published papers from a known Italian university’s lab fish stock. Laboratory conditions can be different. But did the fish species behave the same in natural open waters? Only fisherman would benefit from this knowledge compared to the experimentation in a laboratory aquarium.

  3. mlamphieron 30 Mar 2015 at 6:47 pm

    The problem specifically here was a kind of loophole in the peer review process. Journals sometimes allow the submitting author to suggest who the reviewers might be — it is both a courtesy to the author and a time-saving feature for the editors, since the submitting author probably knows who the experts are more than the editor.

    In this case, the submitting authors employed a “manuscript preparation service” — utilized primarily by authors who are not fluent in English. These services help authors revise their manuscripts and guide them through the submission process (I have some experience doing this for several Asian labs). But there was an additional “happy ending” service offered as well, which was to provide the names of reviewers who would almost always accept your paper.

    The upshot is that this courtesy to potential authors will now most certainly be eliminated in most journals. Authors will still be free to suggest reviewers who should NOT review their paper — these are typically competitors working in the same field or the guy whose wife you made a pass at during social hour.

    The really really unfortunate aspect of this is that every anti-science movement under the sum, anti-AWG, anti-GMO, anti-VAXX, etc. will now cite the headline as evidence that peer review is corrupt, without regard to the actual details.

  4. petrossaon 31 Mar 2015 at 5:27 am

    “I don’t think this will turn out to be a systemic problem” A dutch saying goes: The wish is the father/mother of the belief…

    Now that due to the AGW scandal the diverse self reviewing peer networks have been proven to exist there is absolutely no reason to assume they don’t exist in other branches of science.

    After all they are governed by the same laws of need to publicize, character traits needed to become a top dog and just plain human nature coupled with the downgrading of educational norms.

    Since the 1960’s education became a mass product, and logically it had to adapt to the lowest common denominator. Having 90% fail their grades just isn’t a good advertisement for your education centers. So over the decades with each new assessment levels to make the grade got realigned with the average.

    The result after all the generations of mediocre students which are the norm setting the norm the stakes against a true scholar making it are minimal outclassed by the majority clawing their way up.

    It’s nice, high moral ideas of the right for education for all, but all it does is just making everyone just as incompetent as the the majority.

    If einstein or bohr nowadays applied they’d never ever even got close to publication. To name a few. Good luck science, you’ll need it

  5. RickKon 31 Mar 2015 at 12:40 pm

    Perhaps you can enlighten me, petrossa. I’m trying to recall the period in human history when
    – brilliant people didn’t have to overcome often severe limitations of their society or circumstances;
    – when science (or any complex human institution) didn’t suffer from cliques, rivalries and dishonesty;
    – when one generation didn’t complain about the educational standards of later generations; and
    – when high moral ideas were unimpeded by parochial self-interest.

    Frankly, I think this whole incident is a FANTASTIC example of the effective self-policing within the scientific community.

    The overwhelming majority of science frauds and mistakes are uncovered and exposed by other scientists within the scientific community – across a vast spectrum of fields ranging from cold fusion to fake fossils. Only when the anti-vaxxers, creationists, alt-med or anti-AGW people can stand up and honestly say “we found fraud or flagrant error in our midst, we have outed and ousted the perpetrators, and we have taken steps to ensure it doesn’t happen again” can they hope to take any of the high ground held by the scientific process.

  6. meiguizion 31 Mar 2015 at 3:44 pm

    I cannot attest to the academic culture in terms of tenure and/or peer reviewed publications, but I taught university English in China for four years at universities at varying levels of status and can state that certain cultural practices seem to be more present in university life there then they are here. Cheating among the students was rampant, and one university I worked for (admittedly, a small teaching university in a more remote city) had a fairly established price for getting your child a better grade. In general, there was a lot of pressure from the top level administration to have good good average scores and grades, to the point where grades the foreign teachers handed in were curved after I had already adjusted things to charitably get the distribution where I thought it would be okay.

    That being said, the final university I worked at was one with some national ranking, and is well respected outside of China as a mid-level Technical University. They had a much better culture from what I could discern in terms of academic integrity; there were a few student from one of my classes dismissed from the university for academic indiscretions.

    From my very narrow window, I am not very surprised that peer review was gamed towards the goal of increasing publication rates. That has been an ongoing theme in modern China: authorities set ambitious targets, and lower lever functionaries find ways to get the desired results, even if the means are wholly unfavorable. Look up the multiple Ghost Cities in China as evidence of how economic development goals have been rigged by unneeded construction.

    It does seem to be doing better. I don’t know about with peer reviewed literature, but by most other standards I am familiar with, closer integration with the West, and the aspiration to the status that international recognition brings, seems to be pushing a lot of things in China towards western standards. Slowly.

  7. elmer mccurdyon 02 Apr 2015 at 11:34 am

    This article in Slate give the impression that it is much more of a systemic problem than this post implies.
    http://www.slate.com/articles/health_and_science/science/2015/04/fake_peer_review_scientific_journals_publish_fraudulent_plagiarized_or_nonsense.html

    I haven’t clicked on the biomed announcement yet…

Trackback URI | Comments RSS

Leave a Reply

You must be logged in to post a comment.