I think we just have to face it – humans are cheaters. It’s in our nature.

We are also complex social creatures. The result, according to psychological research, is that most people will engage in small cheats if given the chance and they think they will get away with it. Various studies show, for example, that 75-98% of high-school and college students cheat at least once during their careers.

This makes sense given that we evolved in a resource-limited situation. There was an ever present real risk of starving to death, and so the willingness and ability to sneak a little extra food from the group would have had a distinct survival advantage.

At the same time there was a survival advantage to defending oneself against cheating, and living in groups meant that the group could defend against cheating using social pressure. Humans are therefore conflicted – we want to cheat but we feel bad about it because we also have a feeling of disgust toward cheating in order to pressure others into not cheating. We feel guilty and fear the shame of getting caught. When you balance all these things out, most people will cheat a little when they can get away with it. Those who cheat more are better at rationalizing their own cheating. Increased cheating may also result from greater pressure to perform, overcoming the social pressures against cheating.

Peer-Review and Academic Cheating

As must frequently be pointed out – while the process of science may be optimal for figuring out the world, it has to be executed by flawed humans. There is a culture of honesty, integrity, and openness in the scientific community, because the stakes are so high. The cost of cheating is huge, it can waste tremendous resources, it can slow the progress of science, and can threaten public support.

Every scientist has to rely to some extent on other scientists, and the thought of spending years or decades of one’s career chasing down a scientific result that was the product of another scientist’s cheating is truly frightening. For this reason the scientific community has a harsh no-tolerance policy toward academic cheating, which is completely appropriate.

This culture may be a double-edged sword, however. While it certainly discourages cheating, it may create the naive sense that scientists wouldn’t cheat (except for the occasional aberration). Therefore we can mostly rely on the honor system. I think the evidence shows that this position is untenable.

There can also be tremendous pressure to perform, to publish, and to show interesting results. We often refer to this as researcher “bias” but that may be putting a benign spin on reality. For example, a third of researchers in one survey admitted to engaging in questionable research practices. About 2% admitted to outright fraud, like fabricating data.

Individual cases of outright fraud are always scandalous, and it does seem they are the exception. The most recent case was just reported on in the NEJM. They report:

In August 2015, the publisher Springer retracted 64 articles from 10 different subscription journals “after editorial checks spotted fake email addresses, and subsequent internal investigations uncovered fabricated peer review reports,” according to a statement on their website. The retractions came only months after BioMed Central, an open-access publisher also owned by Springer, retracted 43 articles for the same reason.

They give as one example the case of South Korean researcher Hyung-in Moon:

He gave journals recommendations for peer reviewers for his manuscripts, providing them with names and e-mail addresses. But these addresses were ones he created, so the requests to review went directly to him or his colleagues. Not surprisingly, the editor would be sent favorable reviews — sometimes within hours after the reviewing requests had been sent out.

The Solution

I write frequently about the problems of modern science – problems with bias, pseudoscience, and fraud. I also am very clear to point out, however, that I do not think “science is broken.” My point is mainly to help distinguish between good science and bad science, but also to look at the system to see how it can be improved.

It’s a cliche, but I think it is true that often the first step in solving a problem is admitting there is a problem. The community of scientists can no longer assume that cheating is rare and isolated cases can be dealt with when they pop up. These quaint sensibilities are not adequate to a global scientific community.

There are systems in place to catch cheating, which is why we know about it. The articles mentioned above that were retracted are a result of the system working, at least to an extent. The problem is – those articles should never have been published in the first place.

I do think a lot of the responsibility lies with journal editors and the review process. This is already a burdensome process, and many are reluctant to make it more so, but I think we have no choice.

Some journals are starting to demand raw data, which can be a logistical nightmare but I think this step needs to be tried and explored. Perhaps authors should not be allowed to suggest their own reviewers. Peer-review may need to be more robust on average, and more supervised.

I also think it’s important for journals to make more space to publish exact replications. Replications are perhaps the best way to root out error, bias, and fraud. Flawed results should not replicate.

Stepping back, we can think of this as an attempt to optimize the progress of science. This is a balance – we want science to progress quickly and efficiently, but we need to slow it down for quality control. We may not currently be in the sweet-spot.

I don’t think we are far off, though, and there are some low-hanging fruit and tweaks that will help, while we consider if more sweeping changes are necessary.

There may also be some win-wins – ways to make the process better without making it more onerous. For example, computer algorithms may be able to analyze papers for patterns that suggest bias or fraud, and to detect statistical errors (which are common). This would be similar to using computer analysis to detect plagiarism.

In other words, we may need to make the editorial process smarter, rather than just more burdensome.

Science is largely about data, and computers are great at handling data. We may benefit from an effort to explore how computer automation and standardization can provide a layer of protection from fraud, bias, and error.

Conclusion

The stakes in science are extremely high, and only getting higher. Our civilization increasingly depends upon the reliability of scientific data. Certainly science drives my own profession of medicine. The global warming debate also showcases how important the reliability of data is to making important decisions for our entire civilization.

We may be near the limit of what people can handle in terms of running a global scientific infrastructure that optimizes progress and minimizes error. Relying on cultural pressure and human reviewers may be inadequate. More brute-force may also not be the answer.

Atule Gawande convincingly argued in The Checklist Manifesto that when brute-force is not enough, we need automated systems (the checklist being a simple and effective example).

The scientific community needs to have a serious discussion about what systems can be put in place to discourage and detect bias, fraud, and error in the pre-publication process. We will always need retractions to capture whatever slips through the cracks, but that is a fail-safe, not a primary mechanism.

Meanwhile, for those trying to make sense of published science, this is a reminder that you should never rely on single studies. No single study is ever definitive. Replication is still our best method for determining which scientific results are reliable.