Skeptics are often in a tricky position. We simultaneously are cheerleaders for science, promoting science education, scientific literacy, and the power of science as the best method for understanding the universe.<\/p>\n
At the same time the skeptical approach requires that we explore and discuss all the various flaws, errors, and weaknesses in the institutions and process of science. Science in theory is fantastic, but it is practiced by flawed people with all their cognitive biases and perverse incentives (much like democracy or capitalism).<\/p>\n
I think the best approach to this apparent contradiction is transparency, honesty, to be as constructive as possible, and avoid sliding into nihilism. It’s easy to focus on all the negatives about any institution, and conclude that it’s hopelessly broken. Some institutions are broken and unfixable, so it’s not an inherently unreasonable position. We should strive for a balanced and fair assessment (just like Fox news).<\/p>\n
A recent\u00a0article<\/a>\u00a0published in The Economist is getting a lot of play in scientific and skeptical circles. It reviews what skeptics have been talking about for years. There is a lot of crappy research out there that is unreliable. This means that just because you can find some studies that appear to support your position, it does not mean your position is correct. You cannot know the answer to a question by cherry-picking the studies you want. You have to do a critical analysis of all the research.<\/p>\n John Horgan blogging at Scientific American also discusses the Economist article<\/a>. He adds an interesting perspective, something that is a bit of a running gag on the SGU – looking back at the hyped science news stories of the past reveals a graveyard of failed science. He discusses, for example:<\/p>\n One of my first assignments was profiling Jerrold S. Petrofsky, a biomedical engineer at Wright State University trying to help paralyzed patients walk by electrically stimulating their muscles with a computer-controlled device.<\/p><\/blockquote>\n Horgan notes that Petrofsky’s work was overhyped and never amounted to anything, even after 30 years.<\/p>\n The Economist article and Horgan’s discussion are good overall, but I think a bit too negative. The full articles are worth a read, and regular readers of skeptical blogs will probably recognize many of the points and references, but will also likely learn some new details. Here is my own summary of the major areas of concern regarding the quality and reliability of published scientific research.<\/p>\n Most studies are of poor quality<\/strong>\u00a0\u2013 Doing a large rigorous scientific study is difficult and requires a great deal of resources, including time and money. Researchers therefore conduct many preliminary or exploratory studies, which are small and have only basic controls. Such studies are unreliable, and are useful only in determining if further research is warranted. Preliminary studies, when positive, tend to be false positive (for reasons I will outline separately), and are much more reliable when negative.<\/p>\n Publish or perish<\/strong>\u00a0\u2013 Researchers and institutions are under high pressure to publish studies. This encourages producing a high volume of low quality studies, or in publishing the \u201cleast publishable unit\u201d of ongoing research to maximize the number of papers derived from one\u2019s research.<\/p>\n Researcher bias<\/strong>\u00a0\u2013 Researchers are people who want their ideas to be correct, and studies may be conducted by industry or those with a vested interest in the outcome. Even within accepted methodology, researchers have a great deal of degrees of freedom, or wiggle room. This freedom can be exploited (consciously or subconsciously) to engineer a positive result, even out of completely negative data. Simmons et al\u00a0demonstrated<\/a>\u00a0that a p-value of 0.05 can be created out of negative data 60% of the time just by exploiting researcher degrees of freedom. In surveys\u00a0a third of researchers<\/a>\u00a0admitted to engaging in questionable methods that would exploit degrees of freedom to create positive results.<\/p>\n Publication bias<\/strong>\u00a0\u2013 Journals, whether in print or online, have their own motivations for success. Subscription journals want to maximize their impact factor, and that means publishing exciting studies with new and surprising findings. Such articles, of course, are the very ones that are most likely to be false positives. Open-access journals, on the other hand, that charge researchers a fee are motivated to publish lots of studies, regardless of quality. A recent Science magazine\u00a0article<\/a>\u00a0pointed out the pathetic quality control in this segment of the industry. In addition, researchers themselves are more likely to submit a paper that is positive rather than negative.<\/p>\n Lack of replication<\/strong>\u00a0– Independent replication is the key to the self-correcting nature of science. The problem is, scientists do not do replications enough, and journals do not publish them enough. There is the now famous incident of Psychology Today publishing terrible research by Daryl Bem claiming that subjects could \u201cfeel the future.\u201d Richard Wiseman et al did an\u00a0exact replication<\/a>\u00a0of one of Bem\u2019s studies, with negative results, and submitted it to Psychology Today. Their response? We don\u2019t publish exact replications. Why not? Because they are not sexy enough to boost their impact factor.<\/p>\n