Nov 05 2013

Is Science Broken?

You are currently browsing comments. If you would like to return to the full story, you can read the full entry here: “Is Science Broken?”.

Share

25 responses so far

25 Responses to “Is Science Broken?”

  1. oldmanjenkinson 05 Nov 2013 at 9:12 am

    The lack of negative studies published is a significant problem in pharmaceutical research. Dr Goldacre addresses this in Bad Pharma. The negative studies can even be hidden through various methods even if registered with clinicaltrials.gov. He has taken this and has championed the AllTrials campaign. In order for an informed consent to be given by patients, the practitioners need to be aware of all results both positive and negative study results in order to make an informed decision on how to treat their patients. My father taught me there are no failures, only successes in how not to do something.

  2. dziublaon 05 Nov 2013 at 10:09 am

    While I love the idea of publishing/making available negative studies, and have advocated for it in a number of conversations, I still am not sure the best way to implement it. Is the study negative because it was poorly planned, poorly executed, or was it because it doesn’t work? Do we publish all negative studies or just the ones where we are certain the study was “properly” conducted? How do we know it was properly conducted, as sometimes the mistakes aren’t known until after the fact. Further, once negative results become available, the data mining challenge increases tremendously. I liken the problem to an paleontological dig. To some extent, publishing of negative results would be like reporting on every grain of sand that isn’t a fossil.

  3. The Other John Mcon 05 Nov 2013 at 10:30 am

    Thanks Dr. Novella. A really fascinating case-study in the sometimes ugly, very messy, error-correcting machinery that is modern science, that I saw recently:

    http://en.wikipedia.org/wiki/Faster-than-light_neutrino_anomaly

    Luckily people figured out what was up, and came to the conclusion that relativity wasn’t violated. But this would be a great topic for a philosophy of science class or something similar.

  4. Steven Novellaon 05 Nov 2013 at 10:50 am

    Studies should be published based upon their quality, not their outcome. So high quality negative studies should be published, crappy studies, positive or negative, should not be.

    Preferentially publishing positive studies is data mining. It biases the literature, skewing the results of systematic reviews and meta-analysis.

    While publishing negative studies that would otherwise be neglected may increase the total number of papers published, raising the bar and only publishing quality research would decrease the total number of papers published. Doing both will reduce noise and bias, reduce data mining, and improve the literature as a whole.

  5. sonicon 05 Nov 2013 at 11:15 am

    Isn’t the notion of a ‘negative result’ a bias?

    “We found drug ‘x’ does not work for condition ‘y’ with 95% confidence, is a positive result- isn’t it? Or perhaps I should say- if the study was well done, then we can be more certain (positive) drug ‘x’ doesn’t work for condition ‘y’ than we were before.

    “This new larger study shows the earlier smaller study was in error,” is certainly a positive result- it clears up what might be an important mistake.

    I think the idea of a negative result is an example of bias.

  6. Steven Novellaon 05 Nov 2013 at 11:56 am

    Sonic – I disagree. A positive study finds a statistically significant correlation or effect. A negative study does not – it fails to reject the null hypothesis. These outcomes are not symmetrical with regard to their statistical implications, reliability, and conclusions. The probability of false positive and false negative are not the same. Researcher bias is more likely in the positive than negative direction.

    We need to distinguish positive and negative studies. Both, however, are data and all data should be counted. That is the bias we need to eliminate.

  7. dziublaon 05 Nov 2013 at 12:10 pm

    Sonic,
    I think it can be interpreted that way, but to me negative findings are simply seeing a lack of statistical significance. I always correct students from saying “I have bad data…” Or “I have good data…”. I teach them that an experiment that informs their next question or teaches them something they didn’t know is GOOD data. even if it is something that contradicts our hypothesis or design goals.

    Steve, as for your statement regarding quality, I fully agree. However, the nature of negative findings makes it harder to publish based upon quality. There are many more easy and accurate critiques of a paper that presents a negative result than a positive result. Were the reagents confirmed? Was the reaction controlled for variable X, Y, Z, …? What cell line? What passage? What species animal? Is it appropriate? any “no” or question in those points will result in a rejection because that could explain everything. With a positive result, these questions also arise, but are easier to address with a statement of “Will look at with further studies…”

    An interesting case study on the publishing of negative results is the social sciences studies on same-sex couples and impact on child development. Much of the work has shown no statistical difference between children from same sex couples and group matched heterosexual couples. Yet, the opponents of this work have argued that the authors have selected too rigorous a p value (0.05), which will have a greater likelihood for type II error. Instead, they advocated implementing a p-value of 0.1, which will greatly increase the likelihood of type I errors, the exact errors we wish to minimize.

  8. CWon 05 Nov 2013 at 12:18 pm

    At an upcoming meet-up, the science/skeptic group here in Ann Arbor of which I’m organizer, will be having a guest expert who works for the journal, Science. I’m sure some of the points listed in the summary will be brought up in our conversation.

  9. Cow_Cookieon 05 Nov 2013 at 2:53 pm

    The problem is no one in the cycle has an incentive to take the steps necessary to ensure better science. Journals would suffer from lower subscriptions or paid articles. Scientists and institutions would suffer from less published work.

    The scientific community can talk all it wants about what should happen, but things won’t change unless incentives change since incentives will always trump culture. What we need is another group with competing incentives to provide checks and balances on the existing process.

    I wonder if the best thing to do would be to develop a caste of technicians whose specialty is in analyzing and replicating previous studies. They would knowingly embark on a specific career path out of grad school that would not reward unique research. Instead, it would center entirely on replication and critique of others’ research. Advancement would be based on incentives tailored to the career field — identifying methodological shortcomings, replicating studies, analyzing raw data provided by the original researchers, etc.

    This may sound like a cynical profession, but it’s one that already has counterparts in other industries. Automotive engineers don’t just work for car companies. They also work for the National Highway Traffic Safety Administration to determine when those car companies come up short.

    Of course, funding this would be a challenge. Perhaps the NIH could kick things off by earmarking a set percentage of its grants for replicative studies. Maybe a sustained lobbying effort could even convince Congress to specify that a certain portion of basic research funding must go toward such verification.

    But if there were sufficient funding for this need, institutions and associations would begin offering students the opportunity to train in the career field. Professional associations would form. Perhaps there’d even arise one or two nonprofits whose seal of approval would become the gold standard for researchers looking to prove their methodological quality.

  10. Gareth Priceon 05 Nov 2013 at 2:54 pm

    I agree that it is important to publish negative results when the study is attempting to replicate one where the results were positive.

    However, two things occur to me. Suppose somebody does a high quality study demonstrating that drug X has no effect on something. Nobody else is working on that topic because nobody has particular reason to believe that drug X would have any effect. Now somebody has confirmed what everyone would have suspected had they thought about it. There wouldn’t be much wrong with this but it would be a kind of easy way to get quality research into a high quality journal.

    Secondly, regarding the least publishable unit. I myself have often tried an experiment, got a negative result and pursued the matter no further. It might be useful for other people to know that. Should there be a means of publishing a very short, negative result? It seems that maybe for a negative result the least publishable unit might be smaller than for a positive one.

  11. Enzoon 05 Nov 2013 at 3:15 pm

    I honestly think the solution is going to involve a centralized database where researchers can comment on articles, report replications and report contradictions. So every publication is an evolving piece, constantly being updated and interconnected to relevant work. The NIH has reportedly been working on something like this, but it can’t come soon enough.

    In the end, though, the whole system needs to be reworked. Science is way too competitive now to expect the quality of studies to improve. There needs to be a way to reward good science practices (especially for young investigators). There needs to be incentives to share data with relevant groups (and not greedily protect details that would allow replication, etc.). The journals and raw publication number need to be de-emphasized in terms of advancement. Quality score assessment needs to be stressed, etc.

  12. ccbowerson 05 Nov 2013 at 10:05 pm

    “I think the best approach to this apparent contradiction is transparency, honesty, to be as constructive as possible, and avoid sliding into nihilism.”

    This extends beyond science, but any human endeavour. If someone argues that the process of science is so flawed that it is hopeless to fixed them all, then I will re-frame the situation:

    If the process is so flawed, yet we continue to make significant progress unlike any other human endeavour in our history, imagine what is possible if we make even just incremental improvements to the process. In other words, the worse you think our process is now, the more hope you should have for future improvements. For some reason, people don’t tend to think that way

  13. FacelessManon 06 Nov 2013 at 7:45 am

    @Cow_Cookie I would suggest a different approach where students would try to replicate research as part of their BSc or MSc program (under proper supervision of course), or as an internship. I think this would also serve as a good introduction into research for the students.

    Also I agree with Enzo about the central database. From what I`ve seen many groups try replicating a study (or part of it) before starting work, that would base on the study. But when they get negative results, they just move on a differnt one, because they can`t publish. This pattern then repeats itself across many different groups around the world and a lot of time and money are wasted.

  14. The Other John Mcon 06 Nov 2013 at 9:02 am

    Faceless Person, I am totally in agreement about having bachelor or masters theses be, at least in part, replications of previous work (especially work that is important, and interesting, to the researcher).

    Awesome idea and is what I have previously recommended to fellow students. One of the more interesting outcomes of this idea is that no replication is ever *exactly* exact anyway…

  15. Bruceon 06 Nov 2013 at 9:29 am

    In a previous life as a consultant business analyst I learned very quickly to get my fee or most of it up front for the very reason that if whatever piece of work I did came back with a “no issues” response across the board it would be assumed I had not done my job properly. Obviously, in business it is much more likely to find issues, so I can only imagine what it must be like for scientists to have to find that sexy result to justify their funding.

  16. Cow_Cookieon 06 Nov 2013 at 12:36 pm

    @ FacelessMan and The Other John Mc:

    I’d submit that relegating replication to novices would only reinforce the notion that it’s not important. There may well be value in having students duplicate experiments, but the benefits are going to be more for their own learning than for establishing self-correcting mechanisms. Are students really going to be able to stand up to tenured professors on a consistent basis? There may be exceptions, such as Thomas Herndon, but there’s no way that’s going to become the rule.

  17. steve12on 06 Nov 2013 at 12:47 pm

    Enzo talked about competition and incentives above, and I cannot agree enough with this.

    Right now, the incentive structure for young investigators dictates that you should publish as much middling work as possible. In a sense, much of this is replication + boundary test. If we want to increase the pace of discovery, we need to change the funding structure to favor reasonable but higher risk work.

    This means not only having a repository of null results, but judging scientific productivity differently.

  18. steve12on 06 Nov 2013 at 12:50 pm

    “, I am totally in agreement about having bachelor or masters theses be, at least in part, replications of previous work”

    For the most part, theses at this level are at some level replications, if not straight replications.

  19. The Other John Mcon 06 Nov 2013 at 2:02 pm

    Cow Pattie, I don’t mean to say that replication should be strictly relegated to novices, but that, as steve12 said, it’s a great learning experience in that how to do the experimental design/analysis is already largely laid out, there’s a good a priori idea of what should be found, replication is accomplished, learning is accomplished, and hopefully replication+extensions are accomplished…seems like a win/win all around. But I certainly agree it is problematic that replications are unfortunately ignored or downplayed by journals, there should be more of it, there are some good ideas of how to institutionalize this, etc.

    Great discussion on all fronts though, thanks everyone!

  20. BaSon 06 Nov 2013 at 5:44 pm

    Steve,

    In addition to your list, what about the File Drawer Effect, i.e. the fact that you can conduct 10 studies, hide the 8 negative ones and publish the 2 positive ones. This doesn’t seem to quite rise to Fraud, but it’s definitely Shenanigans and a harmful practice!

  21. daedalus2uon 06 Nov 2013 at 6:54 pm

    The problem is in incentives. I think that most scientists would want to do high quality work, but they can’t for lack of resources. There are unlimited resources for activities that generate “profits”, as our capitalistic system has defined them, as in lottery fees, credit default swaps, and so on.

    If you spend your life learning how to get the most money for the least service, you will become pretty good at it. Those who spend their life learning how to do science won’t have the skills to compete for just making money.

    It isn’t that “science” is broken, it is just that our capitalistic system puts greater value on scams that make money than on good science that doesn’t.

  22. Bill Openthalton 07 Nov 2013 at 3:08 am

    @ TOJM

    it’s a great learning experience in that how to do the experimental design/analysis is already largely laid out,

    And it brings a a fresh (call it naive if you want) view to the experiment. What the teachers have to do is to stimulate and support the questioning this lack of experience brings, to welcome the questions, and to engage the students when explaining the reasons — all too often I hear: “that’s how it’s done”.

    New brooms sweep clean.

  23. steve12on 07 Nov 2013 at 2:26 pm

    You’re exactly right daedalus2u.

    I’m pretty junior, but I feel like I’m spending most of my time chasing cash. More senior people laugh and say it only gets worse.

    And this ties into what I said above about funding safer experiments. With so few resources for basic science and no room for “failure”, the incentives are set up to fund things that will “work”.

  24. NNMon 09 Jan 2014 at 1:30 pm

    There’s too much focus on string theory. It’s like a huge snowball that’s captured so many people, it won’t stop until it finds some more justifications to exist. But it’s purely a mathematical artifact with infinite possibilities.
    Physicists working on alternatives should get more respect and support.
    There are no hidden dimensions. There is no dark matter. Just particles surfing and wiggling on waves and gravitational waves covering our universe.

  25. BillyJoe7on 09 Jan 2014 at 9:40 pm

    Oh dear.

Trackback URI | Comments RSS

Leave a Reply

You must be logged in to post a comment.