Feb 09 2016

The Journal of Failure to Replicate

(Note: We have been having some website issues over the weekend, which is why there has not been a post in a couple of days. All seems to be working now, but we are monitoring closely.)

Science may have a replication problem.

One of the goals of scientific skepticism is to examine the process of science itself, often through the lens of pseudoscience. I find this remarkably helpful, and something that many mainstream scientists often do not understand.

By closely examining pseudoscience as a phenomenon, we can see clear examples of how science goes wrong, how the process of science is subverted, and all the different ways scientists can make mistakes or bias their results. We can then apply this knowledge to legitimate science, flushing out more subtle manifestations of the same problems.

Said another way – if we explore all the reasons that a scientist can come to the conclusion that homeopathy works (when it clearly doesn’t) we will learn much about all the possible ways to fail when people do science (or think they are doing science).

For example, examining pseudoscience has really brought home for me the critical importance of independent replication. We often hear impressive-sounding results from single studies that appear to support one pseudoscience or another. It may not be possible from the published report where the researchers went wrong. The only way to really know is to independently replicate the results. If the researchers were genuine visionaries ready to change science, their results should replicate reliably.

Perhaps the best example of this is the psi research of Daryl Bem. He published a series of studies which he claims provide evidence for precognition, or future events affecting current cognitive processes. This is one of those claims in which it is fair to say, if we know anything in science, we know that this is impossible. This is reversing the arrow of causation. To say that such results are a paradox is an understatement.

Of course, I would be willing to accept such results if they were iron-clad. The results would have to be so robust as to make their falsity more of a paradox than their accuracy. What we got, however, were razor-thin effect sizes with a terrible signal-to-noise ratio, from a researcher who has endorsed questionable research practices. Just a tiny bit of “researcher degrees of freedom” is all that is necessary to explain the results.

The real test of these results, however, came in the replication. Several researchers tried to replicate one or more of Bem’s protocols, with mostly negative results. Not surprising. Far more important than Bem’s unlikely claims and unimpressive research was the reaction of journal editors to these replications.

Richard Wiseman and his colleagues submitting one such replication to the psychology journal that published Bem’s original studies, the Journal of Personality and Social Psychology. Their response was that they do not publish exact replications.

Richard’s response was to create a website where researchers can publish their replications of Bem’s studies.

The response by the journal is the real story here. Journal editors put a low priority on publishing replications of previous studies. They are not exciting. They don’t grab headlines or improve impact factor. That, in turn, decreases the incentive for researchers to carry out replications.

This is a systemic problem. Doing good replications is the only real way to know if a finding is reliable. In addition, with online publishing, journals no longer have the excuse of limited space in a print journal.

To me this is a problem of stoichiometry in science, to use a nerdy metaphor. In order for scientific progress to be optimal we need to have the perfect mix of researchers doing new and speculative research vs doing confirmatory research or applied research. This is like having the right mix of gas and oxygen to produce the hottest flame.

Right now I think the incentives are biased toward new and speculative research and away from confirmatory research. This may mean that we are wasting our time on lots of new ideas that will ultimately lead nowhere, and those ideas hang around longer than they should because we are not confirming them with replications.

This is not just my opinion but an increasingly recognized problem within science. One solution is to dedicate space in existing journals, or even make entirely new journals, for publishing replications. This critical component of science needs to be given a higher priority.

One journal editor is doing just that.

The contradictory results—along with successful confirmations—will be published by F1000Research, an open-access, online-only publisher. Its new“Preclinical Reproducibility and Robustness channel,” launched today, will allow both companies and academic scientists to share their replications so that others will be less likely to waste time following up on flawed findings, says Sasha Kamb, senior vice president for research at Amgen in Thousand Oaks, California.

The journal is the project of biotech company Amgen Inc. and biochemist Bruce Alberts. The journal is a response to evidence that many scientific findings that are still relied upon cannot be replicated.

One recent commentary published in Nature noted that of preclinical cancer research studies they attempted to replicate, only 11% replicated the results.

Another study of psychology studies found that only 39 out of 100 studies were successfully replicated.

Conclusion

I don’t want to overstate the problem. There is a lot of replication going on in science, this is still standard procedure. Typically when I research any medical issue there are multiple studies and we can look to see what the consensus of results show. Eventually replications are done.

But I don’t think we are at an optimal mix, because of perverse publishing incentives. Doing exact replications of studies should be looked upon not as boring but the gold standard of science. I hope we have more journals dedicated to publishing these studies, and a higher priority placed on exact replications by all the major science journals.

49 responses so far

49 thoughts on “The Journal of Failure to Replicate”

  1. Andreas says:

    I think it is important that (non-)replications be published in the same venues as the original work, or at least in competing mainstream journals. Individual journals may have a (misguided) self-interest in not publishing replications of previously published work, but surely competing established publications should seek out such work because it is potentially also highly newsworthy. If such works are relegated so specialized journals, however, they run the risk of being considered fringe and ignored.

    Quite a few fields already have “Journals of Negative Results”, such as the Journal of Negative Results in BioMedicine. In my own field, we had the Journal of Negative Results in Speech and Audio Sciences but it didn’t really catch on because it was outside of the mainstream.

  2. Ian Wardell says:

    Steven Novella referring to precognition said:
    “This is one of those claims in which it is fair to say, if we know anything in science, we know that this is impossible”.

    {sighs}

    No you don’t know that it’s impossible. If you think otherwise you should be able to provide some evidence and/or reasoning which shows this.

  3. Sylak says:

    «(Note: We have been having some website issues over the weekend, which is why there has not been a post in a couple of days. All seems to be working now, but we are monitoring closely.)»

    This also include the SGu site I supposed. since last Thursday I wasn’t able to acces the member section, ( I had to listen to the last podcast with add! lol :-). Which is cool because It reminds me I have a great course plus account ). I was worry that it was my account that was bugged. ( I send a email the webmaster but I supposed he/they were pretty occupy) . So now it’s working, access to premium content and all. Thanks for letting us know!

  4. Robney says:

    @ Ian Wardell

    For pre-cognition to be true our entire model of reality would have to be completely wrong. This is so unlikely to be true that it is equivalent to the probability that objects might start falling upwards tomorrow.

    In the same way we can’t completely rule out the possibility that objects might fall upwards tomorrow, we can’t completely rule out the possibility that pre-cognition might be true. But in either case, the probability of either outcome is so low that we can safely consider them impossibilities in a practical sense. Being an epistemological pedant doesn’t change this.

    But even permitting the possibility that pre-cognition could be true (or even probable), Bem’s research is insufficient to establish it as true. The evidential data for precognition is so weak that is indistinguishable from the data we would expect to see if pre-cognition was not true at all.

    The burden of proof is on you, my friend.

  5. Willy says:

    Ian Wardell claims to be intelligent–and a “philosopher”. I think his post just showed he is clueless and blinded by his own (incorrectly perceived) “brilliance”. What a joke!

  6. Willy says:

    Ian Wardell: Should Dr. Novella be required to show “evidence” that unicorns don’t exist? Zeus? Tinker Bell? The Incredible Hulk? The Easter Bunnt?

    As Robney said, the burden of proof is on you. And…you have bupkis.

  7. BillyJoe7 says:

    “No you don’t know that it’s impossible”

    Yes he does, and so do I and everyone else here, including you.
    No one can possibly believe that something you will do in the future will help you with something you did in the past. The whole idea is preposterous.

    “If you think otherwise you should be able to provide some evidence and/or reasoning which shows this”

    You must have missed the link he provided.

  8. BillyJoe7 says:

    I wonder how long it will take to reference the quack version of QM in support of Bem quackery?

  9. hardnose says:

    Bem’s research was replicated many times. Only Wiseman’s failed attempt was acknowledged by “skeptics.”

    “Perhaps the best example of this is the psi research of Daryl Bem. He published a series of studies which he claims provide evidence for precognition, or future events affecting current cognitive processes. This is one of those claims in which it is fair to say, if we know anything in science, we know that this is impossible. This is reversing the arrow of causation. To say that such results are a paradox is an understatement.”

    Utterly untrue. According to physics, time can be bi-directional. This is well-known and accepted.

    Not everything is obvious to the physical senses. We know that. We have known that for a long time.

  10. hardnose says:

    “For pre-cognition to be true our entire model of reality would have to be completely wrong.”

    In other words, you would have to revise some of your prefabricated dogmatic beliefs.

  11. Ivan Grozny says:

    “For pre-cognition to be true our entire model of reality would have to be completely wrong.”

    I am just curious what do you mean by “our model of reality”? What is that? I agree that the burden of proof is on those who want to prove the phenomenon of precognition is real, but that’s a different and contradictory argument to the statement that the phenomenon is categorically “impossible”. Which scientific principles would be negated if precognition were to be real?

  12. steve12 says:

    “Which scientific principles would be negated if precognition were to be real?”

    For Bem, entropy

  13. hammyrex says:

    Impossible might not be the best word – it’s best to just say there’s no compelling necessity to introduce psi concepts into psychology or physics at this time – they currently do not answer any questions and require a substantial amount of assumptions. Introduction of concepts to a field should have some explanatory power – not just brought in because they’d be really cool.

    Again, one has to be amused at the discrepancy. “HIV does not cause AIDS” – …because we don’t have the one perfect RCT done over 30+ years, so we have to reserve judgement instead of going with the obvious conclusion of hundreds and hundreds of epidemiological studies… but a handful of sloppy experiments with tiny effect sizes that only appear with statistical sorcery prove that humans absolutely have telepathic powers. I think that does a good job of demonstrating where agendas are.

    That being said, it’s hard to pick on Bem too much though – it’s not that his experiments were outrageously bad, it’s that they were the standard for a lot of social psychology… which are outrageously bad.

  14. steve12 says:

    Entropy would be the violated law for any precog (didn’t pay attention that you specifically said precognition).

  15. steve12 says:

    Hammyrex:

    I struggled with the same issue re: the word “impossible”.

    Ultimately it’s semantics, but I think violating entropy is as close as we can get to correctly using that word. There’s nary a more basic principle of our understanding of the universe, and there has never in the history of science been 1 – *not 1!* – instance of it being violated. And that’s a lot of measurements….

    So I’ll go with impossible, as distasteful as it is to put even semantic limits on nature.

  16. hammyrex says:

    I agree – I personally don’t have much a of a problem using impossible as a placeholder for a concept that regards the scientific concepts we have now. I more just wanted to clarify because there’s always going to be that one clown (or clown car) who says “BUT THEY SAID COMPUTARS WERE IMPOSSIBLE TOO THOUGH!” – it’s nice to lay out the semantics every now and then to make sure people “get it”

  17. hardnose says:

    “it’s hard to pick on Bem too much though – it’s not that his experiments were outrageously bad, it’s that they were the standard for a lot of social psychology… which are outrageously bad.”

    He used some accepted and established cognitive psychology experimental paradigms.

    Where did you get the idea that all cognitive psychology experiments are outrageously bad?

  18. hardnose says:

    Bem’s experiments were replicated. Every time a parapsychology experiment succeeds, Wiseman comes along and fails to replicate it. And the “skeptics” choose what they prefer to believe.

  19. steve12 says:

    Hammyrex:

    “That being said, it’s hard to pick on Bem too much though – it’s not that his experiments were outrageously bad, it’s that they were the standard for a lot of social psychology… which are outrageously bad.”

    Bem is a social psychologist employing cognitive paradigms here. The experiments weren’t the problem – the P-Hacking is. This is my field (cognitive) and I have to admit that we have big problems with p-hacking (not as bad as social, but problems nonetheless).

    That’s why all the replications done by anyone with any scientific experience failed. It’s more of a cautionary tale about the methods that we’re employing than anything else.

    But I will say this: anyone who thinks that a non-replicable cognitive psychology experiment should trump the laws of thermodynamics is an idiot.

  20. hardnose says:

    “all the replications done by anyone with any scientific experience failed”

    That is not true.

  21. wellerpond says:

    I’m sure I’m committing some kind of fallacy here, but if there were replicable results that stood out above the noise of standard inquiry, wouldn’t physicists, psychologists, and biologist be clammering to try and replicate? Where are the people who want a Nobel Prize?

    Surely the people who have replicated it would like people to know about their work. Or, is this where conspiracy comes in?

  22. steve12 says:

    Wellerpond:

    Yeah – this is indeed where the conspiracy comes in. As I said above, any labs with any kind of experience doing this type of work failed to replicate at the rate you’d expect if nothing was going on.

    Darryl Bem has been feverishly trying to show that it replicated, but he’s an interested party. He has even gone so far as to block failed replications from being published:

    https://www.theguardian.com/science/2012/mar/15/precognition-studies-curse-failed-replications

    I think that Bem tried to put together a meta-analysis of replications, but only some paper-mill would publish it.

    Even if we say the results are “mixed”, you have to ask youself: should we jetison entropy based on mixed cognitive psychology results from a duplicitous researcher?

    No.

  23. steve12 says:

    “Darryl Bem has been feverishly trying to show that it replicated, but he’s an interested party.”

    And let me make clear: there’s nothing wrong with doing replications of your own experiments (though independent is obviously better). There IS a problem when you’re trying to actively block, as a referee, the publication of failed replications of your work.

    Between this type of behavior and the lax standards of the “journal” Bem’s meta-analysis is published in (see link below), I think it’s fair to ask some questions about Bem’s motivations here.

    http://scholarlykitchen.sspnet.org/2013/01/15/pubmed-and-f1000-research-unclear-standards-applied-unevenly/

  24. steve12 says:

    Here’s links to 2 of the researchers that are in Bem’s meta analysis (accounting for a total of 8 experiments). Mind you, these are the first 2 people I checked):

    Does not have the background required to be able to properly instrument and conduct these types of experiments:
    http://www.univie.ac.at/wissenschaftstheorie/batthyany/cv.pdf

    For all intents and purposes, a parapsychologist:
    http://www.psy.unipd.it/~tressold/cmssimple/

    I mean look at the latter’s “cognitive” publications! That not cognitve psychology, it’s parapsychology.

    Running these types of experiments isn’t running the LHC, but it requires experience. When people who have that experience run the experiments, they don’t replicate.

    I wonder why….

  25. hardnose says:

    90 experiments. You don’t think that’s enough? You think they were all done by amateurs?

    http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2423692

  26. steve12 says:

    The context here is so important considering what we’re throwing overboard – entropy.

    To accept this, we’re throwing away a law of nature that has NEVER been wrong. Ever. If entropy was wrong none of our technology would work. Every day entropy is “replicated” trillions upon trillions of times. Every day.

    Compare that with, at best mixed results in a psychology experiment, where it ‘works’ for parapsychologists and people with little experimental training and DOES NOT WORK in the hands of experienced experimentalists. Hmmm….

    Considering the whole story, I think it’s pretty clear:

    Accepting the Bem experiment = scientific illiteracy.

  27. hardnose says:

    That makes no sense. Time being bi-directional is accepted in physics.

  28. hardnose says:

    “at best mixed results in a psychology experiment”

    ONE psychology experiment? I linked to a review of 90 experiments.

  29. Robney says:

    @ Hardnose,

    “In other words, you would have to revise some of your prefabricated dogmatic beliefs.”

    Yes, if precognition was shown to be true I would have to revise some of my current beliefs. But until you provide any credible evidence for pre-cognition you have no idea whether or not I would dogmatically cling onto my existing beliefs.

    If you remember I acknowledged that precognition was a possibility but based on what we do know, a very improbable one. I’m not sure how you could describe that as dogmatic.

  30. Robney says:

    @ Ivan

    “I am just curious what do you mean by “our model of reality”? What is that?”

    I meant it as a short hand way of saying our current understanding of physics, the universe and causality.

    Many of our current descriptions of reality would be broken if it was possible for effects to precede their cause (or future events affect past events). Its possible but considering the amount of evidence (the sum of all human experience) that is concordant with our current descriptions, it seems unlikely.

    The threshold of evidence to overturn everything we currently know is therefore very high.

  31. hardnose says:

    “Many of our current descriptions of reality would be broken if it was possible for effects to precede their cause (or future events affect past events). Its possible but considering the amount of evidence (the sum of all human experience) that is concordant with our current descriptions, it seems unlikely.”

    The way things appear to our senses is not always how they really are. We have learned that lesson repeatedly thanks to modern science. It seems like the sun goes around the earth. It seems like there are no creatures that are too small to see. Etc.

    As for the sum of all human experience regarding precognition and other “paranormal” phenomena — all of human experience is full of these things. And they are everyday occurrences for many of most people.

    They are only considered impossible or abnormal by materialists. But since you all communicate primarily with each other, you would not know this.

    And in addition to being ordinary experiences, there are many thousands of experiments that demonstrate them.

    Including Bem’s experiments. He wasn’t even a believer, but he became interested. There is no possibility that all his results were errors.

  32. Robney says:

    “The way things appear to our senses is not always how they really are. We have learned that lesson repeatedly thanks to modern science. It seems like the sun goes around the earth. It seems like there are no creatures that are too small to see. Etc”….

    “As for the sum of all human experience regarding precognition and other “paranormal” phenomena — all of human experience is full of these things”

    Your first paragraph undermines your second.

    I should have been more specific. All objectively verifiable human experience confirms our current descriptions of reality.

  33. Ian Wardell says:

    Quite possibly both the future and past exists as well as the present i.e the block Universe hypothesis. Now we know we can perceive the past in the form of our memories. Indeed memories can arguably be considered a form of retrocognition. Given this, then by what reasoning can we claim that “remembering” the future is impossible? Why is remembering the past possible but “remembering” the future is impossible?

  34. Willy says:

    Ian Wardell: Shirley you jest. “Quite possibly”? Really? “Quite”????? “Now we know we can perceive the past…”. Just when is “now”? Did we just discover memories? “Given this…” Isn’t that just a bit of a stretch to be a “given”??? Can I have some of what you are smoking?

    How often do you “remember” things yet to come? Can you help me pick the next hot stock? Can you even help yourself pick the next hot stock? Can you avoid upcoming car wrecks you’d have otherwise been involved in? Probably not, because I’m sure you are far too bright to ever be involved in something as ordinary as a car wreck.

    You have come to enjoy the feeling of blowing smoke up your own derriere just a little bit too much. Sounding profound isn’t being profound. Go back to picking your nose.

  35. Willy says:

    Ian Wardell: Shirley you jest. “Quite possibly”? Really? “Quite”????? “Now we know we can perceive the past…”. Just when is “now”? Did we just discover memories? “Given this…” Isn’t that just a bit of a stretch to be a “given”??? Can I have some of what you are smoking?

    How often do you “remember” things yet to come? Can you help me pick the next hot stock? Can you even help yourself pick the next hot stock? Can you avoid upcoming car wrecks you’d have otherwise been involved in? Probably not, because I’m sure you are far too bright to ever be involved in something as ordinary as a car wreck.

    You have come to enjoy the feeling of blowing smoke up your own derriere just a little bit too much. Sounding profound isn’t being profound. Go back to picking your nose. You are quite pedestrian–and pompous.

  36. Robney says:

    If that was the case why wouldn’t we remember the future with equal fidelity?

    We can remember past events because past events affect current states in our brain. There is nothing to suggest future states affect current states in our brains – which is why we have no demonstrated pre-cognitive knowledge of future events. The arrow of causality seems to point only in one direction. Almost every observation in human experience confirms this.

    Before worrying about the mechanism for pre-cognition you should first establish that it is a real phenomenon.

  37. steve12 says:

    >Now we know we can perceive the past in the form of our memories. Indeed memories can arguably >be considered a form of retrocognition. Given this, then by what reasoning can we claim that >“remembering” the future is impossible?

    oh, Ian! Don’t you ever go changin’ on us!

  38. Charon says:

    “Time being bi-directional is accepted in physics.”

    And yet there’s an arrow of time. Hmm. This is a deep problem – or, I should say, was a deep problem, until Boltzmann solved it in the 19th century. This is that “entropy” thing all the cool kids are talking about. Evolution to more probable states. While entropy can decrease occasionally in very, very tiny systems (random fluctuations), that never happens on macroscopic scales. Arrow of time!

    Ian: also needs to learn about the arrow of time. I’ll pause while the whole class goes and reads Sean Carroll’s From Eternity to Here: The Quest for the Ultimate Theory of Time. If you’re going to opine on physics, you should actually know physics…

  39. Charon says:

    To clarify, time being “bi-directional” is not at all an accepted thing in physics. That our basic laws of physics have time symmetry is (well, kinda – it’s actually charge-parity-time [CPT] symmetry).

    None of this gives any support whatsoever to precognition. And physicists would be all over that. When we briefly thought there might be a faster-than-light neutrino signal at Gran Sasso, the theoretical physicists were adorably happy at the prospect – all sorts of new things to work out! Forget Nobel prizes. Precog would be the sort of thing that gets you remembered alongside Newton, Einstein, Heisenberg…

  40. BillyJoe7 says:

    hardnose: “Time being bi-directional is accepted in physics”

    He can’t even get the terminology right. The equations of physics are time symmetric. Moreover, this does not mean that there is not an arrow of time. In fact, what IS pretty well accepted in physics is that there IS an arrow of time. As someone else has already pointed out, it would violate the second law of thermodynamics.

    Ian: “Quite possibly both the future and past exists as well as the present i.e the block Universe hypothesis”

    Totally irrelevant. The “block universe” is a sort of meta-view of reality based on Relativity Theory. Time does not exist for objects travelling at the speed of light. Past, present, and future exist as a “block”. But the reality in which we live and die has a arrow of time from past to present.

  41. BillyJoe7 says:

    Steven Novella talking about Bem in 2011:

    http://theness.com/neurologicablog/index.php/bems-psi-research/

    “This is, by everything we currently know about physics and the way the universe works, impossible. It is, at least, as close to impossible as we can get in science. It is so massively implausible that only the most solid and reproducible evidence should motivate a rational scientist to even entertain the idea that it could be real”

    Steven Novella this time round:

    “This is one of those claims in which it is fair to say, if we know anything in science, we know that this is impossible. This is reversing the arrow of causation. To say that such results are a paradox is an understatement”

    I, for one, understand and appreciate the hardening of attitude towards nonsense peddlers who misuse physics towards their desired and preconceived conclusions.

  42. Steve Cross says:

    Time to beat my own personal favorite dead horse, the Dunning-Kruger effect. Otherwise known as “Delusions of Competency”.

    Neither Hardnose or Ian know enough to know how much they don’t know. They both love to interject big words that they’ve encountered in their “research”. With the inevitable result being a demonstration that neither has any idea what they’re talking about.

    Pro Tip: If all you ever do is search the literature to find “facts” that you can twist to support your own preconceived notions, then You’re doing it Wrong!!

  43. steve12 says:

    All good points, BUT…

    You guys are really missing out on the artistry it takes to say that memory = precognition.

    Some ideas are so amazing that you need to go grab a hot cup of coffee with your favorite confection, take a break, and just breathe it in. Recharge your soul a bit, ya know?

    So yeah – I just want to applaud Ian on his achievement here

  44. jt512 says:

    @hardnose

    [Bem] used some accepted and established cognitive psychology experimental paradigms.

    Bem made extensive use of researcher degrees of freedom, HARKing, and p-hacking, all of which have a rich history in experimental psychology, so you are correct.

  45. BillyJoe7 says:

    These authors analysed Bem and found him wanting:

    https://dl.dropboxusercontent.com/u/1018886/Bem6.pdf

    “The most important flaws in the Bem experiments, discussed below in detail, are the following: (1) confusion between exploratory and confirmatory studies; (2) insufficient at- tention to the fact that the probability of the data given the hypothesis does not equal the probability of the hypothesis given the data (i.e., the fallacy of the transposed conditional); (3) application of a test that overstates the evidence against the null hypothesis, an unfortu- nate tendency that is exacerbated as the number of participants grows large. Indeed, when we apply a Bayesian t-test (G ̈onen, Johnson, Lu, & Westfall, 2005; Rouder, Speckman, Sun, Morey, & Iverson, 2009) to quantify the evidence that Bem presents in favor of psi, the evidence is sometimes slightly in favor of the null hypothesis, and sometimes slightly in favor of the alternative hypothesis. In almost all cases, the evidence falls in the category “anecdotal”, also known as “worth no more than a bare mention””

    But as others here have said, Bem is not alone:

    “We realize that the above flaws are not unique to the experiments reported by Bem. Indeed, many studies in experimental psychology suffer from the same mistakes. However, this state of affairs does not exonerate the Bem experiments. Instead, these experiments highlight the relative ease with which an inventive researcher can produce significant results even when the null hypothesis is true”

  46. Ian Wardell says:

    Hmm . .the block Universe is irrelevant is it BillyJoe? Well . .the block Universe holds that the future exists (and indeed the past). Thus making it at least *metaphysically* possible to perceive the future. So I suggest — as usual — you have no idea what you’re talking about.

    So, this entropy is interesting. How on earth people imagine it makes it impossible to perceive the future is beyond me. A deck of playing cards is in a certain order. I shuffle them and they become disordered. Therefore it is not possible to “remember” the future . . hmmm.

  47. steve12 says:

    Ian:

    “Thus making it at least *metaphysically* possible”

    Metaphysically possible. That is wonderful. I can’t think of a more meaningless phrase.

    “So, this entropy is interesting. How on earth people imagine it makes it impossible to perceive the future is beyond me.”

    Does your complete lack of understanding of the most basic physics principles give you any pause at all?

    What am I saying? You know physics better than any physicist.

  48. CKava says:

    @ hardnose

    Bem’s metareview has significant issues, which is likely part of the reason it features in the rather odd ‘post-publication’ peer review journal F1000. If you read the post-publication peer reviews at the bottom of the manuscript you will be able to see the serious issues with their analysis and their conclusions. For a readable summary of some of them, see: http://centerforopenscience.github.io/osc/2014/06/25/a-skeptics-review/

    Or you could just read the actual commentaries at the bottom of the F1000 article. Daniel Lakens in depth review in particular gives a clear discussion of the statistical problems:
    “[I] regret to have to conclude that the results are not valid, the analyses are flawed in many respects, and the quality is too low for this meta-analysis to be part of the scientific literature. I believe this manuscript should be rejected.

    Let me be clear that I would have no problem with a well-performed meta-analysis of this literature, regardless of whether it would show a meta-analytic effect size estimate that differed from zero or not. In science, we distinguish between statistical inferences and theoretical inferences (e.g., Meehl, 1990). Even if a meta-analysis would lead to the statistical inference that there is a signal in the noise, there is as of yet no compelling reason to draw the theoretical inference that psi exists, due to the lack of a theoretical framework as acknowledged by the authors. So, a meta-analytical effect size estimate that differs from zero would have to lead to a careful examination of possible confounds in the paradigms that have been used in this literature, and the studies that have been included in this meta-analysis. Such a careful examination has not been done. Therefore, the validity of the measures used to examine psi effects has not been established. Only 18 statistically significant effects have been observed in the last 14 years, as the literature search by the authors reveals, coming from only 7 labs. Only if confounds are sufficiently excluded (preferably in direct replications in different labs) can we start thinking about alternative explanations for the observed data, such as psi. In other words, even if there was robust evidence for a meta-analytic effect size estimate that differed from zero in our statistical inferences, we are far removed from being able to draw any theoretical inferences.”

Leave a Reply