Aug 29 2012

The Power of Replication – Bems Psi Research

Note – This article was also cross-posted at Science-Based Medicine. 

I love reading quotes by the likes of Karl Popper in the scientific literature. A recent replication of Bem’s infamous psi research, Feeling the Future, gives us this quote:

Popper (1959/2002) defined a scientifically true effect as that “which can be regularly reproduced by anyone who carries out the appropriate experiment in the way prescribed.”

The paper is the latest replication of Daryl Bem’s 2011 series of 9 experiments in which he claimed consistent evidence for a precognitive effect, or the ability of future events to influence the present.  The studies were published in The Journal of Personality and Social Psychology, a prestigious psychology journal. All of the studies followed a similar format, reversing the usually direction of standard psychology experiments to determine if future events can affect past performance.

In the 9th study, for example, subjects were given a list of words in sequence on a computer screen. They were then asked to recall as many of the words as possible. Following that they were given two practice sessions with half of the word chosen by the computer at random. The results were then analyzed to see if practicing the words improved the subject’s recall for those words in the past. Bem found that they did, with the largest effect size of any of the 9 studies.

Needles to say, these results were met with widespread skepticism. There are a number of ways to assess an experiment to determine if the results are reliable. You can examine the methods and the data themselves to see if there are any mistakes. You can also replicate the experiment to see if you get the same results.

Replicating Bem

Bem’s studies have not fared well in replication. Earlier this year Ritchie, Wiseman, and French published three independent replications of Bem’s 9th study, all negative. Of further interest is that the journal that originally published Bem’s article had declined to publish Ritchie et al’s paper claiming that they don’t publish replications. This decision (and editorial policy) was widely criticized, as it reflects an undervaluing of replications.

It’s good to see that the journal has relented and agreed to publish a replication. Galak, LeBoeuf, Nelson, and Simmons should be commended, not only on their rigorous replication but their excellent article, which hits all the key points of this entire episode.

The researchers replicated experiments 8 and 9 of Bem (they chose these protocols because they were the most objective). They conducted 7 precise replications involving a total of 3,289 subjects (Bem’s studies involved 950 subjects). Six of the seven studies, when analyzed independently, were negative, while the last was slightly statistically significant. However, when the data are taken together, they are dead negative.  The authors concluded that their experiments found no evidence for psi.

Lessons from Bem

A Bayesian Analysis

At Science-Based Medicine we have advocated taking a more Bayesian approach to scientific data. This involves considering a claim for plausibility and prior probability. Bem and others are fairly dismissive of plausibility arguments and feel that scientists should be open to whatever the evidence states. If we dismiss results because we have already decided the phenomenon is not real, then how will we ever discover new phenomena?

On the other hand, it seems like folly to ignore the results of all prior research and act as if we have no prior knowledge. There is a workable compromise – be open to new phenomena, but put any research results into the context of existing knowledge. What this means is that we make the bar for rigorous evidence proportional to the implausibility of the phenomenon being studied. (Extraordinary claims require extraordinary evidence.)

One specific manifestation of this issue is the nature of the statistical analysis of research outcomes. Some researchers propose that we use a Bayesian analysis of data, which in essence puts the new research data into the context of prior research. A Bayesian approach essentially asks – how much does this new data affect the prior probability that an effect is real?

Wagenmakers et al reanalyzed Bem’s data using a Bayesian analysis and concluded that the data is not sufficient to reject the null hypothesis.  They further claim that the currently in vogue P-value analysis tends to overcall positive results. In reply Bem claims that Wagenmakers used a ridiculously low prior probability in his analysis. In reality, it doesn’t matter what you think the prior probability is, the Bayesian analysis showed that Bem’s data has very little effect on the probability that retrocausal cognition is real.

Galak et al in the new study also perform a Bayesian analysis of their own data and conclude that this analysis strongly favors the null hypothesis. They also specifically cite Wagenmakers’ support for the Bayesian approach.

Criticisms of Bem

It’s possible for a study to “look good on paper” – meaning that the details that get reported in the published paper may make the study look more rigorous than it actually was. There may also be alternate ways to analyze the data that give a different picture Ritchie, Wiseman, and French outline several criticisms of Bem’s methods. They mention the Bayesian issue, but also that an analysis of the data shows an inverse relationship between effect size and subject number. In other words, the fewer the number of subjects the greater the effect size. This could imply a process called optional stopping.

This is potentially very problematic. Related to this is the admission by Bem, according to the article, that he peeked at the data as it was coming in. The reason peeking is frowned upon is precisely because it can result in things like optional stopping, which is stopping the collection of data in an experiment because the results so far are looking positive. This is a subtle way of cherry picking positive data. It is preferred that a predetermined stopping point is chosen to prevent this sort of subtle manipulation of data.

Another issue raised was the use of multiple analysis. Researchers can collect lots of data, by looking at many variables, and then making many comparisons among those variables. Sometimes they only publish the positive correlations, and may or may not disclose that they even looked at other comparisons. Sometimes they publish all the data, but the statistical analysis treats each comparison independently. In short what this means is that if you look at 20 comparisons with a 1/20 chance of reaching statistical significance, by chance one comparison would be significant. You can then declare a real effect. But what should happen is that the statistical analysis is adjusted to account for the fact that 20 different comparisons were made, which can potentially make the positive results negative.

Finally there was a serious issue raised with how the data was handled. Subjects occasionally made spelling error when listing the words they recall. The result may have been a non-word (like ctt for cat) or another word (like car for cat). Researchers had to go through and correct these misspellings manually.

The authors point out that these corrections were done in a non-blinded fashion, creating the opportunity to fudge the data toward the positive by how correction choices are made. Bem countered that even if you removed the corrected words the results would still be positive, but that is still methodologically sloppy and is likely still relevant, for reasons I will now get into.

Researcher Degrees of Freedom

As we see, there were many problems with the methods and statistical analysis of Bem’s original paper. Bem argues that each problem was small and by itself would not have changed the results. This argument, however, misses a critical point, made very clear in another recent paper – one that was also cited and discussed in the Galak paper.

Simmons et al published a paper demonstrating how easy it is to achieve false positive results by exploiting (consciously or unconsciously) so-called “researcher degrees of freedom.” In the abstract they write:

“In this article, we accomplish two things. First, we show that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (! .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis.”

In my opinion this is a seminal paper that deserves wide distribution and discussion among skeptics, scientists, and the public. The paper discusses the fact that researchers make many decisions when designing and executing a study, and analyzing and reporting the data. Each individual decision may have only a small effect on the final outcome. Each decision may be made perfectly innocently, and can be reasonably justified.

However, the cumulative effect of these decisions (degrees of freedom) could be to systematically bias the results of a study toward the positive. The power of this effect is potentially huge, and likely results in a significant bias towards positive studies in the published literature.

But even worse, this effect can also be invisible. As the authors point out – each individual decision can seem quite reasonable by itself. The final published paper may not reflect the fact that the researchers, for example, looked at three different statistical methods of analysis before choosing the one that gave the best results.

The authors lay out some fixes for this problem, such as researchers disclosing their methods prior to collecting data (and no peeking). But another check on this kind of bias in research is replication.

The Power of Replication

The degrees of freedom issue is one big reason that replicating studies, especially precise replications, is so important. A precise replication should have no degrees of freedom, because all the choices were already made by the original study. If the effect being researched is real, then the results should still come out positive. If they were the result of exploiting the degrees of freedom, then they should vanish.

There are also the other recognized benefits of replication. The most obvious is that any unrecognized quirky aspects of study execution or researcher biases should average out over multiple replications. For this reason it is critical for replications to be truly independent.

Another often missed reason why replications are important is simply to look at a fresh set of data. It is possible for a researcher, for example, to notice a trend in data that generates a hypothesis. That trend may have been entirely due to random clustering, however. If the data in which the trend was initially observed is used in a study, then the original random clustering can be carried forward, creating the false impression that the hypothesis is confirmed.

Replication involves gathering an entirely new data set, so any prior random patterns would not carry forward. Only if there is a real effect should the new data reflect the same pattern.

Proper Skepticism

Prominently displayed at the top of the Society for Psychical Research’s website is this quote:

“I shall not commit the fashionable stupidity of regarding everything I cannot explain as a fraud.” – C.G.Jung

Clearly that quote reflects the prevailing attitude among psi researchers of external skepticism of their claims and research. Every skeptic who has voiced their opinion has likely been met with accusations of being dismissive and closed-minded.

But this is a straw man. Skeptics are open to new discoveries, even paradigm-changing revolutionary ideas. Often I am asked specifically – what would it take to make me accept psi claims. I have given a very specific answer – one that applies to any extraordinary claim within medicine. It would take research simultaneously displaying the following characteristics:

1 – Solid methodology (proper blinding, fresh data set, clearly defined end points, etc.)

2 – Statistically significant results

3 – Absolute magnitude of the effect size that is greater than noise level (a sufficient signal to noise ratio)

4 – Consistent results with independent replication.

Most importantly, it would need to display all four of these characteristics simultaneously. Psi research, like most research into alternative medicine modalities like homeopathy and acupuncture,  cannot do that, and that is why I remain skeptical. These are the same criteria that I apply to any claim in science.

In addition, I do think that prior probability should play a role – not in accepting or rejecting any claim a priori, but in setting the threshold for the amount and quality of evidence that will be convincing. This is reasonable – it would take more evidence to convince me that someone hit Bigfoot with their car than that they hit a deer with their car. There is a word for someone who accepts the former claim with a low threshold of evidence.

You can convince me that psi phenomena are real, but it would take evidence that is at least as solid as the evidence that implies that such phenomena are probably not possible.

It is also important to recognize that the evidence for psi is so weak and of a nature that it is reasonable to conclude it is not real even without considering plausibility. But it is probably not a coincidence that we consistently see either poor quality or negative research in areas that do have very low plausibility.

Conclusion

The least important implication of the  recent paper by Galak et al is that it provides further evidence against psi as a real phenomenon, and specifically against the claims of Daryl Bem. Psi is a highly implausible hypothesis that has already been sufficiently refuted, in my opinion, by prior research.

The paper is perhaps a milestone, however, in other important respects:

- It is an admission of sorts by The Journal of Personality and Social Psychology of the importance of precise replication and a reversal of their prior decision not to publish such research.

- The paper highlights the role, and possible superiority, of Bayesian analysis as a method for looking at experimental data.

- It highlights the role of researcher degrees of freedom in generating false positive data, and replication as one solution to this problem.

- This large, rigorous, and negative replication establishes that studies published in peer-reviewed journals with positive and solid-appearing results can still be entirely wrong. It therefore justifies initial skepticism toward any such data, especially when extraordinary claims are involved.

The insights provided by this excellent paper reflect many of the points we have been making at SBM, and should be applied broadly and vigorously to alternative medicine claims.

Share

92 responses so far

92 Responses to “The Power of Replication – Bems Psi Research”

  1. mindmeon 29 Aug 2012 at 8:47 am

    Yes, but what does Alex Tsakiris think?

  2. ConspicuousCarlon 29 Aug 2012 at 2:15 pm

    I am amused (and annoyed) by people who ignore 19 negative results, and when the 20th time randomly looks positive, they think they “found something”. I quote those two words because I think it is a good description of their irrational view of the information. They think and behave not as though they are testing a hypothesis, but as though they were searching for some physical token which, once found, is to be considered discovered regardless of how many rocks they had to turn over before finding it.

    If I weren’t so lazy, I would write an Asimov-style story titled The Twentieth Trial, in which some crank scientist repeatedly conducts tests of a nonsense hypothesis until one day he finally gets statistically significant evidence that the sun is going to explode and public panic ensues.

    Instead of degrees of freedom, in manufacturing it is called “stacked tolerances”. In each individual step, you think you have designed your equipment to be reasonably precise. But in the end, if those tolerances result in too many variations in the same direction, you get parts which don’t fit together.

  3. Bronze Dogon 29 Aug 2012 at 2:46 pm

    They think and behave not as though they are testing a hypothesis, but as though they were searching for some physical token which, once found, is to be considered discovered regardless of how many rocks they had to turn over before finding it.

    I tend to see a more generalized behavior among various woos, and that’s one of them. They latch onto a single study, an anecdote, a graph, or a supposed expert’s name and wave it around as if it was an everlasting totem of power to ward off skeptics.

    They don’t understand that we try to look at broader contexts and how individual bits of evidence fit into the big picture. Replication is a part of that big picture because you need good reasons to exclude a study instead of being allowed to pick your favorite outcome and exclude the rest.

  4. davidsmithon 29 Aug 2012 at 5:41 pm

    Interesting paper. As a proponent who is keen on nullifying null results, it concerns me that many participants in Galak et al.’s experiments (particularly the online ones) may have known about the ‘surprise’ memory test (of course, it’s not a surprise if you know that it’s coming). I am aware of quite a number of people who did indeed know about the memory test and who completed an online session. I guess the experiment may have been circulated around many more friends of these people who were also made aware of the memory test or who just knew about this experiment from the wide media coverage. Why is this important? As Michael Franklin pointed out in response to Ritchie et al.’s failed replication, the original study was supposed to involve incidental encoding during the visualisation phase. However, if participants know that a memory test is coming up then the task is converted from one of incidental encoding to explicit encoding. Perhaps then, conscious/intentional encoding strategies serve to extinguish or reduce this particular effect? Franklin gives verbal overshadowing as an example of this happening in other psychological studies. I only skimmed the Galak paper, but I don’t think this point is mentioned. It is probably the most reasonable explanation for the failure to replicate other than the null is true.

  5. Michael Dugganon 29 Aug 2012 at 9:59 pm

    I have to agree with David Smith. Additionally Bem warned about the likelihood of negative results emerging from online tests. How do you maintain motivation, interest, and belief in success with remote tests? When you remove the online data from the meta-analysis reported by Galak, et al, there are 6 positive significant studies from 15, or a 40% success rate. This rates comparably to other programs in Parapsychology. In fact I have never heard of a successful online study in psi research! Readers are encouraged to investigate the data from Psi research. Some of it is quite impressive IMHO. See here for example: http://www.frontiersin.org/quantitative_psychology_and_measurement/10.3389/fpsyg.2011.00117/full.

    Also, I would like to congratulate Dr Novella on a high quality blog, one of the better ones from the skeptics camp. :-)

  6. Pavelon 30 Aug 2012 at 12:48 am

    Steven Novella: “In addition, I do think that prior probability should play a role – not in accepting or rejecting any claim a priori, but in setting the threshold for the amount and quality of evidence that will be convincing. This is reasonable – it would take more evidence to convince me that someone hit Bigfoot with their car than that they hit a deer with their car. There is a word for someone who accepts the former claim with a low threshold of evidence.”

    I think there’s a great example to support this view that happened just a few months ago: the Opera experiment at CERN showing neutrinos traveling faster than the speed of light. It’s precisely because the claim was so outrageous, the amount of effort to refute the claim was as much astounding. In fact, according to the research director Prof Bertolucci, “One has to realise that the collaboration has never stopped to try to ‘kill’ the measurement (proving that it was erroneous)… Their constant search for systematic (errors) has never stopped, for more than a year.”

    Besides, it also points to the importance of replication of the experiments, which failed to reproduce the results. Sensibly, when all these proper “health checks” were applied, the flaw in the Opera experiment was found and Prof Antonio Ereditato who oversaw the results resigned. If it weren’t for the outrageous skepticism of outrageous claims, the speed of light would have been “broken” several times now. It seems a story like that pops up every few years or so.

  7. Shelleyon 30 Aug 2012 at 7:18 am

    Steven Novella: “In addition, I do think that prior probability should play a role – not in accepting or rejecting any claim a priori, but in setting the threshold for the amount and quality of evidence that will be convincing. This is reasonable – it would take more evidence to convince me that someone hit Bigfoot with their car than that they hit a deer with their car. There is a word for someone who accepts the former claim with a low threshold of evidence.”

    I would agree that a priori probability should play a role. However, when Bem’s article first came out, I found it interesting that when I tried to discuss it with fellow skeptics, the series of studies were dismissed out of hand under the “extraordinary claims . . .” rubric. I think we need to do better than that. We need to be able to identify the flaws in the method, analysis or conclusions if we are truly skeptical and not polemical. This is exactly what we see in the arguments made here. Nice summary of critiques!

  8. Johannon 30 Aug 2012 at 12:30 pm

    I agree with Shelly’s observation. As a proponent, I too have seen much use of the “extraordinary claims” statement to argue against positive findings in parapsychological research. Michael Duggan mentioned an important paper above – the Tressoldi analysis – which discusses this very idea:

    What evidence is there to support the existence of NLP [non-local perception]? If we use the results obtained with the frequentist statistical approach, i.e., P(Data/H0), apart from the results obtained using participants in normal states of consciousness and the free-response protocol, all of the statistics in the remaining meta-analyses lead to the rejection of the null hypothesis, even if the measures of effect size are clearly greater using the free-response protocol. In contrast, if we refer to the results obtained with the Bayesian statistical approach, i.e., P(H0/Data), only for the three meta-analyses which relate to the ganzfeld condition, the RV procedure and anticipatory responses, there is an high probability that H1, the hypothesis supporting the existence of NLP , may be true [...] Are these converging results with these three protocols “extraor-dinary” evidence? Perhaps. Surely these results are well beyond the standards for a “strong recommendation” suggested by the GRADE system [...] Do we need more stringent standards to enable us to accept phenomena that apparently seem to violate our common beliefs regarding physical laws? However, if results analyzed with both frequentist and Bayesian statistical approaches from more than 200 studies conducted by different researchers with more than 6000 participants in total and three different experimental protocols are not considered “extraordinary,” or at least “sufficient” to suggest that the human mind may have quantum-like properties, what standards can possibly apply?

    - Johann

  9. Paul C. Anagnostopouloson 30 Aug 2012 at 12:51 pm

    Davidsmith points out that some subjects may have known that a memory test was coming up after the visualization. Aren’t we kidding ourselves if we believe that subjects presented with a list of words wouldn’t be likely to assume that a test was coming up? It’s a psychology experiment, after all.

    And even if this wasn’t the case when the study was run, isn’t it going to be more and more likely as time goes on? In fact, subjects will know about the study period that follows the test. If there is a decline effect, I suppose we can attribute it to “too much publicity about the Bem experiments.”

    ~~ Paul

  10. Johannon 30 Aug 2012 at 12:52 pm

    Indeed, it has been my personal approach and my aspiration to suspend both belief and disbelief when evaluating the evidence for any phenomenon. I think that our personal beliefs and interpretations cloud the faculties of our reason in the judgement of evidence, and, given our current state of knowledge about the universe, consciousness, and the fundamental constituents of our world, would caution against the belief that our a priori values could be overly helpful in our quest to understand, as Einstein said, “the mind of God”. Essentially, what the ECREE (extraordinary claims…) and Bayesian methods do is they entice us into believing that our knowledge is sufficient to make declarations about what is and what is not possible, when only nature should have such a power.

    Thus, I advocate that we take note of our beliefs and our expectations and use them as tools, but that we do not get too attatched to them, otherwise how are we to follow the advice of Thomas Huxley?

    “Sit down before fact like a little child, and be prepared to give up every preconceived notion. Follow humbly wherever and to whatever abyss Nature leads, or you shall learn nothing.” – TH Huxley

    - Johann

  11. Steven Novellaon 30 Aug 2012 at 1:02 pm

    Johann wrote: “Essentially, what the ECREE (extraordinary claims…) and Bayesian methods do is they entice us into believing that our knowledge is sufficient to make declarations about what is and what is not possible, when only nature should have such a power.”

    I disagree. They simply state that we should not ignore prior scientific knowledge. Science is a cumulative process that builds on itself. To ignore prior probability is to deny all scientific knowledge.

    The best approach is to be skeptical but open. If you want to follow Huxley’s advice you have to sit down before all facts – not just the ones you want. Denying prior probability is denying the facts that tell us a particular claim is extraordinary.

  12. sonicon 30 Aug 2012 at 2:07 pm

    This is an excellent post about a real problem.
    The paper linked to is fantastic and points to some serious problems. The suggested remedies aren’t a bad start. I hope some of this has an impact on the design and statistical analysis of experiments in the future.

    The problem has been known about and with us for some years now– I wonder if the tide would turn if more people would treat the claims backed by statistical analysis with a bit more skepticism. And the problem isn’t just with psi research. I know from experience.

    While the Bayesian approach has its value- I fear we would still be looking for the errors of measurement in the original Michelson- Morley experiments. (Never mind the ‘delayed choice quantum eraser’…) :-)

    The frequentist approach does give more false positives (as opposed to more false negatives). If replication is recognized as being important- then those aren’t really false positives, but rather “interesting results that need further verification.”

    I prefer that approach– but that is an opinion.

    Sorry– I will get off the soap box now.

  13. jt512on 30 Aug 2012 at 2:38 pm

    Shelley wrote:

    I would agree that a priori probability should play a role. However, when Bem’s article first came out, I found it interesting that when I tried to discuss it with fellow skeptics, the series of studies were dismissed out of hand under the “extraordinary claims . . .” rubric. I think we need to do better than that. We need to be able to identify the flaws in the method, analysis or conclusions if we are truly skeptical and not polemical.

    Shelley, you have contradicted yourself. If the prior probability for a hypothesis is infinitesimal, then the rational position is to believe the hypothesis to be false, until such time that there is enough evidence for the hypothesis to start taking it seriously. Therefore, we, as skeptics, do not need to identify the flaws in the research supporting claims like psi.

    Indeed the benefit of the Galak (2012) paper is not that it disproves psi. We already knew (to an insanely high probability) that psi does not exist. The benefit of the paper is that it shows experimental psychology journals editors that they need to be more critical of the research methods employed by experimental psychologists, and that they need to actually publish replication attempts.

    Jay

  14. davidsmithon 30 Aug 2012 at 3:41 pm

    Paul said,

    Aren’t we kidding ourselves if we believe that subjects presented with a list of words wouldn’t be likely to assume that a test was coming up?

    Not necessarily. It is possible that many participants in Bem’s original study did not anticipate a memory test. They were told to visualise each referent of the word after all, not to memorise the word. However, if a significant number of people had explicit knowledge about the memory test in the Galak studies (and I have first hand knowledge of people who did know – you’re one of them!) then a significant number of people would have deliberately tried to memorise the words during the visualisation phase. Whether this is important is up for debate and should be addressed in future studies.

    And even if this wasn’t the case when the study was run, isn’t it going to be more and more likely as time goes on? In fact, subjects will know about the study period that follows the test. If there is a decline effect, I suppose we can attribute it to “too much publicity about the Bem experiments.”

    Yes, that’s my point. It’s possible that intentional encoding of the initial list of words interferes with the effect, and that this is more likely to happen as time goes on as more participants becomre privy to the ‘surprise’ memory test (for example, from reading about the study in science blogs such as this or chatting about it on internet forums). I guess you could take steps to measure the degree to which participants tried to memorise, rather than visualise, the initial list of words – perhaps with a questionnaire.

  15. Johannon 30 Aug 2012 at 3:53 pm

    Steven Novella wrote: “I disagree. They simply state that we should not ignore prior scientific knowledge. Science is a cumulative process that builds on itself. To ignore prior probability is to deny all scientific knowledge.”

    The best approach is to be skeptical but open. If you want to follow Huxley’s advice you have to sit down before all facts – not just the ones you want. Denying prior probability is denying the facts that tell us a particular claim is extraordinary.”

    But all the facts, Steven, are impossible to know – which is my point exactly. As I said, I advocate that we take note of our beliefs and our expectations and use them as tools, but that we do not get too attatched to them, because they are ultimately a reflection only of our state of knowledge (or possible knowledge) – which, in the grand scheme of things, is tiny.

    In contrast to your opinion, I think that when you utilize ECREE, you are not taking into account prior scientific evidence – because the prior scientific evidence for ESP is at least supportive of its existence, according even to major critics – but prior belief, or metaphysical position. Your support of the materialist hypothesis leads you to categorize ESP evidence, naturally, as missing the bar. But from your previous discussions with Bernardo, we know that there are at least plausible alternative metaphysical characterizations of the nature of Mind, such as dualism, idealism, panpsychcism, and monism – all of which are far more compatible with psi than materialism, and which do a fairly decent job of resolving the Hard Problem.

    But even if you believe that all of these are nonsense, even materialism – yes, even materialism – holds some possible explanations for psi via Persinger’s theory of geomagnetic transmission, Stapp’s reformulation of quantum mechanics (allowing for retro-causality), time symmetric quantum mechanics, or others. So we are come to a situation where there are at least plausible arguments for a reality that could incorporate psi, which means that even if we feel greatly attached to prior scientific knowledge as something to allow/deny new findings, we must acknowledge that the probability of psi has risen to a level where it would seem pertinent to reconsider the following question by Tressoldi:

    “… if results analyzed with both frequentist and Bayesian statistical approaches from more than 200 studies conducted by different researchers with more than 6000 participants in total and three different experimental protocols are not considered “extraordinary,” or at least “sufficient” to suggest that the human mind may have quantum-like properties, what standards can possibly apply?”

    In light of all that has been accomplished, let us examine Huxely’s quotation one more time:

    “… be prepared to give up every preconceived notion. Follow humbly wherever and to whatever abyss Nature leads, or you shall learn nothing.”

    A preconceived notion we should consider giving up is that psi is an extraordinary claim.

    - Johann

  16. Shelleyon 30 Aug 2012 at 4:19 pm

    Jay,
    My point is this: we can agree that the prior probability of there being a true effect in Bem’s studies almost certainly approaches zero. However, if the study *appears* to be methodologically sound, it would violate any consideration of scientific openness to fail to give it a reasoned examination.

    If (in what I recall of Baysean terms), we assign a prior likelihood that the hypothesis is true a value of 0, then the probability that the hypothesis is true given the data must also be zero. (Do I recall this correctly? Sorry, it has been a while.) Essentially, we would be saying that no data could ever be convincing. Not a very scientific approach, most particularly considering that proponents of psi likely believe that the probability that psi is true is something closer to one.

    What psi proponents cannot reasonably argue are the methodological and analytical flaws and artifacts that render any “findings” moot. And yes, the Galak paper suggests that reviewers need be critical of the research methods of experimental psychologists.

    Yes. Precisely. I think it useful to do more than say, “extraordinary claim! One need not look further!”

  17. Johannon 30 Aug 2012 at 5:33 pm

    Shelly, why do you consider the prior probability of psi to be near 0? On what knowledge are you basing this idea?

    If you would do me the favor, consider reading my previous post. I am interested to hear what it is in the scheme of our understaning, encompassing science, philosophy, and metaphysics that is so often used to claim that the discovery of psi is “incompatible with what we know” or that it will “overturn our knowledge completely”. What peril is it that others see?

    - Johann

  18. [...] Steven Novella over at Neurologicablog has a nice discussion of our Psi article here. [...]

  19. sonicon 31 Aug 2012 at 5:47 pm

    Both studies (Bem and the attempted replication) are flawed to the point that any conclusions drawn should be very specific.
    What Bem found was that the people he was testing were scoring as if they had a particular ability when he stopped the test. Because he stopped the test in the manner he did, his data isn’t appropriate input into any statistical analysis.

    Therefore any statistical analysis of that data is utterly meaningless and to place any importance on that analysis is completely misguided.

    We can say that Bem seems to have found some sort of an effect that we could test for. That is all.

    The attempt to replicate has serious flaws as well. First, it treats Bem’s analysis as if it were meaningful. They set up the test to see if the effect size was the same as what Bem claimed. That’s how they determine the number of people to test.
    But the analysis that gave the effect size was a meaningless misuse of statistical tools.
    So what the other researchers showed was that the population they tested didn’t show the level of ability that Bem claimed his subjects did. They didn’t show ‘no ability.’

    Also the attempt to replicate also wasn’t conducted on a random sample of any population. Is the manner in which subjects are obtained sufficiently skewed that they might effect the outcome of the trials?
    I think the answer would be yes in this case.

    The mathematics of statistics is based on some very strict premises. The ‘perfect’ test doesn’t actually exist in the real world– it is a mathematical construct.
    The methods of sampling, the determination of when to stop, the manner in which the test is run… often the data is so bad it really shouldn’t be used in the ways it is being used. The data are in violation of the premises on which the analysis is based. GIGO.

    Wow– I’m ranting… :-)

  20. davidsmithon 31 Aug 2012 at 6:12 pm

    Sonic said,

    What Bem found was that the people he was testing were scoring as if they had a particular ability when he stopped the test. Because he stopped the test in the manner he did, his data isn’t appropriate input into any statistical analysis.

    You’re saying he stopped running participants when he saw the data looked good? On what basis are you making this claim?

  21. sonicon 31 Aug 2012 at 8:06 pm

    davidsmith-
    From the post
    “Related to this is the admission by Bem, according to the article, that he peeked at the data as it was coming in.”
    It is arguable– perhaps this didn’t alter the stopping point of the test.
    However, since this is an initial test that will have to be replicated to be meaningful anyway, I would err on the side of caution and call the data interesting but not valid for statistical analysis.

    This is not to call out Bem in particular. I see these problems in many data sets.
    For example– questions I ask include– “What population was randomly sampled for this test?” and “Is the population being tested sufficiently non-random to possibly skew the results?” You can imagine that I think that running tests on sophomore college students who get extra credit for participation is sufficiently non-random to bring about a ‘fail’ in my mind.

    I see lots of results that I think are interesting– but I do get tired of what I think of as abuse of the math. You could say, “Had we run this test in a proper manner and got these results, then we could say…” and give a statistical run down.
    But please– admit the test wasn’t done properly- that it needs replication- and do the replication in a rigorous manner.

  22. Shelleyon 31 Aug 2012 at 11:09 pm

    Johann,
    I am basing my view that the probability that psi is true approaches zero on my past reading of the literature, which does not provide a compelling case for psi. The literature is littered with small effect sizes with weak or deeply flawed methodology or suspect analytical techniques. In addition, i see no clear connection to a literature that supports psi, but an abundant literature that refutes it.

    Having said that, I am not willing to dismiss it outright. I think each study deserves a reasoned examination and not (what I’d consider a flippant dismiss) an ECREE, though I’d still have to see something pretty impressive methodologically and replicable to be convinced.

  23. cremnomaniacon 01 Sep 2012 at 2:44 am

    An interesting review. This goes to the heart of the veracity of research findings, and the all too often publishing of only “significant” findings that, as you are aware, result in publication bias. The replications and the nulls need to be published as well.

    As a graduate of psychology, I find it odd that the basic idea of type I and type II errors is left by the wayside in most of the research I read. FYI: Type I – where a true null hypothesis was incorrectly rejected. Type II – where one fails to reject a false null hypothesis.

    Most readers may know this, some may not, so I apologize for preaching to the choir, but its elementary to statistical analysis. The common P=0.05 in psych studies is arbitrary at best. We could learn something from physics if we demanded a significance level of 5 sigma. It s way out there, but imagine how much crap would be filtered out.

    One caveat, sometimes the most revolutionary findings aren’t significant, statistically speaking. Having just read Carl Sagan’s “Demon Haunted World”, I completely agree that findings need to considered in the context of probability. However, that too should be applied judiciously. As our understanding and knowledge grow along with new discoveries, so does our assessment of probabilities. Probability assessment is based on current understandings, and in that context it has limitations.

  24. BillyJoe7on 01 Sep 2012 at 9:07 am

    It is a measure of the promotional success of parapsychologists that a study like this is taken seriously. To “flippantly dismiss” this study poses a vanishingly small risk of missing something important. I fail to see why any reasonable person should do otherwise. Except, of course, that a prominent journal saw fit to publish the study and so some other reseachers had to waste their valuable time refuting it.
    Oh well…

  25. davidsmithon 01 Sep 2012 at 11:31 am

    Sonic,

    So where in Bem’s article does he say he peeked at the data as it was coming in? I can’t find that bit. Perhaps Steven can clear this up?

  26. davidsmithon 01 Sep 2012 at 11:36 am

    Sonic said,

    For example– questions I ask include– “What population was randomly sampled for this test?” and “Is the population being tested sufficiently non-random to possibly skew the results?” You can imagine that I think that running tests on sophomore college students who get extra credit for participation is sufficiently non-random to bring about a ‘fail’ in my mind.

    That’s only a problem if you were going to generalise the results to the whole population, i.e., to say “yes, we can all precognise”. Testing a sample of students is not a problem if all you want to ask is whether precognition exists.

  27. Shelleyon 01 Sep 2012 at 12:27 pm

    Billyjoe7,
    Hmm. . . So if no reasonable person should waste time examining a study on psi, why is it that as reasonable persons we waste so much of our time examining and critiquing studies on accupuncture, homeopathy, etc?

  28. sonicon 01 Sep 2012 at 1:21 pm

    davidsmith-
    I took the info from the post about Bem’s ‘peeking’. I have no other info about that.

    I agree that any population could be studied to see if those particular people display an ability over the time tested. And one could then say “These specific people apparently demonstrated the ability over the trials run at this particular time, whereas these specific people did not demonstrate the ability over the trials run at a different time.”

    I note that the test “Can a person hit a baseball thrown by a major league pitcher?” gets quite variable results on a daily basis during the season. Does a ‘no-hitter’ prove the other tests (where batters got hits) merely artifacts of chance?
    Perhaps…

    If I am wrong about the ‘peeking’, then my complaint about that in this case is withdrawn.
    The larger point– that many of these tests are run in a manner that makes the use of the math highly suspect stands regardless…

  29. BillyJoe7on 01 Sep 2012 at 5:30 pm

    shelley,

    “So if no reasonable person should waste time examining a study on psi, why is it that as reasonable persons we waste so much of our time examining and critiquing studies on accupuncture, homeopathy, etc?”

    I already gave you the answer in the same comment from which you elected that quote.
    To paraphrase:

    Except, of course, that prominent journals see fit to publish trials of homoeopathy and acupuncture and so some other reseachers have to waste their valuable time refuting it.

    Having said that, the idea that future acts can effect present behaviour is much more appropriately dealt with by derision and sarcasm than a considered response. There is a risk in treating ridiculous claims seriously.

  30. Johannon 01 Sep 2012 at 11:00 pm

    Billyjoe, I can hardly see why studies testing for precognition should merit derision and sarcasm, provided they claim to be conducted scientifically. In time-symmetric quantum mechanics, researchers such as Aharnov, Tollaksen, and Howell have conducted experiments on just this idea – that future events can ripple back to affect the past. Here is a summary of their protocol:

    First the physicists would measure spin in a set of particles at 2 p.m. and again at 2:30 p.m. Then on another day they would repeat the two tests, but also measure a subset of the particles a third time, at 3 p.m. If the predictions of backward causality were correct, then for this last subset, the spin measurement conducted at 2:30 p.m. (the intermediate time) would be dramatically amplified. In other words, the spin measurements carried out at 2 p.m. and those carried out at 3 p.m. together would appear to cause an unexpected increase in the intensity of spins measured in between, at 2:30 p.m.

    So far, their experiments – numbering in the thousands of trials – have appeared to produce positive evidence for the idea that the future can ripple back to affect the past. And while there is still theoretical debate as to the true meaning of weak measurements, these kinds of experiments are no more fascinating, fundamentally, than Bem’s! Such a chance to discover if we ourselves have some ability to experience time symmetry could be derided and thrown away, but at what cost? The loss of knowledge, and the failure to practice science for an obsession with skepticism.

    - Johann

  31. Johannon 01 Sep 2012 at 11:57 pm

    Shelly,

    We must be looking at different subsets of the literature. When I see, for example, the Ganzfeld studies, I see a procedure that has turned up positive evidence for psi in the pre-PRL stage, the PRL stage, and the post-PRL stage, each time defying chance with odds of millions (or billions) to 1. The experiments in the first segment may have contained some artifacts produced by flaws such as having the same envelope sets touched by experimenters and subjects, motivated recording errors, randomization problems, some plausible sensory leakage, etc, but the fact that the PRL database, which did not contain most or all of these errors, displayed a hit rate actually two percent higher than the pre-PRL, seems to rule out that these flaws had much of an effect. Then we see the post-PRL database, widely considered the most rigorous, replicate the PRL with virtually the same hit rate, and odds against chance of more than 40 million to 1. Furthermore, indicators of study quality in this last set appear to correlate positively with effect size, such that the more rigorous and well-conducted the study, the higher the hit rate. Then we have the sub-analyses, which mirror favorably the predictions made by previous ganzfeld experimenters (and common sense), showing that the hit rate for blood relatives across both PRL and post-PRL is 62%, artists (7 studies) at around 40-50%, friends also at around 42%, subjects with at least one selection criteria at around 33%. And, of course, studies without dynamical targets, selection, or that deviated significantly from the standard GZ protocol basically at the null. And a comparison of GZ condition v.s. non-GZ shows that GZ remains highly significant, while non-GZ is at null.

    So the idea that the literature acts as significant disproof of psi to me is very strange; perhaps it could be argued that according to ECREE it is not supportive “enough” (although if your ECREE is supposed to be due to the literature itself, doing so would be circular), but it is certainly supportive. These analyses of the Ganzfeld are not unique either; they are present in other paradigms of research, like Dream telepathy, remote viewing, and psychokinesis. For the Maimonides dream telepathy studies, for example, not only are they highly significant, but meta-analyses of the studies that came after are as well.

    Nowhere can I find a sufficient reason to use the literature as a justification for ECREE; something else must be there, though, as I have argued previously, arguments based on our conception of physical reality are highly tenuous, especially in light of the results from time-symmetric experiments, and our general knowledge of the universe.

    Regards, – Johann

  32. BillyJoe7on 02 Sep 2012 at 4:34 am

    Johann,

    “these kinds of experiments are no more fascinating, fundamentally, than Bem’s!”

    People are not quantum particles.
    What happens at the quantum level cannot be translated to objects the size of humans.
    Weird things happen at the quantum level and what it all means is up for debate, but quantum effects are not seen at the macroscopic level. In the double slit experiments, the effects start disappearing as the size of the particle increases. When the particle reaches the size of about 64 atoms, the effect has already disappeared. The human brain consists of a thousand trillion neural connections. The mind boggles at the number of atoms. I hope you see the problem.

  33. Mlemaon 02 Sep 2012 at 9:44 pm

    BillyJoe, Johann,
    you might be interested in some of the research being done at Cornell with regards to biophotons.

    A Theoretical Mechanism of Szilard Engine Function in Nucleic Acids and the Implications for Quantum Coherence in Biological Systems
    http://arxiv.org/abs/1208.5729

    Emission of Mitochondrial Biophotons and their Effect on Electrical Activity of Membrane via Microtubules
    http://arxiv.org/abs/1012.3371

    one take on this some of this type of research:
    Biophotonic Communications and Information Encoding in Complex Systems – Implications for Consciousness?
    http://integral-options.blogspot.com/2012/07/biophotonic-communications-and.html
    the above links to a number of other interesting papers

    BJ – hope you see the problem with saying that what happens at the quantum level doesn’t effect what’s happening at the macro level

  34. sonicon 02 Sep 2012 at 11:20 pm

    Here is a recent experiment where they got molecules 58- and 114- atoms big to do the ‘wave-particle’ duality in a double-slit situation.
    http://www.livescience.com/19268-quantum-double-slit-experiment-largest-molecules.html

    Nobody knows how big a particle might work– only that 114 is the biggest yet.
    From the article–
    “This is one of the ways particles in the tiny quantum world behave oddly, diverging from the understandable macroscopic, classical world of people and buildings and trees. But scientists have wondered where the boundary between the two is, and if one even exists.”

  35. Johannon 03 Sep 2012 at 12:29 am

    Mlema, many thanks for the links you provided!

    I have been interested in biophotons ever since I learned about them two years ago, and have done some research myself. Among the many models that have been developed as possible explanations for psi, Theresa M. Kelly proposed one based on biophotons in her paper “Models of Extra-sensory Perception; A Quantum Approach to ESP Phenomenology.” Though b-photons may not present the final solution, they are highly involved in the search for quantum processes in living systems. Here is a good article for the layman to begin his/her understanding of biophoton theory:

    http://www.viewzone.com/dnax.html

    Concerning the emissions of b-photons, the following paper provides a rich history of its research, and an intriguing examination of several theories regarding the origin of the emissions:

    http://zeniclinic.com/zen/articles/BiophotonsAndBiocommunication.pdf

    This other paper is by Fritz Albert Popp, the multi-award winning biophysicist who is credited with bringing biophoton research back into conventional practice. It expands on some of the notions of the first paper:

    http://www.anatomyfacts.com/research/PropertiesBioph.pdf

    For those interested, much corroboratory evidence has emerged for the presence of QM effects in living systems. The paper Mlema cites about b-photons in neurons is just one; a 2010 study published in Nature found evidence of entanglement in the photosynthesis of plants, a 2011 study published in The Physical Review found data strongly indicative of entanglement in migrating birds’ beaks, possibly aiding them in sensing and responding to the geomagnetic field, and a study still underway in Japan by Anirban Bandyophadyay found quantum coherence effects in brain microtubles – to name a few.

    It might also be pertinent to mention that the study Sonic cites was undertaken by a widely acclaimed physicist named Anton Zeilinger – a brilliant man – who, among other things, believes that the very basic ideas of QM should be taught to children, in order to encourage the development of a different way of viewing the world.

    I believe the evidence we now have from physics and biology is strongly compatible with a worldview incorporating psi phenomena. Furthermore, although theoretical obstacles still remain as to the complete reconciliation of these two things, it would be utter folly to dismiss the results of parapsychology without serious effort to investigate the subject in a manner that at least resembles an attempt at impartiality.

  36. Johannon 03 Sep 2012 at 12:45 am

    Billyjoe,

    In addition to pursuing the links I provided, you should know that entanglement, fundamentally, is a property of all matter and energy. The seeming loss of entanglement when a QM system is exposed to the classical world results from a process called “decoherence”, or the loss of quantum information to the environment. Decoherence does not break entanglement, however; it merely provides a calculational model for the threshold of entities a system can harbor in an entangled state until the moment when it is impossible, for all practical purpose, to DEMONSTRATE that the entities in that system are entangled. Essentially, the “spooky correlations” provided by entanglement get averaged out over so many objects that they in effect disappear, and no practical purpose can be divined from them (like making QM computers). As I said, the entanglement is not gone; it continues to spread via the interaction of anything with anything else (particles, electromagnetic fields, gravitational fields, etc), indefinitely. So our universe is basically a dynamically re-entangling quantum system; we do not percieve the subtle connections around us (or do we?) because of decoherence. But the question that parapsychology poses is that, if we are all fundamentally interconnected at the subtlest of levels, will we see repercussions of this fact in our lives?

    - Johann

  37. BillyJoe7on 03 Sep 2012 at 8:34 am

    Johann,

    Your paragraph is not news to me.
    The qurestion is: Has any of this actually been demonstrated?
    The answer is: No, it is just pure unbridled non evidence based wild speculation.
    The lesson is: Keep your feet on the ground and your head out of the clouds.

  38. BillyJoe7on 03 Sep 2012 at 8:50 am

    Mlema,

    You are again linking, not to mainstream science, but fringe and pseudo science, or what we might more charitably call wild speculation. It might be interesting, but you can’t hang an argument on it.
    Microtublues were the basis of wild speculation by physicist Roger Penrose. The quantum mind hippy folk have flocked to it like bees to flowers eager for the taste of sweet nectar. But, after a quarter of a century, the idea is still struggling to get out of the quicksand in which it was born.

  39. Johannon 03 Sep 2012 at 3:32 pm

    Billyjoe said,

    “has any of is actually been demonstrated?”

    Absolutely! Entanglement has been demonstrated ever since the Aspect et al experiments in the 1960′s, and the spread of entanglement can be very accurately calculated with decoherence theory. When it spreads too far, the entanglement is said to have collapsed, but the fact that our calculations perfectly describe this process, and that they tell us the entanglement has not disappeared, demonstrates the persistence of entanglement beyond FAPP (for all practical purposes). So we can in fact demonstrate, with a mixture of empirical and theoretical evidence, that fundamental entanglement exists.

    “The lesson is: keep your feet on the ground and your head out of the clouds.”

    I do not very much like walking inclined; from that position, it becomes difficult to see or to notice anything, even that which is right in front of me! No, do not narrow your focus in that way (or your mind); think BIG. Dare to discover, to speculate – even if others call you a crank; it is this very freedom that makes science great.

    A side effecft of skepticism, when taken in large doses, is that it stifles science. Anyone can be skeptical; you only need a prior position, and a strong motivation to defend it. Bible creationists are some of the most skeptical people you can find; their penetrating criticisms of evolution endeavor to uncover every flaw, even the minor ones – yet they are not scientific. They start with a prior certainty (God created the world in this amount of time) and set out to rip to shreds all that opposes it. People are very skeptical of everything, including vaccination, global warming, government promises, conspiracy theories, UFOs, psi, evolution, religion, and the list goes on. And once your skepticism passes a certain threshold, it becomes powerful enough to overwhelm virtually any evidence. This is why I caution caution, why I aspire to suspend both belief and disbelief when evaluating a claim, and why I believe that what we do not know about the universe is more important than what we do when it comes to examining possibilities.

    Over the course of our posts, I have demonstrated three things:

    1. Retrocausality is expected by certain versions of quantum theory, and there is some empirical evidence for it in physics laboratories (as well as parapsychology labs).
    2. Strong quantum coherence has been observed in several living systems, in environments that are hot, wet, and noisy.
    3. The world is fundamentally quantum mechanical in nature, and entanglement is the underlying reality.

    Given all three of these, your thesis that we should deride and dismiss evidence from Parapsych labs claiming to demonstrate retrocausality, should, to the unbiased observer, seem unwise. Perhaps you too will agree, and move to a higher level of discourse?

    - Johann

  40. BillyJoe7on 03 Sep 2012 at 6:01 pm

    Johann,

    Firstly, you confuse scepticism with denialism. In that, you are not alone. For example, climate change scepticism is actually climate change denialism. Scepticism is the scientific attitude: doubt everything untill the evidence is there.

    Secondly, scepticism does not dismiss fringe science. It puts it in its place. When the evidence mounts, then fringe science is taken seriously, and when it becomes convincing, it becomes mainstream, not before. It is important to keep a perspective.

    Thirdly, when I asked “has any of is actually been demonstrated?”, I meant at the macroscopic level of the brain. That answer is still: no. All there are are conflicting theories based on doubtful studies. Also what retrocausality actually means at the quantum level is open to serious debate.

    Lastly, Bem is bunk. Practically zero plausibility and no convincing evidence let alone extraordinary evidence.

  41. Mlemaon 03 Sep 2012 at 7:34 pm

    billyjoe,

    “You are again linking, not to mainstream science, but fringe and pseudo science, or what we might more charitably call wild speculation. It might be interesting, but you can’t hang an argument on it.”

    I have to assume that once again you aren’t actually READING my links. And I guess you’d better tell the folks at Cornell they’re on the fringe.
    If you’d like to dump some qualified and informed derision on something specifically said on one of the links (which I provided simply for consideration), why don’t you go ahead and do that? I challenge you to restrain yourself from simple off-the-cuff insults.

  42. jt512on 03 Sep 2012 at 11:04 pm

    Mlema wrote:

    And I guess you’d better tell the folks at Cornell they’re on the fringe.

    Actually, Daryl Bem is from Cornell.

  43. sonicon 04 Sep 2012 at 12:01 am

    Regarding evidence of psi–

    http://www.psy.unipd.it/~tressold/cmssimple/uploads/includes/MetaFreeResp010.pdf

    It seems that there are numerous well run experiments that can be interpreted as showing the effects of psi (in this case thought transfer).
    It seems that certain conditions yield better results than other conditions (as would be expected from an actual ability).
    It seems that some people demonstrate the effect more often than others (again, what one would expect from an actual ability).
    It appears the experiments have been replicated with similar results over a number of years.

    Certainly these results can be interpreted in a number of ways, but the notion that thought transfer actually happened would have to be included in the possibilities.

    At this point we have no way of measuring thought directly, so the data will be a matter of interpretation. When we can measure thought directly, this may lead to a better understanding of what these experiments are showing.

    (Of course the fact that thought can’t be measured might lead one to think that thought doesn’t actually exist…) ;-)

  44. BillyJoe7on 04 Sep 2012 at 12:14 am

    Mlema,

    I was actually trying not to be insulting.
    And, for your information, many universities do study and teach all sorts of BS.
    Unfortunately, money, popularity, and personality trumps science in many cases.

    jt512: “Actually, Daryl Bem is from Cornell.”

    Thank-you, case in point.

  45. BillyJoe7on 04 Sep 2012 at 12:25 am

    sonic,

    “It seems…
    It seems…
    It seems…
    It appears…”

    Yeah, that sounds like something to hang your argument on.

    “Certainly these results can be interpreted in a number of ways, but the notion that thought transfer actually happened would have to be included in the possibilities”

    With a probability of zero?
    - no transmitting organ
    - no recieving organ
    - no possible mechanism

  46. steve12on 04 Sep 2012 at 1:10 am

    If you wanna know how Bem found Psi, just read the Wagenmaker et al. paper referred to above.

    As that paper points out, many of Bem’s flaws are true of the whole field of psychology. The only wonder re: Bem’s paper is how he couldn’t find a bigger effect size given the amount of fishing he admits doing – let alone all of the casts that reeled in nothing, but that should have been accounted for statistically. And this is w/o getting into selective stopping, abuse of df, etc. When you fish and fish and still can barely get an effect size, that’s crap. And that’s all it ever was.

    So there’s no need to get into a big debate about mechanisms or any of that. in fact, I would say there’s almost no need to replicate because there was nothing TO replicate. It’s just a misunderstanding of / bad practices with statistics that anyone who follows the psychology lit is all too familiar with.

  47. Mlemaon 04 Sep 2012 at 1:52 am

    BillyJoe,

    please look at one of the links from Cornell. The whole article is available as a PDF so you don’t have to “dissect it” as you say:
    http://arxiv.org/abs/1208.5729

    I provided the link as a way to illustrate that what may be happening on the quantum level isn’t irrelevant to what’s happening at the macro level.

    Please tell me exactly what it is in the research that you find “fringe and pseudo science, or what we might more charitably call wild speculation”.

    Also, again, I don’t see that your comments regarding the third link:
    http://integral-options.blogspot.com/2012/07/biophotonic-communications-and.html
    show that you actually read what it said.

    BillyJoe, I enjoy debating with you. But I request that you stick to particulars. Your comment:

    “Microtublues were the basis of wild speculation by physicist Roger Penrose. The quantum mind hippy folk have flocked to it like bees to flowers eager for the taste of sweet nectar. But, after a quarter of a century, the idea is still struggling to get out of the quicksand in which it was born.”

    is very poetic, but doesn’t have a lot to do with the links I provided, and it doesn’t really say anything except that you dislike the ideas of Roger Penrose and you enjoy sarcasm.

    and let me get this straight: The quantitative biology department at Cornell is propagating BS because Bern was a psychology professor there? Which form of deductive reasoning brought you to that conclusion?

  48. Johannon 04 Sep 2012 at 3:16 am

    Billyjoe wrote: “Firstly, you confuse scepticism with denialism. In that, you are not alone. For example, climate change scepticism is actually climate change denialism. Scepticism is the scientific attitude: doubt everything untill the evidence is there.”

    But Billyjoe, it is easy to say that the evidence is not there; any denialist will say that the evidence is not there. And given such a position, it is obvious that doubt should prevail.

    Denialism is simply what happens when skepticism bubbles to an insufferable excess, and the ability to self-critique drops to near-undetectable levels. Anyone can become a denialist, just as anyone can become a skeptic; the difference is merely a matter of degree. And if you believe it impossible to overdo skepticsm on the paranormal and to become a denialist of evidence, I ask you to present to us a position on psi research that is more extreme than your own, and that you think would constitute skepticsm to the point of irrationality.

    “Secondly, scepticism does not dismiss fringe science. It puts it in its place. When the evidence mounts, then fringe science is taken seriously, and when it becomes convincing, it becomes mainstream, not before. It is important to keep a perspective.”

    How can this happen when the guardians of skepticsm refuse to even look at the evidence, in favor of its quick and convenient dismissal?

    “Thirdly, when I asked “has any of is actually been demonstrated?”, I meant at the macroscopic level of the brain. That answer is still: no. All there are are conflicting theories based on doubtful studies. Also what retrocausality actually means at the quantum level is open to serious debate.”

    Clearly; I indicated several times that QM retrocausality was in contention, but the fact that it is a serious, viable contender for the construction of physical reality should instill large amounts of this doubt-thing into the minds of skeptics regarding the impossibility of retrocausality in life. And of course retrocausality has not been conclusively demonstrated in the brain, but some QM effects have, and if retrocausality is a necessary component of QM it is not too much of a stretch to speculate, especially in light of empirical evidence, that our conscious experience may be able to percieve it sometimes, or to a small degree.

    But the question to ask is, even if retrocausality was demonstrated in the brain by a distinguished Cornell professor, would your policy towards evidence for psi allow you to look at it?

    “Lastly, Bem is bunk. Practically zero plausibility and no convincing evidence let alone extraordinary evidence”

    I disagree, naturally. The fact that the amount of significant findings observed in the lab so far (removing the online studies) is 6 out of 15, and that this success rate is associated with odds against chance of 18,939 to 1, gives me clear reason to suspect that something is going on.

    - Johann

  49. BillyJoe7on 04 Sep 2012 at 8:28 am

    Mlema & Johann,

    You are asking a lot of me.

    It’s like asking me to spend a whole heap of time studying complicated multipage writeups of experiments which are said by proponents of homoeopathy to show that homoeopathy works.

    You must first give me a reason to study these papers otherwise, frankly, I’m not going to waste my valuable time. The fact is that there is no reason to believe in retrocausality at the quantum level (let alone that it could “leak through” to the macroscopic level). There are valid explanations other than retrocausality for the outcomes of double slit experiments. In that case why on earth would I, or anyone else, accept retrocausality as the explanation?

    You are living in crazy nutcracker land if you expect me to do that.

    First give me a reason.
    Show me a quantum experiment that unequivocally demonstrates retrocausality.
    And, before you get too excited, I will tell you that you will not be able to do so.

    (The same goes for an unequivocal role for consciousness in the outcome of any quantum experiment, but that is another story and another challenge that cannot be answered)

  50. Johannon 04 Sep 2012 at 12:39 pm

    Billyjoe,

    Finally we get to the heart of the matter. As regards time, it is a non-issue; you can choose to study the evidence for psi at as leisurely a pace as you desire, but merely for having studied it at all, you will be a better man and a better skeptic. It really is not so difficult; think of how much time you already spend on the Internet participating in skeptic forums or blogs like this, having all these general and unspecific conversations with people; you could spend a fraction of it reading a book like “The Conscious Universe” by Dean Radin, “Science and Psychic Phenomena” by Chris Carter, or even “Fringeology” by Steve Volk (a very good, neutral-minded read, highly reccomended). Then, at the pace you choose, you could pursue the original research into the Ganzfeld as directed by these books, read the criticisms, and read the counter-criticsms.

    Myself, I have dedicated countless hours of my time to research and study on these topics, but you need no go so far; just a couple minutes of reading a day will be enough to get you a general and insightful comprehension of the literature. Whatever opinion you hold on psi after this process will be more greatly respected by proponents and skeptics, because you will have taken upon yourself a commendably scientific approach.

    You want a reason study the results of the Ganzfeld? Because the Ganzfeld is widely recognized by proponents and skeptics as the very best evidence for psi, because when considered collectively it returns odds against chance of more than 29 quintillion to 1, because the procedure has been checked, double-checked, and co-created by the best skeptics and parapsychologists to demonstrate psi, because there are consistent predictable variations in the database among different kinds of people, and a whole lot more (see my reply to Shell’s third comment to get a better picture).

    Here’s another reason – a quote by Susan Blackmore:

    “The disbeliever can refuse to look at the positive results. You may think I wouldn’t refuse, but I have to admit that when the Journal of Parapsychology arrives with reports of Helmut Schmidt’s positive findings I begin to feel uncomfortable and am quite apt to put it away ‘to read tomorrow.’…The disbeliever has to take notice of those positive results. I am thinking particularly of the results of Carl Sargent, Charles Honorton, Helmut Schmidt, and Robert Jahn [in experiments on telepathy, remote viewing and psychokinesis]. I suggest that if we think these can easily be dismissed then we are only deluding ourselves. One cannot offer simplistic counter-explanations and throw all these results away. I am not saying that these results may not, in the future, succumb to some normal explanation; they may well do so. But at the moment we do not have such an explanation.”

    Susan Blackmore is a widely acclaimed skeptic, and her advice runs for all skeptics who wish to keep a critical mind.

    Finally, to your retrocausality comments. First, the experiments I talked about were not double slit experiments; they were created specifically to test retrocausality, and they turned up very positive results by more than three investigators. What is under debate is the role of “weak measurement” that they use in their experiments – a procedure commonly accepted to provide highly inaccurate information about QM systems, therefore not perturbing them to the point of collapse. This inaccurate information, however, can be made more accurate through repetition of measurement, and that is what these experimenters do; they repeat the experiment several thousand times to see if a pattern emerges, and it does. But even these results continue to be debated, for it is argued that the role of weak measurement is still not well understood. You must understand, however, that this is a natural consequence of their results; others think they simply have to be wrong, and so now they are debating what measurements actually mean. And they may be wrong, but just the fact that retrocausality is now a serious contender in QM should prompt us to examine other sources of empirical evidence on retrocausality with a less prejudiced eye, for these very experiments count as additional evidence that can be incorporated into the cannon of knwoledge. I also add that if retrocausality occurs in our consciousness it need not be “at the macro level”; it must simply occur in the brain. And we know now that some QM effects do occur in microtubles, and others may take place between the synapses (as physicist Henry Stapp believes).

    So it remains to you to decide what is important in your life, but if you are not going to research the topics in question you should know that it is not intellectually very impressive to go around denying them.

    Finally, I leave you with the question of what skepticism means to you.

    Webster defines it as: “One who is yet undecided as to what is true; one who is looking or inquiring for what is true; an inquirer after facts or reasons.”

    Wikipedia: “In classical philosophy, skepticism refers to the teachings and the traits of the ‘Skeptikoi’, a school of philosophers of whom it was said that they ‘asserted nothing but only opined.’ (Liddell and Scott) In this sense, philosophical skepticism, or Pyrrhonism, is the philosophical position that one should suspend judgment in investigations.[1]”

    And an insightful comment by an anonymous writer: “The original definition of skeptic was a person who questions ALL beliefs, facts, and points-of-view. A healthy perspective in my opinion. Today’s common definition of skeptic is someone who questions any belief that strays outside of the status quo, yet leaving the status quo itself completely unquestioned. Kind of a juvenile and intellectually lazy practice in my opinion.”

    - Johann

  51. steve12on 04 Sep 2012 at 12:55 pm

    “You must first give me a reason to study these papers otherwise, frankly, I’m not going to waste my valuable time. ”

    Exactly. It’s like anti-vaxers who clammer for bench science to find the mechanism of a non-existent effect.

    And frankly, if someone doesn’t know enough about stats to realize that Bem did not reject the null in his experiments, then I wouldn’t really trust that person’s summary of other studies. And that’s where the waste of the most precious resource we have, time, comes in.

  52. sonicon 04 Sep 2012 at 3:15 pm

    BillyJoe7-
    What argument am I making?

    I gave a link to the most recent evaluation I could find of recent experiments published by the American Psychological Association. Did you read the paper?

    If so, then can you give me anything specific that would make you question the analysis?– I believe I gave a reasonable summary.

    If not, then why do you comment on it at all?

  53. Johannon 04 Sep 2012 at 3:40 pm

    Steve12 said: “…if somone doesn’t know enough about stats to realize that Bem did not reject the null in his experiments…”

    Buddy, if Bem had not at least rejected the null in his experiments I do not think the editors at the JPSP would have agreed to publish his paper as having done so. The odds against chance for Bem’s experiments are well beyond a million to 1 from a frequentist perepective, and 9 out of 8 of his experiments independently yielded statistically significant results.

    - Johann

  54. steve12on 04 Sep 2012 at 4:05 pm

    Again, read Wagenmaker et al. – I gave a short mention of some of the problems above. If you can’t get through that paper, then you have no business adjudicating the question of whether Bem did his analyses correctly – and ditto for the other recommendations you give above.

    If you can get through the paper, please refute.

    If Bem’s analyses are done appropriately, the null is well intact. There’s no need for debating mechanisms and all of this. He never showed psi in the first place, he showed the lax standards that are pervasive in psychology today, especially social psychology.

  55. jt512on 04 Sep 2012 at 5:27 pm

    For statistical evidence of selective reporting in Bem (2011), see Francis (2012).

  56. BillyJoe7on 04 Sep 2012 at 5:44 pm

    Johann,

    So you cannot link me to a quantum experiment that unequivocaaly demonstrates retrocausality.
    And you cannot link me to a quantum experiment that unequivocally demonstrates a role for consciousness.

    I will tell that this is the response I always, and without exception, get when I ask these questions.
    It is telling that none of you can seem to do so despite the fact that you base so much of their belief system on this being true.

    Oh well…..

  57. BillyJoe7on 04 Sep 2012 at 5:50 pm

    sonic,

    Sorry, I thought you were providing that link as evidence for your belief in PSI.
    I was merely showing you why I was not interested in reading it.
    “It seems” is not an argument.
    Show me the transmitter and the receiver and a mechanism and I will begin to take PSI seriously.

  58. gsfrancison 04 Sep 2012 at 10:36 pm

    The Galak et al. paper looks to be a fine set of experiments. Given my statistical analysis showing the presence of publication bias in Bem’s studies (Francis, 2012) — contact me if you cannot download a copy– I have to wonder whether it was worth all the effort. Why bother to replicate experiments that are known to be invalid? The burden of proof is on the author making the original claim and flawed experiments do not support any claim at all. The authors probably started their investigations before these details were made public, and there is a lot of interest in Bem’s findings, so probably reporting these new studies was worthwhile.

    Curiously, though the key lesson from the controversy around Bem’s findings may have not been learned. That nice quote

    “Popper (1959/2002) defined a scientifically true effect as that “which can be regularly reproduced by anyone who carries out the appropriate experiment in the way prescribed.”

    is not true for a field like experimental psychology, because an “effect” in psychology is defined statistically. Even if an effect is true in a population, when you select random samples from the population you will sometimes get samples that do not show the effect. Indeed, the absence of such null findings is what provides evidence that the set of reported findings are invalid. The idea that repeated successful replication provides evidence for an effect is simply not true, at least for the kinds of experiments that are typically run in psychology.

    By the way, you can apply the same logic to the experiments in Galak et al. They ran 7 experiments and one of them happened to reject the null hypothesis (p=0.04). Suppose that their conclusion is true and the real effect size for their experiments equals zero. With the 0.05 criterion for determining statistical significance, if the null is true then each experiment has a probability of 0.95 of _not_ rejecting the null hypothesis. Thus, the probability of getting 6 (or more) out of 7 experiments to _not_ reject the null when it really is true is

    (0.95)^7 + 7 x (0.95)^6 x (0.05) = 0.698 + 0.257 = 0.955

    The first term is the probability of all seven experiments not rejecting the null hypothesis if the null is true. The second term computes the probability that in a set of six experiments all do not reject the null and one seventh experiment does reject the null. There are seven ways this could happen. The overall probability of getting the kind of data reported by Galak et al. if there really is no effect is quite believable. In contrast, my analysis suggests that the probability of getting the kind of pattern reported by Bem if the psi effect is really as his experiments suggested is 0.058, which is pretty unbelievable.

    As scientists we remain open to many possibilities. Maybe psi exists, but a researcher cannot provide evidence for the effect with flawed experimental studies. Bem’s investigations do not alter my belief in the existence of psi at all. The Galak et al. study, in contrast, makes a pretty good case that the available data suggests the effect size is very close to (possibly equal to) zero.

  59. Johannon 04 Sep 2012 at 11:10 pm

    Steve12, I have investigated the debate between Bem and Wagenmakers in the past to the best of my ability, but I do need an in-depth understanding of Bayesian methodology to advocate for the existence of psi, or to advise others to look at research instead of deriding and remaining ignorant of it, and I certainly do not need such an understanding to know that when you assign P (H1) = 10−20 (.00000000000000000001) as a prior probability for a hypothesis, you can invalidate any evidence you could possibly imagine. Bem and Utts responded to Wagenmakers in http://dl.dropbox.com/u/8290411/ResponsetoWagenmakers.pdf and their reply I believe is very helpful for understanding how Bayesian statistics can be misused. Wagenmakers replied shortly after, followed by another response from Bem, Utts, and Johnson, and yet another from Wagenmakers et al.

    Myself, I have always been a frequentist; I believe that abandoning objectivity for rampart use of prior probability is an affront to science. So I expressly disagree with Wagenmakers that psychologists need to change the way they analyze their data; in fact, I find this Bayesian/Frequentist debate a rather circular affair. People want to change the whole accepted framework of science because not doing so would mean that they would have to change the whole accepted framework of science (the most common complaint against psi, though, as I mentioned earlier, illegitimate).

    - Johann

  60. Johannon 04 Sep 2012 at 11:27 pm

    Billyjoe,

    You are beating around the bush. Never once did I tell you I had unequivocal proof of anything; all I have to offer you are indicative probabilities and the freedom of uncertainty. I realize that in the end I may be wrong about many things, but I consider my time worthy enough to spend in pursuit of answers (and questions). If you prefer to do other things, or to seek answers and questions in other areas, that is your choice to make, but I think we can both agree that this is an interesting field, if the claims made in it are true, and so that alone is enough to draw my evaluation and attention.

    You heard my offer, now take it or leave it. I can help you, provide you valuable links, and maybe break the shell of your skepticism to a degree – if you are open to that possibility. Otherwise, I decline to continue this conversation, for I feel that your skepticism is not currently genuine – at least in the manner(s) I described in my previous post.

    Regards, – Johann

  61. steve12on 05 Sep 2012 at 1:54 am

    Johann:

    I don’t mention Bayes becasue it’s unnecessary to my point that the null was not falsified (or that he committed a type I, however you wanna look at it). And they talked about a lot of other stuff in that paper. Bem’s use of classical stats is inappropriate because he fished and didn’t penalize himself for it. IF you keep going with almost any dataset and a decent N, you’ll find stat sig for some small difference.

    And if you think those crtiticisms are too speculative, think again.

    Not only does Bem admit fishing (while treating the data as though he didn’t), the Francis (2012) paper cited by jt512 above shows statistical evidence that he fished w/o properly reporting or correcting.

    There’s no “there” there in the Bem study. IT’s only value is pointing out problems in data analysis standards / stat practices in psychology.

  62. BillyJoe7on 05 Sep 2012 at 7:35 am

    Johann,

    “Never once did I tell you I had unequivocal proof of anything; all I have to offer you are indicative probabilities…”

    Fair enough.
    But, when a very good account of the results of these quantum experiments can be made without resorting to consciousness and retrocausality, I have to wonder why you bother with “indicative probabilities” regarding unnecessary hypotheses.

  63. sonicon 05 Sep 2012 at 10:03 am

    BillyJoe7-
    The paper I linked to is the best evaluation of the experimental tests that i am aware of. It covers the difficulties and deals with only the best run tests.

    You want this to be some sort of argument? How bizarre.

    Anyway– If you read the paper, I think you’d agree that my use of the phrase “It seems” is perfectly appropriate to the situation as it now stands.

    “It seems the data gathered by the researchers in the ganzfeld experiments can be seen to be in alignment with the hypothesis that thought transfer actually occurs,” is an appropriate statement– that’s my take- that’s not from the paper.

    But you would have to look at the paper and the evidence it presents to see what I’m talking about.
    Otherwise your comments about that are based on what?

    BTW- what is my belief in psi?
    I can never remember from one minute to the next. :-)

  64. Johannon 05 Sep 2012 at 11:24 am

    Billyjoe,

    A nice article for the layman: http://discovermagazine.com/2010/apr/01-back-from-the-future/article_view?b_start:int=0&-C=

    - Johann

  65. jt512on 05 Sep 2012 at 3:56 pm

    Johann wrote:

    I have investigated the debate between Bem and Wagenmakers in the past to the best of my ability, but I do need an in-depth understanding of Bayesian methodology to advocate for the existence of psi, or to advise others to look at research instead of deriding and remaining ignorant of it, and I certainly do not need such an understanding to know that when you assign P (H1) = 10−20 (.00000000000000000001) as a prior probability for a hypothesis, you can invalidate any evidence you could possibly imagine.

    First of all, prior probabilities do not validate or invalidate evidence. The evidence speaks for itself. The outcome of a Bayesian hypothesis test is a Bayes factor, which is the relative probability of the data under the null and alternative hypothesis. It is independent of the prior probability.

    By calculating Bayes factors for each of the experiments in Bem (2011), Wagenmakers et al showed that taken individually the experiments did not provide strong evidence for psi, and that some actually provided evidence against it. Rouder & Morey (2011) took a Bayesian meta-analytic approach to Bem (2011) and calculated a Bayes factor of about 1/40 for the set of experiments, indicating that the data favor psi over the null hypothesis by a factor of 40. Both analyses took the data at face value, ignoring the likelihood that the Bem had selectively reported studies that favored psi. But the point I want to make is that you can apply these Bayes factors to whatever prior probability about psi you personally have. My prior probability of psi is about one in 10 billion. Applying Rouder’s Bayes factor of 1/40 to that probability only reduces my uncertainty about the existence of psi to one in 280 million, so it’s not nearly enough evidence for me. But YMMV.

    The second point I want to make is that even the tiny prior probability you mention, 10^(-20), is easily overcome if the phenomenon being studied is real. P-values on the order of .0001 are common in fields such as medicine that investigate real effects. The Bayes factor for 10 studies each with 50 subjects reporting a p-value of .0001 is approximately 10^(-23). So if Bem had reported 10 studies with p-values of about .0001 rather than .02, his results (taken at face value) would have overcome a prior of 10^(-20) by a factor of one thousand. What this shows is that even very tiny prior probabilities are easy to overcome if the effects being investigated are real.

    Jay

  66. Johannon 05 Sep 2012 at 4:43 pm

    Jay,

    The fact remains that (from Christensen, Johnson, & Branscum, 2011, discussing an analysis similar to Wagenmakers et al): “the moral of the Lindley–Jeffreys paradox is that if you pick a stupid prior, you can get a stupid posterior”

    Furthermore, Bem et al write: “This raises the question in the present case of how close to 0 the prior probability for the psi alternative would need to be to maintain a posterior probability in favor of the null close to .95. For the knowledge-based prior with a one-sided alternative in favor of psi, one’s prior probability that the psi alternative is true would have to be 10^-8 or lower. Thus, when taking the combined data into account, it would take very strong initial skepticism regarding psi to retain a reasonably high posterior belief in the null. Of course Wagenmakers et al. (2011) admitted that their a priori belief in the psi alternative is, indeed, very close to zero (10^-20), so even the posterior probability of 7.3 x 10^-5 for the null obtained with the knowledge-based prior fails to exceed their threshold for being convinced.”

    Personally, I think a 99,999,999,999,999,999,999 to 1 bias in favor of H0 is not scientific. Something like 100 to 1 is more reasonable, especially in light of the many tediously repeated arguments I made above about our inability to consider knowledge we do not have – which is vast – and that when we consider knowledge we do, it is already borderline inclusive of psi, what with non-locality, retro-causality, and QM effects in living systems.

    Hell, given what we know, it would probably be more reasonable to call it 10 to 1, or 50/50. Or not to assign a probability at all, because quantifying disbelief is a tricky buisness. But then we would have simply returned to frequentist analyses – the objective analyses.

    But hey, even a prior probability of 10^-8 would have left a posterior probability of 7.3 x 10^-7 in favor of psi.

    And I would note that Ganzfeld experiments have been analyzed by Bayesian methods as well as frequentist methods. For a summary of that, simply look at the Tressoldi paper I linked to above in previous responses.

    - Johann

  67. steve12on 05 Sep 2012 at 5:51 pm

    I really don’t think it can be overstated that the Bayesian argument is unnecessary in showing that Bem did not find real stat significance. He performed exploratory analyses but treated them as confirmatory- by his own admission, adn we have statistical evidence that this took place. This is very clearly Type I error a go-go.

    This is not a small detail, either, especially when you’re dealing with small effects. And I don’t want to conflate effect size w/ scientific import. It is quite possible that a particular phenomena may be hard to detect for any number of reasons, leading to a small effect, but if it’s replicable (i.e., real) it may be of great importance. That said, those types of situations make it that much more important that the analyses are done and reported properly, and that you do not have unreported NS test after unreported NS test, then report the significant ones. It is more important when the effect size IS so small, and that’s what I find so objectionable about all of this.

    It’s obvious what happened, so we shouldn’t get lost in debating the most fair prior p, or debating the mechanisms that allowed for something that there is no evidence actually happened.

  68. BillyJoe7on 05 Sep 2012 at 5:59 pm

    Johann,

    “A nice article for the layman: http://discovermagazine.com/2010/apr/01-back-from-the-future/article_view?b_start:int=0&-C=

    I was hoping for a link to a quantum experiment that unequivocally demonstrates retrocausality.
    I don’t think an article for the layman in a popular science magazine makes the cut.
    At this point, and in desperation, I will withdraw the word “unequivocally” and just ask for a link to the quantum experiment that, in your opinion, best demonstrates retrocausality.

  69. jt512on 05 Sep 2012 at 6:47 pm

    Steve12 wrote

    I really don’t think it can be overstated that the Bayesian argument is unnecessary in showing that Bem did not find real stat significance. He performed exploratory analyses but treated them as confirmatory- by his own admission, adn we have statistical evidence that this took place.

    Yes, as I was about to respond to Johann, there is no point about arguing the relative merits of the Wagenmaker vs Utts prior in light of the likelihood that the data upon which both analyses were based are biased. However, I’m not aware (or I have forgotten) that Bem has admitted to doing anything inappropriate in analyzing or reporting his data. Do you have a source for that?

    Jay

  70. steve12on 05 Sep 2012 at 7:00 pm

    Jay – it’s in the Wagenmaker’s paper, pages 3 & 4:

    http://dl.dropbox.com/u/1018886/Bem6.pdf

    There’s noting wrong with his quote (if used appropriately), but it’s pretty clear by the instances from the actual paper pointed out on page 4, plus the unlikely sig rate across all 10 studies given his actual power (the Francis paper you referred to) that he follows his own advice in analyzing his data, but reports it as confirmatory rather than a fishing expedition.

    And let me be clear – he didn’t do anything that doesn’t happen in almost any pyschology lab (especially social). The painful lesson here is that psych, cog neuro, etc need to clean up our acts, because it’s slowing our scientific progress.

  71. steve12on 05 Sep 2012 at 7:07 pm

    “by his own admission” is too strong, actually. By his own philosophy + evidence is more accurate.

  72. jt512on 05 Sep 2012 at 7:56 pm

    Yeah, IIRC, Bem wrote a whole textbook chapter on how to milk a dataset.

  73. sonicon 06 Sep 2012 at 1:00 am

    It seems Bem’s tests are not enough to demonstrate anything.
    While I think they are OK to suggest further testing- it does seem that the data was tested after the fact (noting the difference between men and women on a particular test is interesting– but you then have to retest with an experiment designed specifically for that purpose). So while this is not a completely silly thing to do, it does mean that the results are not valid for statistical interpretation.

    It appears that the attempts to replicate changed the protocols in some ways that might be significant as well.

    This looks to be an example of a series of poorly designed experiments that allow people to interpret them as they see fit. Confirmation bias run amok.

    For a number of years I have hoped that the use of statistics would be constrained to situations where they are appropriate.
    I wish for world peace as well…

    No wonder I feel like Don Quixote sometimes… :-)

  74. BillyJoe7on 06 Sep 2012 at 8:34 am

    Don Quixote:

    “While mostly a rational man of sound reason, his reading of books of chivalry in excess has had a profound effect on him, leading to the distortion of his perception and the wavering of his mental faculties. In essence, he believes every word of these books of chivalry to be true though, for the most part, the content of these books is clearly fiction”

    :|

  75. BillyJoe7on 06 Sep 2012 at 8:37 am

    “At this point, and in desperation, I will withdraw the word “unequivocally” and just ask for a link to the quantum experiment that, in your opinion, best demonstrates retrocausality.”

    Okay, any damn quantum experiment that even remotely demonstrates retrocausality.

  76. Johannon 06 Sep 2012 at 12:49 pm

    Billyjoe,

    I don’t how to put this any better; just read the article. If you want to skip, start on page two.

    - Johan

  77. steve12on 06 Sep 2012 at 2:46 pm

    I do have a particular soft spot for the invocation of QM as a mechanism for “mysteries”. You know – the “the mind is mysterious and QM is mysterious so we need QM to describe the mind” kind of “reasoning”. In this case, it’s being used as a mechanism for a non-existent effect, which is sort of mysterious in and of itself.

    I must have posted it before, but this quote (about “What he Bleep Do we Know?”) from physicist/mathemetician Peter Woit always made me laugh:

    “In this version, he is saying perfectly sensible technical things about quantum mechanics, but they’re embedded in the middle of the nuttiness about QM promoted by the filmmakers (the usual: entanglement=we are all connected, superposition=anything you want to be true is true).”

  78. Johannon 06 Sep 2012 at 3:59 pm

    Steve12,

    There is a greater link between QM and consciousness in the minds of those who make the association than “one mystery must explain another.” In particular, invoking consciousness is the ONLY way to explain the results of the double slit under the orthodox interpretation of QM (also called the Copenhagen interpretation). This is the fact that troubled Eistein, Schrodinger, and Bohr – the possibility of a non-realist interpretation of reality – and was acknowledged by Wigner, Von Neumann, and other physicists that looked deeply into the measurement problem as a potnetial game-changer for our view of the world.

    You are fine to choose other interpretations (I myself lean towards a TSQM, or time symmetric formulation, as it is elegantly defined and strongly consistent with experiments conducted to date), but you would then have to take your pick among infinite universes arising infinitely from every point in time, reversed causality, superdeterminism and other counter-intutive ideas. However you put it, QM clearly has some major implications for our worldview, and it is unclear which interpretation will win the day. Given our uncertainty, and given that several models are compatible with psi, we should stop pretending that the prior probability of the phenomenon can be defined based on our classically Newtonian “common sense physics”, which is now only valid in a restricted domain. Futhermore, the reality of non-locality is almost universely acknowledged by physicists whatever the interpretation, and if one cannot see a potnetial connection between that and psi one needs to look harder.

    Finally, I do not associate myself with What the Bleep; I believe, like many, that it was a misleading one-way interpretation of QM and included lots of other nonsense as well.

    - Johann

  79. steve12on 07 Sep 2012 at 2:11 am

    I’m only addressing whether we need QM phenomena to understand the brain at the ecological level (clearly, as some level, there is a need). Right now there’s the same amount of scientific evidence for this as there is for psi: precisely none (unless one believes that stats are some sort of impressionisstic art form, then I suppose there’s evidence :) )

    Of course, it could be so. But it needs to be shown and has not been. There seems to be evidence that QM is required to explain photosynthesis, so who knows? MAybe there is. LEt’s formulate some predictions and test them validly.

  80. BillyJoe7on 08 Sep 2012 at 6:01 pm

    Johann,

    Unfortunately I’ve had to away unexpectedly for a couple of days so this thread may now be inactive.
    However:

    “I don’t how to put this any better; just read the article.”

    That’s the best you can do? Seriously, you have to be kidding. An article by a science journalist about a maverick scientist is your stand in for a quantum experiment that demonstrates retrocausality? That is supposed to convince me to take retrocausality seriously?

    Well, I won’t bother dissecting the whole piece, I’ll just draw your attention to this particularly revealing exchange between the journalist and the maverick scientist regarding the Rochester experiments. These experiments fail to support retrocausality and so the maverick scientist tries to save his hypothesis by linking it to free will. In other words, he pulls a rabbit out of the hat. But here is the quote:

    [Journalist:] The Rochester experiments seem to demonstrate that actions carried out in the future—in the final, postselection step—ripple back in time to influence and amplify the results measured in the earlier, intermediate step. Does this mean that when the intermediate step is carried out, the future is set and the experimenter has no choice but to perform the later, postselection measurement? It seems not. Even in instances where the final step is abandoned, Tollaksen has found, the intermediate weak measurement remains amplified, though now with no future cause to explain its magnitude at all.

    I put it to Tollaksen straight: This finding seems to make a mockery of everything we have discussed so far.

    Tollaksen is smiling; this is clearly an argument he has been through many times. “The result of that single experiment may be the same”, he explains, “but remember, the power of weak measurements lies in their repetition. No single measurement can ever be taken alone to convey any meaning about the state of reality. Their inherent error is too large. Your pointer will still read an amplified result, but now you cannot interpret it as having been caused by anything other than noise or a blip in the apparatus,” he says.

    In other words, you can see the effects of the future on the past only after carrying out millions of repeat experiments and tallying up the results to produce a meaningful pattern. Focus on any single one of them and try to cheat it, and you are left with a very strange-looking result—an amplification with no cause—but its meaning vanishes. You simply have to put it down to a random error in your apparatus.

    You win back your free will in the sense that if you actually attempt to defy the future, you will find that it can never force you to carry out postselection experiments against your wishes

    In other words he can’t lose!

    If the result of the Rochester experiments had come out as predicted by retrocausality, they would have vindicated his hypothesis. They did not. Let that sink in: these experiments did not demonstrate retrocausality. In fact they were just as you would expect if retrocausality was false. So now he turns if into a win for retrocausality AND free will.

    I truely hope you see the deceit entailed in the above.

    If not, here is the circular logic:
    If you assume retrocausality is true, then the experiments lend support for free will. And, if free will is supported by these experiments, then they (retroactively!) support retrocausality.

    Now, for that link to a quantum experiment that demonstrates retrocausality…. :|

  81. sonicon 08 Sep 2012 at 9:48 pm

    BillyJoe7-
    Bells’ Theorem experiments (e.g. the Aspect experiments) have been shown to rule out all theories of physics which assume locality, time- forward causality, and the existence of an objective real world. (Clauser et. al.)
    http://arxiv.org/pdf/quant-ph/0008036v1.pdf

    Three things that many want to be true– but at least one is not—

    1) the existence of objective reality
    2) locality (no spooky action at a distance– aka ‘magic’)
    3) time moves in one direction.

    We know we have to give up at least one of those three– if you want you can give up two or three of them…
    Which are you most willing to give up?

    Victor Stenger, for one, wanted to keep objective reality. Further, he does not want there to be any magic. He is a skeptic. Therefore time must go both ways–

    http://www.colorado.edu/philosophy/vstenger/Timeless/TimelessSkeptic.pdf

    So if time only goes one-way, then which of these statements is true?
    1) There is no objective reality
    2) One thing can affect another instantaneously at large distances (magic is real)
    3) Both 1) and 2).
    4) there is a fundamental error in physics as it is currently formulated.

    Perhaps there is a combination of the above you prefer.

    Choose your poison—- :-)

    BTW- the experiment most often associated with retrocausality is called ‘the delayed choice quantum eraser’. You can read about it at wikipedia — actually not a bad article…

  82. BillyJoe7on 09 Sep 2012 at 7:02 am

    sonic,

    The quantum eraser experiment does not support retrocausality.

    There are many experiments that demonstate non-locality, and I’m sure Johann would be able to link to one of them at the flick of the wrist, so it’s even more telling that he gets all limp wristed when asked to link to an experiment that demonstrates retrocausality.

    And that’s just a starting point, because he then has to demonstrate retrocausality at the macro level. We all know, Bem is bunk. But Johann has convinced himself that there is something there and that belief is bolstered by his belief that retrocausality has been demonstrated at the micro level and so then why not at the macro level.

    Regarding objective reality:
    If there is no objective reality, it’s pretty strange that there is pretty good agreement amongst scientists about what that reality is and how to manipulate that reality to suit our purposes (the computer in front of you, for example). I don’t suppose you would be willling to challenge objective reality by jumping out of a ten storey window. I would suggest that the path below would tend to make reality feel, shall we say, concrete.

  83. sonicon 10 Sep 2012 at 12:11 am

    BillyJoe7-
    You might be misunderstanding the situation.
    In physics today there are a number of interpretations.
    Each interpretation is capable of explaining the results of the experiments and fits with the mathematics.
    So it would be correct to say that every experiment demonstrates each interpretation- including those that suggest retrocausation.
    You say they show non-locality, but one could just as easily see the exact same experiment and say it shows retrocausality.

    BTW— If you read Stenger’s work (Timeless Reality) I’ll bet you would find the interpretation he is suggesting to be one you would like… here’s a sample–
    http://www.colorado.edu/philosophy/vstenger/Timeless/TimelessSkeptic.pdf

    Seriously, I thought it was crazy until I read the book. Now I understand why someone would prefer that way of looking at things. Seriously, give it a go…

  84. BillyJoe7on 10 Sep 2012 at 12:36 am

    Sonic,

    Interpretations are different.

    You may choose to interpret the quantum laser experiment as being consistent with retrocausality. But this is different from claiming that the experiment demonstrates retrocausality. It does not.

    I will read the Stenger article when I get the chance, but mostly I’m interested in what experiments show, not how they are interpreted. We don’t live at the quantum level so interpretations from our macroscopic viewpoint may actually make no sense at that level.

  85. Johannon 10 Sep 2012 at 2:45 am

    Billyjoe,

    In your selection, you simply bolded the part that was refuted in the second half of your selection; yes, if you conduct a single weak measurement without a future post-selection measurement, you get a spurious-looking result; that is the nature of weak measurments – their inherent error is very large. Only through intensive repetition can a pattern be seen to emerge. Tolakssen is simply making the point that, in a single trial, if we try to cheat the experiment to see it force us to carry out a post-selection step, it will not. Knowledge destroys the system.

    And Aharanov, Tolakssen, and Rueters are not crackpots; each is a well-respected physicist in his own right. Aharanov has published papers in Nature, and his work on the Aharanov-Bohm effect may yet win him a Nobel Prize. We cannot simply give people adverse labels because they advocate a position we consider self-evidently absurd, otherwise we would be putting all of the best physicists on the chopping block – because of our failure of imagination.

    Besides, if I gave you a link to the actual papers for these experiments, do you really think you would be able to understand them?

    - Johann

  86. BillyJoe7on 10 Sep 2012 at 8:47 am

    Johann,

    “if you conduct a single weak measurement without a future post-selection measurement, you get a spurious-looking result”

    That is your account, but this is what the author says:

    “Even in instances where the final step is abandoned…the intermediate weak measurement remains amplified, though now with no future cause to explain its magnitude at all. ”

    In other words, you get the amplified result even without the supposed retrocausation.

    ” Tolakssen is simply making the point that, in a single trial, if we try to cheat the experiment to see it force us to carry out a post-selection step, it will not. ”

    Maybe there’s a typo, but I simply can’t parse this sentence.
    But here is the view of the man who actually designed the Rochester experiments:

    “Andrew Jordan, who designed the Rochester laser amplification experiment with Howell, notes that there is even fundamental controversy over whether his results support Aharonov’s version of backward causality. No one disputes his team’s straightforward experimental results, but “there is much philosophical thought about what weak values really mean, what they physically correspond to—if they even really physically correspond to anything at all,” Jordan says. “My view is that we don’t have to interpret them as a consequence of the future’s influencing the present, but rather they show us that there is a lot about quantum mechanics that we still have to understand.””

    In other words, these experiments do not demonstate retrocausality, some merely interpret them via the lens of retrocausality.

  87. Johannon 10 Sep 2012 at 3:14 pm

    Billyjoe,

    This is the part that came directly after what you bolded:

    “Tollaksen is smiling; this is clearly an argument he has been through many times. “The result of that SINGLE EXPERIMENT may be the same”, he explains, “but remember, the power of weak measurements lies in their REPETITION. No single measurement can ever be taken alone to convey any meaning about the state of reality. Their inherent error is too large. Your pointer will still read an amplified result, but now you cannot interpret it as having been caused by anything other than noise or a blip in the apparatus,” he says.”

    As for the rest of your post, I read the article; I already know the objections. I stated very carefully in numerous posts above that other interpretations could be made about this experiment, and that this is not a conclusive anything. But the results are interesting, and the arguments against them a little tenuous in my opinion. In order to explain them away, the defintion of WQM (weak quantum measurement) must be changed from its so far mainstream interpretation. Also, there is disagreement on everything; the mere fact that other scientists dispute something is not cause to rejct that something as lacking support.

    Frankly, if you cannot see that the results of these experiments are interesting, and that they go some way to make the probability of precognition a little less extreme, that is your loss. I am done here.

    - Johann

  88. BillyJoe7on 10 Sep 2012 at 5:24 pm

    Johann,

    My point all along has been that retrocausality has not been demonstrated at the microscopic level. In effect you have acknowledged this above. My other point is that the “interesting” things that happen at the microscopic level have not been demonstrated at the macroscopic level. In my opinion, this keeps the prior probability of precognition as close to zero as to make no difference. On top of this we have the negative or flawed results of experiments in precognition. Therefore, in my opinion, we can safely ignore precognition as a going concern.

  89. sonicon 10 Sep 2012 at 5:29 pm

    BillyJoe7-
    The result of an experiment might be–
    On Nov. 3, 2009 at 14:37:34 EST detector A made made a click.

    What you say about the click is your interpretation of the data.
    Of course you can say– it was a photon that made the click– but that isn’t what you saw- is it?
    You might even think the photon came from some apparatus that you used to start the experiment– but again, is that an observation or an interpretation?

    What you think the experiment shows is your interpretation.
    Some see QM as showing that real particles that do not have any ‘magical’ power (no action at a distance) move forward and backward in time.
    And they see that in all the experiments that you think show non-local action.
    And if you read what they say it makes as much sense as the other both in terms of the experimental evidence and the mathematics.

    I’m not saying who is correct–
    But you should recognize that your interpretation starts as soon as you start talking about anything beyond an exact description of the exact observed action.

    BTW– I think the use of ‘weak measurement’ will prove to be of little use. They aren’t measurements at all…

  90. BillyJoe7on 11 Sep 2012 at 7:44 am

    sonic,

    Nothing in these experiments looks like retrocausality. Things happen simultaneously. There are no data that provide evidence that one event goes back in time to cause the other event. They just happen at the same time. I would truly love to see a demonstration that this is not the case

  91. sonicon 11 Sep 2012 at 7:58 am

    BillyJoe7-
    Try this–
    http://www.colorado.edu/philosophy/vstenger/Timeless/TimelessSkeptic.pdf

  92. Mlemaon 12 Sep 2012 at 8:45 pm

    sonic:
    you always provide the best links!
    I read them all, but I thought it might be redundant to thank you for each one.

    But i love reading things that make me think in new ways about the universe.
    negative side of the t-axis?
    …very nice…
    :)
    thanks man

Trackback URI | Comments RSS

Leave a Reply

You must be logged in to post a comment.