May 2004
by Steven Novella, MD

Armchair skepticism is fine as far as it goes. It is helpful to be conversant with skeptical philosophy, the principle of Occam’s razor, and the many mechanisms of self-deception. It is important to be able to identify where the burden of proof for a claim properly lays, and to have a healthy disdain for anecdotal evidence. A skeptic can also successfully counter arguments that a given phenomenon is “unexplainable” (and therefore must be paranormal) by offering possible mundane explanations (in addition to pointing out the logical fallacy of making an argument from ignorance). But when confronted with a believer who makes a specific claim to having some paranormal ability or supernatural power, it is difficult to impress the public with such abstract arguments. One carefully conducted test is often much more effective and definitive in challenging such a claim than all the abstract arguments even a well-versed skeptic can muster.

Although most skeptical organizations, the NESS included, profess to investigate paranormal claims, there are only two domestic full-time paranormal investigators of whom I am personally aware: Joe Nickel, the senior investigator for CSICOP, and James Randi. (Outside the US, Massimo Polidoro is an investigator in Italy who trained with Randi and runs the Italian skeptics group: Premanand is the Randi of India and specializes in exposing gurus.)

For years James Randi, one of the skeptical movement’s few celebrities, has confronted the psychics and charlatans of the world with a simple challenge – if you really have the abilities you say then all you have to do is prove it and you will win a large cash reward (not to mention give skeptics everywhere the ultimate smack down). In the humble days of the Randi Psychic Challenge, James carried on his person a check for $10,000. In recent years, however, the James Randi Education Foundation (JREF) has had over $1 million in a special prize account, just waiting for one legitimate conjurer.

Of course, many self-professed psychics and other purveyors of legerdemain do not like being tested. A reasonable person might suspect that this otherwise uncharacteristic shyness might be due to fear that they will not pass a rigorous test of their alleged abilities. The psychics have many more creative reasons. Randi reports that he has been given excuses such as: they are not interested in money (then give the prize away to charity, or stop charging $300 an hour for personal readings), they don’t believe the money really exists (it is in a verifiable account), Randi cheats (test conditions that preclude cheating and to which both sides agree are arranged, even the complete absence of Randi from the testing procedure if desired), and the always popular – their abilities do not function under test conditions (ahem!).

Another common tactic is to agree, publicly and enthusiastically, to the challenge (thereby reaping all the PR benefits of agreeing to be scientifically tested) and then avoid Randi like the plague. Medium Cynthia Browne has practically made a career of this latter strategy. Randi keeps a day count on his website of how many days have passed since Cynthia agreed on the Larry King Live show to be tested by Randi.

The process of the challenge is simple, applicants must state in an unambiguous way what paranormal, supernatural, or occult power they possess. The JREF does not carry out the tests directly, but rather helps design the test and oversees the conditions and procedures. The applicant may participate and approve ahead of time on all aspects of the test.

Actually, the testing is a two step process. Applicants are often first referred to local associates for preliminary testing. This is to weed-out those who cannot get past the stage of outlining what exactly their ability is or how it can be tested, and those who have no ability at all and therefore cannot pass even a simple test. So far, according to the JREF, not a single applicant has passed the preliminary test.

I am proud to say that in New England, the NESS is the JREF associate that performs such preliminary tests. To date several proposals have been sent our way, although we have only carried out one test to its conclusion.

The Coin-Tosser

The first contestant to come our way had a very simple claim—that he could mentally control the outcome of random coin flips. Well, his claim started out as very simple. There is a tendency for such psychic claims to become increasingly complex once the process of designing a rigorous test begins.

In this case, through the course of discussion, I learned that the coin-tosser could not so much “control” the outcome of coin flips as “influence” them. You see, he believes he can increase the percentage of flips that turn out heads or tails, but he can’t control which it will be. He discovered that if he flips a coin hundreds of times either heads or tails will show up more frequently then would be predicted by chance alone. I understand the CIA is considering spending 20 million dollars to see if they can put this impressive ability to the work of espionage. Never-the-less, at least this claim is easily testable.

Although this may be a false dichotomy, we frequently ask ourselves if an individual who claims to have psychic ability is self-deluded or a con-artist. When I finally fully understood the nature of the claim I began to suspect that the coin-tosser was simply a victim of statistical naiveté, a common trait in our species. This belief was confirmed, in my opinion, when I further learned that in order to obtain his statistical results, the coin-tosser must “warm up” for an undetermined amount of time, and his powers wear off after a while. In other words, he was engaging in good-old-fashioned optional starting and stopping.

Optional starting and stopping is a favorite of ESP researhcers. (In fairness, serious researchers have abandoned this method years ago, but it still crops up with less rigorous researchers.) Basically, you run a long series of trials and pick some stretch of data in the middle that, taken by itself, is statistically different from chance results. This, of course, completely invalidates the analysis, because if you can choose any chunk of data you wish from a large sample you can achieve whatever results you desire. Legitimate researchers have a different term for this—fraud.

Our coin-tosser also rediscovered another favorite of ESP researchers—that of counting negative hits. That’s right, sometimes psychics miss the target more often than one would predict by chance (again, only if looked at in isolation, and the decision to count misses is made only after the data is analyzed). The coin-tosser’s equivalent of this strategy was counting heads or tails, and allowing himself the luxury of deciding which only after the fact.

In the final analysis, all of these questionable statistical strategies achieve the same thing. They increase the probability that there will be some random fluctuation in the data that, if looked at in isolation, would seem to defy chance, and they allow for the selection of this subset of data after the fact.

So, I patiently explained this to the coin-tosser and together we designed a protocol that would set out, prior to the acquisition of any data, the exact way the data would be analyzed and the criteria for considering the test positive. Once this was all painstakingly worked out, the coin-tosser decided that he would abandon coin tossing and perhaps he would work on a system of influencing random numbers generated by a computer. I haven’t heard from him yet, I wonder how that’s working for him.

One final testament to the coin-tosser’s inability to grapple with statistics— he was somewhat disappointed with the fact that Randi would not award him the 1 million dollar prize for proving his abilities to a P-value of 0.05 (which basically means that the probability of the results being due to chance alone is 5%). If that’s good enough for medical research, he argued, then why wouldn’t it be good enough to prove his abilities. I tried to explain to him that Randi has no intention of handing 1 million dollars to every 20th person (on average) who takes the challenge. He didn’t get it (literally and figuratively).

The Mind Reader

The next applicant we were asked to screen believes that he has the ability to read minds. The first step in testing such a claim is to get as much information about the applicant as possible. The goal is to get them to commit to as many specific details as possible, and finally to sign off on a testing protocol, stating that it is a fair test of their abilities. We want to avoid having the applicant complain after they fail that they neglected to inform us earlier that their powers do not work on Tuesdays.

So I started by asking the mind reader to tell me, in as much detail as possible, exactly what he can do—being careful to spell out any special conditions or requirements. His first response was fairly straightforward, “I can pretty much tell you exactly what is on your mind,” he assured me confidently. OK, that seemed simple enough. I figured an old-fashioned card reading test would suffice. I asked some standard directed questions to help create a protocol: any limit on distance to the target mind you are attempting to read, any problem with having “skeptics” in the room, do you have to be able to see the target? No problems there.

When I suggested the card reading test, however, he informed me that he does not do well with cards, or with numbers. They are too easy to get confused, he complained. They are also too concrete to misinterpret misses as hits, I thought. After suggesting a few alternatives, he decided that he does best with sentences or phrases. So it was decided that the subject matter of the test would be commonly known phrases or sayings, as would be used, for example, in charades. However, the criteria for counting a hit would be an exact match. This is to avoid claiming after the fact that vague similarities between his guess and the target phrase should be counted as a hit. He agreed, happy with the final protocol I sent him. His only final stipulation was that he be allowed to have a few beers prior to the testing, as this was necessary to relax him. We were happy to oblige.

The downside to using common phrases is that it made statistical analysis impossible, because there is no way to quantify the number of possible phrases to choose from. I suggested that we set the threshold for passing at 25% correct, and the mind reader readily agreed. Frankly, I would have been surprised if he scored even a single hit.

The testing was actually fairly straightforward. Forty target phrases were written on separate pieces of paper prior to the testing. The target subject, a friend who was not known to the applicant prior to the testing, would choose a random phrase, place it in front of them, and concentrate on the phrase for the duration of the trial. The mind reader sat across a long table from the target, with a low barrier in place to prevent him from seeing the paper, but allowing him to see the face of the target. Twenty trials were completed, in each case the target phrase was recorded, and the applicant wrote down his own response without saying it out loud or asking any questions. The entire process was video taped.

When the trials were done and the results tallied, we were not surprised to find that the applicant scored 0 hits out of 20. He was not even close on a single one. We were a bit surprised, however, on just how far off his responses were. In fact, after reviewing all of his responses it became clear that he did not quite understand the nature of the task, even though he proposed it and the details were carefully reviewed with him. His guesses were not so much common phrases as random thoughts, leading us to speculate that perhaps he was mistakenly reading his own mind.

The mind reader, after hearing the correct phrases, asked if he could have another try. We happily agreed, although made it clear that the official testing was over. We ran through another ten trials, since we had plenty of unused written phrases available. The applicant continued to score zero hits.

The applicant was very confused and even apologetic at his failure. But he quickly regained his confidence and asked if he could be tested again. According to the official rules of the psychic challenge applicants must wait one year before applying to be retested.

Conclusion

From discussions with staff at JREF our experience is all too typical of applicants to the psychic challenge, especially recently. In the early years Randi was faced with some applicants who could perform impressive demonstrations, and the trick was in figuring out what conjuring tricks they were employing. Recently, however, applicants have been of an increasingly less impressive sort – more naïve and self-deluded than anything else. I suppose this is due to the fact that anyone claiming to have paranormal abilities who is at all aware of what they are doing has already figured our that they are not going to fool the investigators for the psychic challenge. Therefore, all that is left for the challenge are the self-deluded.

The testing, however, still serves a critical function in the skeptical community. We must have a formal and systematic way of standing up to paranormal claims, and the skills necessary to detect fraud and deception must be maintained and even updated by skeptical activists.

For me, the ultimate goal is to increase the level of scientific literacy in our society. Paranormal claims garner a tremendous amount of public and media attention, however they are usually presented in the context of bad science. But such claims represent an opportunity for the skeptical community to teach the public about the proper methods of science, the pitfalls of illogic and self-deception, and the reality of fraud for self-promotion.

James Randi has played a leading role in establishing first hand testing of paranormal claims center-stage in the skeptical movement, and he is to be admired for his efforts. It is incumbent upon all of us to continue, and even expand upon, his tradition when the sad day comes that we no longer have his light to guide us.

For more information on the Randi Psychic Challenge, go to: http://www.randi.org/research/index.html

Acknowledgements: Thanks to the NESS investigation team, Bob Novella, Perry DeAngelis, and Evan Bernstein, for their efforts in carrying out our investigations. Also thanks to Jacek Chodniki for volunteering to be the target in the mind reading tests.

Filed under: Pseudoscience, Skepticism, paranormal, psychic, Randi