Jan 15 2013
Defending the Million Dollar Challenge
You are currently browsing comments. If you would like to return to the full story, you can read the full entry here: “Defending the Million Dollar Challenge”.
49 Responses to “Defending the Million Dollar Challenge”
Leave a Reply
You must be logged in to post a comment.
No problem. Just tell the applicants we’ll be happy to change the standard of the challenge to p < 0.05, in exchange for an application fee of $50,000. We'll put our money on the line if you will.
Yes – there’s a lot of whining about the challenge from applicants (or those who refuse to apply) – but the bottom line is that they could receive $1 million dollars just for doing what they claim they do every day.
Would a Bayesian approach not be more intuitive here? It seems this ‘floating’ threshold just provides these scammers with yet another excuse.
I read his criticism and some of the comments on his site. Outside of straw man arguments, I think the biggest problem Volk has is that he does not properly understand science or more importantly, the statistics most commonly used in science. For example, he claims that certain applicants and psi types have not gotten enough trials, and commenters point out a need for a big sample. What is entirely missed though are two important things: 1) Bayesian prior probability and 2) statistical power. With regard to the former, making a claim of psychic powers (or something similar) is a fantastic one; one in which we have no objective evidence of having ever occurred. Beysian probability means that we take this into account when formulating our proposed p value, as .05 would be far to generous given the fanatic level of claim (same that we should do with homeopathy and other very unlikely claims). Basically, this means that psychic power claims do not hold the same level of possibility as a claim that a particular known medical treatment. With regard to the latter, we only need more trials for a study or observation when an effect size is very low; this is why studies with very large n’s often publish findings of little clinical or scientific utility, as with increasing sample size, we often find miniscule effects that are often due to statistical noise (particularly the case when multiple studies or multiple statistical analyses are run – which is why is is considered proper to run post-hoc corrections and state hypotheses and directionality apriori).
The main problem is that most people – even some very educated people, lack knowledge of the scientific method and basic statistics. Without this foundation, it is near impossible to have a conversation with them. It is not something you can look up on Wikipedia and form an opinion, it takes multiple science and math courses over time, combined with a willingness to engage in critical thinking. This is one reason why I think the ethics/morality debate in previous posts stagnated after a few initial comments; as those who disagreed with Steve (well the main individual at least) appeared to lack some foundational understanding of science which precluded a discussion in which certain foundational concepts could be agreed upon or properly understood.
My thoughts were that we avoid the whole p-value debacle when we use Bayesian methods, and it would seem that diffuse priors would be more than fair for any claims of psi ability. I think the inherent hierarchy within the Bayesian framework, and the ability to estimate/compare models that have converged upon realistic posterior distributions might lend itself to exposing these frauds…
If I’m not mistaken, even the subtlest claim can be demonstrated to an arbitrary p-value, it just takes a lot more tests. For example, if I claim I can predict a coinflip at least 51% of the time, then we would need to flip a coin [N] times (I can’t do the math right now).
Of course, that’s beside the point when it comes to the central purpose of the Challenge, which is indeed to deal with claims which, if quantified, should be very easy to test. That quantification part is at the core of this, to some extent. “My miracle product works almost all the time! Note that by ‘almost all’, I certainly don’t mean some geeky numbers-y thing like ‘More than 50% of the time.’ “
Lenoxus- you are correct. The point of the criticism that “enough trials” would be impractical, and so the challenge is rigged in that way. They miss the point, as above.
Talk of p-values and Bayesian analysis is good when discussing the scientific study of psi, etc. but all irrelevant to the challenge. The challenge is not a scientific study. The point is to get applicants to say they can do something specific at least a certain amount of the time, and then to test that specific claim. The thresholds are set so that it is unlikely someone will win $1 million dollar by just getting lucky, but also to properly demonstrate the applicant’s powers if they are real.
The whole statistical significance thing is a red herring.
Also,the applicants have to agree in advance that they are satisfied that the test is fairly set up,and at least some of the time (if not always) they get to do some unblinded dry runs to make sure that their “powers” are able to function in that setting.It is only afterward that they almost universally,start finding excuses why they were cheated or what malfunctioned.
Steve, when you say that the purpose is to set a threshold so that is unlikely someone will win due to just getting lucky, this seems similar to making sure we set a significance level so that some random noise won’t be interpreted as a genuine effect. I get that the goal is not to set up a study per se, but the ideas behind setting up a controlled situation rests on similar thinking, no? This is basically a semantic distinction anyway – I agree with your premise set forth above regarding the shenanigans used by Volk and others. I talked about Beyes and effect size, in part, because I had gone over to his blog and read him and the commenters also discussing psi research as if statistics had proven it to exist (I believe they even mention Behm’s stuff without an understanding that most think the studies were due to faulty design and statistical analysis).
“The result is that an applicant can—and did—achieve statistically significant positive results, yet was deemed to “fail” the challenge.”
This was the case of the girl with x-ray vision who claimed 100% accuracy in diagnosing medical symptoms. Her post hoc rationalization for missing the person who had an appendectomy was ‘his appendix grew back.’!!!
‘The whole statistical significance thing is a red herring.’
Exactly, forget about statistical significance, but the probability that any one of these claimants can demonstrate ability better than chance still needs to be framed coherently. How else do we do this?
You’re damned if you do and damned if you don’t. If it isn’t scientific, they’ll run with that when they fail, and if it is scientific, they’ll move the goalposts when they fail.
I say the challenge should give the money to charity…none of these butterheads (lookin’ at you, John Edward) will ever accept a test that will reveal their abject immorality.
Just my 2 cents. Thanks for the post!
If some psychic is gifted with only subtle powers, difficult to statistically detect, then he/she should simply ignore the Randi Challenge and go to Las Vegas instead. There, a tiny nudge of a few percentage points turns the whole city into a money machine.
That’s why there are so many wealthy psychics, right?
The fact that so many people who hold themselves out to be “psychic” or whatever DON’T take the challenge. is a difficult one to explain away. Not just to the skeptically inclined, BUT to ordinary people too.
That’s the real benefit of the challenge, and why charlatans and grifters hate it so much: They can’t explain away why they won’t take the challenge, in a way that passes the man-on-the-street smell test.
Bloviate objections like Volk’s might play well in his echo chamber, but to the the ordinary person, it quacks like a duck.
Even a deluded “true believer” would (and often does) take the challenge. But Volk, and John Edward, and low-lives like them won’t, ever.
There is only one reasonable conclusion to be drawn: That they already know their shtick is bulldust…
.
RickK- Why should a ‘psychic’ waste their time and money in Vegas where gullibility works against them? Their business plan is to get gullibility to work in their favor,and they can do it from their home.
I’m not an “amateur magician” – that might have been corrected…
Also, you can’t say that the MDC involves “statistical analysis.” All tests are designed so that the results are obvious, don’t require such an analysis, and either the required odds are achieved, or not…
The 2 phases — preliminary test and formal test — pretty well insure that a “chance” win could not happen. In any case, no one has ever made it past the preliminary test.
And, we’ve always insisted that there’s no fee, no “bet” involved. If we asked for a fee, as one person suggested, they’d have good reason to not apply, and it’s never been a “bet”… Just do what you say you can do, and you pass the preliminary and then pass on to the formal stage.
*I have so many passwords – some 40+ of them, most with specific, overdone, rules and required inclusions (a capitalized word, a digit, a space, etc.) – that I can’t keep track…
Steven said,
“The purpose of the challenge is not to design and run scientific experiments, and it is not to scientifically prove or disprove the existence of the paranormal or any particular supernatural phenomenon. Randi and the JREF have always been crystal clear about this. Rather, the point of the challenge is to be a public demonstration. There are many people who claim dramatic paranormal abilities. If their claims were anything close to the truth, it should be easy to demonstrate their abilities. As Randi says – he is only asking them to do what they claim to do on a daily basis, just under conditions that make it impossible (or at least very difficult) to cheat. That’s it.”
This is all just weasel words. You’ve often said that the scientific method is the only reliable way to test the truth of a claim. So either the MDC is a scientific test or it isn’t. Full stop. If it isn’t then it is not a reliable way to test these paranormal claims and therefore no merit should be placed on the results of the challenge either way. Furthermore, if the MDC tests are not proper scientific experiments then why are people like yourself, Richard Wiseman, Ray Hyman, etc, drafted in to help “design” them?
Mnestis said,
“The main problem is that most people – even some very educated people, lack knowledge of the scientific method and basic statistics.”
Then the Amazing Randi himself said,
“Also, you can’t say that the MDC involves “statistical analysis.” All tests are designed so that the results are obvious, don’t require such an analysis, and either the required odds are achieved, or not…
The 2 phases — preliminary test and formal test — pretty well insure that a “chance” win could not happen.”
Oh the irony…
david – you are committing a false dichotomy logical fallacy. Experiments are not either completely scientific or unscientific – full stop – as you suggest. There is the full range of scientific rigor from completely uncontrolled to exquisitely designed and executed studies.
The million dollar challenge protocols are scientific in that they are controlled observing conditions, try to isolate the variable in question, and involve multiple trials and some statistical analysis. But they are not designed to be rigorous to the point of being publishable in the peer-reviewed research. They are certainly not designed to detect subtle effects. They are also not designed to answer a scientific hypothesis, but rather to test a specific claim by an individual.
What Randi means is that they don’t calculate p-values or anything like that. They do enough statistical analysis, however, to calculate the odds of achieving the predetermined threshold by chance (not the kind of analysis done in a scientific paper, but useful for the purpose of the challenge).
The methods used are valid and scientific and sufficient for the purpose of the challenge, they are just not designed to serve as scientific research. They are not designed to answer “is psi real” but rather “can this person do what they claim they can do.”
“david – you are committing a false dichotomy logical fallacy. Experiments are not either completely scientific or unscientific – full stop – as you suggest. Experiments are not either completely scientific or unscientific – full stop – as you suggest. There is the full range of scientific rigor from completely uncontrolled to exquisitely designed and executed studies. ”
Steven – I’m not committing a false dichotomy logical fallacy. Obviously, anything that claims to be a “test” of something doesn’t automatically qualify as being a scientific experiment merely by virtue of being labelled a “test”. So clearly, tests can be categorised as scientific and unscientific by definition. Those that apply the scientific method are defined as scientific tests and can vary on the dimension of experimental rigor. So the issue is whether or not the MDC tests are meant to apply the scientific method or not. And we have a clear statement from you saying that, yes, they ARE meant to apply the scientific method, thus qualifying them as scientific tests. Whether or not they are designed as good or sloppy scientific tests is a secondary issue.
This is, of course, in complete contradiction to your previous statement that “the purpose of the challenge is not to design and run scientific experiments”. I realise you are claiming the MDC tests are not meant to be rigourous, publishable in scientific journals, or even test a specific hypothesis. So in summary, you are essentially saying that the MDC tests are poor scientific experiments. If that’s the case, just say so.
They are not designed to answer “is psi real” but rather “can this person do what they claim they can do.”
Yes, I realise that, and as a modus operandi for investigating claims of the paranormal it’s about as myopic as it gets. It’s like only being interested in testing someone’s claim that they can lift a 10 tonne weight with their bare hands and then claiming that nothing remarkable has happened when they lift 2 tonnes instead.
I am an applicant for the Million Dollar Challenge. I announced my claim at Tam 8 in Las Vegas. I used to be a pharmacist and by accident discovered an ability that satisfies the criteria for the MDC. I am working with Dr. Novella and others to develop proper protocol. I have an ability to restore normal feeling and skin sensation to people that have permanent post-operative numbness and sensitivity from certain types of surgeries. The types of surgeries that often result in permanent numbness and adhesions that I have done this process on most frequently are: c-sections, hysterectomies, gall bladder, appendix and breast cancer. The process takes a few minutes. I stand 4-6 feet away from the person and simply ask where the affected area is and do a process. A bystander usually cannot tell that I am acting any differently. This may also be done over the phone or on skype. The tests are to be performed on subjects that are not neurologically compromised because of medication or chemicals they may be using. These results are medically provable using tests and equipment that a doctor or neurologist would use to assess neurological function and tissue dynamics to verify that it is not a subjective interpretation or placebo effect. Many medical devices exist to evaluate touch, pain, vibration or temperature, skin sensitivity, changes in sensation with light and deep pressure, and changes in tissue pliability from adhesions and scar tissue. I do this for free and have videoed this hundreds of times on a large diversity of conditions. There are also other tissue and function changes that may be measured scientifically. I believe that science should be open and transparent to enhance discovery. I consider the MDC as a science grant offered for furthering innovative research that I have been doing. I thank James Randi and the JREF for this opportunity.
Steve, I believe you are being Zached.
Rick,
any psychic powers greater than the percentage of the house in Vegas would be easy to detect.
The house odds have to cover cheating, and in the better places, complementary food and drink.
No probably if you can outperform chance only marginally you are best going into stock markets, or other financial markets. Clearly the place to look for those with supernatural powers, although my friend who retired well before 40, went from theoretical physics (not profitable) to derivative trading, and attributes it all to his abilities at mathematics not magic.
Simon,
Of all the people who went into derivative trading, someone had to get lucky.
David – you are just doubling down on your initial logical fallacy. Tests are not scientific or unscientific – that is a blatant false dichotomy. There are many features of a “test” that would make it scientific. There is no magic threshold of these features at which a test becomes scientific. Your attempt to label this as quality (a good vs poor scientific test) simply uses semantics to distract from the fact that there is a spectrum and your contention remains a false dichotomy.
You also completely missed the point about the MDC – It is scientific in the methods that it uses, it is not scientific in the kind of questions it asks (not “is this hypothesis valid” but “can this person perform this specific task as claimed”). These are two separate criteria, the latter not having anything to do with rigor, and so your conclusion that this is equal to a poor scientific test is not valid. Rigorous methods can still be applied – and they are.
Finally, the 2 ton vs 10 ton analogy is not valid for two reasons. One – it has nothing to do with the probability of guessing correctly. You can’t lift 2 tons by accident. Second, the threshold of success for trials is set by mutual agreement of the MDC and the applicant. It is specifically set low enough so that if any genuine paranormal ability were demonstrated it would count as a success (eliminating false negatives). For the preliminary test the threshold is set as low as possible while still minimizing the chance of random success. This varies from test to test, but 1/100 to 1/1000 is typical. In the example linked to in the article the threshold was set at 1/50. That is pretty generous.
Tests are not scientific or unscientific – that is a blatant false dichotomy.
Ok, let’s use an example. I want to test whether this crystal substance in front of me is sodium chloride so I taste it and declare that, yes it is, because it tastes like salt. Is that test scientific or unscientific?
There are many features of a “test” that would make it scientific.
So if those features are present in the MDC tests then one can regard the test as scientific. Of course, there will be disagreement on what particular features to include, there always is.
There is no magic threshold of these features at which a test becomes scientific. Your attempt to label this as quality (a good vs poor scientific test) simply uses semantics to distract from the fact that there is a spectrum and your contention remains a false dichotomy.
Well, there sort of is a magic threshold. That threshold is set by agreed upon definitions of what constitutes the scientific method. If we intend to use that method in our test, then it can be regarded as a scientific test. I understand your point that a test can vary on the degree to which the scientific method is adhered to. There are good and sloppy tests and different kinds of methods can be used in different kinds of tests answering different kinds of question.
You also completely missed the point about the MDC – It is scientific in the methods that it uses, it is not scientific in the kind of questions it asks (not “is this hypothesis valid” but “can this person perform this specific task as claimed”).
Actually, both types of question are regularly asked in science and you know it. HM, for example, claimed to have lost his ability to form new memories so that claim was tested by running tasks and observing his performance under controlled conditions. But of course, scientists did not simply test whether HM’s perception of his own memory ability was accurate and reliable (which is as far as the MDC would go). Rather, scientists actually thought about the situation and ran a battery of tests that were designed to find out the real extent of his memory performance. This isn’t done when the JREF “investigate”. They are only interested in testing whether the claimant’s perception of their own ability is accurate. This is an extremely myopic approach to investigating claims of the paranormal.
These are two separate criteria, the latter not having anything to do with rigor, and so your conclusion that this is equal to a poor scientific test is not valid. Rigorous methods can still be applied – and they are.
Sorry, I can’t extract much meaning from this. The latter question has nothing to do with rigor yet rigorous methods can still be applied? What are you trying to say?
Finally, the 2 ton vs 10 ton analogy is not valid for two reasons…
You misunderstand the point of my analogy. It was to highlight how short sighted it is to investigate a paranormal claim merely by examining whether the claimant is accurate in their own perception of their ability. A much more interesting question is whether the claimant has any ability at all. I suspect the JREF adopts this approach from a biased experience of dealing with truly deluded people. However, there are people out there with genuinely mysterious experiences that usually happen spontaneously and who don’t claim any special ability to control these experiences at will. I’m not sure why the JREF is not interested in going after those people like parapsychologists try to do.
One final comment. Previously, you said this about the MDC tests:
They are not designed to answer “is psi real”
and later you said:
the threshold of success for trials is set by mutual agreement of the MDC and the applicant. It is specifically set low enough so that if any genuine paranormal ability were demonstrated it would count as a success
The bolded part contradicts the previous statement.
davidsmith,
I think you are the one with the myopic view.
Forget about the little pieces and stand back and get the whole picture, then it should become clear.
Of course, you have an agenda, so this may not work for you.
One little example:
Steve said the MDC is not designed to test: “is PSI real”.
He also said that “if any genuine paranormal ability were demonstrated it would count as a success”.
These statements are both correct.
The MDC is designed to test: “can this individual do what he says he can do (that seems to be an aspect of PSI)”
In other words, the focus is on this individual and what he says he can do.
Obviously, they would not be interested in testing his claims if his claims were not of a paranormal nature and, if he succeeded, this would lend cedibility to PSI, but that is not the focus of the MDC.
The MDC was set up because there were, and still are, many in the public spotlight who claim paranormal ability, thereby giving the public the impression that the paranormal exists. The purpose of the MDC was to demonstrate that they really cannot do what they think they can do, thereby giving some pause for thought amongst the public inclined to believe that the paranormal exists.
“You misunderstand the point of my analogy. It was to highlight how short sighted it is to investigate a paranormal claim merely by examining whether the claimant is accurate in their own perception of their ability”
But that’s exactly what they set out to do (test a paranormal claim), so what better way than test that paranormal claim? How is that shortsighted? They’re just doing what they set out to do.
“there are people out there with genuinely mysterious experiences that usually happen spontaneously and who don’t claim any special ability to control these experiences at will. I’m not sure why the JREF is not interested in going after those people like parapsychologists try to do.”
That is a bold claim.
I hope you have evidence to back up that claim.
Meantime, let us just leave it to the parapsychologists to do what they are interested in doing, and the JREF to do what they are interested in doing.
BillyJoe7,
Most of us have an agenda. Mine is to find out the truth, what’s yours?
Steve said the MDC is not designed to test: “is PSI real”.
He also said that “if any genuine paranormal ability were demonstrated it would count as a success”.
These statements are both correct .
They contradict each other. If the test is designed so that a particular result would count as a success if genuine paranormal abilities were demonstrated, then it logically follows that the test has demonstrated that psi is real (if we’re defining “psi” and “genuine paranormal abilities” as the same thing).
The MDC is designed to test: “can this individual do what he says he can do (that seems to be an aspect of PSI)” In other words, the focus is on this individual and what he says he can do. Obviously, they would not be interested in testing his claims if his claims were not of a paranormal nature and, if he succeeded, this would lend cedibility to PSI, but that is not the focus of the MDC.
The JREF states that the purpose of the MDC is for someone to show, under proper observing conditions, evidence of any paranormal, supernatural, or occult power or event. Therefore, it follows that according to the JREF, a successful MDC test would provide evidence for the notion that psi is real.
How is that shortsighted? They’re just doing what they set out to do.
Because, here, “what they set out to do” only considers testing whether a claimant’s perception of their own ability is accurate. See my HM example. (However, we also find that it isn’t really what they set out to do, hence the contradictions)
That is a bold claim.
I hope you have evidence to back up that claim.
Yes I do, but I certainly don’t want to spend page after page arguing about how to interpret spontaneous case studies when the end result will be no different than when we started. In any case, are you genuinely interested or just being flippant?
Meantime, let us just leave it to the parapsychologists to do what they are interested in doing, and the JREF to do what they are interested in doing.
Oh right, so I can’t comment about what I think of the JREF’s approach now?
A scientific test is designed to answer a specific hypothesis to a specific degree of statistical significance. The correct way to calculate the statistics is with Bayesian analysis, which also depends on the prior probability. For a skeptic such as myself, the prior probability of psi powers is very low, a reasonable estimate would be one chance in 10^100 that it is correct.
A test that would have significance on that level would require something like 350 consecutive coin flips predicted with 99.999 (total of 100 ’9′s) accuracy. That is what a “scientific” test requires to change my prior of whether psi exists or not, because my estimate of the prior plausibility that there is psi is so low.
A 1 in 100 or 1 in 50 for a preliminary test is a gigantic relaxation of that. Even if someone won the MDC, it would not change my prior plausibility estimate unless it was done with a very high degree of scrutiny and to very high statistical significance, on the order of 1 in 10^100.
Davidsmith,
My agenda here was to correct your misconceptions about the MDC. I have no stake in the MDC. In fact, I think it has just about outlived its usefulness. I also do not agree that they should relax the odds. With odds of 1 in 100, someone will eventually win the preliminary and claim victory and cry foul when not presented with the million dollars. Or others will cry foul for them. The public misperception that is likely to follow will be very hard to put right.
But, yes, I have never seen any evidence of paranormal phenomena, so I don’t hold a belief that paranormal phenomena exist.
” If the test is designed so that a particular result would count as a success if genuine paranormal abilities were demonstrated, then it logically follows that the test has demonstrated that psi is real”
There are several things wrong with that statement.
Firstly, the test is not designed so that a particular result would count as a success if genuine paranormal abilities. The test is designed to demonstrate if a particular individual can do what he claims he can do. The claim must be something that the JREF considers to be a paranormal claim, otherwise they would not be interested in testing him. But they are interested in testing this particular individual’s particular claim They get him to describe in sufficient detail exactly what he thinks he can do. Then they design a protocol to test that claim, and they do this with sufficient input and full agreement of that individual.
Secondly, a successful test would not demonstrate that PSI is real. It would provide evidence in support of PSI. But, because the prior probability of PSI is so low, if would not raise the probability of PSI being real by much. If the odds of a successful result was 1 in 50 , even 1 in 100, a successful result would not shift the probability that PSI is real hardly at all.
Thirdly, a successful result would mean that this particular individual was able to pass a test of his claimed abilities with a 1 in 100 chance that this was just a lucky result.
“Because, here, “what they set out to do” only considers testing whether a claimant’s perception of their own ability is accurate”
But this has nothing to do with being short-sighted. They have no intention of taking on the whole paranormal field. They have decided that they will attempt to counteract the public perception that there are people with paranormal ability by testing these individuals under controlled conditions and offering a million dollars to induce them to take part. Put upmor shut up. The failure of prominent public figures who claim paranormal ability to take up the challenge and the failure of those taking on the challenge to take out the million dollars and the ridiculous reasons they give afterwards for their failures are clear successes of the MDC in counteracting the public’s perception that PSI is real.
“I certainly don’t want to spend page after page arguing about how to interpret spontaneous case studies when the end result will be no different than when we started”
You don’t sound very confident that you can make your case that PSI is real.
“In any case, are you genuinely interested or just being flippant?”
No, I would seriously be interested in what you would consider good evidence that PSI is real. Remember that the prior probability of PSI is very low, so the evidence has to be pretty convincing. Something barely discernible above the noise is not convincing evidence.
“Oh right, so I can’t comment about what I think of the JREF’s approach now?”
Of course you can. I’m saying the JREF is not likely to be interested in what you think about how they should spend their time and money and what you think their purpose should be, just like you are apparently not be interested in typing page after page defending PSI. They have their own purpose and you have yours. Each to their own. Steven Novella either hasn’t the time or inclination to respond to your post, or doesn’t think it worth his while to do so. I have. Each to his own. I might think it’s not worth while continuing. You might think I should. But why should I care what you think I should do if it doesn’t serve my purposes. And why should I continue just because it serves your purpose if it doesn’t serve mine?
BillyJoe7
My agenda here was to correct your misconceptions about the MDC.
That’s ok, I’ll try to help you correct that error.
” If the test is designed so that a particular result would count as a success if genuine paranormal abilities were demonstrated, then it logically follows that the test has demonstrated that psi is real”
There are several things wrong with that statement….
It would help if you read my posts carefully. My statement paraphrased previous statements made by Steven and was meant to address an internal contradiction in what he wrote. It did not express my views on what the MDC is able to show. But anyway, I’ll try to help you understand the point I was making. Note the use of the word “if” in my statement. This introduces a conditional clause which was a direct quote from one of Steven’s previous statements on the MDC. From that the consequence follows. So let me know when you find an argument against the point that I actually made instead of just providing irrelevant personal opinion about the purpose of the challenge. Unless you would like me to pick apart your own interpretation of the MDC?
They have no intention of taking on the whole paranormal field.
I don’t expect them to take on the whole field. Most parapsychologists don’t even do that. However, paranormal claims deserve thoughtful investigation aimed at finding out whether people really possess paranormal ability, not short sighted investigation that merely tests whether a claimant has an accurate perception of their ability.
You don’t sound very confident that you can make your case that PSI is real.
I was talking about spontaneous paranormal experiences that are genuinely mysterious enough to serve as starting points for serious investigations. But from what else you said, I believe that a conversation with you about the experimental evidence for psi would also be a waste of time. This has nothing to do with the level of confidence I have in the evidence by the way.
Of course you can. I’m saying the JREF is not likely to be interested in what you think about how they should spend their time and money and what you think their purpose should be, just like you are apparently not be interested in typing page after page defending PSI. They have their own purpose and you have yours. Each to their own. Steven Novella either hasn’t the time or inclination to respond to your post, or doesn’t think it worth his while to do so. I have. Each to his own. I might think it’s not worth while continuing. You might think I should. But why should I care what you think I should do if it doesn’t serve my purposes. And why should I continue just because it serves your purpose if it doesn’t serve mine?
My god what is this nonsense. You start trying to assert that I should leave the JREF alone and then ask me whether you should care about this conversation? Continue the conversation if you want. I’ll try to continue pointing out your errors.
Davidsmith,
“My god what is this nonsense”
Well, this is a little embarrassing…
That last paragraph was to make it absolutely clear that I’m not taking this conversation seriously. And, you know, my way of saying “get a life”. Sorry, it was probably not as clear as I thought it would be. But, frankly, your objections to what Steven Novella said and what the JREF is doing are not really worth the time of day to discuss. And, sorry, I’m past caring about paranormal phenomena. There is no evidence that PSI exists. Been there, done that, nothing to see, move on please. But it seems someone has got you by the short and curlies and you’re enjoying it….ah, sorry, there I go again….I will stop now.
If someone claimed that they can walk through a solid wall (doors excepted of course
) then I would be genuinely impressed (after ruling out trickery ). But if that person claimed that after bumping into the wall 100 or so tries,that they partially went through the wall at a barely detectable level,that was hard to replicate,then yeah,I think it would be hard to believe. Call me crazy.
That would imply that you would know the truth when you found it. Evidence suggests otherwise.
@davidsmith, If you are so concerned with finding people with some ability, even if it is not the ability they proclaim to have, then I would challenge you to take all those who are outside the p value and put them through a rigorous test. If they then continue to perform slightly better than chance they can then advertise themselves as “Performs Slightly Better than Chance”.
What you also fail to realise is that people who take the challenge are explicitly saying they will “beat” the challenge as the challenge is set.
davidsmith wrote: “Rather, scientists actually thought about the situation and ran a battery of tests that were designed to find out the real extent of his memory performance. This isn’t done when the JREF “investigate”. They are only interested in testing whether the claimant’s perception of their own ability is accurate.”
Exactly. This is my point. The JREF is using valid scientific methods, but their scope is limited. They are not doing all the follow up experiments that would be necessary to answer the bigger question – is psi real. Certainly, as has been pointed out, if someone could pass the million dollar challenge that would “suggest” the possibility of psi, but a lot of follow up would be required to make a scientific case.
Second – my statement was not an internal contradiction. You simply misinterpreted it. Let me clarify.
If, in the hypothetical situation that someone possessed genuine psi ability, the MDC tests are designed to give a true positive, rather than a false negative. This does not mean they are designed to test for psi (see reasons above).
Exactly. This is my point. The JREF is using valid scientific methods, but their scope is limited. They are not doing all the follow up experiments that would be necessary to answer the bigger question – is psi real. Certainly, as has been pointed out, if someone could pass the million dollar challenge that would “suggest” the possibility of psi, but a lot of follow up would be required to make a scientific case.
Of course, in science, the question “is psi real” is not answered by one experiment. The JREF website states that the MDC is:
a one-million-dollar prize to anyone who can show, under proper observing conditions, evidence of any paranormal, supernatural, or occult power or event
So, according to the JREF, a successful test is evidence for a paranormal event. This follows from the description of the challenge. Clearly, a test of a normal ability is not of interest to them. This means that the JREF has to have some notion of what is paranormal and what isn’t. Therefore, when you actually examine the stated purpose of the challenge, it is not just a test of whether a claimant has accurate perception of their own ability. It is an attempt to carry out a scientific test for a paranormal event with a positive result interpreted as evidence of such.
I don’t have a problem with how that stands in principle. However, the challenge is actually implemented as a test of whether someone can “do what they say they can do”. We agree that this is a very limited approach. Like I said before, it’s like testing someone’s claim that they can lift more than 10 tonnes while not being interested when they lift 2. It’s in the spirit of inquiry where the JREF is unscientific.
If, in the hypothetical situation that someone possessed genuine psi ability, the MDC tests are designed to give a true positive, rather than a false negative .
More contradictions. If you’re hypothesising that genuine psi ability existed and predicting a particular result from that, you are testing a hypothesis – one of the hallmarks of a scientific experiment. Yet you insist that the MDC is not a test of a scientific hypothesis. And you wonder why some people are pointing out how confused all of this is?
This does not mean they are designed to test for psi (see reasons above).
They clearly are designed to provide evidence for a paranormal event (psi) – the JREF says so.
In a way, the MDC is kind of a scam. In reality, it’s myopic and poorly executed research with very little thought going into what it’s trying to achieve. But it is masquerading as something else, with proponents falsely claiming it isn’t supposed to be science in an attempt to draw attention away from it’s serious scientific shortcomings.
Bye.
@davidsmith
I think you’re confused on what a scientific hypothesis is. A hypothesis in science is a preliminary explanation of an observed natural phenomenon. The scientific process first requires verifiable physical data that can then be further measured and tested.
So, if someone comes along and claims that they have some never before observed ability, especially one that would contradict what is already known about the physical universe, then the first step would be to observe and verify the proposed evidence. That is what JREF is doing. What they are doing isn’t a scientific experiment, but rather verification of the presence of the claimed physical evidence. Science cannot test what is not testable or measurable. The methods used to determine whether there is any physical phenomenon present may employ scientific strategies and procedures, but that is the step taken before the scientific process can be used.
What the claimants propose to JREF aren’t hypotheses. Those testing the claims aren’t testing hypotheses. “Does psi exist” doesn’t even qualify as a hypothesis, because there is no observed evidence present that is testable. If one observed someone actually bending spoons or walking through walls, then one possible hypothesis could then be “psi is occurring”, though I submit that that would be irresponsible of any good scientist who wouldn’t exhaust more plausible explanations first (Occam’s Razor).
That’s easy – pretty much anything that directly contradicts what we know to be natural. It’s not that they have some notion of what the supernatural or paranormal are, but that they would be phenomena outside of what is known to be natural.
@davidsmith You are just the kind of person I hated playing D&D with.
@ Murmur
Haha, you totally hit it on the head!
@davidsmith
I’m not really even sure what position you’re advocating anymore.
-If you think the MDC is not conducting science experiments, you’re right (they’re just using scientific/skeptical thinking)
-If you think they’re masquerading as science, than I can see your point but disagree with you.
-If you think the MDC is a waste of time and a distraction, then I think you have an interesting point but may have muddled your argument.
The bottom line is that if you want to classify the kind of “experiments” the MDC is doing they can, at best, be described as pilot studies. If you wanted to prove that Psi is real, you’d need to acquire a huge body of evidence that tested the existence of Psi abilities on a large scale, precisely define Psi and offer some kind of mechanism by which Psi works.
One last note:
You could never really use science to prove Psi exists. At the end of the day, even if you gathered mountains of evidence, you’d end up with a Theory of Psi. In other words, you would be able to say “Given the available evidence, this Theory of Psi is the best explanation we have to explain these phenomena”
rezistnzisfutl,
A hypothesis in science is a preliminary explanation of an observed natural phenomenon.
It looks like you are confusing mechanistic hypotheses with scientific hypotheses per se. For a hypothesis to be scientific, it merely has to be testable. For example, in the MDC test of Natasha Demkina, the “paranormal hypothesis” was that there would be a relationship between her responses and the independent variable. The null hypothesis was that there would be no relationship. The JREF may well have used different terminology but that is what they were doing at the end of the day (one remaining issue was, of course, agreement on the prestated threshold for rejection of the null).
What they are doing isn’t a scientific experiment, but rather verification of the presence of the claimed physical evidence.
Setting up controlled conditions where you want to observe whether a relationship exists in nature is still a scientific experiment. The null hypothesis is that the relationship does not exist while the alternative is that is does. This is very much part of the scientific process as it is practiced today. Read psychology journals for examples.
“Does psi exist” doesn’t even qualify as a hypothesis, because there is no observed evidence present that is testable.
The relationship you wish to test the existence of does not need to have been observed before (whether under controlled or uncontrolled conditions) for the scientific process to work. Nevertheless, there are literally thousands of reported experiences that we label as “extrasensory experiences” that have occurred under uncontrolled conditions. The point of the scientific experiment in this context is to test whether similar relationships occur under controlled conditions.
That’s easy – pretty much anything that directly contradicts what we know to be natural. It’s not that they have some notion of what the supernatural or paranormal are, but that they would be phenomena outside of what is known to be natural.
There is a distinction between ‘supernatural’ and ‘paranormal’. The term ‘supernatural’ means outside of the natural. This places anything deemed supernatural beyond scientific inquiry. Therefore, if you are trying to scientifically test for the existence of a particular relationship in nature, then by definition, you can’t call it a supernatural phenomenon. ‘Paranormal’ on the other hand means outside of what is known to be normal (I prefer – currently unexplained by present knowledge derived from the scientific method).
test – to try get rid of italics error
CivilUnrest,
I’m not really even sure what position you’re advocating anymore.
-If you think the MDC is not conducting science experiments, you’re right (they’re just using scientific/skeptical thinking)
-If you think they’re masquerading as science, than I can see your point but disagree with you.
-If you think the MDC is a waste of time and a distraction, then I think you have an interesting point but may have muddled your argument.
I think that the MDC is an attempt at scientific experiments but, with regards to tests of “psi” abilities, the way the hypotheses are constructed is short sighted and thoughtless, lacking the true spirit of scientific inquiry in my opinion. However, proponents of the MDC often try to pass it off as “not supposed to be science” which is nonsense. I think this is an attempt to draw attention away from the scientific shortcomings of the challenge. If you think I have muddled my argument, it would help if you pointed out why you think so.
you are using the nirvana fallacy, that because the MDC is not a perfect test of all modes of paranormal, psi or supernatural activities simultaneously it is a scam.
https://en.wikipedia.org/wiki/Nirvana_fallacy
A better analogy than the 10 tons 2 tons would be someone claims to be able to lift 10 tons, but when that test fails, claims to be able to walk on water, then when that test fails, claims to be able to predict the future, then when that test fails claims distant seeing, then when that fails claims distance healing.
Essentially a hunt for anomalies unrelated to the claimed ability when the claimed ability can’t be produced. Choosing anomalies post hoc is a perfect way to find anomalies. Every sequence of 100 coin flips is extremely unlikely. But if you flip a coin 100 times, you will get a sequence with 100% certainty. If you wait until after you know what the flips are to decide if the resulting sequence is an anomaly, you will find lots of anomalies.
I think it comes down to this:
The JREF are testing indivdual paranormal claims scientifically but say that are not really doing what could be described as proper scientific experiments. Davidsmith says they ARE trying to do proper scientific experiments but are not really doing their proper scientific experiments properly.
“the way the hypotheses are constructed is short sighted and thoughtless, lacking the true spirit of scientific inquiry in my opinion”
This is his real beef. They are not doing tests that will give a chance for a signal to rise above the noise every now and then.
you are using the nirvana fallacy, that because the MDC is not a perfect test of all modes of paranormal, psi or supernatural activities simultaneously it is a scam.
No, I said the MDC is kind of a scam because the tests are, in actuality, scientific experiments but the JREF tries to pass them off as “not supposed to be science”, I suspect in an attempt to deflect legitimate criticism about the scientific shortcomings of the tests. So the scam bit is about use of the “not supposed to be science” claim. You can disagree with me on that, of course, but I’m not using the Nirvana fallacy.
Essentially a hunt for anomalies unrelated to the claimed ability when the claimed ability can’t be produced. Choosing anomalies post hoc is a perfect way to find anomalies. Every sequence of 100 coin flips is extremely unlikely. But if you flip a coin 100 times, you will get a sequence with 100% certainty. If you wait until after you know what the flips are to decide if the resulting sequence is an anomaly, you will find lots of anomalies.
I don’t understand. Is this supposed to be what the JREF do when they run the challenge?
@davidsmith
For a hypothesis to be scientific, it first has to have physical evidence to test. The hypothesis is simply a preliminary explanation of the observed physical phenomenon, nothing more. It is you who is confused on what a scientific hypothesis is because science ONLY deals with the physical universe. Therefore, the MDC experiments are not scientific because they don’t involve the scientific process.
Verifying that the claimed evidence exists at all in the first place is not science, even if the procedures used to verify are identical to procedures used in an actual scientific experiment or study. If JREF were able to positively determine that the claimed evidence actually exists, they could THEN formulate a hypothesis on the phenomenon because they have actual physical evidence to test.
So, testing the validity of a claim of evidence is not the same as testing a hypothesis of evidence already verified to exist.
Actually, yes, in order for something to be qualified as a hypothesis, evidence for its existence must first be confirmed, whether or not one actually observed it directly. For instance, we have strong evidence for a Big Bang event, even though no one ever observed it before; that evidence being physical, measurable, and independently verified phenomenon. The claims of psi, however, have never been verified to exist – they cannot even withstand the first step of the scientific process – so no hypotheses can be made about them.
Claims of psi existence, no matter how many people make it, are not hypotheses and therefore testing these claims isn’t science, even if the testing procedure is identical to scientific experimentation.
I’ll grant that there are definitional distinctions between the supernatural and the paranormal. Unfortunately, many people use the terms colloquially interchangeably, but in a strict sense, one can have “paranormal” results, or results that are outside what is accepted as normal, and it still not be “supernatural”, or events that lie outside what is known about the physical universe. In this case, psi, and many other of the claims leveled in the MDC, are both paranormal and supernatural in nature. If one claims that they are able to physically lift 10 tons, that could be considered a paranormal claim. If one says that angels help him lift 10 tons when he can normally only lift 2 tons, that would be both paranormal and supernatural, because his ability to lift 2 tons is outside the norm, but angels are supernatural in nature.
rezistnzisfutl,
For a hypothesis to be scientific, it first has to have physical evidence to test.
This is incorrect. There is no logical reason why a hypothesis cannot test for the presence of a relationship in nature that has never before been observed. In fact, we do this all the time in science when a theory is used to predict novel observations. However, a theoretical precedent is also not necessary for the process of hypothesis testing to work. That said, we don’t normally form hypotheses without any input from theory or previous observations for pragmatic reasons. It is clearly more sensible to let hypotheses be guided by current observation, but not logically necessary.
The hypothesis is simply a preliminary explanation of the observed physical phenomenon, nothing more.
Again, I think you are confusing mechanistic hypotheses with scientifically testable hypotheses per se. Mechanistic hypotheses are a subset of scientifically testable hypotheses. For example, the hypothesis “men are generally taller than women” is a testable hypothesis and amenable to scientific inquiry. However, there is no explanatory component. Rather, it is a hypothesis that states there is a certain relationship that exists in the natural world.
It is you who is confused on what a scientific hypothesis is because science ONLY deals with the physical universe.
I didn’t say otherwise did I? Hypotheses can certainly make predictions about what has yet to be observed (physically).
Verifying that the claimed evidence exists at all in the first place is not science, even if the procedures used to verify are identical to procedures used in an actual scientific experiment or study.
There are many examples of published scientific studies that test hypotheses stating that certain relationships exist in nature. This is very much part of the scientific process as it is practiced today. Read psychology journals for examples.
Actually, yes, in order for something to be qualified as a hypothesis, evidence for its existence must first be confirmed, whether or not one actually observed it directly.
You are confused. It is not the hypothesis that is hypothesised to exist in nature. It is physical relationships that are hypothesised to exist. Furthermore, if physical relationships that are hypothesised to exist in nature must “first be confirmed to exist”, it would be impossible to make predictions about novel observations – one of the driving forces of science.
For instance, we have strong evidence for a Big Bang event, even though no one ever observed it before; that evidence being physical, measurable, and independently verified phenomenon.
This contradicts what you have been claiming. If nobody had ever observed these physical, measureable and independently verified phenomena before, then clearly, hypotheses can be made about the existence of physical relationships that have not yet been observed. You are in agreement with me here.
The claims of psi, however, have never been verified to exist – they cannot even withstand the first step of the scientific process – so no hypotheses can be made about them.
Again, this comes down to a mix-up between mechanistic hypotheses and scientifically testable hypotheses per se. See above for reasons why (bear in mind that I am not disputing whether the claims of psi have actually been verified to exist, but rather, I am disputing your claim that hypotheses cannot be made that state such relationships exist in nature).
You’re simply not getting it, or are not reading my post thoroughly. A scientific hypothesis must have some sort of observable physical evidence first before it can be formed. Evidence must be directly observed, whether or not the hypothesis created for the phenomenon in question can be directly observed or not. Simply dreaming up a claim about the existence of something without having evidence for the claim is not a hypothesis. Transversely, without evidence first being observed, there is no basis for a hypothesis. Evidence MUST precede hypotheses. That is why scientists verify the presence of the evidence first in the peer-review process before reviewing the hypothesized explanations of the evidence.
That relationship in nature that “has never been observed” must have some sort of corroborating physical, verifiable evidence first before a scientific hypothesis about it can be formed (aka physical relationships). Simply stating “psi exists” isn’t a hypothesis, it’s an assertion. If one were able to bend spoons in their hand without any sort of mechanical assistance, and demonstrate that ability to others, then a valid scientific hypothesis can be formulated that “psi is what caused the spoon to bend” (although it would be irresponsible of a scientist not to rule out more likely, physical explanations first). The problem is, no one has ever been able to demonstrate the ability to bend spoons in their hand without mechanical help or trickery. The same goes with all claims of psi that have ever been presented.
In the case of psi, the “evidence” people claim to present isn’t really evidence at all, because there is nothing to present. Their claims are untestable, therefore unfalsifiable.
The relationships in nature must first be observed to exist before a hypothesis can be created. “Men are generally taller than women” is simply an observation, that’s all. It is NOT a hypothesis because it doesn’t explain why they are taller, it just describes the physical attribute. That’s what a hypothesis is, a preliminary explanation of why it is that men are taller than women. So, a hypothesis would be something like ,”men are taller because they evolved with higher levels of testosterone and growth hormone.” Then the scientists would go about testing whether that is true or not.
I challenge you to cite one peer-reviewed scientific publication that uses a descriptive observation (relationships in nature) as its hypothesis. A psychologist simply stating “some patients diagnosed with clinical depression respond to a certain anti-depressant better than others” is simply an observation, not a hypothesis. A psychologist would have to then go on to posit an explanation as to WHY the anti-depressant works. So, an actual hypothesis would be something like ,”We believe, based on the evidence, that the responsive patients have a higher number of receptors for the drug in question than the non-responders.”
To try to use an observation as the hypothesis is useless. That’s like stating that my desk is black, and that’s my hypothesis. There is nothing to test just knowing that my desk is black. If I were to then explain that my desk is black because it absorbs nearly all electromagnetic light, that is a testable hypothesis.
Those who do the MDC are making existence claims, not hypotheses. They are describing a claimed phenomenon “I can walk through walls”. That is merely an description of on observation that their solid body can pass through other solid matter. Simply stating that is not explanatory in any way as to how or why, so it’s not a hypothesis. It goes without saying that without demonstrating that the person can actually walk through walls, obviously no hypothesis can be formed.