Mar 01 2010

Armchair Skeptics

I am often asked if skeptics and skeptical organizations should undertake first-hand investigations. Of course, it depends upon what your goals are. But I think the question can be re-phrased to mean – is there any value or benefit to first hand investigation, and to this the answer is a definite “yes.”

But this is not to denigrate the value of skeptical review from the comfort of your computer chair. This kind of activity has sometimes been referred to as “armchair skepticism” – meant to be derogatory. While I see the value in going out into the field, armchair skepticism has a valuable and complementary role to play.

In fact, these two activities mirror what real scientists do, and are roughly analogous to peer-review vs experimental replication.

Armchair Skepticism

The community of scientists keep each other honest, and keep the process of science grinding forward, in various ways – only one of which is going into the lab to replicate a study or do follow up research. When a colleague publishes a paper, or presents a paper at a meeting, his colleagues provide analysis and criticism. Ideas are examined for logic, internal consistency, and plausibility. Other options, perhaps neglected by the researcher, are explored. And existing research, perhaps not taken into account by the researcher, is brought up and discussed.

This feedback is provided without ever doing any actual investigation. When skeptics perform the exact same service to paranormal or fringe claims, this should not be denigrated at all, but seen as providing in our area of expertise the same kind of analysis that scientists provide in theirs.

This “peer-review” takes several forms. First, the term “peer-review” is often used to refer to the formal process of reviewing a paper that has been submitted for publication. I am not referring to this formal peer-review (which I do not think has any analogy in skeptical activity), but rather to the informal peer-review that collectively refers to all the efforts of the scientific community to hammer errors and flaws out of scientific thinking.

Informal peer-review has various manifestations, all of which have analogies in skeptical activism. For example, scientists will often dissect a specific published paper, analyzing it for weaknesses of methodology, strength of the outcome, how well the authors interpret their own findings, and putting it into the context of plausibility and other published research. This analysis may be published as a letter or commentary in the same journal as the original study, or incorporated into a talk at a meeting. Skeptics will often do the exact same thing, but these days published as a blog or article in the skeptical literature.

Scientists will also publish systematic reviews or topic reviews, reviewing all the published evidence and arriving at a bottom-line assessment of the state of the science. Skeptics do this as well. This activity does involve “research,” but not laboratory or field research – rather it involves researching the literature. It is getting easier and more viable to perform this activity sitting in front of a computer with internet access, as there are online libraries of published research, and many journals and news outlets have online versions.

Depending upon the topic and the depth of one’s investigation, it may be necessary to venture into a physical library, but this is getting less and less necessary. This is the level of research one would do when writing a book, but not for a daily blog post.

Armchair skeptics therefore provide a valuable service, similar to the activity of working scientists. We can analyze specific claims, topics, or published research for quality, plausibility, and historical and scientific context. We can then tailor our writing to communicate with our colleagues, the public, or both simultaneously. We can also provide reference material (like The Skeptic’s Dictionary), which fills the role of a science textbook or reference website.

Skeptical Investigation

While I am a strong advocate of armchair skepticism, if you have the opportunity to go into the field and do first-hand investigation, you will likely find the experience very illuminating. Theory and book-learning does only go so far, and there are aspects to paranormal investigations that you would simply not imagine until you are there to see it first hand.

I and my colleagues have performed a number of investigations – mostly haunted houses, but also EVP, channelers, exorcism, and psychics. Most of these were with other (less-than-skeptical) groups. What was always very striking was how unimpressive the paranormal investigators or claimants were. We always gave them more credit than they deserved, and were surprised at how easy it was to analyze their evidence.

For example, when we investigated a channeler in Connecticut who claimed to channel the 600 year old spirit of a man from Nepal, I was prepared to have the Nepalese analyzed to see if it was modern or appropriate to the claimed time period. However, the channeler did not speak any dialect spoken in Nepal, just English with a cheesy regional accent (it sounded Indian to me).

We also went on an EVP (electronic voice phenomenon) recording session with some local ghost-hunters. EVP is basically an exercise in audio pareidolia – listen to static long enough and you will make out words. We sat in the attic of an allegedly haunted restaurant. Even though it was relatively quiet in the location, I was struck by how much background noise there was. When you listen carefully, you can hear voices in the street, the rumblings of any building, and blowing of the fan, and other noises of unclear source. It was far more noisy than I had anticipated, providing a rich source of raw material for later imaginative listening.

I also actually sat through hours of recorded exorcisms. This is not an activity I am eager to repeat, but it certainly gave me a more thorough perspective on what goes on in such exorcisms – basically nothing. They are incredibly boring non-events.

I will also relay the experience of Susan Blackmore, a former ESP researcher who eventually gave up ESP research as fruitless. She noted that there is only so much you can infer about the quality of another researcher’s methodology from the published report. When you actually go into their lab and examine the methods first hand, you get a much better idea of the quality, and may identify flaws that were not apparent from the published description.

And of course, Joe Nickell, who does investigations full time, has many tales of paranormal claims and stories that could not be definitively resolved without on-the-site investigation.

Conclusion

Both armchair skepticism and first-hand investigation are important to skeptical activism. They are complementary, each filling different needs. Both also are analogous to activity that working scientists perform in the process of peer-review, new investigations, and replicating previous research.

In the last decade or so I feel that the skeptical community has honed its ability to perform meaningful peer-review of paranormal claims, and communicate the results of that review to the public. It is also my sense that our overall activism would benefit from increased efforts to perform more first hand investigations.

I would even go beyond replicating the typical haunted house investigation, and do some real hypothesis testing. This of course takes time and resources, but would be well worth it.

Share

12 responses so far

12 Responses to “Armchair Skeptics”

  1. Sc00teron 01 Mar 2010 at 10:00 am

    I agree, even if it’s to do research and give a presentation at a Skeptics in the Pub meeting. That’s a very different experience than writing blog posts on the web. But the important thing is to do your research and be professional.

    Granite State Skeptics did a haunted house investigation on Halloween last year and while it was a success, there were a lot of things I would have done different looking back on it. That’s to be expected of course, but the resources are limited. There’s tons of woo books on how to do investigations but nothing really scientific or skeptical that’s easy to find. But we got press, good information, and exposure to the public.

  2. Fred Cunninghamon 01 Mar 2010 at 10:13 am

    Would it be possible to collect a volunteers list to participate as assistants in your investigations? Another possibility would be to have workshops on investigating claims.

  3. RickKon 01 Mar 2010 at 10:37 am

    I’d love to see a formal “Skeptical Seal of Approval” label that can be applied to a new claim on a website. This would indicate that a review has taken place, and while the claims of the website or paper may or may not be accurate, no apparent magic was used in their formulation.

    And it’s too bad we can’t paste banners on websites (like Maloney’s) that say “Warning: Material on this site may harm your brain”

  4. Sc00teron 01 Mar 2010 at 10:42 am

    @RickK – That would be hard to enforce and there’s nothing really stopping the woo-ers from making their own kind of seal of approval and just use that. Most people don’t check.

  5. ccbowerson 01 Mar 2010 at 12:30 pm

    If only everyone was an armchair skeptic.

  6. jacksonskepticalsocietyon 01 Mar 2010 at 6:02 pm

    I go to a lot of dubious health lectures and seminars and whatnot (only free ones) and I have to say, “non-event” is about as good as it gets. Most of ‘em aren’t even worth blogging about, even if I’ve got a lack of internet-based blag-terial.

  7. tmac57on 01 Mar 2010 at 9:41 pm

    Echoing ccbowers point, it should be obvious to whoever might denigrate ” armchair skepticism”, that that is far better than “armchair gullibility”.

  8. bluedevilRAon 01 Mar 2010 at 9:54 pm

    speaking of armchair skepticism…i launched my blog in the spirit of Neurologica and SBM:

    http://dfreedm.wordpress.com/

    i would definitely appreciate the criticisms of dedicated neurologicals and SBMers!

  9. ccbowerson 01 Mar 2010 at 10:28 pm

    I guess I should correct the grammar of my original statement for the subjunctive (before someone else does) and finish the second half of of what my statement should have been…

    If everyone were an armchair skeptic, there would be no need for any other kind.

  10. Michael Meadonon 02 Mar 2010 at 3:18 am

    Excellent points, Steve. (And spot on tmac57).

    Honestly, Joe Nickell’s critiques of armchair skepticism has always rather annoyed me, for exactly the reasons you give. While it may be pragmatically useful to present oneself (as Nickell does) as a ‘neutral investigator’ who has no ‘preconceived notions’, clearly, prior-probability (in Bayes’ sense) has a role to play. It’s a bit like the difference between evidence- and science-based medicine: sure, Joe can go around looking at his n-th ‘haunted house’ only to find no evidence of the paranormal, but it’s perfectly acceptable for the rest of us to say ‘there are no good reasons to believe in ghosts, you have the onus, so go away until you bring some evidence to the table’.

    Still, I’d love to do some skeptical investigating. Apparently South Africa (where I’m from) has its own paranormal society now (see this for a critique). I’m planning to tag along at some point…

  11. davidsmithon 02 Mar 2010 at 5:10 am

    “This feedback is provided without ever doing any actual investigation. When skeptics perform the exact same service to paranormal or fringe claims, this should not be denigrated at all, but seen as providing in our area of expertise the same kind of analysis that scientists provide in theirs.”

    I completely agree. Like you say, this is what happens within all universities. Someone presents their latest research and other scientists, sometimes from completely different research fields, contribute to critical appraisal of the methodology and conclusions made by the reseacher presenting her evidence.

    Within the field of parapsychology, there are many excellent commentaries, criticisms and constructive advice that comes from the sceptical community (I don’t like using that phrase – aren’t the people who apply science yet publish positive results also part of the sceptical community?). This can come in the form of blog entries and other online writings. However, the writings of sceptics sometimes go beyond respectfull scholarly criticism, stray into ad hominem and ridicule, and are sometimes so off the mark as to wonder if there are ulterior motives beyond an attempt to provide sound critical review. One notable example would be the Skeptic’s Dictionary. As far as serious and fair commentary on academic parapsychological issues goes, that source is terrible. I dare say it fails in other areas I know less about.

    Having said that I agree with most of this blog post. I just think that some archair sceptics spoil it for the rest of us by resorting to unscholarly behaviour.

    “I would even go beyond replicating the typical haunted house investigation, and do some real hypothesis testing. This of course takes time and resources, but would be well worth it.”

    I think this is essential. Doing haunted house investigations or exposing the folly of ‘EVP studies’ is more akin to an excercise in how to promote critical thinking to the credulous public. Doing serious investigations into paranormal experiences is another thing entirely. The sceptical community should try to replicate parapsychology studies that have already claimed positive results and get the studies published in peer reviewed parapsychology journals. Like you say, that takes money and time but it would be worth it. You never know, you may even come away with positive results too!

  12. derekcbarton 02 Mar 2010 at 9:42 pm

    The Independent Investigations Group (http://www.iigwest.org) has been doing armchair and in-the-field skeptical research for 10 years now.

    This year the IIG is expanding and it is working with the Center For Inquiry on creating IIG Affiliates at local Centers. The first affiliate is being set up in Washington, DC the weekend of March 13, 2010.

    It will start with a lecture by Ben Radford entitled “Scientific Paranormal Investigations: How to Solve Unexplained Mysteries”
    http://www.centerforinquiry.net/dc/events/igg_seminar_-_scientific_paranormal_investigations/

    After that James Underdown and Brian Hart will present the “Independent Investigations Group Kick-Off”
    http://www.centerforinquiry.net/dc/events/independent_investigations_group_kick-off/

    If you are located near Washington, DC I hope that you will check it out and join the new IIG-DC.

    Thanks.

    -Derek

Trackback URI | Comments RSS

Leave a Reply

You must be logged in to post a comment.