Dec 13 2010

The Decline Effect

While there is a complex spectrum of attitudes toward science, there are three clusters worth pointing out, specifically in reference to the provocative New Yorker article by Jonah Lehrer called The Truth Wears Off. The first group are those with an overly simplistic or naive sense of how science functions. This is a view of science similar to those films created in the 1950s and meant to be watched by students, with the jaunty music playing in the background. This view generally respects science, but has a significant underappreciation for the flaws and complexity of science as a human endeavor. Those with this view are easily scandalized by revelations of the messiness of science.

The second cluster is what I would call scientific skepticism – which combines a respect for science and empiricism as a method (really “the” method) for understanding the natural world, with a deep appreciation for all the myriad ways in which the endeavor of science can go wrong. Scientific skeptics, in fact, seek to formally understand the process of science as a human endeavor with all its flaws. It is therefore often skeptics pointing out phenomena such as publication bias, the placebo effect, the need for rigorous controls and blinding, and the many vagaries of statistical analysis. But at the end of the day, as complex and messy the process of science is, a reliable picture of reality is slowly ground out.

The third group, often frustrating to scientific skeptics, are the science-deniers (for lack of a better term). They may take a postmodernist approach to science – science is just one narrative with no special relationship to the truth. Whatever you call it, what the science-deniers in essence do is describe all of the features of science that the skeptics do (sometimes annoyingly pretending that they are pointing these features out to skeptics) but then come to a different conclusion at the end – that science (essentially) does not work.

I often feel that those in this third group – the science deniers – started out in the naive group, and then were so scandalized by the realization that science is a messy human endeavor that the leap right to the nihilistic conclusion that science must therefore be bunk.

But scientists themselves are generally in the middle group (at least they should be), and so news about the various foibles of science is no surprise.

The New Yorker Article

The article by Lehrer falls generally into this third category. He is discussing what has been called “the decline effect” – the fact that effect sizes in scientific studies tend to decrease over time, sometime to nothing. This term was first applied to the parapsychological literature, and was in fact proposed as a real phenomena of ESP – that ESP effects literally decline over time. Skeptics have criticized this view as magical thinking and hopelessly naive – Occam’s razor favors the conclusion that it is the flawed measurement of ESP, not ESP itself, that is declining over time.  Lehrer, however, applies this idea to all of science, not just parapsychology. He writes:

And this is why the decline effect is so troubling. Not because it reveals the human fallibility of science, in which data are tweaked and beliefs shape perceptions. (Such shortcomings aren’t surprising, at least for scientists.) And not because it reveals that many of our most exciting theories are fleeting fads and will soon be rejected. (That idea has been around since Thomas Kuhn.) The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.

This paragraph sums up what I was describing above – Lehrer is ultimately referring to aspects of science that skeptics have been pointing out for years (as a way of discerning science from pseudoscience), but Lehrer takes it to the nihilistic conclusion that it is difficult to prove anything, and that ultimately “we still have to choose what to believe.” Bollocks!

Lehrer is describing the cutting edge or the fringe of science, and then acting as if it applies all the way down to the core. I think the problem is that there is so much scientific knowledge that we take for granted – so much so that we forget it is knowledge that derived from the scientific method, and at one point was not known. Many conclusions in science do not decline over time – they strengthen, and they become so overwhelmingly confirmed that they can be treated as established facts (although  always open to revision). The theory of gravity, relativity, evolution, the DNA basis of genetics, the germ theory of infectious disease – these are not illusions that evaporate under closer scrutiny, but fundamental aspects of reality that science has elucidated.

It is telling that Lehrer uses as his primary examples of the decline effect studies from medicine, psychology, and ecology – areas where the signal to noise ratio is lowest in the sciences, because of the highly variable and complex human element. We don’t see as much of a decline effect in physics, for example, where phenomena are more objective and concrete.

The Decline Effect

If the truth itself does not “wear off”, as the headline of Lehrer’s article provocatively states, then what is responsible for this decline effect? The answer can be found in the previous articles of this blog and many others, like science-based medicine. But to quickly review – it is no surprise that effect science in preliminary studies tend to be positive. This can be explained on the basis of experimenter bias – scientists want to find positive results, and initial experiments are often flawed or less than rigorous. It takes time to figure out how to rigorously study a question, and so early studies will tend not to control for all the necessary variables. There is further publication bias in which positive studies tend to be published more than negative studies.

Further, some preliminary research may be based upon chance observations – a false pattern based upon a quirky cluster of events. If these initial observations are used in the preliminary studies, then the statistical fluke will be carried forward. Later studies are then likely to exhibit a regression to the mean, or a return to more statistically likely results (which is exactly why you shouldn’t use initial data when replicating a result, but should use entirely fresh data – a mistake for which astrologers are infamous).

All of these effects, and more, are why skeptics are frequently cautioning against new or preliminary scientific research. Don’t get excited by every new study touted in the lay press, or even by a university’s press release. Most new findings turn out to be wrong. In science, replication is king. Consensus and reliable conclusions are built upon multiple independent lines of evidence, replicated over time, all converging on one conclusion. But such conclusions are possible, and occur so often we take them for granted. While initial conclusions that turn out to be wrong are sensational – as is turning such events into an indictment of science itself.

Conclusion

Lehrer does make some good points in his article, but they are points that skeptics are fond of making. In order to have a  mature and functional appreciation for the process and findings of science, it is necessary to understand how science works in the real world, as practiced by flawed scientists and scientific institutions. This is the skeptical message.

But at the same time reliable findings in science are possible, and happen frequently – when results can be replicated and when they fit into the expanding intricate weave of the picture of the natural world being generated by scientific investigation.

But apparently it is more provocative to focus on the edges of science where results are preliminary and replication calls them into question. It is true that often preliminary results are seized upon, and in medicine even acted upon, before they are adequately confirmed. That is an important lesson. But it does not mean that science doesn’t work. It just means that science grinds more slowly than perhaps we would like. Science is self-corrective, and perhaps there is more to correct than we would like. But in the end the process of science works itself out. It is not just another narrative, and we don’t have to arbitrarily choose what to believe.

Share

17 responses so far

17 Responses to “The Decline Effect”

  1. CrookedTimberon 13 Dec 2010 at 9:33 am

    Nicely summarized. I am generally a fan of Lehrer’s writing, but was left scratching my head at this article. I think he made some interesting points but greatly overstated his conclusion.

  2. SARAon 13 Dec 2010 at 12:16 pm

    Deniers are offended by the messy lack of certainty that is the hallmark of our existence. They lambaste it as though it were a sin, rather than a neutral reality.
    Instead of identifying the reality of uncertainty as inherent, they attack science for not creating certainty.

  3. Juanon 13 Dec 2010 at 12:52 pm

    I’ve been attending art schools and have been in the “art world’ for about ten years now and the denier noise becomes overwhelming too often, too quickly. I started listening to “Nonsense on Stilts” right now and combined with this article I keep thinking about the need for literature like this, similar to the way Hitchens, Dawkins and Hecht made their atheist wave a few years ago, but targeting academic rather than religious deniers. I remember being in a lecture and not knowing how to counter lecturer on his postmodern rant until some time later when I came across an article like this. It’s upsetting that the majority of the lectures slant more towards the postmodern conclusion than the scientific-skepticism one, unless you want the nazi rants to begin.

  4. Michael Meadonon 13 Dec 2010 at 1:58 pm

    While I think Lehrer is wrong here (as far as I can tell – I don’t have access to the New Yorker) I think you’re being rather harsh Steve. I don’t think he belongs to the science denier crowd – he has a long record (e.g. on his blog) of doing responsible science communication. He’s no postmodernist.

    I think he’s in the second group – but could use a couple of (intellectual) slaps to sort out some confusion. Maybe this piece will be it…

  5. Eternally Learningon 13 Dec 2010 at 2:16 pm

    I think another factor in the tendency of some to dimiss science when they find out how messy it is, is the confusion about the nature of truth and knowledge. One of the things that took me a while to learn on my journey away from religion was that knowledge of the truth was not black and white, but a spectrum. Religion claims to have knowledge completely on the truth end of the spectrum and that anything else is on the falsehood end. When you compare that to science saying that it’s 76% certain of the truth (just paraphrasing of course) it seems like science comes up short in the truth department.

    For myself, even after I rejected the concept of the supernatural, it took a while to get past this black and white view of knowledge. Once I realized that it was a spectrum and that where you were on the spectrum was determined by how much and how strong the evidence you had was, I saw where religion was on that spectrum compared to science and it was clear which way worked better.

  6. B Hitton 13 Dec 2010 at 4:29 pm

    Great post. I think one problem Lehrer may be having here is looking at the issue through the somewhat narrow lens of a science journalist. When he talks about science, he is really talking about newsworthy science, which is inevitably where a lot of the messiness happens (cutting edge, fringe). By the time something survives the gauntlet of replication and refinement, it’s no longer news.

  7. cwfongon 13 Dec 2010 at 5:06 pm

    Lehrer tries to put an O.Henry twist to all of his articles and more often than not falls short.

  8. BillyJoe7on 14 Dec 2010 at 5:05 am

    Here is the original article posted as a pdf:

    http://crayz.org/science.pdf

    If you cut off the head (“The truth wears off: Is there something wrong with the scientific method?”) and the tail (“When the experiments are done, we still have to choose what to believe”), you are left with the body of the article which is actually not all that bad.

    Which makes me wonder if he actually understood what he was being told by the various scientists he interviewed. It seems he got lost in the detail and missed the big picture.

  9. elmer mccurdyon 14 Dec 2010 at 12:29 pm

    “This is a view of science similar to those films created in the 1950s and meant to be watched by students, with the jaunty music playing in the background…”

    Love the sneering caricatures.

  10. elmer mccurdyon 14 Dec 2010 at 12:41 pm

    Seriously, I have learned from bitter experience that the sort of attitude displayed there and elsewhere on this blog is a cue to run, don’t walk, away from from the doctor in question.

  11. BillyJoe7on 14 Dec 2010 at 4:01 pm

    elmer mccurdy,

    You are an idiot.

    There, did you get the reaction you were after?
    Sneering caricature, my ass.
    Let us both look in the mirror and see if we come up roses.

  12. dwayneon 14 Dec 2010 at 11:22 pm

    Funny, I don’t fall any more slowly when I slip on ice than when I was younger. Water seems to boil in about the same amount of time. My wife’s pregnancy with my daughter lasted about as long as my mother’s when she had me.

    Also, it’s ironic that whenever someone wishes to prove the inadequacy of scientific assumptions, they use as evidence … the findings of science.

    But I’m sympathetic to their plight, because, after all, what other way is there?

  13. kloxon 15 Dec 2010 at 1:42 am

    I read this review several days before actually reading the article, and by then I forgot I already read the review. Then my GF restumbled upon this!

    Anyways, what I am curious about is Schooler’s testing of the decline effect, i.e. the precognition experiment mentioned in the essay. Was Schooler’s protocol changing between testing attempts? It is suggested that it can be explained with regression to the mean, but why is it generally a really “high” response in the first attempt that decreases later? It seems like something about the experiment has actually changed. Better controls, better observation, more/less bias (either could change results), etc would seem like a better explanation than just regression to the mean. If it is just regression to the mean, then wouldn’t that imply that these people are simply applying bad statistics? They aren’t having tight enough error-bars, etc., to sufficiently show the effect is real?

    So what is really going on with modern studies that are demonstrating the decline effect (ignoring the supernatural ones)?

    Thanks,

    Jacob Block

  14. semioticuson 20 Dec 2010 at 12:14 am

    This is a definitely interesting read. I agree with Lehrer’s assertions about publication bias, yet it should not be forgotten that Lehrer’s article also is subject to publication bias as well. A mere assertion of the common belief would not be published or discussed, so Lehrer, maybe unwillingly, feels the urge to overstate his conclusions.

    His article takes off Schooler’s evaluation of his own experiment. Yet here is another article that asserts Schooler’s observations have indeed been repeated by other researchers and there was some common ground between those further researches.

    http://digitalcommons.utep.edu/cgi/viewcontent.cgi?article=1007&context=christian_meissner

    I disagree with the statement that ubiquity of the results weaken the “regression to the mean” hypothesis. Here is a question: Let’s say Schooler’s initial experiment suggested a result that was in line with the previous conclusions about verbal overshadowing, that there was no such thing. Would he feel the need to repeat his experiment? I don’t think so. We observe a “decline effect” which is a fact, yet the reason there is no “rise effect” is because the assumption that is not contrary does not get reevaluated. Even if regression to the mean exists, for us to be able to observe it there needs to be a decline effect, and that seems to weaken the hypothesis while it does not mean that it is weakened.

    Besides that, I fully agree with the conclusion of Steven Novella’s article. I think it says everything I want to say about the second part of Lehrer’s article.

    Thanks.

  15. neurobonkerson 03 Jan 2011 at 2:53 pm

    FYI The original article is no longer paywalled

    http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer

  16. zen_arcadeon 07 Jan 2011 at 2:10 pm

    I enjoyed reading Lehrer’s article but was, like many of you, puzzled and annoyed by the bogus conclusions drawn from phenomena that most good scientists and lovers of science acknowledge. The broad postmodernist, anti-science conclusion of the last few sentences I found especially troubling: “Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.” Yep, it’s all about “narrative,” all truth is socially constructed, right? Ugh.

  17. Aldebaranon 16 Mar 2011 at 3:59 pm

    What is most gratifying to me about Steve’s and many other reactions to Lehrer’s article is to see, by the replies, just how deeply he has gotten under your skin. If all of you were really so sure of your position as you claim, then his article would not induce so much disproportionate anger, in reply. Therefore, Steve’s dispassionate, analytically rigorous response (“Bollocks!”), and his wistfully naive profession of faith (“But in the end the process of science works itself out”) merely adds to my sense that Lehrer is indeed on to something.

    Also, with respect to this:

    “but the[y] [skeptics of science] come to a different conclusion at the end – that science (essentially) does not work.”

    What utter nonsense. No skeptic of science I know claims that science “does not work”. They merely claim, as Steve will also admit, that science is flawed, and, more importantly, that science does not arrive at perfectly objective, absolute truth, nor does it hold primacy of place as the only valid way of viewing and interpreting phenomena.

    OK, point made. So, Steve and the Amen Corner: Get your flame-throwers ready, consult your “Ad Hominem Thesaurus”, and go nuts!

Trackback URI | Comments RSS

Leave a Reply

You must be logged in to post a comment.