Sep 30 2008

Scientific Consensus

Published by under Uncategorized
Comments: 14

I often refer to the “consensus of scientific opinion” and was asked to elaborate on exactly what that is and, more importantly, how it is determined. From a practical point of view, how can the average citizen get a handle on what the scientific consensus is on any given topic? For some burning questions, like whether or not there is significant anthropogenic global warming, much of the debate centers around whether or not there is a consensus and what it means. For others, like should we invest in biofuel from corn, a consensus seems elusive.

The Role of Consensus

For anyone trying to take a scientific approach to knowledge about the world, we must rely heavily upon experts, or those who are more knowledgable than we are. There is no choice – there is simply too much specialized scientific knowledge for anyone to be an expert in everything, or even a significant portion of scientific disciplines.

Further, being an educated layperson is usually not enough to form your own opinions on specific scientific questions. Forming a reliable opinion often requires a level of detailed knowledge that only an expert in the field can obtain. Even experts can be wrong, of course, and since lay opinions are likely to span all possibilities, some are bound to be correct. Experts, however, are far more likely to have an opinion that accurately reflects the evidence and to understand how to incorporate new evidence as it comes in.

There are plenty of examples, however, of experts coming to conclusions that were profoundly wrong. How is the public supposed to rely upon expert opinion, then? Proper scientific authority does not rely in any individual. Individuals are quirky and may have biases and influences that lead them astray.

Scientific authority, rather, lies in the consensus of scientific opinion. When many experts look at the data and come to the same conclusion, it is more likely to be accurate than anomalous. A consensus of many experts is simply more reliable than the quirky opinions of a single expert.

It is therefore the consensus of opinion (or should be) that determines what goes in the science textbooks, what is taught is science classrooms, how applied sciences are regulated, and how society spends its resources.

When Consensus is Wrong

There are, of course, examples of when the scientific consensus proved to be wrong.  In fact, this happens all the time – whenever evidence points in one direct but later evidence reveals a different answer or (more likely) reveals a deeper reality. It is the nature of science that is constantly changes in response to new evidence, and so the consensus of opinion is a moving target.

But not all scientific consensus is created equal.

We therefore need another assessment in addition to what the consensus is on a given question – we also need to consider how solid the consensus is. There are some question for which the scientific consensus is so solid (reflecting an overwhelming amount of evidence) that it would be perversely absurd to deny it. The earth is an oblate spheroid. DNA is the molecule that carries hereditary information. Life on earth is the result of common descent. Infectious illnesses are caused by microscopic organisms.

Other conclusions are solid but not beyond the possibility of revision. Still others are probable but preliminary. And some scientific questions are genuine controversies, without a clear consensus. The more solid a consensus is, the less likely it is to be overturned in the future. It’s not impossible for a consensus to be overturned – it’s just progressively unlikely as the consensus becomes more solid.

When is a consensus of scientific opinion not reliable? If the scientific process is working properly, then never. So the real question is, when does the scientific process break down. Here are what I think are the read flags for a supposed consensus of which you should be skeptical:

– The consensus seems premature.  If we have only been studying a problem for a short time, the overall amount of evidence is small, or there has not been time for proper replication of experiments to occur, then scientific opinions are likely premature.

– If the consensus emerges from a highly politically or ideologically charged atmosphere.

– If the consensus exists only within a subculture, such as a fringe group looking to promote a predetermined conclusoin.

– If the consensus is dominated by industry self-interest.

As I said – these situations do not represent genuine scientific consensus, but rather a breakdown in the system or an ideologically or otherwise motivated subgroup looking to masquerade as a scientific consensus.

So What Is The Consensus

With all this in mind, how can the average person go about figuring out what the consensus is on any scientific question and how solid it is? A good place to start is to see if any relevant institution has put together an expert panel to review the evidence and make a consensus statement. If the American Academy of Neurology puts out a position paper on a neurological issue, it probably is a good reflection of the consensus.

Of course, panel decisions are only as good as the panel. You have to ask whether or not the panel was representative and objective. When Clinton put together a presidents panel on alternative medicine and packed it with alternative medicine providers, that is not exactly what I would consider a reflection of the scientific consensus. If you cherry pick the opinions you choose – that skews the results, and does not reflect the true consensus.

Also, for a consensus numbers matter (by definition). You can always find experts to support any belief – but what do most scientists think? Surveys (when well done) are useful for this purpose. Over 98% of scientists accept the fact that life on earth is the product of organic evolution and displays features of common descent. That is an overwhelming consensus that has endured for decades.

Percentages are more important that absolute numbers. Creationists try to bamboozle the public by trotting out as many creationist scientists as they can – distracting from the more important fact that >98% of scientists disagree with them.

Another good source of information is systematic reviews of the literature. This is a mechanism that experts use to develop their consensus, in fact. If multiple independent reviews all come to the same conclusion, that is a good indication of the consensus.

For politically controversial topics the process of determining the consensus can be challenging. This is because ideological groups have discovered how to best muddy the waters to obscure the consensus. Many have formed their own institutes (like the Discovery Institute for Intelligent Design) with the specific purpose of creating the impression of a scientific controversy where none exists.

Industry has discovered that they can pollute the scientific literature by funding poorly designed studies designed to produce a favorable outcome. Ideologically aligned organizations have formed to fund research to promote their agenda.

The worst manifestation of distorting the process are groups like antivaccinationists who have used bully tactics and intimidation to attack scientists who disagree with their agenda, or simply when they don’t like the outcome of their research.

Conclusion

The institutions of science are still strong and for most questions it is not difficult to find a reliable reflection of the consensus of scientific opinion, once you know what to look for. For most questions it is simply a matter of looking for statements put out by institutions or reviewing a broad cross-section of the scientific community, and not putting much weight on the opinions of a lone scientist – no matter what their credentials.

But the institutions of science can also be very precarious. There are actually many systematic attacks on the institutions and practice of science from a variety of directions – but generally from those with an ideological axe to grind. There is no shortage of well-funded groups looking to distort the process of science, and therefore the impression of consensus, to rig the game in their favor, or ensure a favorable outcome. Short of that they wish to muddy the waters – to at least sow confusion to keep as many people from understanding the scientific consensus as possible.

For this reason science needs constant defense and support, and ideological groups distorting science need watchdogs.

14 responses so far

14 Responses to “Scientific Consensus”

  1. zntneoon 30 Sep 2008 at 9:08 am

    Great post steve i was wondering I’ve heard, maybe on SGU, about Cochrane reviews. Would that be a good place to go for good reviews about medical topics?

  2. zntneoon 30 Sep 2008 at 9:09 am

    I remember now where i read about them. I read about them in the book “snake-oil science”

  3. Brian Eganon 30 Sep 2008 at 11:33 am

    Hi Steven,

    Perhaps a bit off topic, but could you clue me into the amount of scientific consensus there is around the alteration of the brain due to technology?

    I work at a library as their in-house web designer, and feel as though my generation has really been under attack from all sides lately. From “Dumbest Generation: How the Digital Age Stupefies Young Americans” by Mark Bauerlein to “iBrain: Surviving the Technological Alteration of the Modern Mind” by Gary Small, a “prominent neuroscientist” according to the Chronicle of Higher Ed tech blog.

    Bauerlein is an English professor, and despite claiming that the young generation engages in the research path of least resistance, as a person with a Sociology degree I can catch a number of his sloppy statistical errors and cherry-picking with studies. In fact, I would really applaud you in helping me understand the scientific method, because I think it’s quite clear Bauerlein started with his thesis (kids today are dumb) and then justified it with a number of studies (but never established that his generation was really all that intelligent to begin with). I don’t think he’s entirely off base that my generation could be smarter, but I think there are a number of things he leaves out, and felt angry rather than enlightened after reading this book.

    I haven’t had a chance to read iBrain, but I thought you might have some initial thoughts based on your reading on the impact of digital techology on our thought processing — is there a consensus as to what technology is doing to our brains, or are there interesting preliminary results that are being spun into pop-psychology self help books?

    Any thoughts would be appreciated from a young skeptic!

  4. themightylearton 30 Sep 2008 at 12:42 pm

    Don’t you just love it when IDiots throw around their 700+ list of scientist that doubt evolution? Even if we grant them the benefit of the doubt and assume their list is correct, Steve is right that they always fail to mention the other 700,000 who think the above mentioned 700 are morons!

    Someone once said something to the effect that “Galileo was a rebel, but not all rebels are Galileo”. Of course all those douchebags think/pretend they’re Galileo.

    Andrew Wakefield was an expert too, but look at all the stupidity and trouble he’s caused with his anti MMR crusade.

    And Jenny McCarthy…well she’s an expert all right but not in medicine or science for that matter.

  5. sonicon 30 Sep 2008 at 1:27 pm

    How might tell when scientific consensus is largely determined by ideological considerations?

    Consider physics- there are more experimental physicists trying to disprove theories than there are doing experiments to support existing theories. The goal is to find what is wrong with existing theory and produce a better one. If the new collider can’t find the higgs boson, for example, the new theories will be forth coming and each will be discussed and it may take years for a consensus to form.

    Consider the example from biology. From a recent article from Nature

    Michael J. Sanderson, “Phylogenetic Signal in the Eukaryotic Tree of Life,” Science, 4 July 2008: Vol. 321. no. 5885, pp. 121-123, DOI: 10.1126/science.1154449.

    An unbiased reading of the evidence would indicate that the evidence thus far amassed would be in favor of a theory that did not include a ‘tree of life’. (Mr. Sanderson assumes a tree of life and then tries to fit the data to it- his project is not going well. Under a null hypothesis, one would consider a different hypothesis)

    But anyone looking for evidence to disprove the notion of a ‘tree of life’ is subject to all sorts of Ad Homminem attack.

    In general, I would argue, science that is motivated by the attempt to overthrow the existing theory (discovery) is more reliable and less likely to be motivated by bias, than science that assumes the answer and tries to fit the data to agree with existing theory.
    Also any area of science in which ad hominem attack is a regular form of discussion should be questioned.

    Unfortunately we live in a time when biology is ruled by the arguments of ‘believers’ and athiests and the science suffers on both accounts.

  6. daedalus2uon 30 Sep 2008 at 3:23 pm

    sonic, I think you are misreading and/or misunderstanding what the paper says and means.

    There was no data in that paper or alluded to that is inconsistent with the hypothesis of a “tree of life”. The paper attempts to link 14,289 phylogenies using only 2.6 million sequences. That is less that a couple hundred sequences per phylogeny, some phylogenies are completely sequenced, some are very sparsely sequenced.

    If the hypothesis of a “tree of life” were false, then there would be “leaves” unlinked to “branches”, unlinked to “trunks”, unlinked to “roots”. None are observed, none have been observed.

    What the author is saying is that the precise details of the fine structure of the “tree of life” are not completely certain, but there has been much progress and there are still some gaps to be filled in.

    Being able to fit all extant organisms into a “tree of life” is a prediction of the hypothesis of common descent. A single organism that didn’t fit on the tree of life would falsify the hypothesis of common descent. That would be front page news in every biology science journal in the world. That would be a guaranteed “ticket to Stockholm” for whoever discovered it. That would be an extraordinary finding, and would require extraordinary evidence. A complete sequence of the organism’s genome showing no homology would be sufficient. Until someone does produce the sequence of an organism that is not homologous to other organisms in the “tree of life”, the scientific consensus is that all organisms on Earth are related, share a common ancestor and can be fit somewhere on a very large “tree of life”.

    In physics the situation is different. There is consensus in the physics community that the current understanding is incomplete and that quantum mechanics and general relativity are incompatible and both cannot be correct. How either or both are wrong is unknown, but because they are incompatible they both cannot be correct.

    So far there is no data incompatible with the “tree of life”.

  7. wastrelon 30 Sep 2008 at 3:59 pm

    So is there a percentage of the consensus available for AGW? Doe it approach evolution’s consensus?

  8. sonicon 30 Sep 2008 at 5:04 pm

    daedalus2u,
    From the article:

    “Across eukaryotes, however, although phylogenetic evidence is very broadly distributed, for the average species in the database it is equivalent to less than one well-supported gene tree.”

    If there is less than one well-supported gene tree- that means there is no supported gene tree- right? So, the majority of the organisms under study have no ‘tree of life’ that they fit into. This has not made any headlines.

    The researchers conclusion-

    ” This analysis shows that a stronger sampling effort aimed at genomic depth, in addition to taxonomic breadth, will be required to build high-resolution phylogenetic trees at this scale.”

    I might agree that further study would yield a tree. But the current evidence does not support the notion. It would not be unusual in physics that such a study would yield the question- what is wrong with this theory?

    Homology could be the result of things other than evolution as it is currently formulated.

  9. daedalus2uon 30 Sep 2008 at 6:30 pm

    No, what is being said that for the average species, there is not enough genetic information to create a high resolution “tree of life” for that species. That is not a surprise. Most species have not been sequenced to a large extent. If the genes that are homologous back to the root are not sequenced for a particular species, that species cannot be fit into the tree of life back to the root yet.

    There is no suggestion that there are any species that won’t fit into the tree of life once more of their genome is sequenced, but precisely where they do fit can only be known after the sequencing and comparing is done.

    The only other suggestion for homology that I am aware of is Intelligent Design which posits that a nearly omnipotent entity generated all extant species de novo with their genomes exactly configured so as to mimic the homology that would result from common descent. This is far more difficult than generating all extant species de novo. Creating the DNA being used so as to mimic common descent in all organisms simultaneously is quite a challenge. It requires not just attention to function, but attention to comparisons of all DNA with all other DNA and requiring that there be a mutational path that joins all of them simultaneously while retaining function. However this variant of ID makes no predictions that are different than the hypothesis of common descent.

  10. sonicon 01 Oct 2008 at 3:15 am

    daedalus2u,
    I don’t think we disagree about what the paper says.

    Look at it from a Bayesian perspective. I’m saying that unless one has a prior probability of ‘common descent’ very high, then the evidence as currently presented would bring about the question, ‘what is wrong with this theory?’ (Afterall, the vast majority of life forms under study have no tree they fit into as of the current data)

    And to get back to my original point-
    Both ‘believers’ and ‘non-believers’ have deep seated priors when it comes to evolution and common descent. These priors have little to do with evidence and this is an unfortunate reality in the world of biology today.

    And this has something to do with how I look at the ‘scientific consensus’ in this area.

  11. sadunkalon 01 Oct 2008 at 6:51 am

    I think sonic has a point. I can’t comment directly on the paper, but I got the impression that there is a lack of complete objectivity.

    And I completely agree with this argument:

    “Science that is motivated by the attempt to overthrow the existing theory (discovery) is more reliable and less likely to be motivated by bias, than science that assumes the answer and tries to fit the data to agree with existing theory.”

    And I believe that today only a very few, extremely privileged scientific fields are free from that bias and the 4 red flags Mr. Novella mentioned unfortunately:

    ” – The consensus is premature. (Not enough experiments)

    – A highly politically or ideologically charged atmosphere.

    – If the consensus exists only within a subculture, such as a fringe group looking to promote a predetermined conclusoin.

    – If the consensus is dominated by industry self-interest. ”

    I think those four points + sonic’s argument are interconnected too. Many -including the vast majority of the skeptics community- fail to see outside influences in modern science. Mainly corporations(aka. politics) determine the path scientific research will take, and they also have a big influence over the outcome of any research. Even the peer review system has significant weaknesses under such conditions.

    So I personally have a lack of trust for scientific authorities nowadays, especially when it comes to big issues. I wasn’t always like that, I got smarter this year. And I’m willing to do something about it actually, I’m trying to get more people to question those authorities, and that very fundamentally.

  12. daedalus2uon 01 Oct 2008 at 7:43 am

    No sonic, you do not understand what the paper says. The paper says we are not able to identify the unique “tree” that a particular species fits into. The paper does not say that any particular species does not fit into any tree.

    There are many possible trees that they fit into; the data simply isn’t of high enough resolution to tell which tree in complete detail. With more data, there is every confidence that there will be enough resolution to fit every species into its own unique tree of life. That simply means that for that species there is a unique path back to the last common ancestor. We already know there is a path from that last common ancestor to each extant species because each extant species has genes which derive from that last common ancestor.

    There was a complete “tree” from the last common ancestor to each extant individual alive today. In principle each individual alive today could be traced back through each generation, through each set of parents, through each ancestor back to the last common ancestor. There isn’t enough data available to do that (and very likely never will be). We know that each individual did have ancestors. Simply because that information has been lost doesn’t change that it happened. The alternative idea, that some individuals poofed into existence with no ancestors has no data to support it. Can it be proven that it never happened? Probably not. Has it ever been observed to happen? No it hasn’t. Is there any data that is only explained by individuals poofing into existence? No, there isn’t. What is the a priori probability of individuals poofing into existence? I think it is very low. If you have some data to suggest otherwise I would be happy to hear it, but I don’t think there is any.

    Every single bit of data collected so far is completely consistent with common descent. There is no datum (yes, singular form) that is inconsistent with common descent. That there is no datum inconsistent with common descent is not a “scientific consensus”, it is an observed fact, it is data. Observed facts (aka data) trump scientific ideas, theories, laws and consensus.

  13. DevilsAdvocateon 01 Oct 2008 at 10:14 am

    Thanks for an excellent ost on an important aspect of scientific discovery. One factor I’d include is that while Science holds the message, Science is rarely the messenger presenting it to the people. That requires the media. The problem is that most media are no less a corporate animal than the more obviously biased and self-interested corporations involved in the process, as well as the bias that can creep in when an academic entity relies on certain findings to keep research funds streaming.

    Ultimately, for folks like myself, a layman in virtually all but my particular field, it becomes exceedingly difficult to determine whether the media, the messenger, is presenting the message accurately and without bias. It’s a whole second level of difficulty.

    Excellent post.

  14. b_calderon 06 Oct 2008 at 10:43 am

    There is a reasonable amount of research and publication on the sociology of science that outlines both how the scientific method works and how scientists relate with in the social framework. Kuhn’s Structure of Scientific Revolutions and Collins’ Sociology of Philosophies are what I used besides seeing it in action. Don’t you love the way the blood drains from the face of a presenter when certain people start to ask questions after a paper is presented?

    Brian – I would concentrate of the researchers who do good work and ignore the bad ones. Let someone else shoot down the bloated blimps of fake ed research. Education research is particularly poor and we all know it, particularly after Bush effectively castrated the NSF Education Directorate moving funding to cretin monkey territory that would dependably echo his abstinence-only sex ed line. The MacArthur Foundation has this:
    http://spotlight.macfound.org/
    – among them is danah boyd (who evidently doesn’t capitalize) and is at M$’s Boston research lab now.
    http://www.readwriteweb.com/archives/microsoft_makes_key_hire_in_researcher_danah_boyd.php
    Her personal website is now offline, let’s hope she hasn’t become part of the Borg.

Trackback URI | Comments RSS

Leave a Reply

You must be logged in to post a comment.