Because I am an activist skeptic I am often asked specific questions about how to be a better skeptic. This is obviously a complex question, and I view skepticism (like all knowledge) as a journey not a destination. I am still trying to work out how to be a better skeptic.

One recent question, however, took a great approach to the issue of practical skepticism – what questions should a skeptic ask themselves when confronted with a news item? Here is my process:

1 – How plausible is the claim?

This is admittedly a tricky question that requires a lot of judgment. The risk is that you will think any claim that already aligns with your beliefs as being plausible and anything that contradicts them as being implausible. This is not as bad as it sounds, however, if your current beliefs are based on logic and evidence. To the extent that your beliefs (by which I mean the model of reality that you construct in your head) are based on ideology and subjective perspective, the notion of plausibility can be self-fulfilling.

I say “can be” because it does not have to be. This is partly because this first question regarding plausibility is the first question, not the only question. You should not reject implausible claims out of hand. The purpose of evaluating plausibility is to determine the appropriate bar of evidence needed to accept the claim. This is essentially, “Extraordinary claims require extraordinary evidence.”

If a news item claims that astronomers have found an interesting new exoplanet, I can accept this claim as long as I have a decent reference, preferably the primary reference.  If someone claims they have discovered the remains of Bigfoot, I will not accept that claim until the evidence has been thoroughly independently vetted and other more plausible interpretations have been adequately ruled out.

The quality and amount of evidence has to be in proportion to the claim, which means you need to have some sense of how plausible the claim is. Regardless, however, I can accept any claim as long as the evidence is sufficient.

2 – What is the source?

Not all sources of information are equivalent. One of the primary problems with the way information is shared on social media is that it tends to scrub information of its sources. Everything becomes a generic news item spread on Facebook or Twitter. People already have what psychologists call “source amnesia,” meaning we tend to remember facts but not where those facts come from. How many times have you heard or said, “I read somewhere that…”?

Apparently remembering the source of information was of insufficient evolutionary advantage to dedicate significant gray matter to.

In our complex modern world, however, knowing the source of information is critical. For any news source I have three basic questions: what is their overall quality, what is their overall bias, and how primary is the source. The best sources, obviously, are those who have the highest journalistic standards and make a genuine attempt to be ideologically and politically neutral.

If a source has a clear ideological agenda, that taints them as a source. I don’t trust an environmental group to give me objective information about a controversial environmental issue. That does not mean they are worthless as a source, but I cannot rely upon them to be objective. I assume they bias their analysis and which information they choose to convey to fit their agenda. The same is true for information that comes from any vested interest. People know not to trust companies with information about their own products. It helps to look at all information as a product.

I addition to quality and bias, there is also primary vs secondary sources. You should always go back to the primary source when possible, or as close to it as possible. For scientific studies, this means going to the original study. If you are not a scientists or academic with access to journal articles, you can usually at least access the abstract for free. That is a quick summary of the research with the bottom line conclusion. If written well it is helpful in evaluating what the research actually did and found. This is not sufficient for a full evaluation of the research by an expert, but is helpful for a layperson to tell if the reporting is at least accurately portraying the research.

3 – What is the consensus?

Beyond analyzing individual sources, it is helpful to determine if the claim represents a consensus of those who would likely know or who have the proper expertise.  A good rule of thumb is never to trust any single source. So, when I find an interesting news item, I will quickly search to see what other independent sources are saying about the same news.

One pitfall here is that often a single news source will be reprinted multiple times by different outlets. Often I find that most or even all news outlets are just reprinting the same press release. Many journalists today are independent and submit their work to news organizations like the Associated Press. A news outlet can then get their stories from the AP rather than having their own journalists, especially for national or international news.

For science items it is not uncommon to see the press release reprinted, often without any change, by multiple outlets.

It often takes digging, searching on the people involved or the general topic, to find independent sources of information.

I also always want to find out what outlets with different editorial biases are saying. Until you know what the critics are saying and why you don’t really have a good sense of the claims being made. This means searching for the contrary opinion, or a skeptical analysis. Eventually you want to get to the point that you have a fairly thorough understanding of the various perspectives, their arguments, and their evidence, and how they reply to claims and criticisms from the other side. Only then can you really get a feel for which side has the better case.

If you already have an opinion on an issue, you should seek out contrary opinions, and deal with the most charitable version of opposing positions.

4 – What is the context?

Information and claims do not exist in a vacuum. Understanding the background to a story will help to interpret any new claim. In science, this means understanding what else we know and how the claim fits into the overall scientific picture of reality.

It is important to understand if there is a controversy over a topic, and what the history of that controversy is. If you didn’t know, for example, the relationship between libertarian ideology and the fossil fuel industry, or the organic food industry and anti-GMO propaganda, you would not be able to put a specific claim into the broader context.

For a scientific study, that study does not exist in isolation, but as part of an ongoing research program exploring certain questions. There are often different lines of evidence with different strengths and weaknesses. There may be different scientific disciplines with different approaches. Geologists and paleontologists use different tools to explore the question of what happened 65 million years ago that resulted in the extinction of non-avian dinosaurs.

5 – What are your own biases?

Asking the above questions is a process, but it is not a cookbook or algorithm. You can’t just go through the steps and get to the right answer. The process requires judgement, and there may be no absolutely right answer in the end. There may be unknowns or subjective elements.

Whenever judgement is required, it is a good assumption that your own biases are going to have an effect. The ultimate question, therefore, is – what are your own biases? You should be the most skeptical of information that plays right into your existing belief, especially if you are emotionally invested or they are a source of identity. It is reasonable to try to be as neutral and unbiased as possible, but you will never achieve this state, so at least you can be aware of your biases.

This also means trying to back up from any claim and understand what your narrative is. The most subtle and seductive intellectual trap is to have a narrative and not know it. As vigilant as I try to be about this, I still find myself falling into this trap. There are skeptical narratives, for example, such as the notion that con artists deceive people deliberately for nefarious purposes. While this certainly happens, it becomes an easy narrative that we can accept too quickly. Reality may be more complex.

In addition to biases, there are all the cognitive pitfalls that I write about frequently here. Memory and perception are flawed, and human reasoning tends to take certain pathways of least resistance. We also are emotional creatures and our cognition is biased to reduce cognitive dissonance and enhance our emotional comfort.

Conclusion

There is no formula for evaluating a claim, but there is a process. Of course, not everyone has the time or resources to do a deep dive on every single claim. Life is too short. However, if you have not looked deeply into a topic then perhaps you should reserve judgment, or at least acknowledge that you do not have a solid basis for a strong opinion.

You don’t always have to put your nickle down. It is OK to say that you simply do not have a strong opinion because you recognize a question is complex and you have not taken the time to look into it thoroughly.

It is also reasonable to find sources that you generally trust. In essence, you can put the work into finding reliable sources with relatively low bias, or at least you understand their bias, and rely on them to summarize much of the above process for you. A great science communicator, like Neil Tyson, can serve this function. But I would not rely too heavily on any one source, as I said above. Having a few reliable high quality sources to get you up to speed on a topic is not a bad idea.

But there is no substitute for going through the process yourself. The process is also open-ended and is never done. This includes developing a basic scientific literacy and critical thinking skills.

It would be nice if most people came out of basic public education with these skills. Alas the evidence clearly shows this is not the case.