Continued from Part I

5) How to Analyze a Scientific Study

I don’t expect a non-scientist, or even a scientist far outside their area of expertise, to be able to do a detailed analysis of the strengths and weaknesses of a study. That is what peer-review is for. However, there are some basic rules of thumb that could give even a lay person a rough idea how seriously they should take a study. Always ask at least the following questions:

Is the study controlled in some way? Was the treatment group compared to a control group, or was the alleged effect compared to some baseline?

Is the study blinded? Were the primary measurements or assessments performed by someone who was blinded to whether or not the alleged effect is supposed to be present?

Are the outcomes being measured subjective or objective? How are they being measured? What do they really mean?

How large is the study? Studies with small numbers of subjects or measurements (less than 50 per group is a good rule of thumb) are considered small and unreliable.

Is the study an observation or an experiment? Are they just looking for some correlation  (in which it is difficult to make statements about cause and effect), or are they controlling for variables and isolating the one factor of interest?

What is the reaction of the scientific community to the study? Are experts generally critical or excited about the results?

6) Most Studies are Wrong or Incomplete

Individual scientific studies vary on a spectrum from garbage to highly rigorous. No individual study is definitive or is the final answer to any scientific question. Researchers have unconscious bias. Rigorous studies are extremely difficult to pull off and scientists almost always make compromises or trade-offs. Scientists also make mistakes or fail to consider subtle effects or sources of error.

Therefore, don’t base any conclusions on individual studies. They just don’t tell us that much.

Studies need to be independently replicated. That is the only way to really know if a phenomenon is real or just illusion or error.

When trying to answer a scientific question, look to systematic reviews or expert summaries that evaluate all of the evidence and come to an overall conclusion.

7) Consensus Matters

When a community of experts largely agrees that the scientific evidence leads us to a certain conclusion, take that conclusion very seriously. Scientific consensus is not infallible (and again, look to the confidence level or error bars) but it is the best we have.

If as a non-expert you do not agree with or understand the scientific consensus you need to understand that it is overwhelmingly likely that you are wrong, rather than the majority of experts who have dedicated their careers to studying a topic in depth are wrong. Seriously – scientists are generally pretty smart people, they are thoughtful, they spend their time thinking about their area of expertise, reading about it, discussing it with their colleagues, debating minor technical points, going to conferences, and doing real research.

The difference between a non-expert’s level of understanding and an expert’s level of understanding is vast. Chances are you don’t even know enough to have a real grasp of the vastness of your own ignorance. It is reasonable to assume they are probably right. If you have thought of a potential problem, then they have thought of that problem. Listen to what they have to say.

It is also important to understand that individual scientists can be wrong and can have quirky opinions and biases. Don’t trust the opinions of one scientist; they may be in the minority. Within a community of scientists, however, individual biases will tend to average out and reliability goes way up.

8) Understand Pseudoscience

There is no clear line of demarcation between science and pseudoscience, but it is important to recognize the features of pseudoscience and why they are pathological. Here are some quick red flags to help recognize pseudoscience:

– An individual scientist or group is an outlier of the scientific community. This is not a guarantee they are wrong, but there are hundreds if not thousands of cranks for every visionary, so if you’re playing the odds, be very skeptical of outliers.

– Scientist is hostile to criticism. Criticism is part of science, it is how the community hammers out what is likely to be true. Extreme hostility to the normal constructive criticism that is part of science is a huge red flag.

– Related to the above point, pseudoscientists tend not to be integrated into the relevant scientific community. They are working alone on the fringe, and not testing their ideas with those who are in the best position to judge them and give them critical feedback.

– Claims seem to be way out of proportion to the evidence. Proponents may claim a huge paradigm shift based on the most preliminary of evidence, reject competing theories without adequate justification, rely exclusively on their own research, and are not skeptical of their own claims.

– It seems as if they are working backward from their desired conclusion, rather than following the evidence. Related to this, it may seem as if they are working under an obvious ideology, and looking for evidence to support that ideology.

9) Understand Denialism

There are various common strategies for denying well-established science. They include:

– Setting impossible standards for evidence, and then moving the goalpost when new evidence comes to light.

– Dismissing the consensus as a conspiracy, dismissing proponents of the real science as shills, and dismissing any evidence as flawed regardless of quality. Deniers will also engage in witch hunts in which the normal everyday connections that scientists and academics make as part of their job are used as evidence for a conflict of interest or a real conspiracy.

– Change definitions as needed to sow confusion, evade conclusions, and dismiss evidence.

10) Understand that humans are flawed and biased. 

This is admittedly a big category, and contains much of what we call skeptical philosophy. However, it is important to understand the basic fact that we cannot rely on casual observation or thinking because human brains are biased and flawed. We need rigorous logic and methods of observation to compensate for those flaws and biases, and that, essentially, is what science is.

Here is a quick list of the most important concepts:

– Confirmation bias. We tend to notice, remember, and accept bits of information that support our current beliefs or narrative, and ignore or reject any information that contradicts our beliefs. This can create the powerful illusion that the facts support our position.

– Memory is flawed and constructed. Every time we remember something we are reconstructing the memory, and adjusting it to fit our current beliefs. Our brains are more concerned with internal consistency than accuracy.

– Our perceptions are constructed fictions that evolved to be useful, but not necessarily accurate.

– Motivated reasoning. We are very good at inventing reasons to support our beliefs. This is why it is critical to challenge your own beliefs (not just seek to support them), to consider alternatives, and to seek out and be open to differing views.

– Placebo effects, superstitious thinking, and pattern recognition. These phenomena are related in that they all reflect a tendency to assume that two or more things that appear to be associated are causally related. If we take a treatment then feel better, we assume the treatment made us better. If we wear our favorite sweater and our sports team wins the playoff game, it was because we wore the sweater. If weird or coincidental things occur, there must have been an underlying reason.

– Logical fallacies. There are many formal and informal logical fallacies, too many to discuss here, but here is a good overview. Just be aware that there are valid and invalid ways of thinking and we need to be careful about our own arguments.

Conclusion

Scientific literacy means not only having a working understanding of the big ideas of science, but also understanding critical thinking and how science works. This list of the basic components of the latter two is certainly incomplete, and I welcome feedback about what else should be included.

These ten categories of literacy regarding the scientific process are meant to be a quick overview of the basics, what I consider to be the minimum that anyone should know in order to be truly scientifically literate.