One of the basic skills of critical thinking in our modern society is, first, the realization that not all scientific studies are created equal. Studies range from worthless to rigorous, and not all “studies” are even actual studies, but rather just surveys or reviews of prior research.

Further, it’s helpful to be able to look at a study and evaluate it for basic issues of quality. There will often be technical details that only an expert in the field will recognize, and statistical analysis is a specialty unto itself. But basic research concepts could apply to any study and give you at least an idea of how reliable it is. There are other generic factors as well, such as the quality of the journal in which it was published, the funding source, the history of the researchers, and whether or not the paper was ever retracted. Finally, any individual study needs to be put into the context of the overall literature, and not just cherry-picked.

Vaccinated vs Unvaccinated Study

Recently the anti-vaccine community has been sending around a study on social media that purports to show that vaccinated children have a higher rate of neurodevelopmental disorders (NDD) than unvaccinated children. This contradicts a wealth of prior studies that show that the only health difference between vaccinated and less vaccinated (fewer vaccines and/or given later) is that the less vaccinated children have more infections, especially vaccine-preventable diseases. 

Orac gives a good history and analysis of this study. The study was first published in a low-rent open access journal, Frontiers in Public Health. Actually, for some reason only the abstract was published. Then, in the wake of criticism of the quality of the study, it was retracted. The author has apparently been shopping it around since then and has found and even lower rent journal who will publish it, the Journal of Translational Research.

The lead author of the study, Anthony Mawson, is clearly sympathetic to anti-vaccine views, and the text of the study also gives away this bias. Funding for the study was coordinated by anti-vaccine activists.

So it’s ticking all the red flags for a dubious and biased low-quality study, but the actual quality of the study is what matters, so let’s take a look. Here is a description of the methods:

The study was designed as a cross-sectional survey of homeschooling mothers on their vaccinated and unvaccinated biological children ages 6 to 12. As contact information on homeschool families was unavailable, there was no defined population or sampling frame from which a randomized study could be carried out, and from which response rates could be determined. However, the object of our pilot study was not to obtain a representative sample of homeschool children but a convenience sample of unvaccinated children of sufficient size to test for significant differences in outcomes between the groups.

We proceeded by selecting 4 states (Florida, Louisiana, Mississippi, and Oregon) for the survey (Stage 1). NHERI compiled a list of statewide and local homeschool organizations, totaling 84 in Florida, 18 in Louisiana, 12 in Mississippi and 17 in Oregon. Initial contacts were made in June 2012. NHERI contacted the leaders of each statewide organization by email to request their support. A second email was then sent, explaining the study purpose and background, which the leaders were asked to forward to their members (Stage 2). A link was provided to an online questionnaire in which no personally identifying information was requested. With funding limited to 12 months, we sought to obtain as many responses as possible, contacting families only indirectly through homeschool organizations. Biological mothers of children ages 6-12 years were asked to serve as respondents in order to standardize data collection and to include data on pregnancy-related factors and birth history that might relate to the children’s current health. The age-range of 6 to 12 years was selected because most recommended vaccinations would have been received by then.

The obvious problems should be immediately apparent. The fact that they decided to focus on homeschool families is odd, which they explain as a “convenience” sample. However, this opens the possibility that the study subjects are not representative of the general population.

The biggest problem, however, is that this is a self-selective survey. Rigorous study design is all about controlling for variables, what scientists call confounding factors. So, if we want to know what the overall health effect is of being fully vaccinated vs partially or entirely unvaccinated, we need to isolate vaccination as a variable and control for all other possible variables.

The best way to do this is by choosing subjects at random to either be vaccinated or not. However, this study will never be done because vaccines are already part of the standard of care, and it is grossly unethical to randomize a research subject into an arm that deprives them of standard care. Without randomization, parents are choosing whether or not to vaccinate their children and there are many variables that might influence this decision, from opportunity to holding anti-vaccine views.

While this ideal study will never be done, there are ways to get reasonably good data with careful methods. In such studies, however, it is especially important to use rigorous and thorough methods of acquiring data, and have very thorough data, so that possible confounding variables can be controlled for as best as possible.

That is not what this study does, however. This is a survey of a non-representative population of undetermined size. So, parents will decide whether or not they will bother responding to the survey, and they are entirely responsible for the information provided. We don’t know the response rate because we don’t know how many parents received the request. The lower the response rate the greater the potential effect of self-selection.

A self-selective self report of information of non-randomized subjects in a nonrepresentative population of undetermined size. All of that adds up to – worthless crap. The information in this study is literally worthless. Individually these are horrendous flaws, cumulatively they are fatal. That is why the study was retracted from an open-access journal with a less-than-stellar reputation.

And what did this study find? Exactly what the authors and funders of the study wanted to find. Bias is always an issue, but if a study is transparent and rigorous enough, I don’t worry as much about the funding source. When a study has methods this bad, there is no reason to think that the results are anything other than a reflection of bias.

This study does show one thing, though – the anti-vaccine community is willing to cherry-pick any study no matter how worthless as long as it shows what they want.

Meanwhile they continue to deny the existence of  vaccinated vs unvaccinated studies (and are even touting this as the first such study). That is simply not true. Here is a summary of previous studies comparing these populations. They show that vaccinated children are healthier than unvaccinated children and have no increase in NDD or any other adverse outcomes.

That is one of the hallmarks of pseudoscience and denial, ignoring better quality studies while flogging lower quality studies because they show what you want.