People generally suck at statistics. Our innate sense of how likely something is does not accord very well with reality, especially for large numbers.

But don’t worry, this just means you have to think a little harder about how likely things are. David Hand writes about this in his 2014 book: The Improbability Principle: Why Coincidences, Miracles, and Rare Events Happen Every Day. This is making the rounds again in the media because of the recent “rare” astronomical events.

Yesterday the Summer Solstice coincided with the Strawberry Moon – the first full moon in June. The last time this happened was in 1967. Recently we have seen “rare” transits of Mercury and Venus across the sun.

These events are not that rare, and I really don’t see what the fuss is all about (I guess the media is desperate for anything they can hype.) Don’t get me wrong, I love astronomical events, it is their rarity that I think is overhyped.

Even more impressive are the times that someone wins the lottery twice. This seems like it should be so unlikely that it would never happen, but it happens all the time.

Hand and others have pointed out what is going on here, which can help you think a little more critically about such events.

The Law of Large Numbers

First it is important to recognize that when you have lots of opportunities for unlikely things to happen, they are bound to happen by chance. As I like to say, in New York City, which has a population of over 8 million people, a 1 in 8 million coincidence should happen every day.

There are over 7 billion people on the planet. There has to be some damn unusual stories out there. Because of how connected the world is, and the amount of mass and social media, you are very likely to hear about the unlikely things that happen to people.

Data Mining

Regardless of population size, the number of opportunities for chance alignments is greater than you think – even if just considering one person. There is another way to dramatically increase the probability of an event that, taken in isolation, is rare, by mining lot of data.

For example, what if you dream of a friend you have not seen in 20 years, and then the next day another friend of your mentions the friend about whom you dreamt. That would seem like a very unlikely coincidence – perhaps evidence of a psychic connection.

However, what you are failing to consider are all the details contained in your dreams, and all the details of your daily life. You are unconsciously mining a vast amount of data to come up with one alignment of details.

This confusion is often referred to as the lottery fallacy, which is considering the odds of John Smith winning the lottery after the fact that he did win, rather than considering the odds of anyone winning the lottery. The former is very unlikely, the latter is a certainty.

Cold readers depend upon this phenomenon as well, throwing out details and hoping that the target of the reading with mine the data of their lives and find some connection, and then consider that specific connection to be unlikely.

Open Ended Criteria

Probabilities can also be altered dramatically by using open ended criteria. My favorite example of that is the notion that celebrities die in threes. For people who believe this, it seems like the evidence always supports their belief.

The problem is that the criteria are open ended – what constitutes a celebrity, and over what time frame? A time frame is never stipulated, and so once a celebrity dies all you have to do is wait for two more to die, and bingo, the rule of threes is validated.

Hand discusses this with respect to the recent “coincidence” of the June full moon and the Summer Solstice. What time frame is the moon considered “full.” The peak fullness of the moon is a moment in time (7:02 am Monday, Eastern time). How much time on either side do we consider the moon “full?” Twelve hours, one day, two days? The same is true of the Solstice – it is a moment in time, but generally we consider the entire day the Solstice.

This comes up a lot in medical research. What are the criteria for being sick, for getting better, for having a response to the treatment? This is one of the fatal weaknesses of anecdotal evidence, the criteria for endpoints are not specified. If you think vitamin C prevents colds, you may not count those “sniffles” as a real cold.

Conclusion

We can unconsciously increase the odds of a “rare” event occurring by post hoc analysis, open ended criteria (wiggle room), and mining large sets of data. In fact, we are inherently good at doing all of these things. We might call the overall process pattern recognition. Our brains evolved to make connections, to see patterns, and then to imbue them with meaning.

In fact, once our brains barely sees a possible pattern, it tries to bring it out, to make it more obvious. It does this by adjusting our perception to highlight the apparent pattern. Visually we call this pareidolia (seeing a face in random noise, for example), but it is a more general mental phenomenon.

Memory then kicks in, altering the record of what happened to further enhance the apparent pattern. Did you really dream of that friend you hadn’t heard of in 20 years, or was it someone else, or just a vague dream character. Then your memory of the dream shifted and aligned itself with the distant friend once their name was mentioned the next day.

Our memory also shifts in specific ways, such as remembering two events as happening closer together if we think they are causally related.

Confirmation bias is part of the process also – we notice, accept, and remember data points which support the pattern, and we ignore, reject, or forget those data points that contradict the pattern. Eventually we are absolutely certain the pattern is real, because we think we have experienced it first hand. The data seems overwhelming, because it has been systematically and unconsciously mined from a vast set of potential data.

If you think rare events are happening more often than they should by chance, do not look for some explanation in the universe. The explanation is in your brain.