There is a general consensus that people overall do not have a good intuitive grasp of statistics. In addition, there are multiple biases filtering our perception and memory. Therefore we tend to engage in a biased evaluation of biased data. The entire gambling industry depends on this fact.

As an example, consider the famous Monty Hall problem. In the classic game show, Let’s Make a Deal, the host Monty Hall would often show three doors to the contestant. In this problem Monty Hall says that behind two of the doors there are goats and behind one there is a new car. The contestant chooses one of the three doors. Monty (who knows where the car is) then opens one of the doors the contestant did not choose to reveal one of the two goats. He then offers the contestant the opportunity to change their pick to the other remaining door. Should they switch?

The answer is unequivocally yes. Yet many people have a very difficult time grasping the statistics behind the explanation. In fact, statistics can be so counter-intuitive that the world’s psychology researchers may in fact perpetuate an error for thirty years before someone realizes it. That is what two researchers (Miller and Sanjurjo) are now claiming.

Hot Hands

The current debate surrounds the question of whether or not there is a phenomenon known as hot hands. Intuitively, many basketball players (and others engaged in similar activities) believe in hot hands, the notion that hits and misses come in streaks. When you are in the groove you are “feeling it” and your confidence is high, you will shoot well. Alternately, after one or more misses you may be choking, feeling the pressure, and your confidence is low resulting in more misses. Many players swear they observe this phenomenon in themselves and other players.

In 1985, however, Gilovich, Vallone, and Tversky published a classic paper in which they showed that the hot hands phenomenon did not exist. They observed real life basketball outcomes, and did experiments in which subjects threw basketballs in controlled conditions. They found no hot hands.

They approached the question this way – if you score one or more hits, what is the likelihood that the next shot will be a hit or a miss? The hot hands hypothesis predicts that the probability will be above the average hit rate for a player, but the researchers found that the probability was the same.

In other words, the data look as if each shot at the basket is an independent event that does not depend upon what happened previously. Gilovich et al argued that the hot hands phenomenon was nothing but a cognitive illusion. This sparked the great hot hands debate which has raged ever since. The problem is that the statistical result is so counter-intuitive.

In general, Gilovich’s result has held up for the past thirty years. There are some papers that show, at best, a very minor hot hands phenomenon in the data examined, but nothing close to what is subjectively observed. There remains a significant disconnect between casual observation and rigorous statistical analysis.

Miller and Sanjurjo, ironically, are now arguing that Gilovich and other hot hands researchers are all also victims of a cognitive bias in the way they have viewed the statistics. In their paper (which is explained further here) they argue that previous researchers have been looking at the problem incorrectly. The proper way to look at the data, they argue, is to consider how many opportunities there are for a hit to follow a hit, and then calculate the percentage of successful opportunities.

They show that, if you have a coin-flip situation where there are independent events with a 50/50 probability, when you look at runs of data, in only 40% of the situations in which there is an opportunity for a hit to be followed by another hit does this actually occur (rather than the 50% that everyone has assumed). Therefore, if actual data shows that 50% of the time there is a hit, that is actual evidence for a hot hands phenomenon, not evidence against it.

If you are having a hard time grasping this, that is the point I am making. Just as our intuitive feelings about the Monty Hall problem are wrong, it may be the case that our intuitive feelings about how to analyze streaks of events are also wrong.

Conclusion

The paper is published online and has been open reviewed. However it has not been peer-reviewed as yet. So far it seems to be holding up, but it’s still early.

What is almost guaranteed to happen is that this paper will spark another round of hot-hands debate. Is this the proper way to analyze the data? If you use this analysis to relook at previous data, does a hot-hands phenomenon emerge? How big is the phenomenon? It may turn out to be statistically present, but still a tiny phenomenon that no one should be able to observe casually, meaning that the original point of Gilovich et al is still valid – the observed hot-hands is largely an illusion.

I understand what Miller and Sanjurjo are saying, but I haven’t really grappled with it intuitively, meaning I don’t feel confident I understand why their analysis is correct. It’s possible, from my perspective, that it is they who are committing a subtle error in the way they are approaching the data, introducing some new bias they are missing.

I leave it to the experts to sort it out, and I just have to hope that someone is able to explain it to me in a way I can understand.

For me this paper does not change an important bottom line – statistics is often more complex and subtle than our intuition. Caution and humility are still necessary.