Or – how knowledge is fungible.
We recently received the following letter
Hi SGU team.
The following may interest you.
I work in a pharmaceutical laboratory and we pride ourselves on using logic, careful experimental design and hypothesis testing to solve testing issues. We recently came across a problem that was eventually only solved by identifying that we had inadvertently made a “Selection Bias” logical fallacy and we (I) only worked this out because I listen to your show and absorbed at least some of the brain training you guys so kindly give out.
Until recently the commonly accepted wisdom in our lab was that we were experiencing a long-standing and ongoing instrument issue which somehow caused the first standard of an analysis to generally read lower results than subsequent standards. This was reported by most analysts and was accepted as a real issue.
We spent a good deal of time trying to problem solve the issue by investigating the instruments, the sample matrix, the vials, the analyst technique, environmental conditions, etc (the list goes on) but to no avail – the problem seemed to continue despite our best efforts.
Finally my brain kicked in and I suggested to my colleagues that we may be experiencing “Selection Bias” – what if there is nothing wrong with the first standard and there is in fact no issue? Dismissed out of hand I went ahead anyway and statistically sampled a good random selection of result sets and normalized them. Low and behold the first standard, on average, agrees with the nominal mean – ie there was no statistically significant issue to begin with!
In short we were accepting anecdotal evidence as a true and real phenomenon and went straight to looking for the cause instead of first verifying that an actual, statistically significant, issue existed in the first place. Doh!
Every subsequent statistical analysis reconfirms the first one I did yet there are still die-hard’s who are sure that we are seeing a real phenomenon.
I think I am reasonably intelligent but this experience shows how easy it is for anyone to fall victim to fallacious thinking.
Thanks for the heads-up – thought you would like to know you are making a difference – in my world at least.
I just wanted to share this as an excellent example of how knowledge is fungible – in other words, you never can tell how knowledge in one area will apply to another. Often we are asked what the utility is in examining and “debunking” silly claims like Bigfoot, UFOs, and crop circles. This is why – because you can learn a great deal from a careful examination of any claims.
Thanks for the letter, Barry.