Apr 21 2010

Computer Models for Medical Diagnosis

One of the premises of The Checklist Manifesto by Atul Gawande is that modern society is too complex for our humble monkey brains to handle. We cannot keep track of all the complexity that modern technology has created – tasks such as flying jumbo jets or doing surgery are accidents waiting to happen. So we should, he argues, use a mental crutch – the humble checklist.

I agree, and I would add medical diagnosis to the list of those tasks that tend to overwhelm our cognitive capacity. It takes years of practice, and a vast fund of ever-changing knowledge, to be an effective diagnostician, and even a low rate of failure can lead to unacceptable outcomes. As our medical knowledge explodes, we are becoming the victims of our own success. Systems have been put into place to help – such as requiring maintenance of certification for specialty boards, and continuing medical education, but it is still a struggle.

Computer technology, in my opinion, is the best way forward. Computers and information devices hold tremendous promise – doctors can now hold in their hands a searchable database of information that they can access at the point of patient care – right when they need the information. Don’t be put off if your doctor starts searching Google during your consultation. That’s a good thing – it means they know how to access information and are not afraid to do so, even if just as a check on their memory, or to make sure they haven’t forgotten anything or missed any new and relevant publications.

In addition to accessing information, computers can be used to process information. These applications are called expert systems (because they are meant to be used by experts), and while such applications have been developed for more than two decades, their use in practice is still minimal. Part of this is simply integration – the doctor cannot use what they do not have access to. But also these systems have been a bit cumbersome and doctors and hospitals need to learn how best to incorporate them into the workflow of treating patients. (That final step of implementation is all important, and often where such systems break down, because neither the doctors nor the computer programmers know how best to do it. )

But I am optimistic that we will increasingly incorporate expert systems into medical practice to aid in making proper diagnoses. In many diagnostic situations, the number of variables to consider and give proper weight is simply overwhelming. Doctors are affected too much by their quirky recent experience. They rely upon heuristics which are useful shortcuts but often inaccurate. And there are gaps in their knowledge. Doctors rely upon some combination of using a systematic algorithmic approach and gestalt diagnostic judgment. But we all suffer from the fact that humans are inherently poor at statistics, and our intuitions lead us astray. For example, we overestimate the predictive value of clinical signs that may be present with a specific disease, but also common in the healthy population.

Computers are far better at weighing multiple variables using precise evidence-based statistics – that is, once the proper model has been developed.

A recent study published in the BMJ reports on the development of one such model. The researchers reviewed thousands of cases of children presenting to the emergency room with a fever. Most of the time such fevers are benign and self-limiting, but 7% of the time they are the result of a serious bacterial infection (pneumonia, meningitis, urinary tract infection). How good are doctors at deciding who to treat with antibiotic and who to send home? In this study 66-81% of children with bacterial infections were treated with antibiotics on first presentation. This is not great. Of those who were not treated, about two-thirds eventually were. Part of the reason for the low initial treatment is that physicians waited until the cultures came back to start antibiotics – not unreasonable, but not optimal. The authors note:

We identified two major, potentially correctable, difficulties in the current diagnostic decision making process. Firstly, in combining the demographic items and clinical symptoms and signs related to febrile illness, the physicians tended to underestimate the likelihood of serious bacterial infection. There are too many relevant signs and symptoms for doctors to assimilate effectively; instead, they tended to discount the information and underestimate the probability of serious disease. As such, the full diagnostic value of current clinical tests was often not reached. Secondly, where near patient tests were available, such as urinalysis for urinary tract infection and chest radiograph for pneumonia, errors in interpretation meant that serious bacterial infection was left untreated at the initial presentation.

Basically – there are too many variables for doctors to consider, so they tend to simplify things by discounting some variables, and this leads to error. Computers, however, don’t get overwhelmed, they just do what you tell them. So the researchers used the information from these thousands of cases to develop a computer model to determine who has a serious infection and who doesn’t. They then validated that model with the same data set, and found that it did as well or better than the physicians. The model still needs to be validated with fresh data – how well does it predict new cases and how does it effect final outcome? But the results are promising.

This does not mean that computers will be practicing medicine anytime soon. Physicians areĀ  needed to gather data from patients, and human judgment is needed for things like – how sick does this person look. Also, clinical decision making is still beyond current software technology – that level of artificial intelligence is not here. Expert systems are meant to aid experts, not replace them.

Overall, in my opinion, this technology is underutilized. I think a push is needed to develop more expert systems and incorporate them into patient care workflow. These systems can help reduce mistakes and optimize care. In the new era of cost control in medicine it should also be considered that such systems can potentially save a great deal of health care dollars. The Obama administration and others have focused on electronic medical records. I am a big fan of EMR, but the evidence so far shows that they do not save money. We need to look at expert systems – I think they may have more potential.

For many tasks, our humble gray matter is simply not sufficient, and we need help, such as checklists. And for other tasks, such as complex diagnostic situations, even checklists are insufficient – we need computing power. These systems exist and they work, and I hope to see more of them incorporated into practice.

15 responses so far