Apr 21 2010

Computer Models for Medical Diagnosis

One of the premises of The Checklist Manifesto by Atul Gawande is that modern society is too complex for our humble monkey brains to handle. We cannot keep track of all the complexity that modern technology has created – tasks such as flying jumbo jets or doing surgery are accidents waiting to happen. So we should, he argues, use a mental crutch – the humble checklist.

I agree, and I would add medical diagnosis to the list of those tasks that tend to overwhelm our cognitive capacity. It takes years of practice, and a vast fund of ever-changing knowledge, to be an effective diagnostician, and even a low rate of failure can lead to unacceptable outcomes. As our medical knowledge explodes, we are becoming the victims of our own success. Systems have been put into place to help – such as requiring maintenance of certification for specialty boards, and continuing medical education, but it is still a struggle.

Computer technology, in my opinion, is the best way forward. Computers and information devices hold tremendous promise – doctors can now hold in their hands a searchable database of information that they can access at the point of patient care – right when they need the information. Don’t be put off if your doctor starts searching Google during your consultation. That’s a good thing – it means they know how to access information and are not afraid to do so, even if just as a check on their memory, or to make sure they haven’t forgotten anything or missed any new and relevant publications.

In addition to accessing information, computers can be used to process information. These applications are called expert systems (because they are meant to be used by experts), and while such applications have been developed for more than two decades, their use in practice is still minimal. Part of this is simply integration – the doctor cannot use what they do not have access to. But also these systems have been a bit cumbersome and doctors and hospitals need to learn how best to incorporate them into the workflow of treating patients. (That final step of implementation is all important, and often where such systems break down, because neither the doctors nor the computer programmers know how best to do it. )

But I am optimistic that we will increasingly incorporate expert systems into medical practice to aid in making proper diagnoses. In many diagnostic situations, the number of variables to consider and give proper weight is simply overwhelming. Doctors are affected too much by their quirky recent experience. They rely upon heuristics which are useful shortcuts but often inaccurate. And there are gaps in their knowledge. Doctors rely upon some combination of using a systematic algorithmic approach and gestalt diagnostic judgment. But we all suffer from the fact that humans are inherently poor at statistics, and our intuitions lead us astray. For example, we overestimate the predictive value of clinical signs that may be present with a specific disease, but also common in the healthy population.

Computers are far better at weighing multiple variables using precise evidence-based statistics – that is, once the proper model has been developed.

A recent study published in the BMJ reports on the development of one such model. The researchers reviewed thousands of cases of children presenting to the emergency room with a fever. Most of the time such fevers are benign and self-limiting, but 7% of the time they are the result of a serious bacterial infection (pneumonia, meningitis, urinary tract infection). How good are doctors at deciding who to treat with antibiotic and who to send home? In this study 66-81% of children with bacterial infections were treated with antibiotics on first presentation. This is not great. Of those who were not treated, about two-thirds eventually were. Part of the reason for the low initial treatment is that physicians waited until the cultures came back to start antibiotics – not unreasonable, but not optimal. The authors note:

We identified two major, potentially correctable, difficulties in the current diagnostic decision making process. Firstly, in combining the demographic items and clinical symptoms and signs related to febrile illness, the physicians tended to underestimate the likelihood of serious bacterial infection. There are too many relevant signs and symptoms for doctors to assimilate effectively; instead, they tended to discount the information and underestimate the probability of serious disease. As such, the full diagnostic value of current clinical tests was often not reached. Secondly, where near patient tests were available, such as urinalysis for urinary tract infection and chest radiograph for pneumonia, errors in interpretation meant that serious bacterial infection was left untreated at the initial presentation.

Basically – there are too many variables for doctors to consider, so they tend to simplify things by discounting some variables, and this leads to error. Computers, however, don’t get overwhelmed, they just do what you tell them. So the researchers used the information from these thousands of cases to develop a computer model to determine who has a serious infection and who doesn’t. They then validated that model with the same data set, and found that it did as well or better than the physicians. The model still needs to be validated with fresh data – how well does it predict new cases and how does it effect final outcome? But the results are promising.

This does not mean that computers will be practicing medicine anytime soon. Physicians are  needed to gather data from patients, and human judgment is needed for things like – how sick does this person look. Also, clinical decision making is still beyond current software technology – that level of artificial intelligence is not here. Expert systems are meant to aid experts, not replace them.

Overall, in my opinion, this technology is underutilized. I think a push is needed to develop more expert systems and incorporate them into patient care workflow. These systems can help reduce mistakes and optimize care. In the new era of cost control in medicine it should also be considered that such systems can potentially save a great deal of health care dollars. The Obama administration and others have focused on electronic medical records. I am a big fan of EMR, but the evidence so far shows that they do not save money. We need to look at expert systems – I think they may have more potential.

For many tasks, our humble gray matter is simply not sufficient, and we need help, such as checklists. And for other tasks, such as complex diagnostic situations, even checklists are insufficient – we need computing power. These systems exist and they work, and I hope to see more of them incorporated into practice.

Share

15 responses so far

15 Responses to “Computer Models for Medical Diagnosis”

  1. superdaveon 21 Apr 2010 at 9:36 am

    I have friends who work in the area of training computer to analyze MRI images for aiding diagnoses. This isn’t quite as sexy as telling a computer your symptoms and getting a disease back but is an important incremental step that should be mentioned.

  2. JChizmaron 21 Apr 2010 at 11:21 am

    This is an excellent article. My current research is very similar to what you describe here, but perhaps in a less generalized approach.

    We have the ability now to collect so much data from patients, but the skill needed to interpret it and conclude a diagnosis takes many years to learn. Digital signal processing and mathematical models are the perfect tools for analyzing multiple sets of correlated data to produce a very accurate diagnosis that the physician can then use to direct treatment.

  3. Watcheron 21 Apr 2010 at 11:21 am

    I might be a bit worried if my physician started googling during my visit. I probably wouldn’t if they were doing a pubmed search though ;)

  4. ccbowerson 21 Apr 2010 at 12:05 pm

    I assume by expert systems are something like intelligent diagnosis-specific order sets plus decision trees? It appear that we are moving closer to something like this, which is great, but this will be done unevenly. We already have pretty wide variations in how healthcare is implemented in different areas, and in some cases this could result in discrepancies in healthcare quality. Not that that’s a reason to not go ahead and implement expert systems (after validating them)… in the long run this will benefit all.

    Done properly and selectively I think that expert systems could have a huge impact on optimizing care for many diseases, paricularly ones that are seen often (certain infections, ceratain cardiovasular events, etc). But wouldnt it be much easier and efficient to implement such things with EMR (if not already done)? I don’t see how this is an either/ or senario… you need both

  5. Gallenodon 21 Apr 2010 at 1:12 pm

    In addition to diagnosis, computer databases should be particularly useful in determining potential drug interactions.

    Any time you mix two or more prescription drugs, over-the-counter products, or certain foods (e.g. I had to give up drinking grapefruit juice because it has an enzyme that reacts with medication I take) there is some potential that they will react with each other in some way. Given the myriad of drugs available and the huge number of potential combinations, I cannot see how the average GP could keep up these days without some database that tracks how drugs work in combination.

    The last time I went in for a physical, my GP had access to a recently installed medical records database that did just that. We went through the process of entering in my prescriptions (I take six, at the moment) and the resulting report showed no known interaction issues, which was of course reassuring.

    However, having done some database work over the years, I know the output of such a system is only as good as the input. The medical records system used locally, which includes the prescription management application, is called PRISM. Does anyone know who controls the data entry into systems like these and what research reports they may include or ignore in the data approval process?

  6. zorrobanditoon 21 Apr 2010 at 1:47 pm

    I’d agree that the use of computerized information is helpful. However, it is not a substitute for actually looking at and listening to the patient.

    My doctor, an older guy, is obviously overwhelmed by the task of learning how to use the computer’s database. The last time I was in there, he spent something like 80% of the time peering at his computer screen; we were 2/3 of the way through the appointment before he made eye contact.

    The computer will not give you the diagnosis in a vacuum. It needs information, input, and you can only get that from the patient. That, in turn, only works if you realize that you HAVE a patient, that you’re not in there just to work with your little computer.

  7. eeanon 21 Apr 2010 at 5:17 pm

    @zorrobandito: well Steven is clear that experts are needed to run these expert systems.

    My mom used to be a nurse practitioner at a clinic where she (and other NPs and MDs) would spend an hour or two at the end day doing dictation about each patient so that summaries could be typed up and included in the file. Apparently this was cost effective to the clinic since typed-up notes lead to less problems with Medicare and Medicaid (which was most patients). And they were all being paid salaries and not hourly wages anyways.

    But I imagine the job would’ve been much nicer if she could have just filled out a EMR right there in the clinic room. But I guess the advantage of letting doctors finish all their paperwork in the examine room is also the disadvantage that doctors can finish all the paperwork in the examine room. :D

  8. Steven Novellaon 21 Apr 2010 at 5:45 pm

    zorro – also, we are all adapting to the new information systems in medicine. Doctors have to learn how to use the computer and still face the patient and make eye-contact, etc. It takes a certain comfort level. I can see someone who is uncomfortable with the computer or having problems with it getting their attention drawn.

    Also, if patients want the advantages of an EMR such as better record keeping and instant access to a wealth of information, such as test results, then they have to put up with the downside as well. There are lots of trade-offs.

  9. orbitzon 21 Apr 2010 at 7:00 pm

    I know a few people who are writing tools to help make these types of medical decisions. One really neat thing is one can use a technique (one of many) called Decision Trees, where the computer learns a series of questions to ask, and then based on the responds what the next question to ask is, and eventually gets to a decision.

    So you could feed it a patient’s record and it might say

    - “well what is their blood pressure in normal range?”
    –”yes? then is their heart rate normal?”
    –”no? then could be a heart attack, check (some other symptom of
    a heart attack)”.

    The machine can even suggest tests to do as it decides to help it get better information. If done well it will only suggest tests that NEED to be done so doctors aren’t throwing every test they have at the person, some of which could be harmful, all of which are probably expensive. The really need part, though, is when the machine has come to a decision it can tell you the path on the tree that it took to get to that decision so the doctor can then look at it. Even if the machine is wrong, it might give insight to the doctor by narrowing down the diagnosis.

  10. tudzaon 21 Apr 2010 at 8:25 pm

    Expert systems are called such because they are meant to be used by experts? Do you mean that is the ideal, or has the definition for such things changed?

    I thought an expert system was meant to represent the knowledge and decision process of an expert to make a useful tool for others. Wikipedia seems to be describing what I think of when this term is used:

    http://en.wikipedia.org/wiki/Expert_system

  11. Scepticonon 21 Apr 2010 at 9:12 pm

    Hopefully this (mostly) related paper is interesting and not unwelcome:

    http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0010265

    “Clinical Reasoning in the Real World Is Mediated by Bounded Rationality: Implications for Diagnostic Clinical Practice Guidelines”

    Abstract Top
    Background

    Little is known about the reasoning mechanisms used by physicians in decision-making and how this compares to diagnostic clinical practice guidelines. We explored the clinical reasoning process in a real life environment.
    Method

    This is a qualitative study evaluating transcriptions of sixteen physicians’ reasoning during appointments with patients, clinical discussions between specialists, and personal interviews with physicians affiliated to a hospital in Brazil.
    Results

    Four main themes were identified: simple and robust heuristics, extensive use of social environment rationality, attempts to prove diagnostic and therapeutic hypothesis while refuting potential contradictions using positive test strategy, and reaching the saturation point. Physicians constantly attempted to prove their initial hypothesis while trying to refute any contradictions. While social environment rationality was the main factor in the determination of all steps of the clinical reasoning process, factors such as referral letters and number of contradictions associated with the initial hypothesis had influence on physicians’ confidence and determination of the threshold to reach a final decision.
    Discussion

    Physicians rely on simple heuristics associated with environmental factors. This model allows for robustness, simplicity, and cognitive energy saving. Since this model does not fit into current diagnostic clinical practice guidelines, we make some propositions to help its integration.

  12. toniclarkon 22 Apr 2010 at 9:09 am

    I don’t think that expert systems are called such “because they are meant to be used by experts.” Rather, it’s because they quickly bring to bear on a problem expertise that would normally require one or more human experts and take a lot of time to acquire. Expert systems, knowledge-based applications of AI, simulate the performance of an expert and are less prone to human error. They generally comprise a knowledge base and some kind of reasoning engine.

    I have long worked for Probem-Knowedge Couplers, Inc., or PKC (http://www.pkc.com/), founded by a renowned pioneer in the field of medical IT, Lawrence L. (Larry) Weed, MD. Weed is the originator of the Problem Oriented Medical Record (POMR) and of Problem-Knowledge Couplers, problem-based software that couples an individual’s signs, symptoms, lab findings, and medical history to a knowledge base (KNet) of medical information drawn from both textbooks, practice guidelines, and the latest clinical research. The program then presents a list of possible diagnoses and options for further diagnostic tests. What Steve says about the limitations of human information processing, Larry Weed has been saying for over 30 years.

    Steve also says: “I am optimistic that we will increasingly incorporate expert systems into medical practice to aid in making proper diagnoses.” I’m optimistic, too. In the past, there’s seemed to be a lot of resistance to computer-based diagnostic tools in the medical community. I do think that’s changing. People coming out of med school now have grown up with computers and assume that they’ll be using them. Couplers and other computer-based diagnostic programs are not meant to replace docs, but to serve them as powerful tools. It’s true that these tools are still a bit cumbersome and hard to integrate into an existing practice — but it’s been done successfully.

    Some articles you might like to see:

    http://www.bio-itworld.com/hitw/newsletters/2006/12/21/24393

    http://xnet.kp.org/permanentejournal/spr09/health_care.html

    http://www.bangordailynews.com/detail/49407.html

    http://www.boston.com/news/globe/reprints/071402_whenyourdoc/

    Steve, come and visit us anytime. We’re located in Burlington, VT. http://www.pkc.com/

    Toni Clark
    Medical Editor, PKC

  13. TeddyBreamon 23 Apr 2010 at 12:18 am

    The biggest impediment to the use of expert systems is not the system. My experience is that most large organizations, hospitals included, have an already existing set of protocols and procedures that can be readily coded into a computer.

    Sure the resulting system isn’t particularly expert, but does fit the definition of an expert system and moreover can produce the expected productivity gains and reduction in human error. It’s not pretty but it works. Efficient payroll systems are basically based on this idea.

    The biggest impediment to the implementation of expert systems is cultural and rests on how professionals relate to their work and how they see the value in what they produce.

    Most experts, along with most people, see their job as the collection of activities they do rather than as the goal they are trying to achieve. The result being that when portions of their job that are automated they resist as they percieve the computer system reducing their value to the organisation (in fact the opposite is occuring).

    Compound this with groups of people who are used to doing things their way (academics, managment, doctors, most professionals in fact) and the usual problems of large IT implementations and it’s easy to see why these systems are not more widely spread.

    I wish they were more wide spread because the gains can be immense, but without a change in the way people think about their work and how they add value to an organisation it’s just too hard to do on large scales.

    I point you to the failed INCIS policing expert system as an excellent example. There is a good wikipedia page on this including links to the relvant inquiries and the resulting multimillion dollar law suits. There are claims that the failure of this project, in particular the exceptionally long emergency phone number call waits, resulted in deaths.

    Thanks as always Steve for a great post.

  14. Spurllon 13 May 2010 at 10:37 am

    An excellent post, as always, Dr. Novella. I, too, am optimistic about the future of such systems—although I graduated with distinction from the U of Manitoba with a specialisation in artificial intelligence, so it would be unseemly if I were not optimistic, methinks.

    I was slightly worried by this statement, however:

    So the researchers used the information from these thousands of cases to develop a computer model to determine who has a serious infection and who doesn’t. They then validated that model with the same data set, and found that it did as well or better than the physicians.

    I have quite a lot of experience training predictive models, and one of the first things that I learned is that this sort of straightforward in-sample validation is not a good idea. It seems like they had a fairly large sample-size; they could have used k-fold cross-validation (or even simply divided the set into separate training and test sets). I skimmed the article, and it was unclear to me whether they had in fact simply done in-sample testing—but I’ll take Dr. Novella’s word on it.

  15. jcbmackon 14 Jul 2010 at 5:06 am

    Computer models can be very useful, but I do suggest caution as well.

Trackback URI | Comments RSS

Leave a Reply

You must be logged in to post a comment.