I was slightly worried by this statement, however:

So the researchers used the information from these thousands of cases to develop a computer model to determine who has a serious infection and who doesn’t. They then validated that model with the same data set, and found that it did as well or better than the physicians.

I have quite a lot of experience training predictive models, and one of the first things that I learned is that this sort of straightforward in-sample validation is not a good idea. It seems like they had a fairly large sample-size; they could have used k-fold cross-validation (or even simply divided the set into separate training and test sets). I skimmed the article, and it was unclear to me whether they had in fact simply done in-sample testing—but I’ll take Dr. Novella’s word on it.

Sure the resulting system isn’t particularly expert, but does fit the definition of an expert system and moreover can produce the expected productivity gains and reduction in human error. It’s not pretty but it works. Efficient payroll systems are basically based on this idea.

The biggest impediment to the implementation of expert systems is cultural and rests on how professionals relate to their work and how they see the value in what they produce.

Most experts, along with most people, see their job as the collection of activities they do rather than as the goal they are trying to achieve. The result being that when portions of their job that are automated they resist as they percieve the computer system reducing their value to the organisation (in fact the opposite is occuring).

Compound this with groups of people who are used to doing things their way (academics, managment, doctors, most professionals in fact) and the usual problems of large IT implementations and it’s easy to see why these systems are not more widely spread.

I wish they were more wide spread because the gains can be immense, but without a change in the way people think about their work and how they add value to an organisation it’s just too hard to do on large scales.

I point you to the failed INCIS policing expert system as an excellent example. There is a good wikipedia page on this including links to the relvant inquiries and the resulting multimillion dollar law suits. There are claims that the failure of this project, in particular the exceptionally long emergency phone number call waits, resulted in deaths.

Thanks as always Steve for a great post.

I have long worked for Probem-Knowedge Couplers, Inc., or PKC (http://www.pkc.com/), founded by a renowned pioneer in the field of medical IT, Lawrence L. (Larry) Weed, MD. Weed is the originator of the Problem Oriented Medical Record (POMR) and of Problem-Knowledge Couplers, problem-based software that couples an individual’s signs, symptoms, lab findings, and medical history to a knowledge base (KNet) of medical information drawn from both textbooks, practice guidelines, and the latest clinical research. The program then presents a list of possible diagnoses and options for further diagnostic tests. What Steve says about the limitations of human information processing, Larry Weed has been saying for over 30 years.

Steve also says: “I am optimistic that we will increasingly incorporate expert systems into medical practice to aid in making proper diagnoses.” I’m optimistic, too. In the past, there’s seemed to be a lot of resistance to computer-based diagnostic tools in the medical community. I do think that’s changing. People coming out of med school now have grown up with computers and assume that they’ll be using them. Couplers and other computer-based diagnostic programs are not meant to replace docs, but to serve them as powerful tools. It’s true that these tools are still a bit cumbersome and hard to integrate into an existing practice — but it’s been done successfully.

Some articles you might like to see:

http://www.bio-itworld.com/hitw/newsletters/2006/12/21/24393

http://xnet.kp.org/permanentejournal/spr09/health_care.html

http://www.bangordailynews.com/detail/49407.html

http://www.boston.com/news/globe/reprints/071402_whenyourdoc/

Steve, come and visit us anytime. We’re located in Burlington, VT. http://www.pkc.com/

Toni Clark
Medical Editor, PKC

http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0010265

“Clinical Reasoning in the Real World Is Mediated by Bounded Rationality: Implications for Diagnostic Clinical Practice Guidelines”

Abstract Top
Background

Little is known about the reasoning mechanisms used by physicians in decision-making and how this compares to diagnostic clinical practice guidelines. We explored the clinical reasoning process in a real life environment.
Method

This is a qualitative study evaluating transcriptions of sixteen physicians’ reasoning during appointments with patients, clinical discussions between specialists, and personal interviews with physicians affiliated to a hospital in Brazil.
Results

Four main themes were identified: simple and robust heuristics, extensive use of social environment rationality, attempts to prove diagnostic and therapeutic hypothesis while refuting potential contradictions using positive test strategy, and reaching the saturation point. Physicians constantly attempted to prove their initial hypothesis while trying to refute any contradictions. While social environment rationality was the main factor in the determination of all steps of the clinical reasoning process, factors such as referral letters and number of contradictions associated with the initial hypothesis had influence on physicians’ confidence and determination of the threshold to reach a final decision.
Discussion

Physicians rely on simple heuristics associated with environmental factors. This model allows for robustness, simplicity, and cognitive energy saving. Since this model does not fit into current diagnostic clinical practice guidelines, we make some propositions to help its integration.

I thought an expert system was meant to represent the knowledge and decision process of an expert to make a useful tool for others. Wikipedia seems to be describing what I think of when this term is used:

http://en.wikipedia.org/wiki/Expert_system

So you could feed it a patient’s record and it might say

- “well what is their blood pressure in normal range?”
–”yes? then is their heart rate normal?”
–”no? then could be a heart attack, check (some other symptom of
a heart attack)”.

The machine can even suggest tests to do as it decides to help it get better information. If done well it will only suggest tests that NEED to be done so doctors aren’t throwing every test they have at the person, some of which could be harmful, all of which are probably expensive. The really need part, though, is when the machine has come to a decision it can tell you the path on the tree that it took to get to that decision so the doctor can then look at it. Even if the machine is wrong, it might give insight to the doctor by narrowing down the diagnosis.

Also, if patients want the advantages of an EMR such as better record keeping and instant access to a wealth of information, such as test results, then they have to put up with the downside as well. There are lots of trade-offs.

My mom used to be a nurse practitioner at a clinic where she (and other NPs and MDs) would spend an hour or two at the end day doing dictation about each patient so that summaries could be typed up and included in the file. Apparently this was cost effective to the clinic since typed-up notes lead to less problems with Medicare and Medicaid (which was most patients). And they were all being paid salaries and not hourly wages anyways.

But I imagine the job would’ve been much nicer if she could have just filled out a EMR right there in the clinic room. But I guess the advantage of letting doctors finish all their paperwork in the examine room is also the disadvantage that doctors can finish all the paperwork in the examine room.

My doctor, an older guy, is obviously overwhelmed by the task of learning how to use the computer’s database. The last time I was in there, he spent something like 80% of the time peering at his computer screen; we were 2/3 of the way through the appointment before he made eye contact.

The computer will not give you the diagnosis in a vacuum. It needs information, input, and you can only get that from the patient. That, in turn, only works if you realize that you HAVE a patient, that you’re not in there just to work with your little computer.