ABSTRACT
Clinical decision-making is challenging mainly because of two factors: (1) patient conditions are often complicated with partial and changing information; (2) people have cognitive biases in their decision-making and information-seeking. Consequentially, misdiagnoses and ineffective use of resources may happen.
To better support clinical decision-making, a framework named HypothesisDriven Story Building (HDSB) was proposed to address the information challenges during clinical diagnosis. When given partial information, HDSB generates a list of hypotheses that can explain the current information and rank the hypotheses based on their likelihoods. If more information is needed, HDSB recommends a list of possible actions for information seeking, ranked in their effectiveness in differentiating the potential hypotheses. Whenever new information arrives, the HDSB framework updates the potential hypotheses accordingly.
The HDSB framework was built based on Multi-Layer Bayesian Network (MLBN), which is an extension of standard Bayesian network with relational representation on each node, enabling the probabilistic causal inferences for relations based on variable bindings. In MLBN, different conditional probability tables can be defined for different variable bindings so that the Bayesian inferences can be specialized or personalized and abductive reasoning can be conducted.
A web-based clinical diagnostic decision support prototype SRCAST-Diagnosis was developed based on the HDSB framework. In a given scenario, SRCAST-Diagnosis will display patient conditions, recommend differential diagnoses, and rank lab tests to the user. It was evaluated through a controlled experiment conducted at Hershey Medical Center. Participants including nurses, residents, and physicians were divided into a control group and an experimental group. Their actions (ordering lab tests and making diagnoses) were recorded.
The result showed that SRCAST-Diagnosis can significantly improve the diagnosis accuracy and reduce the cost of resources overall, although the performances for different role players may vary. The data also showed that the tool helped more decision-makers who made the wrong initial diagnosis eventually find the correct diagnosis (counteracting anchoring heuristics) and helped them quickly figure out the correct diagnosis with significantly less resource cost (counteracting confirmation biases). It can be concluded that misdiagnosis and ineffective use of resources are associated with human cognitive biases, and a well-designed decision support system is able to improve diagnosis accuracy and resource efficiency by counteracting the cognitive biases.
Chapter 1
INTRODUCTION
“…we’ve estimated that most of this plan can be paid for by finding savings within the existing health care system, a system that is currently full of waste and abuse. Right now, too much of the hard-earned savings and tax dollars we spend on health care don’t make us any healthier.”
– President Barack Obama
REMARKS TO A JOINT SESSION OF CONGRESS ON HEALTH CARE
September 9, 2009
The health care system in the United States has long been a controversial issue, and its high cost is one of the major reasons. In 2009, it was estimated that the spending on healthcare in the U. S. was approximately 16.2% of its GDP, or $7,681 per capita (CMS, 2008). A recent study (Dalen, 2009) found that medical expenditures were a significant factor that contributed to 62% of the country’s personal bankruptcies in 2007. The proposed health care reform, therefore, aims to offer quality and affordable health care to all Americans. Significant milestones have been made with two bills—the Patient Protection and Affordable Care Act and the Health Care and Education Reconciliation Act—passed in 2010 by Congress and signed by President Barack Obama.
In the overall cost of health care, a great portion is related to medical liability and defensive medicine. According to 2006 analysis conducted by Price Waterhouse Coopers, this cost is about 10% of the overall health care cost. Medical liability and defensive medicine are rooted in medical malpractices, which are the failures of health care providers to meet the professional standards by which patient injuries or deaths are caused (Danzon, 1985). Many factors can cause malpractices. Among them, misdiagnosis is one of the major causes. Studies have disclosed that one third of all malpractice claims concerned misdiagnosis (Phillips et al., 2004), making it one of the fundamental reasons for defensive medicine and its related cost.
Defensive medicine involves the practice of doctors’ diagnostic or therapeutic measures primarily (but not necessarily or solely) to reduce their exposure to malpractice liability, instead of ensuring the health of patients (Congress, 1994). One typical action that a physician may engage in to reduce malpractice liability is to order extra tests or procedures, which is considered positive defensive medicine. The fear of malpractice has lead to significant changes in emergency department decision-making and has been associated with the increased hospitalization of low-risk patients as well as the increased use of diagnostic tests (Katz et al., 2005). In another study, it was reported that “nearly all” (93%) of the physicians who participated in the study had practiced defensive medicine (Studdert et al., 2005). For instance, 43% participants reported using image technology in clinically unnecessary circumstances, as an “assurance behavior” (Studdert et al., 2005; Manner, 2007). The practice of ordering unnecessary tests in order to reduce the risk of malpractice litigation is known as “defensive testing” (DeKay & Asch, 1998).
There is a strong link between misdiagnosis and the high cost of health care. The misdiagnosis-cost relationship has been the dilemma: if health care providers are constrained to reduce the number of tests to control the cost, they may potentially be exposed to the risk of increasing the misdiagnosis rate, which may eventually cost them the medical liability awards; if health care providers are encouraged to practice defensive medicine and conduct as many lab tests as they think are clinically assured (but more than are necessary), the entire society and ultimately each individual must share the cost in the long run, even though a specific patient or physician may receive a short-term benefit.
In his 2009 speech to the Joint Session of Congress, President Barack Obama proposed to “put patient safety first and let doctors focus on practicing medicine” while at the same time bringing down the cost of health care. His passion and ambition motivated me to explore the possibility of breaking the misdiagnosis-cost dilemma.
1.1 THE MOTIVATION
Misdiagnosis and its consequences, including medical malpractice litigation and defensive medicine, have contributed a substantial portion to the high cost of health care in the United States. Therefore, it is critical to understand the causes of misdiagnosis before potential methods can be explored for solving the misdiagnosis-cost dilemma.
A misdiagnosis is an error in diagnosis. It may be in various formats, such as a wrong diagnosis, a missed diagnosis, a delayed diagnosis, or missed complications. A misdiagnosis is the erroneous result of a physician’s cognitive assessment of a patient’s condition. Therefore, it is the common effect of complicated patient conditions
(externally) and physicians’ cognitive constraints (internally).
1.1.1 COMPLEX PATIENT CONDITIONS
A misdiagnosis is most likely to occur with patients having complex situations, especially in those whose symptoms can be explained by more than one disease, and whose conditions can change over time. A typical patient diagnosis process usually faces three challenges:
- Patient conditions are only partially known.
Patient conditions are only partially known most of the time during the diagnosis process. A diagnosis is an effort of assessing the true patient condition and the actual cause of the patient’s current situation based on limited information such as symptoms, vital signs, physical examination results, and lab results. With the accumulation of information, the assessment will become more and more accurate, but it will still be a best guess rather than the truth. Thus, incomplete information is the first challenge to making a correct diagnosis.
- Patient conditions can be obtained only if they are requested.
Some patient conditions or symptoms can be observed by physicians or nurses; however, other conditions can be obtained only if they are requested, such as lab test results, and will remain unknown until the lab test is explicitly ordered. Resources for lab tests are oftentimes costly or tightly scheduled. Ordering redundant lab tests is not only economically inefficient for the patient but also may deprive other patients with urgent needs of the opportunities to use the lab test resources. Therefore, the second challenge for a physician is to order the right amount of information for making a correct diagnosis.
- Patient conditions can change.
The conditions of a patient may be unstable during the diagnostic process. A symptom may appear or disappear. A vital sign may have a significant jump or drop. Some of these changes may have a significant impact on a suspected diagnosis while others may not. Whenever a condition changes, it must be integrated into the current differential diagnosis. The significance level of the change and its consequence must be determined: should the list of differential diagnoses be changed, should only their ranking be changed, or should no change be made? Thus the third challenge for a physician is to adjust his differential diagnosis based on new information in a timely manner.
These challenges highlight some of the difficulties in making a correct medical diagnosis. All three are related to information use and seeking. However, human cognitive biases make a medical diagnosis process more error-prone.
1.1.2 HUMAN COGNITIVE BIASES
Human decision-makers’ activities when dealing with these three challenges are affected by their cognitive biases. A cognitive bias is a person’s tendency to make errors in judgment based on cognitive factors, a phenomenon studied in cognitive science and social psychology (Gilovich, Griffin, & Kahneman, 2002). Researchers have found that human cognitive biases will prevent individuals from making better decision choices (Hogarth, 1987; Reason, 1990). Within the healthcare domain, studies have shown the effects of human cognitive biases on clinical quality (Bornstein & Emler, 2001; Milstein
& Adler, 2003; Redelmeier, 2005).
Among the various types of cognitive biases that affect decision-makers’ judgment, some are related to information using and seeking, which make the (above) three challenges more difficult. For example, during a diagnosis, a physician may overtrust or over emphasize one particular piece of information based on his past experiences. Consequently, he may stick to an initial suspected diagnosis made based on this previous information and overlook other possibilities (Tversky & Kahneman, 1974). Furthermore, when the decision-maker needs additional information to verify the diagnosis, he may again selectively pay more attention to information that will further support the favorable diagnosis than to other information (Plous, 1993).
Human cognitive biases are the systematic cognitive tendencies of human beings caused by the constraints of human brains, not errors made due to individual mistakes (Gilovich & Griffin, 2002). Therefore, the reward-penalty mechanism (which led to the practice of defensive medicine) may not work to reduce rate of misdiagnosis attributed to human cognitive biases, but may significantly increase the cost of health care.
1.1.3 RESEARCH GAPS
Many efforts have been made to reduce misdiagnosis rates and to control costs. However, most of these have focused only on one element instead of taking them into account jointly. For example, diagnostic decision support systems were designed primarily to reduce the misdiagnosis rate by offering diagnostic recommendations, but they did little to address the problems of defensive medicine and cost control. On the other hand, Fries et al. (1993) suggested that measurements on preventing disease, reducing risky behaviors, improving self-management, and applying healthcare promotion programs at work could be made for analyzing potential cost reductions. These suggestions would be useful to reduce the need for medical services before a patient becomes ill, but none of them address the issue of cost control during the patient diagnosis process.
In summary, there are two important gaps in the current research on reducing misdiagnosis and controlling cost:
- The factor of cognitive biases is missing from the design of diagnostic decision support systems.
Although studies have shown the relationship between human cognitive biases and misdiagnosis (Bornstein & Emler, 2001; Milstein & Adler, 2003; Redelmeier, 2005), few efforts have addressed human cognitive biases in clinical diagnostic decision support systems as a fundamental measurement of reducing misdiagnosis. So far, efforts related to diagnostic decision support systems have focused on the design of algorithms, the representation of knowledge, incorporating more types of diseases, improving user interfaces, and offering links to searchable databases. Each of these activities will, no doubt, improve the accuracy of recommended diagnosis; however, none of the decision support systems have shown evidence that their improvements in diagnostic accuracy are associated with any counteraction of cognitive biases.
- Cost control is rarely considered when a diagnostic decision support system recommends possible diagnosis.
Many types of clinical decision support systems have been developed for different purposes and applied in different areas. Some focus on suggesting possible diagnoses that match a patient’s signs and symptoms. Others focus on cost reduction. However, cost control exists in relatively simple format: it is achieved by monitoring medication orders, and by avoiding duplicate and unnecessary tests (Perreault & Metzger, 1999). Currently, no list of possible lab tests is given, ranked in importance level to differentiate the differential diagnoses. Requesting labs in the order of their importance level could be expected to improve the overall efficiency of lab resources.
The research gaps set the goal of my research: to break the misdiagnosis-cost dilemma (i.e., to reduce health care cost by increasing the effective use of lab resources) and, at the same time, to improve or retain diagnosis accuracy. This goal shaped my research questions.
1.2 RESEARCH QUESTIONS
To address the research gaps, I raised the following four questions which will be investigated in the remainder of this dissertation.
- What model can be developed to describe physicians’ hypothesis reasoning and information seeking process in clinical diagnosis?
- Can a decision support tool, built based on the proposed model, reduce misdiagnosis?
- Can the same tool reduce the cost of lab tests while reducing misdiagnosis?
- Are these improvements caused by the counteraction of cognitive biases?
The first question was aimed at setting up the requirements for a general framework to model the process by which a physician (or possibly a computer-based decision support system) makes diagnostic decisions, especially when facing incomplete and changing patient conditions. The framework must address: how the decision-maker generates differential diagnoses when information is only partially known; how the decision-maker identifies and collects missing information; and how the decision-maker revises previously made tentative decisions when given new information.
Questions 2 and 3 were aimed at evaluating the usefulness of the decision support tool in reducing both misdiagnosis and cost. The tool needs to show its capability for reducing both misdiagnosis and cost at the same time, demonstrating its potential to break the misdiagnosis-cost dilemma.
Question 4 was aimed at understanding how these improvements were possible. It must show whether the effects of certain human cognitive biases would be counteracted by the tool so that a decision-maker’s behaviors and choices would potentially be changed.
The next section will outline the approaches taken to answer the research questions.
1.3 RESEARCH APPROACH
To answer the first question, I developed the Hypothesis-Driven Story Building (HDSB) framework which describes, in a general perspective, a decision-maker’s activities while moving toward his final decision. Diagnosis is modeled as a story building activity, which aims to find an explanation for the observed information. A story building process begins by generating hypotheses, then proceeds to seeking information and revising hypotheses, and ends by making a final decision. The HDSB framework was built on the Multi-Layer Bayesian Network (MLBN), an extension of the classical Bayesian network, to enable probabilistic inferences with predicate sentences.
To answer the second and third questions, a controlled experiment was designed and conducted. Participants with certain levels of medical decision-making experience were recruited for the experiment. Participants were divided into an experimental group and a control group. Those in the experimental group were backed up by the decision support tool while making their decisions, while those in the control group made decisions based on their own judgment. Each participant’s decisions about final diagnosis and lab tests were recorded. An analysis on the diagnosis accuracy and cost efficiency of each group was conducted.
To answer the fourth question, I tracked the intermediate behaviors of each participant’s decision-making process during the experiment. Whenever a decision-maker wanted to order lab tests, he was asked to select the diagnosis that he most suspected at that time. This allowed a comparison of the accuracy of the initial diagnosis, both with and without the decision support tool, and also tracked the time at which a participant switched to a correct diagnosis from an initially incorrect one. These data reflected the cognitive tendency in decision-making, especially the anchoring heuristics and confirmation biases. Thus, by tracking the intermediate behaviors and choices, an analysis of these two cognitive biases can be conducted.
This approach is composed of stages of design, implementation, experimentation, and analysis; therefore, it is a holistic approach. Through it, not only are the research questions answered, but also insights for future development are generated.
1.4 DISSERTATION ROADMAP
The remainder of this dissertation is structured as follows:
Chapter 2 introduces related studies on cognitive biases, clinical decision-making, diagnostic decision support systems, and technologies for developing the decision support tool (abductive reasoning, Bayesian networks, value of information in decision theory, and entropy of information.)
Chapter 3 introduces the Multi-Layer Bayesian Network, which is fundamental to the HDSB framework.
Chapter 4 presents the Hypothesis-Driven Story Building framework, including the methodology for each specific step: hypothesis generation, hypothesis evaluation, hypothesis-driven information seeking, and hypothesis revision.
Chapter 5 describes the implementation of the HDSB-enabled decision support tool, built on the R-CAST platform but using a new MLBN component, and incorporating an HDSB module. It also presents an agent interface and a web-based application interface.
Chapter 6 discusses the experiment design used to evaluate the tool.
Chapter 7 provides the analysis of the experiment data. Diagnosis accuracy and cost efficiency are analyzed, and the relationship between the performance improvement and the counteraction of cognitive biases are discussed.
Chapter 8 offers a discussion of remaining issues, including cognitive biases, other potential use of the tool, and the limitations of the study.
Chapter 9 summarizes the contributions of my work and its possible future extension.
HYPOTHESIS-DRIVEN STORY BUILDING: COUNTERACTING HUMAN COGNITIVE BIASES TO IMPROVE MEDICAL DIAGNOSIS SUPPORT