This issue of the Journal of Hospital Medicine describes 2 research and quality improvement demonstration projects funded by the Gordon and Betty Moore Foundation. Early detection is central to both projects. This introductory article does not provide a global review of the now voluminous literature on rapid response teams (RRTs), sepsis detection systems, or treatment protocols. Rather, it takes a step back and reassesses just what early detection and quantification of critical illness are. It then examines the implications of early detection and its quantification.

CONCEPTUAL FRAMEWORK

We define severe illness as the presence of acute disease such that a person can no longer expect to improve without dedicated hospital treatment but which is not inevitably associated with mortality, postdischarge morbidity, or major loss of autonomy. In contrast, we define critical illness as acute disease with high a priori risk of mortality, postdischarge morbidity, and major (possibly total) loss of autonomy. We accept that the boundaries between ordinary illness, severe illness, and critical illness are blurred. The basic assumption behind all efforts at early detection is that these edges can be made sharp, and that the knowledge base required to do so can also lead to improvements in treatment protocols and patient outcomes. Further, it is assumed that at least some forms of critical illness can be prevented or mitigated by earlier detection, identification, and treatment.

Research over the last 2 decades has provided important support for this intuitive view as well as making it more nuanced. With respect to epidemiology, the big news is that sepsis is the biggest culprit, and that it accounts for a substantial proportion of all hospital deaths, including many previously considered unexpected hospital deaths due to in‐hospital deterioration.[1] With respect to treatment, a number of studies have demonstrated that crucial therapies previously considered to be intensive care unit (ICU) therapies can be initiated in the emergency department or general medicalsurgical ward.[2]

Figure 1 shows an idealized framework for illness presenting in the emergency department or general medicalsurgical wards. It illustrates the notion that a transition period exists when patients may be rescued with less intense therapy than will be required when condition progression occurs. Once a certain threshold is crossed, the risk of death or major postdischarge morbidity rises exponentially. Unaided human cognition's ability to determine where a given patient is in this continuum is dangerously variable and is highly dependent on the individuals training and experience. Consequently, as described in several of the articles in this issue as well as multiple other publications, health systems are employing comprehensive electronic medical records (EMRs) and are migrating to algorithmic approaches that combine multiple types of patient data.[3, 4] Although we are still some distance from being able to define exact boundaries between illness, severe illness, and critical illness, current EMRs permit much better definition of patient states, care processes, and short‐term outcomes.

Figure 1

Whereas our ability to quantify many processes and short‐term outcomes is expanding rapidly, quantification of the possible benefit of early detection is complicated by the fact that, even in the best of circumstances, not all patients can be rescued. For some patients, rescue may be temporary, raising the prospect of repeated episodes of critical illness and prolonged intensive care without any hope of leaving the hospital. Figure 2 shows that, for these patients, the problem is no longer simply one of preventing death and preserving function but, rather, preserving autonomy and dignity. In this context, early detection means earlier specification of patient preferences.[5, 6]

Figure 2

JUST WHAT CONSTITUTES EARLY DETECTION (AND HOW DO WE QUANTIFY IT)?

RRTs arose as the result of a number of studies showing thatin retrospectin‐hospital deteriorations should not have been unexpected. Given comprehensive inpatient EMRs, it is now possible to develop more rigorous definitions. A minimum set of parameters that one would need to specify for proper quantification of early detection is shown on Figure 3. The first is specifying a T₀, that is, the moment when a prediction regarding event X (which needs to be defined) is issued. This is different from the (currently unmeasurable) biologic onset of illness as well as the first documented indication that critical illness was present. Further, it is important to be explicit about the event time frame (the time period during which a predicted event is expected to occur): we are predicting that X will occur within E hours of the T₀. The time frame between the T₀ and X, which we are referring to as lead time, is clinically very important, as it represents the time period during which the response arm (eg, RRT intervention) is to be instituted. Statistical approaches can be used to estimate it, but once an early detection system is in place, it can be quantified. Figure 3 is not restricted to electronic systems; all components shown can be and are used by unaided human cognition.

Figure 3

Figure 4

It is essential to specify what data are used to generate probability estimates as well as the time frames used, which we refer to as the look‐back time frames. Several types of data could be employed, with some data elements (eg, age or gender) being discrete data with a 1:1 fixed correspondence between the patient and the data. Other data have a many‐to‐1 relationship, and an exact look‐back time frame must be specified for each data type. For example, it seems reasonable to specify a short (1224 hours) look‐back period for some types of data (eg, vital signs, lactate, admission diagnosis or chief complaint), an intermediate time period (13 days) for information on the current encounter, and a longer (months to years) time period for preexisting illness or comorbidity burden.

Because many events are rare, traditional measures used to assess model performance, such as the area under the receiver operator characteristic curve (C statistic), are not as helpful.[7] Consequently, much more emphasis needs to be given to 2 key metrics: number needed to evaluate (or workup to detection ratio) and threshold‐specific sensitivity (ability of the alert to detect X at a given threshold). With these, one can answer 3 questions that will be asked by the physicians and nurses who are not likely to be researchers, and who will have little interest in the statistics: How many patients do I need to work up each day? How many patients will I need to work up for each possible outcome identified? For this amount of work, how many of the possible outcomes will we catch?

Data availability for the study of severe and critical illness continues to expand. Practically, this means that future research will require more nuanced ontologies for the classification of physiologic derangement. Current approaches to severity scoring (collapsing data into composite scores) need to be replaced by dynamic approaches that consider differential effects on organ systems as well as what can be measured. Severity scoring will also need to incorporate the rate of change of a score (or probability derived from a score) in predicting the occurrence of an event of interest as well as judging response to treatment. Thus, instead of at time of ICU admission, the patient had a severity score of 76, we may have although this patient's severity score at the time of admission was decreasing by 4 points per hour per 10 mL/kg fluid given, the probability for respiratory instability was increasing by 2.3% per hour given 3 L/min supplemental oxygen. This approach is concordant with work done in other clinical settings (eg, in addition to an absolute value of maximal negative inspiratory pressure or vital capacity, the rate of deterioration of neuromuscular weakness in Guillain‐Barr syndrome is also important in predicting respiratory failure[8]).

Electronic data also could permit better definition of patient preferences regarding escalation of care. At present, available electronic data are limited (primarily, orders such as do not resuscitate).[9] However, this EMR domain is gradually expanding.[10, 11] Entities such as the National Institutes of Health could develop sophisticated and rapid questionnaires around patient preferences that are similar to those developed for the Patient Reported Outcomes Measurement Information System.[12] Such tools could have a significant effect on our ability to quantify the benefits of early detection as it relates to a patient's preferences (including better delineation of what treatments they would and would not want).

ACTIVATING A RESPONSE ARM

Early identification, antibiotic administration, fluid resuscitation, and source control are now widely felt to constitute low‐hanging fruit for decreasing morbidity and mortality in severe sepsis. All these measures are included in quality improvement programs and sepsis bundles.[13, 14, 15] However, before early interventions can be instituted, sepsis must at least be suspected, hence the need for early detection. The situation with respect to patient deterioration (for reasons other than sepsis) in general medical surgical wards is less clear‐cut. Reasons for deterioration are much more heterogenous and, consequently, early detection is likely necessary but not sufficient for outcomes improvement.

The 2 projects described in this issue describe nonspecific (indicating elevated risk but not specifying what led to the elevation of risk) and sepsis‐specific alerting systems. In the case of the nonspecific system, detection may not lead to an immediate deployment of a response arm. Instead, a secondary evaluation process must be triggered first. Following this evaluation component, a response arm may or may not be required. In contrast, the sepsis‐specific project essentially transforms the general medicalsurgical ward into a screening system. This screening system then also triggers specific bundle components.

Neither of these systems relies on unaided human cognition. In the case of the nonspecific system, a complex equation generates a probability that is displayed in the EMR, with protocols specifying what actions are to be taken when that probability exceeds a prespecified threshold. With respect to the sepsis screening system, clinicians are supported by EMR alerts as well as protocols that increase nursing autonomy when sepsis is suspected.

The distinction between nonspecific (eg, acute respiratory failure or hemodynamic deterioration) and specific (eg, severe sepsis) alerting systems is likely to disappear as advances in the field occur. For example, incorporation of natural language processing would permit inclusion of semantic data, which could be processed so as to prebucket an alert into one that not just gave a probability, but also a likely cause for the elevated probability.

In addition, both types of systems suffer from the limitation of working off a limited database because, in general, current textbooks and training programs primary focus remains that of treatment of full‐blown clinical syndromes. For example, little is known about how one should manage patients with intermediate lactate values, despite evidence showing that a significant percentage of patients who die from sepsis will initially have such values, with 1 study showing 63% as many deaths with initial lactate of 2.5 to 4.0 mmol/L as occurred with an initial lactate of >4.0 mmol/L.[16] Lastly, as is discussed below, both systems will encounter similar problems when it comes to quantifying benefit.

QUANTIFYING BENEFIT

Whereas the notion of deploying RRTs has clearly been successful, success in demonstrating unequivocal benefit remains elusive.[17, 18, 19] Outcome measures vary dramatically across studies and have included the number of RRT calls, decreases in code blue events on the ward, and decreases in inpatient mortality.[20] We suspect that other reasons are behind this problem. First is the lack of adequate risk adjustment and ignoring the impact of patients near the end of life on the denominator. Figure 4 shows recent data from 21 Kaiser Permanente Northern California (KPNC) hospitals, which can now capture care directive orders electronically,[21] illustrates this problem. The majority (53%) of hospital deaths occur among a highly variable proportion (range across hospitals, 6.5%18.0%) of patients who arrive at the hospital with a restricted resuscitation preference (do not resuscitate, partial code, and comfort care only). These patients do not want to die or crash and burn but, were they to trigger an alert, they would not necessarily want to be rescued by being transferred to the ICU either; moreover, internal KPNC analyses show that large numbers of these patients have sepsis and refuse aggressive treatment. The second major confounder is that ICUs save lives. Consequently, although early detection could lead to fewer transfers to the ICU, using the end point of ICU admission is very problematic, because in many cases the goal of alerting systems should be to get patients to the ICU sooner, which would not affect the outcome of transfer to the ICU in a downward direction; in fact, such systems might increase transfer to the ICU.

The complexities summarized in Figure 4 mean that it is likely that formal quantification of benefit will require examination of multiple measures, including balancing measures as described below. It is also evident that, in this respectlack of agreement as to what constitutes a good outcomethe issues being faced here are a reflection of a broader area of disagreement within our profession and society at large that extends to medical conditions other than critical illness.

POTENTIAL HARMS OF EARLY DETECTION

Implementation of early detection and rapid response systems are not inherently free of harm. If these systems are not shown to have benefit, then the cost of operating them is moving resources away from other, possibly evidence‐based, interventions.[22] At the individual level, alerts could frighten patients and their families (for example, some people are very uncomfortable with the idea that one can predict events). Physicians and nurses who work in the hospital are already quite busy, so every time an alert is issued, it adds to the demand on their already limited time, hence, the critical importance of strategies to minimize false alarms and alert fatigue. Moreover, altering existing workflows can be disruptive and unpopular.

A potentially more quantifiable problem is the impact of early detection systems on ICU operations. For example, if an RRT decides to transfer a patient from the ward to the ICU as a preventive measure (soft landing) and this in turn ties up an ICU bed, that bed is then unavailable for a new patient in the emergency department. Similarly, early detection systems coupled with structured protocols for promoting soft landings could result in a change in ICU case mix, with greater patient flow due to increased numbers of patients with lower severity and lower ICU length of stay. These considerations suggest the need to couple early detection with other supportive data systems and workflows (eg, systems that monitor bed capacity proactively).

Lastly, if documentation protocols are not established and followed, early detection systems could expose both individual clinicians as well as healthcare institutions to medicallegal risk. This consideration could be particularly important in those instances where an alert is issued and, for whatever reasons, clinicians do not take action and do not document that decision. At present, early detection systems are relatively uncommon, but they may gradually become standard of care. This means that in‐house out of ICU deteriorations, which are generally considered to be bad luck or due to a specific error or oversight, may then be considered to be preventable. Another possible scenario that could arise is that of plaintiffs invoking enterprise liability, where a hospital's not having an early detection system becomes considered negligent.

ARTICLES IN THIS ISSUE

In this issue of the Journal of Hospital Medicine, we examine early detection from various perspectives but around a common theme that usually gets less attention in the academic literature: implementation. The article by Schorr et al.[23] describes a disease‐specific approach that can be instantiated using either electronic or paper tools. Escobar et al.[24] describe the quantitative as well as the electronic architecture of an early warning system (EWS) pilot at 2 hospitals that are part of an integrated healthcare delivery system. Dummett et al.[25] then show how a clinical rescue component was developed to take advantage of the EWS, whereas Granich et al.[26] describe the complementary component (integration of supportive care and ensuring that patient preferences are respected). The paper by Liu et al.[27] concludes by placing all of this work in a much broader context, that of the learning healthcare system.

FUTURE DIRECTIONS: KEY GAPS IN THE FIELD

Important gaps remain with respect to early detection and response systems. Future research will need to focus on a number of areas. First and foremost, better approaches to quantifying the costbenefit relationships of these systems are needed; somehow, we need to move beyond a purely intuitive sense that they are good things. Related to this is the need to establish metrics that would permit rigorous comparisons between different approaches; this work needs to go beyond simple comparisons of the statistical characteristics of different predictive models. Ideally, it should include comparisons of different approaches for the response arms as well. We also need to characterize clinician understanding about detection systems, what constitutes impending or incipient critical illness, and the optimum way to provide early detection. Finally, better approaches to integrating health services research with basic science work must be developed; for example, how should one test new biomarkers in settings with early detection and response systems?

The most important frontier, however, is how one can make early detection and response systems more patient centered and how one can enhance their ability to respect patient preferences. Developing systems to improve clinical management is laudable, but somehow we need to also find ways to have these systems make a better connection to what patients want most and what matters most to them, something that may need to include new ways that sometimes suspend use of these systems. At the end of the day, after early detection, patients must have a care experience that they see as an unequivocal improvement.

This issue of the Journal of Hospital Medicine describes 2 research and quality improvement demonstration projects funded by the Gordon and Betty Moore Foundation. Early detection is central to both projects. This introductory article does not provide a global review of the now voluminous literature on rapid response teams (RRTs), sepsis detection systems, or treatment protocols. Rather, it takes a step back and reassesses just what early detection and quantification of critical illness are. It then examines the implications of early detection and its quantification.

CONCEPTUAL FRAMEWORK

We define severe illness as the presence of acute disease such that a person can no longer expect to improve without dedicated hospital treatment but which is not inevitably associated with mortality, postdischarge morbidity, or major loss of autonomy. In contrast, we define critical illness as acute disease with high a priori risk of mortality, postdischarge morbidity, and major (possibly total) loss of autonomy. We accept that the boundaries between ordinary illness, severe illness, and critical illness are blurred. The basic assumption behind all efforts at early detection is that these edges can be made sharp, and that the knowledge base required to do so can also lead to improvements in treatment protocols and patient outcomes. Further, it is assumed that at least some forms of critical illness can be prevented or mitigated by earlier detection, identification, and treatment.

Research over the last 2 decades has provided important support for this intuitive view as well as making it more nuanced. With respect to epidemiology, the big news is that sepsis is the biggest culprit, and that it accounts for a substantial proportion of all hospital deaths, including many previously considered unexpected hospital deaths due to in‐hospital deterioration.[1] With respect to treatment, a number of studies have demonstrated that crucial therapies previously considered to be intensive care unit (ICU) therapies can be initiated in the emergency department or general medicalsurgical ward.[2]

Figure 1 shows an idealized framework for illness presenting in the emergency department or general medicalsurgical wards. It illustrates the notion that a transition period exists when patients may be rescued with less intense therapy than will be required when condition progression occurs. Once a certain threshold is crossed, the risk of death or major postdischarge morbidity rises exponentially. Unaided human cognition's ability to determine where a given patient is in this continuum is dangerously variable and is highly dependent on the individuals training and experience. Consequently, as described in several of the articles in this issue as well as multiple other publications, health systems are employing comprehensive electronic medical records (EMRs) and are migrating to algorithmic approaches that combine multiple types of patient data.[3, 4] Although we are still some distance from being able to define exact boundaries between illness, severe illness, and critical illness, current EMRs permit much better definition of patient states, care processes, and short‐term outcomes.

Figure 1

Whereas our ability to quantify many processes and short‐term outcomes is expanding rapidly, quantification of the possible benefit of early detection is complicated by the fact that, even in the best of circumstances, not all patients can be rescued. For some patients, rescue may be temporary, raising the prospect of repeated episodes of critical illness and prolonged intensive care without any hope of leaving the hospital. Figure 2 shows that, for these patients, the problem is no longer simply one of preventing death and preserving function but, rather, preserving autonomy and dignity. In this context, early detection means earlier specification of patient preferences.[5, 6]

Figure 2

JUST WHAT CONSTITUTES EARLY DETECTION (AND HOW DO WE QUANTIFY IT)?

RRTs arose as the result of a number of studies showing thatin retrospectin‐hospital deteriorations should not have been unexpected. Given comprehensive inpatient EMRs, it is now possible to develop more rigorous definitions. A minimum set of parameters that one would need to specify for proper quantification of early detection is shown on Figure 3. The first is specifying a T₀, that is, the moment when a prediction regarding event X (which needs to be defined) is issued. This is different from the (currently unmeasurable) biologic onset of illness as well as the first documented indication that critical illness was present. Further, it is important to be explicit about the event time frame (the time period during which a predicted event is expected to occur): we are predicting that X will occur within E hours of the T₀. The time frame between the T₀ and X, which we are referring to as lead time, is clinically very important, as it represents the time period during which the response arm (eg, RRT intervention) is to be instituted. Statistical approaches can be used to estimate it, but once an early detection system is in place, it can be quantified. Figure 3 is not restricted to electronic systems; all components shown can be and are used by unaided human cognition.

Figure 3

Figure 4

It is essential to specify what data are used to generate probability estimates as well as the time frames used, which we refer to as the look‐back time frames. Several types of data could be employed, with some data elements (eg, age or gender) being discrete data with a 1:1 fixed correspondence between the patient and the data. Other data have a many‐to‐1 relationship, and an exact look‐back time frame must be specified for each data type. For example, it seems reasonable to specify a short (1224 hours) look‐back period for some types of data (eg, vital signs, lactate, admission diagnosis or chief complaint), an intermediate time period (13 days) for information on the current encounter, and a longer (months to years) time period for preexisting illness or comorbidity burden.

Because many events are rare, traditional measures used to assess model performance, such as the area under the receiver operator characteristic curve (C statistic), are not as helpful.[7] Consequently, much more emphasis needs to be given to 2 key metrics: number needed to evaluate (or workup to detection ratio) and threshold‐specific sensitivity (ability of the alert to detect X at a given threshold). With these, one can answer 3 questions that will be asked by the physicians and nurses who are not likely to be researchers, and who will have little interest in the statistics: How many patients do I need to work up each day? How many patients will I need to work up for each possible outcome identified? For this amount of work, how many of the possible outcomes will we catch?

Data availability for the study of severe and critical illness continues to expand. Practically, this means that future research will require more nuanced ontologies for the classification of physiologic derangement. Current approaches to severity scoring (collapsing data into composite scores) need to be replaced by dynamic approaches that consider differential effects on organ systems as well as what can be measured. Severity scoring will also need to incorporate the rate of change of a score (or probability derived from a score) in predicting the occurrence of an event of interest as well as judging response to treatment. Thus, instead of at time of ICU admission, the patient had a severity score of 76, we may have although this patient's severity score at the time of admission was decreasing by 4 points per hour per 10 mL/kg fluid given, the probability for respiratory instability was increasing by 2.3% per hour given 3 L/min supplemental oxygen. This approach is concordant with work done in other clinical settings (eg, in addition to an absolute value of maximal negative inspiratory pressure or vital capacity, the rate of deterioration of neuromuscular weakness in Guillain‐Barr syndrome is also important in predicting respiratory failure[8]).

Electronic data also could permit better definition of patient preferences regarding escalation of care. At present, available electronic data are limited (primarily, orders such as do not resuscitate).[9] However, this EMR domain is gradually expanding.[10, 11] Entities such as the National Institutes of Health could develop sophisticated and rapid questionnaires around patient preferences that are similar to those developed for the Patient Reported Outcomes Measurement Information System.[12] Such tools could have a significant effect on our ability to quantify the benefits of early detection as it relates to a patient's preferences (including better delineation of what treatments they would and would not want).

ACTIVATING A RESPONSE ARM

Early identification, antibiotic administration, fluid resuscitation, and source control are now widely felt to constitute low‐hanging fruit for decreasing morbidity and mortality in severe sepsis. All these measures are included in quality improvement programs and sepsis bundles.[13, 14, 15] However, before early interventions can be instituted, sepsis must at least be suspected, hence the need for early detection. The situation with respect to patient deterioration (for reasons other than sepsis) in general medical surgical wards is less clear‐cut. Reasons for deterioration are much more heterogenous and, consequently, early detection is likely necessary but not sufficient for outcomes improvement.

The 2 projects described in this issue describe nonspecific (indicating elevated risk but not specifying what led to the elevation of risk) and sepsis‐specific alerting systems. In the case of the nonspecific system, detection may not lead to an immediate deployment of a response arm. Instead, a secondary evaluation process must be triggered first. Following this evaluation component, a response arm may or may not be required. In contrast, the sepsis‐specific project essentially transforms the general medicalsurgical ward into a screening system. This screening system then also triggers specific bundle components.

Neither of these systems relies on unaided human cognition. In the case of the nonspecific system, a complex equation generates a probability that is displayed in the EMR, with protocols specifying what actions are to be taken when that probability exceeds a prespecified threshold. With respect to the sepsis screening system, clinicians are supported by EMR alerts as well as protocols that increase nursing autonomy when sepsis is suspected.

The distinction between nonspecific (eg, acute respiratory failure or hemodynamic deterioration) and specific (eg, severe sepsis) alerting systems is likely to disappear as advances in the field occur. For example, incorporation of natural language processing would permit inclusion of semantic data, which could be processed so as to prebucket an alert into one that not just gave a probability, but also a likely cause for the elevated probability.

In addition, both types of systems suffer from the limitation of working off a limited database because, in general, current textbooks and training programs primary focus remains that of treatment of full‐blown clinical syndromes. For example, little is known about how one should manage patients with intermediate lactate values, despite evidence showing that a significant percentage of patients who die from sepsis will initially have such values, with 1 study showing 63% as many deaths with initial lactate of 2.5 to 4.0 mmol/L as occurred with an initial lactate of >4.0 mmol/L.[16] Lastly, as is discussed below, both systems will encounter similar problems when it comes to quantifying benefit.

QUANTIFYING BENEFIT

Whereas the notion of deploying RRTs has clearly been successful, success in demonstrating unequivocal benefit remains elusive.[17, 18, 19] Outcome measures vary dramatically across studies and have included the number of RRT calls, decreases in code blue events on the ward, and decreases in inpatient mortality.[20] We suspect that other reasons are behind this problem. First is the lack of adequate risk adjustment and ignoring the impact of patients near the end of life on the denominator. Figure 4 shows recent data from 21 Kaiser Permanente Northern California (KPNC) hospitals, which can now capture care directive orders electronically,[21] illustrates this problem. The majority (53%) of hospital deaths occur among a highly variable proportion (range across hospitals, 6.5%18.0%) of patients who arrive at the hospital with a restricted resuscitation preference (do not resuscitate, partial code, and comfort care only). These patients do not want to die or crash and burn but, were they to trigger an alert, they would not necessarily want to be rescued by being transferred to the ICU either; moreover, internal KPNC analyses show that large numbers of these patients have sepsis and refuse aggressive treatment. The second major confounder is that ICUs save lives. Consequently, although early detection could lead to fewer transfers to the ICU, using the end point of ICU admission is very problematic, because in many cases the goal of alerting systems should be to get patients to the ICU sooner, which would not affect the outcome of transfer to the ICU in a downward direction; in fact, such systems might increase transfer to the ICU.

The complexities summarized in Figure 4 mean that it is likely that formal quantification of benefit will require examination of multiple measures, including balancing measures as described below. It is also evident that, in this respectlack of agreement as to what constitutes a good outcomethe issues being faced here are a reflection of a broader area of disagreement within our profession and society at large that extends to medical conditions other than critical illness.

POTENTIAL HARMS OF EARLY DETECTION

Implementation of early detection and rapid response systems are not inherently free of harm. If these systems are not shown to have benefit, then the cost of operating them is moving resources away from other, possibly evidence‐based, interventions.[22] At the individual level, alerts could frighten patients and their families (for example, some people are very uncomfortable with the idea that one can predict events). Physicians and nurses who work in the hospital are already quite busy, so every time an alert is issued, it adds to the demand on their already limited time, hence, the critical importance of strategies to minimize false alarms and alert fatigue. Moreover, altering existing workflows can be disruptive and unpopular.

A potentially more quantifiable problem is the impact of early detection systems on ICU operations. For example, if an RRT decides to transfer a patient from the ward to the ICU as a preventive measure (soft landing) and this in turn ties up an ICU bed, that bed is then unavailable for a new patient in the emergency department. Similarly, early detection systems coupled with structured protocols for promoting soft landings could result in a change in ICU case mix, with greater patient flow due to increased numbers of patients with lower severity and lower ICU length of stay. These considerations suggest the need to couple early detection with other supportive data systems and workflows (eg, systems that monitor bed capacity proactively).

Lastly, if documentation protocols are not established and followed, early detection systems could expose both individual clinicians as well as healthcare institutions to medicallegal risk. This consideration could be particularly important in those instances where an alert is issued and, for whatever reasons, clinicians do not take action and do not document that decision. At present, early detection systems are relatively uncommon, but they may gradually become standard of care. This means that in‐house out of ICU deteriorations, which are generally considered to be bad luck or due to a specific error or oversight, may then be considered to be preventable. Another possible scenario that could arise is that of plaintiffs invoking enterprise liability, where a hospital's not having an early detection system becomes considered negligent.

ARTICLES IN THIS ISSUE

In this issue of the Journal of Hospital Medicine, we examine early detection from various perspectives but around a common theme that usually gets less attention in the academic literature: implementation. The article by Schorr et al.[23] describes a disease‐specific approach that can be instantiated using either electronic or paper tools. Escobar et al.[24] describe the quantitative as well as the electronic architecture of an early warning system (EWS) pilot at 2 hospitals that are part of an integrated healthcare delivery system. Dummett et al.[25] then show how a clinical rescue component was developed to take advantage of the EWS, whereas Granich et al.[26] describe the complementary component (integration of supportive care and ensuring that patient preferences are respected). The paper by Liu et al.[27] concludes by placing all of this work in a much broader context, that of the learning healthcare system.

FUTURE DIRECTIONS: KEY GAPS IN THE FIELD

Important gaps remain with respect to early detection and response systems. Future research will need to focus on a number of areas. First and foremost, better approaches to quantifying the costbenefit relationships of these systems are needed; somehow, we need to move beyond a purely intuitive sense that they are good things. Related to this is the need to establish metrics that would permit rigorous comparisons between different approaches; this work needs to go beyond simple comparisons of the statistical characteristics of different predictive models. Ideally, it should include comparisons of different approaches for the response arms as well. We also need to characterize clinician understanding about detection systems, what constitutes impending or incipient critical illness, and the optimum way to provide early detection. Finally, better approaches to integrating health services research with basic science work must be developed; for example, how should one test new biomarkers in settings with early detection and response systems?

The most important frontier, however, is how one can make early detection and response systems more patient centered and how one can enhance their ability to respect patient preferences. Developing systems to improve clinical management is laudable, but somehow we need to also find ways to have these systems make a better connection to what patients want most and what matters most to them, something that may need to include new ways that sometimes suspend use of these systems. At the end of the day, after early detection, patients must have a care experience that they see as an unequivocal improvement.

This issue of the Journal of Hospital Medicine describes 2 research and quality improvement demonstration projects funded by the Gordon and Betty Moore Foundation. Early detection is central to both projects. This introductory article does not provide a global review of the now voluminous literature on rapid response teams (RRTs), sepsis detection systems, or treatment protocols. Rather, it takes a step back and reassesses just what early detection and quantification of critical illness are. It then examines the implications of early detection and its quantification.

CONCEPTUAL FRAMEWORK

We define severe illness as the presence of acute disease such that a person can no longer expect to improve without dedicated hospital treatment but which is not inevitably associated with mortality, postdischarge morbidity, or major loss of autonomy. In contrast, we define critical illness as acute disease with high a priori risk of mortality, postdischarge morbidity, and major (possibly total) loss of autonomy. We accept that the boundaries between ordinary illness, severe illness, and critical illness are blurred. The basic assumption behind all efforts at early detection is that these edges can be made sharp, and that the knowledge base required to do so can also lead to improvements in treatment protocols and patient outcomes. Further, it is assumed that at least some forms of critical illness can be prevented or mitigated by earlier detection, identification, and treatment.

Research over the last 2 decades has provided important support for this intuitive view as well as making it more nuanced. With respect to epidemiology, the big news is that sepsis is the biggest culprit, and that it accounts for a substantial proportion of all hospital deaths, including many previously considered unexpected hospital deaths due to in‐hospital deterioration.[1] With respect to treatment, a number of studies have demonstrated that crucial therapies previously considered to be intensive care unit (ICU) therapies can be initiated in the emergency department or general medicalsurgical ward.[2]

Figure 1 shows an idealized framework for illness presenting in the emergency department or general medicalsurgical wards. It illustrates the notion that a transition period exists when patients may be rescued with less intense therapy than will be required when condition progression occurs. Once a certain threshold is crossed, the risk of death or major postdischarge morbidity rises exponentially. Unaided human cognition's ability to determine where a given patient is in this continuum is dangerously variable and is highly dependent on the individuals training and experience. Consequently, as described in several of the articles in this issue as well as multiple other publications, health systems are employing comprehensive electronic medical records (EMRs) and are migrating to algorithmic approaches that combine multiple types of patient data.[3, 4] Although we are still some distance from being able to define exact boundaries between illness, severe illness, and critical illness, current EMRs permit much better definition of patient states, care processes, and short‐term outcomes.

Figure 1

Whereas our ability to quantify many processes and short‐term outcomes is expanding rapidly, quantification of the possible benefit of early detection is complicated by the fact that, even in the best of circumstances, not all patients can be rescued. For some patients, rescue may be temporary, raising the prospect of repeated episodes of critical illness and prolonged intensive care without any hope of leaving the hospital. Figure 2 shows that, for these patients, the problem is no longer simply one of preventing death and preserving function but, rather, preserving autonomy and dignity. In this context, early detection means earlier specification of patient preferences.[5, 6]

Figure 2

JUST WHAT CONSTITUTES EARLY DETECTION (AND HOW DO WE QUANTIFY IT)?

RRTs arose as the result of a number of studies showing thatin retrospectin‐hospital deteriorations should not have been unexpected. Given comprehensive inpatient EMRs, it is now possible to develop more rigorous definitions. A minimum set of parameters that one would need to specify for proper quantification of early detection is shown on Figure 3. The first is specifying a T₀, that is, the moment when a prediction regarding event X (which needs to be defined) is issued. This is different from the (currently unmeasurable) biologic onset of illness as well as the first documented indication that critical illness was present. Further, it is important to be explicit about the event time frame (the time period during which a predicted event is expected to occur): we are predicting that X will occur within E hours of the T₀. The time frame between the T₀ and X, which we are referring to as lead time, is clinically very important, as it represents the time period during which the response arm (eg, RRT intervention) is to be instituted. Statistical approaches can be used to estimate it, but once an early detection system is in place, it can be quantified. Figure 3 is not restricted to electronic systems; all components shown can be and are used by unaided human cognition.

Figure 3

Figure 4

It is essential to specify what data are used to generate probability estimates as well as the time frames used, which we refer to as the look‐back time frames. Several types of data could be employed, with some data elements (eg, age or gender) being discrete data with a 1:1 fixed correspondence between the patient and the data. Other data have a many‐to‐1 relationship, and an exact look‐back time frame must be specified for each data type. For example, it seems reasonable to specify a short (1224 hours) look‐back period for some types of data (eg, vital signs, lactate, admission diagnosis or chief complaint), an intermediate time period (13 days) for information on the current encounter, and a longer (months to years) time period for preexisting illness or comorbidity burden.

Because many events are rare, traditional measures used to assess model performance, such as the area under the receiver operator characteristic curve (C statistic), are not as helpful.[7] Consequently, much more emphasis needs to be given to 2 key metrics: number needed to evaluate (or workup to detection ratio) and threshold‐specific sensitivity (ability of the alert to detect X at a given threshold). With these, one can answer 3 questions that will be asked by the physicians and nurses who are not likely to be researchers, and who will have little interest in the statistics: How many patients do I need to work up each day? How many patients will I need to work up for each possible outcome identified? For this amount of work, how many of the possible outcomes will we catch?

Data availability for the study of severe and critical illness continues to expand. Practically, this means that future research will require more nuanced ontologies for the classification of physiologic derangement. Current approaches to severity scoring (collapsing data into composite scores) need to be replaced by dynamic approaches that consider differential effects on organ systems as well as what can be measured. Severity scoring will also need to incorporate the rate of change of a score (or probability derived from a score) in predicting the occurrence of an event of interest as well as judging response to treatment. Thus, instead of at time of ICU admission, the patient had a severity score of 76, we may have although this patient's severity score at the time of admission was decreasing by 4 points per hour per 10 mL/kg fluid given, the probability for respiratory instability was increasing by 2.3% per hour given 3 L/min supplemental oxygen. This approach is concordant with work done in other clinical settings (eg, in addition to an absolute value of maximal negative inspiratory pressure or vital capacity, the rate of deterioration of neuromuscular weakness in Guillain‐Barr syndrome is also important in predicting respiratory failure[8]).

Electronic data also could permit better definition of patient preferences regarding escalation of care. At present, available electronic data are limited (primarily, orders such as do not resuscitate).[9] However, this EMR domain is gradually expanding.[10, 11] Entities such as the National Institutes of Health could develop sophisticated and rapid questionnaires around patient preferences that are similar to those developed for the Patient Reported Outcomes Measurement Information System.[12] Such tools could have a significant effect on our ability to quantify the benefits of early detection as it relates to a patient's preferences (including better delineation of what treatments they would and would not want).

ACTIVATING A RESPONSE ARM

Early identification, antibiotic administration, fluid resuscitation, and source control are now widely felt to constitute low‐hanging fruit for decreasing morbidity and mortality in severe sepsis. All these measures are included in quality improvement programs and sepsis bundles.[13, 14, 15] However, before early interventions can be instituted, sepsis must at least be suspected, hence the need for early detection. The situation with respect to patient deterioration (for reasons other than sepsis) in general medical surgical wards is less clear‐cut. Reasons for deterioration are much more heterogenous and, consequently, early detection is likely necessary but not sufficient for outcomes improvement.

The 2 projects described in this issue describe nonspecific (indicating elevated risk but not specifying what led to the elevation of risk) and sepsis‐specific alerting systems. In the case of the nonspecific system, detection may not lead to an immediate deployment of a response arm. Instead, a secondary evaluation process must be triggered first. Following this evaluation component, a response arm may or may not be required. In contrast, the sepsis‐specific project essentially transforms the general medicalsurgical ward into a screening system. This screening system then also triggers specific bundle components.

Neither of these systems relies on unaided human cognition. In the case of the nonspecific system, a complex equation generates a probability that is displayed in the EMR, with protocols specifying what actions are to be taken when that probability exceeds a prespecified threshold. With respect to the sepsis screening system, clinicians are supported by EMR alerts as well as protocols that increase nursing autonomy when sepsis is suspected.

The distinction between nonspecific (eg, acute respiratory failure or hemodynamic deterioration) and specific (eg, severe sepsis) alerting systems is likely to disappear as advances in the field occur. For example, incorporation of natural language processing would permit inclusion of semantic data, which could be processed so as to prebucket an alert into one that not just gave a probability, but also a likely cause for the elevated probability.

In addition, both types of systems suffer from the limitation of working off a limited database because, in general, current textbooks and training programs primary focus remains that of treatment of full‐blown clinical syndromes. For example, little is known about how one should manage patients with intermediate lactate values, despite evidence showing that a significant percentage of patients who die from sepsis will initially have such values, with 1 study showing 63% as many deaths with initial lactate of 2.5 to 4.0 mmol/L as occurred with an initial lactate of >4.0 mmol/L.[16] Lastly, as is discussed below, both systems will encounter similar problems when it comes to quantifying benefit.

QUANTIFYING BENEFIT

Whereas the notion of deploying RRTs has clearly been successful, success in demonstrating unequivocal benefit remains elusive.[17, 18, 19] Outcome measures vary dramatically across studies and have included the number of RRT calls, decreases in code blue events on the ward, and decreases in inpatient mortality.[20] We suspect that other reasons are behind this problem. First is the lack of adequate risk adjustment and ignoring the impact of patients near the end of life on the denominator. Figure 4 shows recent data from 21 Kaiser Permanente Northern California (KPNC) hospitals, which can now capture care directive orders electronically,[21] illustrates this problem. The majority (53%) of hospital deaths occur among a highly variable proportion (range across hospitals, 6.5%18.0%) of patients who arrive at the hospital with a restricted resuscitation preference (do not resuscitate, partial code, and comfort care only). These patients do not want to die or crash and burn but, were they to trigger an alert, they would not necessarily want to be rescued by being transferred to the ICU either; moreover, internal KPNC analyses show that large numbers of these patients have sepsis and refuse aggressive treatment. The second major confounder is that ICUs save lives. Consequently, although early detection could lead to fewer transfers to the ICU, using the end point of ICU admission is very problematic, because in many cases the goal of alerting systems should be to get patients to the ICU sooner, which would not affect the outcome of transfer to the ICU in a downward direction; in fact, such systems might increase transfer to the ICU.

The complexities summarized in Figure 4 mean that it is likely that formal quantification of benefit will require examination of multiple measures, including balancing measures as described below. It is also evident that, in this respectlack of agreement as to what constitutes a good outcomethe issues being faced here are a reflection of a broader area of disagreement within our profession and society at large that extends to medical conditions other than critical illness.

POTENTIAL HARMS OF EARLY DETECTION

Implementation of early detection and rapid response systems are not inherently free of harm. If these systems are not shown to have benefit, then the cost of operating them is moving resources away from other, possibly evidence‐based, interventions.[22] At the individual level, alerts could frighten patients and their families (for example, some people are very uncomfortable with the idea that one can predict events). Physicians and nurses who work in the hospital are already quite busy, so every time an alert is issued, it adds to the demand on their already limited time, hence, the critical importance of strategies to minimize false alarms and alert fatigue. Moreover, altering existing workflows can be disruptive and unpopular.

A potentially more quantifiable problem is the impact of early detection systems on ICU operations. For example, if an RRT decides to transfer a patient from the ward to the ICU as a preventive measure (soft landing) and this in turn ties up an ICU bed, that bed is then unavailable for a new patient in the emergency department. Similarly, early detection systems coupled with structured protocols for promoting soft landings could result in a change in ICU case mix, with greater patient flow due to increased numbers of patients with lower severity and lower ICU length of stay. These considerations suggest the need to couple early detection with other supportive data systems and workflows (eg, systems that monitor bed capacity proactively).

Lastly, if documentation protocols are not established and followed, early detection systems could expose both individual clinicians as well as healthcare institutions to medicallegal risk. This consideration could be particularly important in those instances where an alert is issued and, for whatever reasons, clinicians do not take action and do not document that decision. At present, early detection systems are relatively uncommon, but they may gradually become standard of care. This means that in‐house out of ICU deteriorations, which are generally considered to be bad luck or due to a specific error or oversight, may then be considered to be preventable. Another possible scenario that could arise is that of plaintiffs invoking enterprise liability, where a hospital's not having an early detection system becomes considered negligent.

ARTICLES IN THIS ISSUE

In this issue of the Journal of Hospital Medicine, we examine early detection from various perspectives but around a common theme that usually gets less attention in the academic literature: implementation. The article by Schorr et al.[23] describes a disease‐specific approach that can be instantiated using either electronic or paper tools. Escobar et al.[24] describe the quantitative as well as the electronic architecture of an early warning system (EWS) pilot at 2 hospitals that are part of an integrated healthcare delivery system. Dummett et al.[25] then show how a clinical rescue component was developed to take advantage of the EWS, whereas Granich et al.[26] describe the complementary component (integration of supportive care and ensuring that patient preferences are respected). The paper by Liu et al.[27] concludes by placing all of this work in a much broader context, that of the learning healthcare system.

FUTURE DIRECTIONS: KEY GAPS IN THE FIELD

Important gaps remain with respect to early detection and response systems. Future research will need to focus on a number of areas. First and foremost, better approaches to quantifying the costbenefit relationships of these systems are needed; somehow, we need to move beyond a purely intuitive sense that they are good things. Related to this is the need to establish metrics that would permit rigorous comparisons between different approaches; this work needs to go beyond simple comparisons of the statistical characteristics of different predictive models. Ideally, it should include comparisons of different approaches for the response arms as well. We also need to characterize clinician understanding about detection systems, what constitutes impending or incipient critical illness, and the optimum way to provide early detection. Finally, better approaches to integrating health services research with basic science work must be developed; for example, how should one test new biomarkers in settings with early detection and response systems?

The most important frontier, however, is how one can make early detection and response systems more patient centered and how one can enhance their ability to respect patient preferences. Developing systems to improve clinical management is laudable, but somehow we need to also find ways to have these systems make a better connection to what patients want most and what matters most to them, something that may need to include new ways that sometimes suspend use of these systems. At the end of the day, after early detection, patients must have a care experience that they see as an unequivocal improvement.

Patients who deteriorate outside highly monitored settings and who require unplanned transfer to the intensive care unit (ICU) are known to have high mortality and morbidity.[1, 2, 3, 4, 5] The notion that early detection of a deteriorating patient improves outcomes has intuitive appeal and is discussed in a large number of publications.[6, 7, 8, 9, 10] However, much less information is available on what should be done after early detection is made.[11] Existing literature on early warning systems (EWSs) does not provide enough detail to serve as a map for implementation. This lack of transparency is complicated by the fact that, although the comprehensive inpatient electronic medical record (EMR) now constitutes the central locus for clinical practice, much of the existing literature comes from research institutions that may employ home‐grown EMRs, not community hospitals that employ commercially available systems.

In this issue of the Journal of Hospital Medicine, we describe our efforts to bridge that gap by implementing an EWS in a pair of community hospitals. The EWS's development and its basic statistical and electronic infrastructure are described in the articles by Escobar and Dellinger and Escobar et al.[2, 12, 13] In this report, we focus on how we addressed clinicians' primary concern: What do we do when we get an alert? Because it is described in detail by Granich et al.[14] elsewhere in this issue of the Journal of Hospital Medicine, a critical component of our implementation process (ensuring that patient preferences with respect to supportive care are honored) is not discussed.

Our article is divided into the following sections: rationale, preimplementation preparatory work, workflow development, response protocols, challenges and key learnings, and concluding reflections.

RATIONALE

Much of the previous work on the implementation of alarm systems has focused on the statistics behind detection or on the quantification of processes (eg, how many rapid response calls were triggered) or on outcomes such as mortality. The conceptual underpinnings and practical steps necessary for successful integration of an alarm system into the clinicians' workflow have not been articulated. Our theoretical framework was based on (1) improving situational awareness[15] (knowing what is going on around you and what is likely to happen next) and (2) mitigating cognitive errors.

An EWS enhances situational awareness most directly by earlier identification of a problem with a particular patient. As is detailed by Escobar et al.[16] in this issue of the Journal of Hospital Medicine, our EWS extracts EMR data every 6 hours, performs multiple calculations, and then displays 3 scores in real time in the inpatient dashboard (known as the Patient Lists activity in the Epic EMR). The first of these scores is the Laboratory‐Based Acute Physiologic Score, version 2 (LAPS2), an objective severity score whose retrospective version is already in use in Kaiser Permanente Northern California (KPNC) for internal benchmarking.[13] This score captures a patient's overall degree of physiologic instability within the preceding 72 hours. The second is the Comorbidity Point Score, version 2 (COPS2), a longitudinal comorbidity score based on the patient's diagnoses over the preceding 12 months.[13] This score captures a patient's overall comorbidity burden. Thus, it is possible for a patient to be very ill (high COPS2) while also being stable (low LAPS2) or vice versa. Both of these scores have other uses, including prediction of rehospitalization risk in real time,[17] which is also being piloted at KPNC. Finally, the Advanced Alert Monitoring (AAM) score, which integrates the LAPS2 and COPS2 with other variables, provides a 12‐hour deterioration risk, with a threshold value of 8% triggering response protocols. At or above this threshold, which was agreed to prior to implementation, the system achieves 25% sensitivity, 98% specificity, with a number needed to evaluate of 10 to 12, a level of workload that was felt to be acceptable by clinicians. Actions triggered by the EWS may be quite different from those one would take when being notified of a code blue, which is called at the time an event occurs. The EWS focuses attention on patients who might be missed because they do not yet appear critically ill. It also provides a shared, quantifiable measure of a patient's risk that can trigger a standardized plan of action to follow in evaluating and treating a patient.[15]

In addition to enhancing situational awareness, we intended the alarms to produce cognitive change in practitioners. Our goal was to replace medical intuition with analytic, evidence‐based judgment of future illness. We proceeded with the understanding that replacing quick intuition with slower analytic response is an essential skill in developing sound clinical reasoning.[18, 19, 20] The alert encourages physicians to reassess high‐risk patients facilitating a cognitive shift from automatic, error‐prone processing to slower, deliberate processing. Given the busy pace of ward work, slowing down permits clinicians to reassess previously overlooked details. Related to this process of inducing cognitive change is a secondary effect: we uncovered and discussed physician biases. Physicians are subject to potential biases that allow patients to deteriorate.[18, 19, 20] Therefore, we addressed bias through education. By reviewing particular cases of unanticipated deterioration at each hospital facility, we provided evidence for the problem of in‐hospital deterioration. This framed the new tool as an opportunity for improving treatment and encouraged physicians to act on the alert using a structured process.

INTERVENTIONS

Preimplementation Preparatory Work

Initial KPNC data provided strong support for the generally accepted notion that unplanned transfer patients have poor outcomes.[2, 4, 5] However, published reports failed to provide the granular detail clinicians need to implement a response arm at the unit and patient level. In preparation for going live, we conducted a retrospective chart review. This included data from patients hospitalized from January 1, 2011 through December 31, 2012 (additional detail is provided in the Supporting Information, Appendix, in the online version of this article). The key findings from our internal review of subjective documentation preceding deterioration are similar to those described in the literature and summarized in Figure 1, which displays the 5 most common clinical presentations associated with unplanned transfers.

Figure 1

The chart review served several major roles. First, it facilitated cognitive change by eliminating the notion that it can't happen here. Second, it provided considerable guidance on key clinical components that had to be incorporated into the workflow. Third, it engaged the rapid response team (RRT) in reviewing our work retrospectively to identify future opportunities. Finally, the review provided considerable guidance with respect to structuring documentation requirements.

As a result of the above efforts, other processes detailed below, and knowledge described in several of the companion articles in this issue of the Journal of Hospital Medicine, 3 critical elements, which had been explicitly required by our leadership, were in place prior to the go‐live date: a general consensus among hospitalists and nurses that this would be worth testing, a basic clinical response workflow, and an automated checklist for documentation. We refined these in a 2‐week shadowing phase preceding the start date. In this phase, the alerts were not displayed in the EMR. Instead, programmers working on the project notified selected physician leaders by phone. This permitted them to understand exactly what sort of patients were reaching the physiologic threshold so that they could better prepare both RRT registered nurses (RNs) and hospitalists for the go‐live date. This also provided an opportunity to begin refining the documentation process using actual patients.

The original name for our project was Early Detection of Impending Physiologic Deterioration. However, during the preparatory phase, consultation with our public relations staff led to a concern that the name could be frightening to some patients. This highlights the need to consider patient perceptions and how words used in 1 way by physicians can have different connotations to nonclinicians. Consequently, the system was renamed, and it is now referred to as Advance Alert Monitoring (AAM).

Workflow Development

We carefully examined the space where electronic data, graphical user interfaces, and clinical practice blend, a nexus now commonly referred to as workflow or user experience.[21] To promote situational awareness and effect cognitive change, we utilized the Institute for Health Care Improvement's Plan‐Do‐Study‐Act model.[22, 23] We then facilitated the iterative development of a clinician‐endorsed workflow.[22, 23, 24, 25] By adjusting the workflow based on ongoing experience and giving clinicians multiple opportunities to revise (a process that continues to date), we ensured clinicians would approach and endorse the alarm system as a useful tool for decision support.

Table 1 summarizes the work groups assembled for our implementation, and Table 2 provides a system‐oriented checklist indicating key components that need to be in place prior to having an early warning system go live in a hospital. Figure 2 summarizes the alert response protocols we developed through an iterative process at the 2 pilot sites. The care path shown in Figure 2 is the result of considerable revision, mostly due to actual experience acquired following the go live date. The diagram also includes a component that is still work in progress. This is how an emergency department probability estimate (triage support) will be integrated into both the ward as well as the ICU workflows. Although this is beyond the scope of this article, other hospitals may be experimenting with triage support (eg, for sepsis patients), so it is important to consider how one would incorporate such support into workflows.

Table 1.Workgroups Established for Early Warning System Rollout

Workgroup	Goals
NOTE: Abbreviations: POLST, physician orders for life‐sustaining treatment.
Clinical checklist	Perform structured chart review of selected unplanned transfer patients and near misses
	Develop a checklist for mitigation strategies given an alert
Develop documentation standards given an alert
Develop escalation protocol given an alert
Workload and threshold	Determine threshold for sensitivity of alerts and resulting impact on clinician workload
Patient preferences	Prepare background information to be presented to providers regarding end‐of‐life care and POLST orders
	Coordinate with clinical checklist workgroup to generate documentation templates that provide guidance for appropriate management of patients regarding preferences on escalation of care and end‐of‐life care
Electronic medical record coordination	Review proposed electronic medical record changes
	Make recommendation for further changes as needed
Develop plan for rollout of new and/or revised electronic record tools
Designate contact list for questions/emssues that may arise regarding electronic record changes during the pilot
Determine alert display choices and mode of alert notification
Nursing committee	Review staffing needs in anticipation of alert
	Coordinate with workload and threshold group
Develop training calendar to ensure skills necessary for successful implementation of alerts
Make recommendations for potential modification of rapid response team's role in development of a clinical checklist for nurses responding to an alert
Design educational materials for clinicians
Local communication strategy	Develop internal communication plan (for clinical staff not directly involved with pilot)
	Develop external communication plan (for nonclinicians who may hear about the project)

Table 2.Hospital System‐Wide Go Live Checklist

Level	Tasks
Administration	Obtain executive committee approval
	Establish communication protocols with quality assurance and quality improvement committees
Review protocols with medicallegal department
Communication	Write media material for patients and families
	Develop and disseminate scripts for front‐line staff
Develop communication and meet with all relevant front‐line staff on merits of project
Educate all staff on workflow changes and impacts
Clinical preparation	Conduct internal review of unplanned transfers and present results to all clinicians
	Determine service level agreements, ownership of at‐risk patients, who will access alerts
Conduct staff meetings to educate staff
Perform debriefs on relevant cases
Determine desired outcomes, process measures, balancing measures
Determine acceptable clinician burden (alerts/day)
Technology	Establish documentation templates
	Ensure access to new data fields (electronic medical record security process must be followed for access rights)
Workflows	Workflows (clinical response, patient preferences, supportive care, communication, documentation) must be in place prior to actual go live
Shadowing	Testing period (alerts communicated to selected clinicians prior to going live) should occur

Figure 2

RESPONSE PROTOCOLS

At South San Francisco, the RRT consists of an ICU nurse, a respiratory care therapist, and a designated hospitalist; at Sacramento, the team is also augmented by an additional nurse (the house supervisor). In addition to responding to the AAM alerts, RRT nurses respond to other emergency calls such as code blues, stroke alerts, and patient or patient‐familyinitiated rapid response calls. They also expedite time sensitive workups and treatments. They check up on recent transfers from the ICU to ensure continued improvement justifying staying on the ward. Serving as peer educators, they assist with processes such as chest tube or central line insertions, troubleshoot high‐risk medication administration, and ensure that treatment bundles (eg, for sepsis) occur expeditiously.

The RRT reviews EWS scores every 6 hours. The AAM score is seen as soon as providers open the chart, which helps triage patients for evaluation. Because patients can still be at risk even without an elevated AAM score, all normal escalation pathways remain in place. Once an alert is noted in the inpatient dashboard, the RRT nurse obtains a fresh set of vital signs, assesses the patient's clinical status, and informs the physician, social worker, and primary nurse (Figure 2). Team members work with the bedside nurse, providing support with assessment, interventions, plans, and follow‐up. Once advised of the alert, the hospitalist performs a second chart review and evaluates the patient at the bedside to identify factors that could underlie potential deterioration. After this evaluation, the hospitalist documents concerns, orders appropriate interventions (which can include escalation), and determines appropriate follow‐up. We made sure the team knew that respiratory distress, arrhythmias, mental status changes, or worsening infection were responsible for over 80% of in‐hospital deterioration cases. We also involved palliative care earlier in patient care, streamlining the process so the RRT makes just 1 phone call to the social worker, who contacts the palliative care physician and nurse to ensure patients have a designated surrogate in the event of further deterioration.

Our initial documentation template consisted of a comprehensive organ system‐based physician checklist. However, although this was of use to covering physicians unfamiliar with a given patient, it was redundant and annoying to attending providers already familiar with the patient. After more than 30 iterations, we settled on a succinct note that only documented the clinicians' clinical judgment as to what constituted the major risk for deterioration and what the mitigation strategies would be. Both of these judgments are in a checklist format (see Supporting Information, Appendix, in the online version of this article for the components of the physician and nurse notes).

Prior to the implementation of the system, RRT nurses performed proactive rounding by manually checking patient labs and vital signs, an inefficient process due to the poor sensitivity and specificity of individual values. Following implementation of the system, RRT RNs and clinicians switched to sorting patients by the 3 scores (COPS2, LAPS2, AAM). For example, patients may be stable at admission (as evidenced by their AAM score) but be at high risk due to their comorbidities. One approach that has been employed is to proactively check such patients to ensure they have a care directive in place, as is described in the article by Granich et al.[14] The Supportive Care Team (detailed in Granich et al.) assesses needs for palliative care and provides in‐hospital consultation as needed. Social services staff perform chart reviews to ensure a patient surrogate has been defined and also works with patients and their families to clarify goals of care.

CHALLENGES AND KEY LEARNINGS

One challenge that arose was reconciling the periodic nature of the alert (every 6 hours) with physicians' availability, which varied due to different rounding workflows at the 2 sites. Consequently, the alert cycle was changed; at the first site, the cycle was set to 1000‐1600‐2200‐0400, whereas the second site chose 0800‐1400‐2000‐0200.

One essential but problematic component of the clinical response is the issue of documentation. Inadequate documentation could lead to adverse outcomes, clinician malpractice exposure, and placing the entire hospital at risk for enterprise liability when clinical responses are not documented. This issue is complicated by the fact that overzealous efforts could lead to less or no documentation by making it too onerous for busy clinicians. We found that the ease with which data can populate progress notes in the EMR can lead to note bloat. Clearly, no documentation is not enough, and a complete history and physical is too much. Paradoxically, 1 of the issues underlying our problems with documentation was the proactive nature of the alerts themselves; because they are based on an outcome prediction in the next 12 hours, documenting the response to them may lack (perceived) urgency.

Shortly after the system went live, a patient who had been recently transferred out to the ward from the ICU triggered an alert. As a response was mounted, the team realized that existing ward protocols did not specify which physician service (intensivist or hospitalist) was responsible for patients who were transitioning from 1 unit to another. We also had to perform multiple revisions of the protocols specifying how alerts were handled when they occurred at times of change of shift. Eventually, we settled on having the combination of a hospitalist and an RRT nurse as the cornerstone of the response, with the hospitalist service as the primary owner of the entire process, but this arrangement might need to be varied in different settings. As a result of the experience with the pilot, the business case for deployment in the remaining 19 hospitals includes a formal budget request so that all have properly staffed RRTs, although the issue of primary ownership of the alert process for different patient types (eg, surgical patients) will be decided on a hospital‐by‐hospital basis. These experiences raise the intriguing possibility that implementation of alert systems can lead to the identification of systemic gaps in existing protocols. These gaps can include specific components of the hospital service agreements between multiple departments (emergency, hospital medicine, ICU, palliative care, surgery) as well as problems with existing workflows.

In addition to ongoing tweaking of care protocols, 3 issues remain unresolved. First is the issue of documentation. The current documentation notes are not completely satisfactory, and we are working with the KPNC EMR administrators to refine the tool. Desirable refinements include (1) having the system scores populate in more accessible sectors of the EMR where their retrieval will facilitate increased automation of the note writing process, (2) changing the note type to a note that will facilitate process audits, and (3) linking the note to other EMR tools so that the response arm can be tracked more formally. The second issue is the need to develop strategies to address staff turnover; for example, newer staff may not have received the same degree of exposure to the system as those who were there when it was started. Finally, due to limited resources, we have done very limited work on more mechanistic analyses of the clinical response itself. For example, it would be desirable to perform a formal quantitative, risk‐adjusted process‐outcome analysis of why some patients' outcomes are better than others following an alert.

Finally, it is also the case that we have had some unexpected occurrences that hint at new uses and benefits of alert systems. One of these is the phenomenon of chasing the alert. Some clinicians, on their own, have taken a more proactive stance in the care of patients in whom the AAM score is rising or near the alert threshold. This has 2 potential consequences. Some patients are stabilized and thus do not reach threshold instability levels. In other cases, patients reach threshold but the response team is informed that things are already under control. A second unexpected result is increased requests for COPS2 scores by clinicians who have heard about the system, particularly surgeons who would like to use the comorbidity scores as a screening tool in the outpatient setting. Because KPNC is an integrated system, it is not likely that such alternatives will be implemented immediately without considerable analysis, but it is clear that the system's deployment has captured the clinicians' imagination.

CONCLUSIONS AND FUTURE DIRECTIONS

Our preparatory efforts have been successful. We have found that embedding an EWS in a commercially available EMR is acceptable to hospital physicians and nurses. We have developed a coordinated workflow for mitigation and escalation that is tightly linked to the availability of probabilistic alerts in real time. Although resource limitations have precluded us from conducting formal clinician surveys, the EWS has been discussed at multiple hospital‐wide as well as department‐specific meetings. Although there have been requests for clarification, refinements, and modifications in workflows, no one has suggested that the system be discontinued. Further, many of the other KPNC hospitals have requested that the EWS be deployed at their site. We have examined KPNC databases that track patient complaints and have not found any complaints that could be linked to the EWS. Most importantly, the existence of the workflows we have developed has played a major role in KPNC's decision to deploy the system in its remaining hospitals.

Although alert fatigue is the number 1 reason that clinicians do not utilize embedded clinical decision support,[26] simply calibrating statistical models is insufficient. Careful consideration of clinicians' needs and responsibilities, particularly around ownership of patients and documentation, is essential. Such consideration needs to include planning time and socializing the system (providing multiple venues for clinicians to learn about the system as well as participate in the process for using it).

We anticipate that, as the system leaves the pilot stage and becomes a routine component of hospital care, additional enhancements (eg, sending notifications to smart phones, providing an alert response tracking system) will be added. Our organization is also implementing real‐time concurrent review of inpatient EMRs (eg, for proactive detection of an expanded range of potential process failures), and work is underway on how to link the workflows we describe here with this effort. As has been the case with other systems,[27] it is likely that we will eventually move to continuous scanning of patient data rather than only every 6 hours. Given that the basic workflow is quite robust and amenable to local modifications, we are confident that our clinicians and hospitals will adapt to future system enhancements.

Lastly, we intend to conduct additional research on the clinical response itself. In particular, we consider it extremely important to conduct formal quantitative analyses on why some patients' outcomes are better than others following an alert. A key component of this effort will be to develop tools that can permit an automatedor nearly automatedassessment of the clinical response. For example, we are considering automated approaches that would scan the EMR for the presence of specific orders, notes, vital signs patterns, and laboratory tests following an alert. Whereas it may not be possible to dispense with manual chart review, even partial automation of a feedback process could lead to significant enhancement of our quality improvement efforts.

Patients who deteriorate outside highly monitored settings and who require unplanned transfer to the intensive care unit (ICU) are known to have high mortality and morbidity.[1, 2, 3, 4, 5] The notion that early detection of a deteriorating patient improves outcomes has intuitive appeal and is discussed in a large number of publications.[6, 7, 8, 9, 10] However, much less information is available on what should be done after early detection is made.[11] Existing literature on early warning systems (EWSs) does not provide enough detail to serve as a map for implementation. This lack of transparency is complicated by the fact that, although the comprehensive inpatient electronic medical record (EMR) now constitutes the central locus for clinical practice, much of the existing literature comes from research institutions that may employ home‐grown EMRs, not community hospitals that employ commercially available systems.

In this issue of the Journal of Hospital Medicine, we describe our efforts to bridge that gap by implementing an EWS in a pair of community hospitals. The EWS's development and its basic statistical and electronic infrastructure are described in the articles by Escobar and Dellinger and Escobar et al.[2, 12, 13] In this report, we focus on how we addressed clinicians' primary concern: What do we do when we get an alert? Because it is described in detail by Granich et al.[14] elsewhere in this issue of the Journal of Hospital Medicine, a critical component of our implementation process (ensuring that patient preferences with respect to supportive care are honored) is not discussed.

Our article is divided into the following sections: rationale, preimplementation preparatory work, workflow development, response protocols, challenges and key learnings, and concluding reflections.

RATIONALE

Much of the previous work on the implementation of alarm systems has focused on the statistics behind detection or on the quantification of processes (eg, how many rapid response calls were triggered) or on outcomes such as mortality. The conceptual underpinnings and practical steps necessary for successful integration of an alarm system into the clinicians' workflow have not been articulated. Our theoretical framework was based on (1) improving situational awareness[15] (knowing what is going on around you and what is likely to happen next) and (2) mitigating cognitive errors.

An EWS enhances situational awareness most directly by earlier identification of a problem with a particular patient. As is detailed by Escobar et al.[16] in this issue of the Journal of Hospital Medicine, our EWS extracts EMR data every 6 hours, performs multiple calculations, and then displays 3 scores in real time in the inpatient dashboard (known as the Patient Lists activity in the Epic EMR). The first of these scores is the Laboratory‐Based Acute Physiologic Score, version 2 (LAPS2), an objective severity score whose retrospective version is already in use in Kaiser Permanente Northern California (KPNC) for internal benchmarking.[13] This score captures a patient's overall degree of physiologic instability within the preceding 72 hours. The second is the Comorbidity Point Score, version 2 (COPS2), a longitudinal comorbidity score based on the patient's diagnoses over the preceding 12 months.[13] This score captures a patient's overall comorbidity burden. Thus, it is possible for a patient to be very ill (high COPS2) while also being stable (low LAPS2) or vice versa. Both of these scores have other uses, including prediction of rehospitalization risk in real time,[17] which is also being piloted at KPNC. Finally, the Advanced Alert Monitoring (AAM) score, which integrates the LAPS2 and COPS2 with other variables, provides a 12‐hour deterioration risk, with a threshold value of 8% triggering response protocols. At or above this threshold, which was agreed to prior to implementation, the system achieves 25% sensitivity, 98% specificity, with a number needed to evaluate of 10 to 12, a level of workload that was felt to be acceptable by clinicians. Actions triggered by the EWS may be quite different from those one would take when being notified of a code blue, which is called at the time an event occurs. The EWS focuses attention on patients who might be missed because they do not yet appear critically ill. It also provides a shared, quantifiable measure of a patient's risk that can trigger a standardized plan of action to follow in evaluating and treating a patient.[15]

In addition to enhancing situational awareness, we intended the alarms to produce cognitive change in practitioners. Our goal was to replace medical intuition with analytic, evidence‐based judgment of future illness. We proceeded with the understanding that replacing quick intuition with slower analytic response is an essential skill in developing sound clinical reasoning.[18, 19, 20] The alert encourages physicians to reassess high‐risk patients facilitating a cognitive shift from automatic, error‐prone processing to slower, deliberate processing. Given the busy pace of ward work, slowing down permits clinicians to reassess previously overlooked details. Related to this process of inducing cognitive change is a secondary effect: we uncovered and discussed physician biases. Physicians are subject to potential biases that allow patients to deteriorate.[18, 19, 20] Therefore, we addressed bias through education. By reviewing particular cases of unanticipated deterioration at each hospital facility, we provided evidence for the problem of in‐hospital deterioration. This framed the new tool as an opportunity for improving treatment and encouraged physicians to act on the alert using a structured process.

INTERVENTIONS

Preimplementation Preparatory Work

Initial KPNC data provided strong support for the generally accepted notion that unplanned transfer patients have poor outcomes.[2, 4, 5] However, published reports failed to provide the granular detail clinicians need to implement a response arm at the unit and patient level. In preparation for going live, we conducted a retrospective chart review. This included data from patients hospitalized from January 1, 2011 through December 31, 2012 (additional detail is provided in the Supporting Information, Appendix, in the online version of this article). The key findings from our internal review of subjective documentation preceding deterioration are similar to those described in the literature and summarized in Figure 1, which displays the 5 most common clinical presentations associated with unplanned transfers.

Figure 1

The chart review served several major roles. First, it facilitated cognitive change by eliminating the notion that it can't happen here. Second, it provided considerable guidance on key clinical components that had to be incorporated into the workflow. Third, it engaged the rapid response team (RRT) in reviewing our work retrospectively to identify future opportunities. Finally, the review provided considerable guidance with respect to structuring documentation requirements.

As a result of the above efforts, other processes detailed below, and knowledge described in several of the companion articles in this issue of the Journal of Hospital Medicine, 3 critical elements, which had been explicitly required by our leadership, were in place prior to the go‐live date: a general consensus among hospitalists and nurses that this would be worth testing, a basic clinical response workflow, and an automated checklist for documentation. We refined these in a 2‐week shadowing phase preceding the start date. In this phase, the alerts were not displayed in the EMR. Instead, programmers working on the project notified selected physician leaders by phone. This permitted them to understand exactly what sort of patients were reaching the physiologic threshold so that they could better prepare both RRT registered nurses (RNs) and hospitalists for the go‐live date. This also provided an opportunity to begin refining the documentation process using actual patients.

The original name for our project was Early Detection of Impending Physiologic Deterioration. However, during the preparatory phase, consultation with our public relations staff led to a concern that the name could be frightening to some patients. This highlights the need to consider patient perceptions and how words used in 1 way by physicians can have different connotations to nonclinicians. Consequently, the system was renamed, and it is now referred to as Advance Alert Monitoring (AAM).

Workflow Development

We carefully examined the space where electronic data, graphical user interfaces, and clinical practice blend, a nexus now commonly referred to as workflow or user experience.[21] To promote situational awareness and effect cognitive change, we utilized the Institute for Health Care Improvement's Plan‐Do‐Study‐Act model.[22, 23] We then facilitated the iterative development of a clinician‐endorsed workflow.[22, 23, 24, 25] By adjusting the workflow based on ongoing experience and giving clinicians multiple opportunities to revise (a process that continues to date), we ensured clinicians would approach and endorse the alarm system as a useful tool for decision support.

Table 1 summarizes the work groups assembled for our implementation, and Table 2 provides a system‐oriented checklist indicating key components that need to be in place prior to having an early warning system go live in a hospital. Figure 2 summarizes the alert response protocols we developed through an iterative process at the 2 pilot sites. The care path shown in Figure 2 is the result of considerable revision, mostly due to actual experience acquired following the go live date. The diagram also includes a component that is still work in progress. This is how an emergency department probability estimate (triage support) will be integrated into both the ward as well as the ICU workflows. Although this is beyond the scope of this article, other hospitals may be experimenting with triage support (eg, for sepsis patients), so it is important to consider how one would incorporate such support into workflows.

Table 1.Workgroups Established for Early Warning System Rollout

Workgroup	Goals
NOTE: Abbreviations: POLST, physician orders for life‐sustaining treatment.
Clinical checklist	Perform structured chart review of selected unplanned transfer patients and near misses
	Develop a checklist for mitigation strategies given an alert
Develop documentation standards given an alert
Develop escalation protocol given an alert
Workload and threshold	Determine threshold for sensitivity of alerts and resulting impact on clinician workload
Patient preferences	Prepare background information to be presented to providers regarding end‐of‐life care and POLST orders
	Coordinate with clinical checklist workgroup to generate documentation templates that provide guidance for appropriate management of patients regarding preferences on escalation of care and end‐of‐life care
Electronic medical record coordination	Review proposed electronic medical record changes
	Make recommendation for further changes as needed
Develop plan for rollout of new and/or revised electronic record tools
Designate contact list for questions/emssues that may arise regarding electronic record changes during the pilot
Determine alert display choices and mode of alert notification
Nursing committee	Review staffing needs in anticipation of alert
	Coordinate with workload and threshold group
Develop training calendar to ensure skills necessary for successful implementation of alerts
Make recommendations for potential modification of rapid response team's role in development of a clinical checklist for nurses responding to an alert
Design educational materials for clinicians
Local communication strategy	Develop internal communication plan (for clinical staff not directly involved with pilot)
	Develop external communication plan (for nonclinicians who may hear about the project)

Table 2.Hospital System‐Wide Go Live Checklist

Level	Tasks
Administration	Obtain executive committee approval
	Establish communication protocols with quality assurance and quality improvement committees
Review protocols with medicallegal department
Communication	Write media material for patients and families
	Develop and disseminate scripts for front‐line staff
Develop communication and meet with all relevant front‐line staff on merits of project
Educate all staff on workflow changes and impacts
Clinical preparation	Conduct internal review of unplanned transfers and present results to all clinicians
	Determine service level agreements, ownership of at‐risk patients, who will access alerts
Conduct staff meetings to educate staff
Perform debriefs on relevant cases
Determine desired outcomes, process measures, balancing measures
Determine acceptable clinician burden (alerts/day)
Technology	Establish documentation templates
	Ensure access to new data fields (electronic medical record security process must be followed for access rights)
Workflows	Workflows (clinical response, patient preferences, supportive care, communication, documentation) must be in place prior to actual go live
Shadowing	Testing period (alerts communicated to selected clinicians prior to going live) should occur

Figure 2

RESPONSE PROTOCOLS

At South San Francisco, the RRT consists of an ICU nurse, a respiratory care therapist, and a designated hospitalist; at Sacramento, the team is also augmented by an additional nurse (the house supervisor). In addition to responding to the AAM alerts, RRT nurses respond to other emergency calls such as code blues, stroke alerts, and patient or patient‐familyinitiated rapid response calls. They also expedite time sensitive workups and treatments. They check up on recent transfers from the ICU to ensure continued improvement justifying staying on the ward. Serving as peer educators, they assist with processes such as chest tube or central line insertions, troubleshoot high‐risk medication administration, and ensure that treatment bundles (eg, for sepsis) occur expeditiously.

The RRT reviews EWS scores every 6 hours. The AAM score is seen as soon as providers open the chart, which helps triage patients for evaluation. Because patients can still be at risk even without an elevated AAM score, all normal escalation pathways remain in place. Once an alert is noted in the inpatient dashboard, the RRT nurse obtains a fresh set of vital signs, assesses the patient's clinical status, and informs the physician, social worker, and primary nurse (Figure 2). Team members work with the bedside nurse, providing support with assessment, interventions, plans, and follow‐up. Once advised of the alert, the hospitalist performs a second chart review and evaluates the patient at the bedside to identify factors that could underlie potential deterioration. After this evaluation, the hospitalist documents concerns, orders appropriate interventions (which can include escalation), and determines appropriate follow‐up. We made sure the team knew that respiratory distress, arrhythmias, mental status changes, or worsening infection were responsible for over 80% of in‐hospital deterioration cases. We also involved palliative care earlier in patient care, streamlining the process so the RRT makes just 1 phone call to the social worker, who contacts the palliative care physician and nurse to ensure patients have a designated surrogate in the event of further deterioration.

Our initial documentation template consisted of a comprehensive organ system‐based physician checklist. However, although this was of use to covering physicians unfamiliar with a given patient, it was redundant and annoying to attending providers already familiar with the patient. After more than 30 iterations, we settled on a succinct note that only documented the clinicians' clinical judgment as to what constituted the major risk for deterioration and what the mitigation strategies would be. Both of these judgments are in a checklist format (see Supporting Information, Appendix, in the online version of this article for the components of the physician and nurse notes).

Prior to the implementation of the system, RRT nurses performed proactive rounding by manually checking patient labs and vital signs, an inefficient process due to the poor sensitivity and specificity of individual values. Following implementation of the system, RRT RNs and clinicians switched to sorting patients by the 3 scores (COPS2, LAPS2, AAM). For example, patients may be stable at admission (as evidenced by their AAM score) but be at high risk due to their comorbidities. One approach that has been employed is to proactively check such patients to ensure they have a care directive in place, as is described in the article by Granich et al.[14] The Supportive Care Team (detailed in Granich et al.) assesses needs for palliative care and provides in‐hospital consultation as needed. Social services staff perform chart reviews to ensure a patient surrogate has been defined and also works with patients and their families to clarify goals of care.

CHALLENGES AND KEY LEARNINGS

One challenge that arose was reconciling the periodic nature of the alert (every 6 hours) with physicians' availability, which varied due to different rounding workflows at the 2 sites. Consequently, the alert cycle was changed; at the first site, the cycle was set to 1000‐1600‐2200‐0400, whereas the second site chose 0800‐1400‐2000‐0200.

One essential but problematic component of the clinical response is the issue of documentation. Inadequate documentation could lead to adverse outcomes, clinician malpractice exposure, and placing the entire hospital at risk for enterprise liability when clinical responses are not documented. This issue is complicated by the fact that overzealous efforts could lead to less or no documentation by making it too onerous for busy clinicians. We found that the ease with which data can populate progress notes in the EMR can lead to note bloat. Clearly, no documentation is not enough, and a complete history and physical is too much. Paradoxically, 1 of the issues underlying our problems with documentation was the proactive nature of the alerts themselves; because they are based on an outcome prediction in the next 12 hours, documenting the response to them may lack (perceived) urgency.

Shortly after the system went live, a patient who had been recently transferred out to the ward from the ICU triggered an alert. As a response was mounted, the team realized that existing ward protocols did not specify which physician service (intensivist or hospitalist) was responsible for patients who were transitioning from 1 unit to another. We also had to perform multiple revisions of the protocols specifying how alerts were handled when they occurred at times of change of shift. Eventually, we settled on having the combination of a hospitalist and an RRT nurse as the cornerstone of the response, with the hospitalist service as the primary owner of the entire process, but this arrangement might need to be varied in different settings. As a result of the experience with the pilot, the business case for deployment in the remaining 19 hospitals includes a formal budget request so that all have properly staffed RRTs, although the issue of primary ownership of the alert process for different patient types (eg, surgical patients) will be decided on a hospital‐by‐hospital basis. These experiences raise the intriguing possibility that implementation of alert systems can lead to the identification of systemic gaps in existing protocols. These gaps can include specific components of the hospital service agreements between multiple departments (emergency, hospital medicine, ICU, palliative care, surgery) as well as problems with existing workflows.

In addition to ongoing tweaking of care protocols, 3 issues remain unresolved. First is the issue of documentation. The current documentation notes are not completely satisfactory, and we are working with the KPNC EMR administrators to refine the tool. Desirable refinements include (1) having the system scores populate in more accessible sectors of the EMR where their retrieval will facilitate increased automation of the note writing process, (2) changing the note type to a note that will facilitate process audits, and (3) linking the note to other EMR tools so that the response arm can be tracked more formally. The second issue is the need to develop strategies to address staff turnover; for example, newer staff may not have received the same degree of exposure to the system as those who were there when it was started. Finally, due to limited resources, we have done very limited work on more mechanistic analyses of the clinical response itself. For example, it would be desirable to perform a formal quantitative, risk‐adjusted process‐outcome analysis of why some patients' outcomes are better than others following an alert.

Finally, it is also the case that we have had some unexpected occurrences that hint at new uses and benefits of alert systems. One of these is the phenomenon of chasing the alert. Some clinicians, on their own, have taken a more proactive stance in the care of patients in whom the AAM score is rising or near the alert threshold. This has 2 potential consequences. Some patients are stabilized and thus do not reach threshold instability levels. In other cases, patients reach threshold but the response team is informed that things are already under control. A second unexpected result is increased requests for COPS2 scores by clinicians who have heard about the system, particularly surgeons who would like to use the comorbidity scores as a screening tool in the outpatient setting. Because KPNC is an integrated system, it is not likely that such alternatives will be implemented immediately without considerable analysis, but it is clear that the system's deployment has captured the clinicians' imagination.

CONCLUSIONS AND FUTURE DIRECTIONS

Our preparatory efforts have been successful. We have found that embedding an EWS in a commercially available EMR is acceptable to hospital physicians and nurses. We have developed a coordinated workflow for mitigation and escalation that is tightly linked to the availability of probabilistic alerts in real time. Although resource limitations have precluded us from conducting formal clinician surveys, the EWS has been discussed at multiple hospital‐wide as well as department‐specific meetings. Although there have been requests for clarification, refinements, and modifications in workflows, no one has suggested that the system be discontinued. Further, many of the other KPNC hospitals have requested that the EWS be deployed at their site. We have examined KPNC databases that track patient complaints and have not found any complaints that could be linked to the EWS. Most importantly, the existence of the workflows we have developed has played a major role in KPNC's decision to deploy the system in its remaining hospitals.

Although alert fatigue is the number 1 reason that clinicians do not utilize embedded clinical decision support,[26] simply calibrating statistical models is insufficient. Careful consideration of clinicians' needs and responsibilities, particularly around ownership of patients and documentation, is essential. Such consideration needs to include planning time and socializing the system (providing multiple venues for clinicians to learn about the system as well as participate in the process for using it).

We anticipate that, as the system leaves the pilot stage and becomes a routine component of hospital care, additional enhancements (eg, sending notifications to smart phones, providing an alert response tracking system) will be added. Our organization is also implementing real‐time concurrent review of inpatient EMRs (eg, for proactive detection of an expanded range of potential process failures), and work is underway on how to link the workflows we describe here with this effort. As has been the case with other systems,[27] it is likely that we will eventually move to continuous scanning of patient data rather than only every 6 hours. Given that the basic workflow is quite robust and amenable to local modifications, we are confident that our clinicians and hospitals will adapt to future system enhancements.

Lastly, we intend to conduct additional research on the clinical response itself. In particular, we consider it extremely important to conduct formal quantitative analyses on why some patients' outcomes are better than others following an alert. A key component of this effort will be to develop tools that can permit an automatedor nearly automatedassessment of the clinical response. For example, we are considering automated approaches that would scan the EMR for the presence of specific orders, notes, vital signs patterns, and laboratory tests following an alert. Whereas it may not be possible to dispense with manual chart review, even partial automation of a feedback process could lead to significant enhancement of our quality improvement efforts.

Patients who deteriorate outside highly monitored settings and who require unplanned transfer to the intensive care unit (ICU) are known to have high mortality and morbidity.[1, 2, 3, 4, 5] The notion that early detection of a deteriorating patient improves outcomes has intuitive appeal and is discussed in a large number of publications.[6, 7, 8, 9, 10] However, much less information is available on what should be done after early detection is made.[11] Existing literature on early warning systems (EWSs) does not provide enough detail to serve as a map for implementation. This lack of transparency is complicated by the fact that, although the comprehensive inpatient electronic medical record (EMR) now constitutes the central locus for clinical practice, much of the existing literature comes from research institutions that may employ home‐grown EMRs, not community hospitals that employ commercially available systems.

In this issue of the Journal of Hospital Medicine, we describe our efforts to bridge that gap by implementing an EWS in a pair of community hospitals. The EWS's development and its basic statistical and electronic infrastructure are described in the articles by Escobar and Dellinger and Escobar et al.[2, 12, 13] In this report, we focus on how we addressed clinicians' primary concern: What do we do when we get an alert? Because it is described in detail by Granich et al.[14] elsewhere in this issue of the Journal of Hospital Medicine, a critical component of our implementation process (ensuring that patient preferences with respect to supportive care are honored) is not discussed.

Our article is divided into the following sections: rationale, preimplementation preparatory work, workflow development, response protocols, challenges and key learnings, and concluding reflections.

RATIONALE

Much of the previous work on the implementation of alarm systems has focused on the statistics behind detection or on the quantification of processes (eg, how many rapid response calls were triggered) or on outcomes such as mortality. The conceptual underpinnings and practical steps necessary for successful integration of an alarm system into the clinicians' workflow have not been articulated. Our theoretical framework was based on (1) improving situational awareness[15] (knowing what is going on around you and what is likely to happen next) and (2) mitigating cognitive errors.

An EWS enhances situational awareness most directly by earlier identification of a problem with a particular patient. As is detailed by Escobar et al.[16] in this issue of the Journal of Hospital Medicine, our EWS extracts EMR data every 6 hours, performs multiple calculations, and then displays 3 scores in real time in the inpatient dashboard (known as the Patient Lists activity in the Epic EMR). The first of these scores is the Laboratory‐Based Acute Physiologic Score, version 2 (LAPS2), an objective severity score whose retrospective version is already in use in Kaiser Permanente Northern California (KPNC) for internal benchmarking.[13] This score captures a patient's overall degree of physiologic instability within the preceding 72 hours. The second is the Comorbidity Point Score, version 2 (COPS2), a longitudinal comorbidity score based on the patient's diagnoses over the preceding 12 months.[13] This score captures a patient's overall comorbidity burden. Thus, it is possible for a patient to be very ill (high COPS2) while also being stable (low LAPS2) or vice versa. Both of these scores have other uses, including prediction of rehospitalization risk in real time,[17] which is also being piloted at KPNC. Finally, the Advanced Alert Monitoring (AAM) score, which integrates the LAPS2 and COPS2 with other variables, provides a 12‐hour deterioration risk, with a threshold value of 8% triggering response protocols. At or above this threshold, which was agreed to prior to implementation, the system achieves 25% sensitivity, 98% specificity, with a number needed to evaluate of 10 to 12, a level of workload that was felt to be acceptable by clinicians. Actions triggered by the EWS may be quite different from those one would take when being notified of a code blue, which is called at the time an event occurs. The EWS focuses attention on patients who might be missed because they do not yet appear critically ill. It also provides a shared, quantifiable measure of a patient's risk that can trigger a standardized plan of action to follow in evaluating and treating a patient.[15]

In addition to enhancing situational awareness, we intended the alarms to produce cognitive change in practitioners. Our goal was to replace medical intuition with analytic, evidence‐based judgment of future illness. We proceeded with the understanding that replacing quick intuition with slower analytic response is an essential skill in developing sound clinical reasoning.[18, 19, 20] The alert encourages physicians to reassess high‐risk patients facilitating a cognitive shift from automatic, error‐prone processing to slower, deliberate processing. Given the busy pace of ward work, slowing down permits clinicians to reassess previously overlooked details. Related to this process of inducing cognitive change is a secondary effect: we uncovered and discussed physician biases. Physicians are subject to potential biases that allow patients to deteriorate.[18, 19, 20] Therefore, we addressed bias through education. By reviewing particular cases of unanticipated deterioration at each hospital facility, we provided evidence for the problem of in‐hospital deterioration. This framed the new tool as an opportunity for improving treatment and encouraged physicians to act on the alert using a structured process.

INTERVENTIONS

Preimplementation Preparatory Work

Initial KPNC data provided strong support for the generally accepted notion that unplanned transfer patients have poor outcomes.[2, 4, 5] However, published reports failed to provide the granular detail clinicians need to implement a response arm at the unit and patient level. In preparation for going live, we conducted a retrospective chart review. This included data from patients hospitalized from January 1, 2011 through December 31, 2012 (additional detail is provided in the Supporting Information, Appendix, in the online version of this article). The key findings from our internal review of subjective documentation preceding deterioration are similar to those described in the literature and summarized in Figure 1, which displays the 5 most common clinical presentations associated with unplanned transfers.

Figure 1

The chart review served several major roles. First, it facilitated cognitive change by eliminating the notion that it can't happen here. Second, it provided considerable guidance on key clinical components that had to be incorporated into the workflow. Third, it engaged the rapid response team (RRT) in reviewing our work retrospectively to identify future opportunities. Finally, the review provided considerable guidance with respect to structuring documentation requirements.

As a result of the above efforts, other processes detailed below, and knowledge described in several of the companion articles in this issue of the Journal of Hospital Medicine, 3 critical elements, which had been explicitly required by our leadership, were in place prior to the go‐live date: a general consensus among hospitalists and nurses that this would be worth testing, a basic clinical response workflow, and an automated checklist for documentation. We refined these in a 2‐week shadowing phase preceding the start date. In this phase, the alerts were not displayed in the EMR. Instead, programmers working on the project notified selected physician leaders by phone. This permitted them to understand exactly what sort of patients were reaching the physiologic threshold so that they could better prepare both RRT registered nurses (RNs) and hospitalists for the go‐live date. This also provided an opportunity to begin refining the documentation process using actual patients.

The original name for our project was Early Detection of Impending Physiologic Deterioration. However, during the preparatory phase, consultation with our public relations staff led to a concern that the name could be frightening to some patients. This highlights the need to consider patient perceptions and how words used in 1 way by physicians can have different connotations to nonclinicians. Consequently, the system was renamed, and it is now referred to as Advance Alert Monitoring (AAM).

Workflow Development

We carefully examined the space where electronic data, graphical user interfaces, and clinical practice blend, a nexus now commonly referred to as workflow or user experience.[21] To promote situational awareness and effect cognitive change, we utilized the Institute for Health Care Improvement's Plan‐Do‐Study‐Act model.[22, 23] We then facilitated the iterative development of a clinician‐endorsed workflow.[22, 23, 24, 25] By adjusting the workflow based on ongoing experience and giving clinicians multiple opportunities to revise (a process that continues to date), we ensured clinicians would approach and endorse the alarm system as a useful tool for decision support.

Table 1 summarizes the work groups assembled for our implementation, and Table 2 provides a system‐oriented checklist indicating key components that need to be in place prior to having an early warning system go live in a hospital. Figure 2 summarizes the alert response protocols we developed through an iterative process at the 2 pilot sites. The care path shown in Figure 2 is the result of considerable revision, mostly due to actual experience acquired following the go live date. The diagram also includes a component that is still work in progress. This is how an emergency department probability estimate (triage support) will be integrated into both the ward as well as the ICU workflows. Although this is beyond the scope of this article, other hospitals may be experimenting with triage support (eg, for sepsis patients), so it is important to consider how one would incorporate such support into workflows.

Table 1.Workgroups Established for Early Warning System Rollout

Workgroup	Goals
NOTE: Abbreviations: POLST, physician orders for life‐sustaining treatment.
Clinical checklist	Perform structured chart review of selected unplanned transfer patients and near misses
	Develop a checklist for mitigation strategies given an alert
Develop documentation standards given an alert
Develop escalation protocol given an alert
Workload and threshold	Determine threshold for sensitivity of alerts and resulting impact on clinician workload
Patient preferences	Prepare background information to be presented to providers regarding end‐of‐life care and POLST orders
	Coordinate with clinical checklist workgroup to generate documentation templates that provide guidance for appropriate management of patients regarding preferences on escalation of care and end‐of‐life care
Electronic medical record coordination	Review proposed electronic medical record changes
	Make recommendation for further changes as needed
Develop plan for rollout of new and/or revised electronic record tools
Designate contact list for questions/emssues that may arise regarding electronic record changes during the pilot
Determine alert display choices and mode of alert notification
Nursing committee	Review staffing needs in anticipation of alert
	Coordinate with workload and threshold group
Develop training calendar to ensure skills necessary for successful implementation of alerts
Make recommendations for potential modification of rapid response team's role in development of a clinical checklist for nurses responding to an alert
Design educational materials for clinicians
Local communication strategy	Develop internal communication plan (for clinical staff not directly involved with pilot)
	Develop external communication plan (for nonclinicians who may hear about the project)

Table 2.Hospital System‐Wide Go Live Checklist

Level	Tasks
Administration	Obtain executive committee approval
	Establish communication protocols with quality assurance and quality improvement committees
Review protocols with medicallegal department
Communication	Write media material for patients and families
	Develop and disseminate scripts for front‐line staff
Develop communication and meet with all relevant front‐line staff on merits of project
Educate all staff on workflow changes and impacts
Clinical preparation	Conduct internal review of unplanned transfers and present results to all clinicians
	Determine service level agreements, ownership of at‐risk patients, who will access alerts
Conduct staff meetings to educate staff
Perform debriefs on relevant cases
Determine desired outcomes, process measures, balancing measures
Determine acceptable clinician burden (alerts/day)
Technology	Establish documentation templates
	Ensure access to new data fields (electronic medical record security process must be followed for access rights)
Workflows	Workflows (clinical response, patient preferences, supportive care, communication, documentation) must be in place prior to actual go live
Shadowing	Testing period (alerts communicated to selected clinicians prior to going live) should occur

Figure 2

RESPONSE PROTOCOLS

At South San Francisco, the RRT consists of an ICU nurse, a respiratory care therapist, and a designated hospitalist; at Sacramento, the team is also augmented by an additional nurse (the house supervisor). In addition to responding to the AAM alerts, RRT nurses respond to other emergency calls such as code blues, stroke alerts, and patient or patient‐familyinitiated rapid response calls. They also expedite time sensitive workups and treatments. They check up on recent transfers from the ICU to ensure continued improvement justifying staying on the ward. Serving as peer educators, they assist with processes such as chest tube or central line insertions, troubleshoot high‐risk medication administration, and ensure that treatment bundles (eg, for sepsis) occur expeditiously.

The RRT reviews EWS scores every 6 hours. The AAM score is seen as soon as providers open the chart, which helps triage patients for evaluation. Because patients can still be at risk even without an elevated AAM score, all normal escalation pathways remain in place. Once an alert is noted in the inpatient dashboard, the RRT nurse obtains a fresh set of vital signs, assesses the patient's clinical status, and informs the physician, social worker, and primary nurse (Figure 2). Team members work with the bedside nurse, providing support with assessment, interventions, plans, and follow‐up. Once advised of the alert, the hospitalist performs a second chart review and evaluates the patient at the bedside to identify factors that could underlie potential deterioration. After this evaluation, the hospitalist documents concerns, orders appropriate interventions (which can include escalation), and determines appropriate follow‐up. We made sure the team knew that respiratory distress, arrhythmias, mental status changes, or worsening infection were responsible for over 80% of in‐hospital deterioration cases. We also involved palliative care earlier in patient care, streamlining the process so the RRT makes just 1 phone call to the social worker, who contacts the palliative care physician and nurse to ensure patients have a designated surrogate in the event of further deterioration.

Our initial documentation template consisted of a comprehensive organ system‐based physician checklist. However, although this was of use to covering physicians unfamiliar with a given patient, it was redundant and annoying to attending providers already familiar with the patient. After more than 30 iterations, we settled on a succinct note that only documented the clinicians' clinical judgment as to what constituted the major risk for deterioration and what the mitigation strategies would be. Both of these judgments are in a checklist format (see Supporting Information, Appendix, in the online version of this article for the components of the physician and nurse notes).

Prior to the implementation of the system, RRT nurses performed proactive rounding by manually checking patient labs and vital signs, an inefficient process due to the poor sensitivity and specificity of individual values. Following implementation of the system, RRT RNs and clinicians switched to sorting patients by the 3 scores (COPS2, LAPS2, AAM). For example, patients may be stable at admission (as evidenced by their AAM score) but be at high risk due to their comorbidities. One approach that has been employed is to proactively check such patients to ensure they have a care directive in place, as is described in the article by Granich et al.[14] The Supportive Care Team (detailed in Granich et al.) assesses needs for palliative care and provides in‐hospital consultation as needed. Social services staff perform chart reviews to ensure a patient surrogate has been defined and also works with patients and their families to clarify goals of care.

CHALLENGES AND KEY LEARNINGS

One challenge that arose was reconciling the periodic nature of the alert (every 6 hours) with physicians' availability, which varied due to different rounding workflows at the 2 sites. Consequently, the alert cycle was changed; at the first site, the cycle was set to 1000‐1600‐2200‐0400, whereas the second site chose 0800‐1400‐2000‐0200.

One essential but problematic component of the clinical response is the issue of documentation. Inadequate documentation could lead to adverse outcomes, clinician malpractice exposure, and placing the entire hospital at risk for enterprise liability when clinical responses are not documented. This issue is complicated by the fact that overzealous efforts could lead to less or no documentation by making it too onerous for busy clinicians. We found that the ease with which data can populate progress notes in the EMR can lead to note bloat. Clearly, no documentation is not enough, and a complete history and physical is too much. Paradoxically, 1 of the issues underlying our problems with documentation was the proactive nature of the alerts themselves; because they are based on an outcome prediction in the next 12 hours, documenting the response to them may lack (perceived) urgency.

Shortly after the system went live, a patient who had been recently transferred out to the ward from the ICU triggered an alert. As a response was mounted, the team realized that existing ward protocols did not specify which physician service (intensivist or hospitalist) was responsible for patients who were transitioning from 1 unit to another. We also had to perform multiple revisions of the protocols specifying how alerts were handled when they occurred at times of change of shift. Eventually, we settled on having the combination of a hospitalist and an RRT nurse as the cornerstone of the response, with the hospitalist service as the primary owner of the entire process, but this arrangement might need to be varied in different settings. As a result of the experience with the pilot, the business case for deployment in the remaining 19 hospitals includes a formal budget request so that all have properly staffed RRTs, although the issue of primary ownership of the alert process for different patient types (eg, surgical patients) will be decided on a hospital‐by‐hospital basis. These experiences raise the intriguing possibility that implementation of alert systems can lead to the identification of systemic gaps in existing protocols. These gaps can include specific components of the hospital service agreements between multiple departments (emergency, hospital medicine, ICU, palliative care, surgery) as well as problems with existing workflows.

In addition to ongoing tweaking of care protocols, 3 issues remain unresolved. First is the issue of documentation. The current documentation notes are not completely satisfactory, and we are working with the KPNC EMR administrators to refine the tool. Desirable refinements include (1) having the system scores populate in more accessible sectors of the EMR where their retrieval will facilitate increased automation of the note writing process, (2) changing the note type to a note that will facilitate process audits, and (3) linking the note to other EMR tools so that the response arm can be tracked more formally. The second issue is the need to develop strategies to address staff turnover; for example, newer staff may not have received the same degree of exposure to the system as those who were there when it was started. Finally, due to limited resources, we have done very limited work on more mechanistic analyses of the clinical response itself. For example, it would be desirable to perform a formal quantitative, risk‐adjusted process‐outcome analysis of why some patients' outcomes are better than others following an alert.

Finally, it is also the case that we have had some unexpected occurrences that hint at new uses and benefits of alert systems. One of these is the phenomenon of chasing the alert. Some clinicians, on their own, have taken a more proactive stance in the care of patients in whom the AAM score is rising or near the alert threshold. This has 2 potential consequences. Some patients are stabilized and thus do not reach threshold instability levels. In other cases, patients reach threshold but the response team is informed that things are already under control. A second unexpected result is increased requests for COPS2 scores by clinicians who have heard about the system, particularly surgeons who would like to use the comorbidity scores as a screening tool in the outpatient setting. Because KPNC is an integrated system, it is not likely that such alternatives will be implemented immediately without considerable analysis, but it is clear that the system's deployment has captured the clinicians' imagination.

CONCLUSIONS AND FUTURE DIRECTIONS

Our preparatory efforts have been successful. We have found that embedding an EWS in a commercially available EMR is acceptable to hospital physicians and nurses. We have developed a coordinated workflow for mitigation and escalation that is tightly linked to the availability of probabilistic alerts in real time. Although resource limitations have precluded us from conducting formal clinician surveys, the EWS has been discussed at multiple hospital‐wide as well as department‐specific meetings. Although there have been requests for clarification, refinements, and modifications in workflows, no one has suggested that the system be discontinued. Further, many of the other KPNC hospitals have requested that the EWS be deployed at their site. We have examined KPNC databases that track patient complaints and have not found any complaints that could be linked to the EWS. Most importantly, the existence of the workflows we have developed has played a major role in KPNC's decision to deploy the system in its remaining hospitals.

Although alert fatigue is the number 1 reason that clinicians do not utilize embedded clinical decision support,[26] simply calibrating statistical models is insufficient. Careful consideration of clinicians' needs and responsibilities, particularly around ownership of patients and documentation, is essential. Such consideration needs to include planning time and socializing the system (providing multiple venues for clinicians to learn about the system as well as participate in the process for using it).

We anticipate that, as the system leaves the pilot stage and becomes a routine component of hospital care, additional enhancements (eg, sending notifications to smart phones, providing an alert response tracking system) will be added. Our organization is also implementing real‐time concurrent review of inpatient EMRs (eg, for proactive detection of an expanded range of potential process failures), and work is underway on how to link the workflows we describe here with this effort. As has been the case with other systems,[27] it is likely that we will eventually move to continuous scanning of patient data rather than only every 6 hours. Given that the basic workflow is quite robust and amenable to local modifications, we are confident that our clinicians and hospitals will adapt to future system enhancements.

Lastly, we intend to conduct additional research on the clinical response itself. In particular, we consider it extremely important to conduct formal quantitative analyses on why some patients' outcomes are better than others following an alert. A key component of this effort will be to develop tools that can permit an automatedor nearly automatedassessment of the clinical response. For example, we are considering automated approaches that would scan the EMR for the presence of specific orders, notes, vital signs patterns, and laboratory tests following an alert. Whereas it may not be possible to dispense with manual chart review, even partial automation of a feedback process could lead to significant enhancement of our quality improvement efforts.

In the landmark Best Care at Lower Cost report, the Institute of Medicine presents a compelling vision of a US healthcare system where science, information technology, incentives, and care culture are brought together seamlessly to produce high‐quality healthcare.[1] At the center of this transformation is the learning healthcare system, a system characterized by its ability to leverage data arising from care provision to drive rapid improvements in care delivery.[2] When steeped within the right organizational milieu, these data help to close the virtuous cycle of continuous learning moving from science to evidence to care and back to new science. The anticipated end result is a healthcare system that can provide Americans with superior care at lower cost.

Hospital‐based practitioners will recognize the inpatient setting as an ideal demonstration opportunity for continuous learning. Hospital care is costly, accounting for more than 30% of all US healthcare costs[3]; intensive care alone accounts for a notable proportion of the US gross domestic product.[4] Inpatient care is associated with significant mortality and morbidity, and its use is often greatly increased in patients' last days.[5, 6] Fortunately, the inpatient setting also offers an ideal opportunity to leverage high‐quality data to help inform and improve care. The digitization of medicine means that far more data are now available through electronic health records, medical devices, and tests.[7] This is particularly true for inpatients, for whom a large volume of data are produced even over relatively short hospital stays.

Whereas the challenge to improve hospital care is daunting, there is an incredible opportunity to advance the quality of inpatient care through realizing the vision of the learning hospital system. In the sections that follow, we use an object lessonsepsis care within hospitals of the Kaiser Permanente Northern California (KPNC) integrated healthcare delivery systemto evaluate the challenges and insights gleaned from working toward building a learning hospital system. Then, we describe further steps that could enhance the use of inpatient data to drive improved care.

THE FRAMEWORK OF A LEARNING HEALTHCARE SYSTEM

Best Care at Lower Cost notes a fundamental paradox in US healthcare: although we have witnessed a dramatic expansion in biomedical knowledge, innovative therapies and surgical procedures, and clinical treatments to extend survival, US healthcare persistently falls short on the basic dimensions of quality, outcomes, cost, and equity.[1] The proposed path forward lies in building the learning healthcare system, a system characterized by continuous knowledge development, improvement, and application. Figure 1 shows the critical nodes in the framework for continuous learning, which include: (1) the development of new scientific knowledge (science), (2) the translation of science into clinical evidence of efficacy (evidence), and (3) the application of efficacious interventions through effective care delivery (care). In healthcare today, transitions between these nodes are rife with missed or wasted opportunities like delays in applying high‐quality evidence or poorly managed insights arising from scientific discovery. If such opportunities could be recovered, however, the quality of healthcare could be improved dramatically.[8]

Figure 1

The pursuit of continuous learning is aided by rapid changes in the quality and quantity of biomedical data available over the past decade, especially through the use of electronic health records, novel biomolecular tools, and digital sensors.[2, 7, 9] The Internet has ushered in a new era of data connectivity, for example, allowing for highly engaged communication between patients and providers as well as collaboration between professional or citizen scientists on data of unprecedented scale.[10] New methodologic approaches, including data mining and machine learning, increasingly leverage commodity hardware to conduct previously computationally intractable analyses.[9] Moreover, the development of domain ontologies fosters the discovery of meaningful insights from data of heterogeneous types.[11]

Ultimately, however, improvements in data alone are inadequate to achieve continuous learning. As shown in Figure 1, whereas data form the channels that allow for transitions from science to evidence to care, novel insights need to be steeped within the right culture, motivated by the right incentives, and supported by the right leaders.[1, 12] Within the sustainable learning healthcare system, knowledge generation feeds practice change with the support and guidance of system leadership; improved practice, in turn, generates new knowledge and completes the virtuous cycle of learning.

THE PROMISE OF CONTINUOUS LEARNING IN HOSPITAL SETTINGS

The hospital is an ideal setting in which to foster continuous learning because advances in inpatient care have the potential to substantially improve healthcare quality and value.[8] Americans were hospitalized roughly 37 million times in 2012; in total, these episodes cost $378 billion.[3] Over 700,000 patients die in US hospitals annually, with reports showing that many patients utilize greatly increased inpatient and critical care services near the end of their lives in a manner that appears misaligned with their preferences.[11, 13] Hospital care is also highly variable in quality and cost; this heterogeneity is not closely associated with improved outcomes.[14, 15] Preventable harm and medical injury occur commonly in hospitals and are now recognized to be a leading cause of inpatient death.[16] Finally, emerging research illuminates the substantial toll that acute care has on patients and families resulting in new comorbidity, functional or neuropsychiatric impairment, rehospitalization, and financial burden that persist long after patients are discharged.[17]

Fortunately, inpatient care also exhibits several qualities that improve the likelihood that continuous learning can be achieved. Although it is clear that hospitalizations occur within the arc of a patient's larger health trajectory, these distinct episodes offer the potential to observe patient trajectories and treatments evolving within relatively compressed time intervals; over that same interval, a large volume of data are produced. Stored within comprehensive electronic health records, these granular data now allow inpatient episodes to be digitally recapitulated with high fidelity, bolstering their use in driving care improvements.[18]

AN OBJECT LESSON IN THE LEARNING FRAMEWORK: SEPSIS CARE

Translating Science to Evidence in Sepsis

Although sepsis has attracted great attention in modern hospital care, sepsis was described long ago by Hippocrates to describe the process by which wounds fester.[19] Recast after the confirmation of germ theory, sepsis came to be known primarily as the blood poisoning resulting from pathogenic organisms.[20] However, with the advent of antibiotics, numerous scientific studies now recognize that sepsis actually results from the dysregulated host immune response to systemic infection, which can also cause organ dysfunction.[21] Based on this knowledge, landmark translational and clinical studies in the 2000s provided strong evidence that early identification of sepsis patients and aggressive infection control and resuscitation were associated with improved mortality (Figure 2, step 1).[22]

Figure 2

Translating Evidence to Care in Sepsis at KPNC

In 2007, the leadership of KPNC initiated a regional effort to improve the quality of care and reduce the variability in performance at its medical centers (Table 1).[23] Reviewing data from nearly 1000 inpatientsthe last 50 consecutive hospital deaths from each of 19 medical centersa mortality diagnostic based on Institute for Healthcare Improvement recommendations[24] revealed that sepsis had a major impact on hospital outcomes. For example, even though sepsis patients were still relatively under‐recognized at the time, accounting for fewer than 3% of hospitalizations, they contributed to one‐quarter of hospital deaths. In light of these compelling data, senior regional leadership identified reducing sepsis mortality as a key performance improvement goal (Figure 2, step 2).

Table 1.Timeline of Elements in the Kaiser Permanente Northern California Sepsis Performance Improvement Effort

Time Period	Event Summary
NOTE: Specific elements related to data infrastructure and analysis are indicated in italics. Abbreviations: ARISE, Australasian Resuscitation in Sepsis Evaluation; CMS SEP‐1, Centers for Medicare and Medicaide Services Severe Sepsis and Septic Shock: Management Bundle; ED, emergency department; EGDT, early goal‐directed therapy; KP, Kaiser Permanente; ProCESS, Protocolized Care for Early Septic Shock; ProMISe, Protocolized Care for Early Septic Shock.
2007	Variability in hospital standardized mortality ratio observed, indicating an opportunity to drive improved outcomes.
	Initiation of staggered implementation of unified electronic medical record across all KP sites (starting in 2006 and ending in 2009).
Spring 2008	Mortality diagnostic chart review completed identifying sepsis and infection‐related causes as key factors in hospital outcomes.
May 2008	Regional Mortality Summit held with a focus on patient safety and mortality reduction efforts through performance improvement. Executive regional and local leadership alignment to focus on sepsis performance improvement.
Summer 2008	Sepsis Steering Committee evaluates best available evidence, develops treatment algorithms, and plans for medical center pilots.
Fall 2008	Pilot intervention deployed at 2 medical centers.
November 2008	First Regional Sepsis Summit: development of sepsis performance improvement playbook, training materials, implementation plans, and measurement strategy.
November 2008	All medical centers begin to form multidisciplinary sepsis teams and performance improvement committees, obtain equipment and supplies including assembly of a sepsis cart. Multidisciplinary teams included ED physician champion, ED nurse champion, improvement advisor, hospitalists, intensivists, quality improvement personnel, nurse educators, and even resident physicians.
January 2009	Performance data collection begins on EGDT processes and outcomes. Initiation of 2 key elements to enhance screening for and detection of sepsis: (1) concomitant ordering of serum lactic acid along with blood cultures, and (2) definition of lactate >2.0 as a critical lab value.
	Use of manual chart review for case finding and central database entry because of ongoing implementation of electronic medical record and limited sepsis‐specific data infrastructure.
March 2009	Regional train the trainer sessions occur and local educational spread efforts begin including: collaborative calls, in‐person training events, and medical center site visits.
August 2009	Grant funding from the Gordon and Betty Moore Foundation begins with a planned 2‐year duration providing funding for improvement advisors with performance improvement expertise and data infrastructure development.
November 2009	Second Regional Sepsis Summit. Identification of intermediate lactate sepsis patients having significant mortality.
January 2010	Initiate measurement of performance for intermediate lactate sepsis patients with a focus on lactate clearance as an outcome measure of interest.
2010	Development of an intranet Web‐based data abstraction tool to identify cases and auto‐populate specific fields for review. Facilities were responsible for review of cases at the local level to foster rapid feedback cycles for local performance improvement. Standardized data query tools were deployed to foster local medical center engagement and system‐level evaluation.
	Accompanying development of a sepsis performance improvement scorecard allowing for comparison of longitudinal performance metrics across all facilities. Scorecard elements included: proportion of lactates drawn following ED blood culture, EGDT‐specific bundle elements (ie, number of EGDT cases, antibiotics within 1 hour, first central venous pressure within 2 hours of EGDT start, target mean arterial pressure achievement), repeat lactate elements, balancing measures for central line placement (ie, pneumothorax, central line infection), and overall sepsis statistics.
April 2011	Third Regional Sepsis Summit. Refinement of EGDT bundle and further development of intermediate lactate bundle approach, including piloting specific treatment bundles targeting this population. Collaborative performance improvement environment in which successful strategies at 1 site were rapidly disseminated to other sites including the Sepsis Alert and the Sepsis Clock.
May 2012	Research analysis of fluid volume and lactate clearance in intermediate lactate sepsis population begins.
February 2013	Fourth Regional Sepsis Summit. Regional spread of intermediate lactate bundle including the use of fluids, antibiotics, and repeat lactate measurements.
May 2013	Research analysis of the contribution of sepsis to hospital deaths (within KP and in national sample) as well as post‐sepsis resource utilization and mortality
March 2014	Publication of ProCESS randomized clinical trial, requiring systemic reevaluation of EGDT‐based sepsis strategy. Subsequent publications of ARISE and ProMISe trials confirming findings from ProCESS. Updated approach under consideration and informally disseminated to practitioners.
October 2014	Updated sepsis treatment guidelines and data capture strategy fully implemented moving away from a catheter‐based strategy for all EGDT‐eligible patients.
October 2015	Sixth Regional Sepsis Summit held to adjust sepsis treatment and data measurement strategy to align more closely with CMS SEP‐1 guidelines.

Based on the principles of performance improvement methodology, clinical and operational leaders established an environment with aligned culture, incentives, and leadership around sepsis care. The effort was launched in late 2008 at a Sepsis Summit, bringing together a multidisciplinary group of stakeholders (eg, hospitalist, emergency department, and intensive care chiefs of staff and nursing managers; medical center and nursing executive and operational leadership) and providing sepsis care pathways based on the best available evidence.[23] Regional investments in the digital infrastructure to support implementation resulted in the provision of granular data within monthly sepsis scorecards quantifying each medical center's performance and trends for a diverse set of sepsis bundle metrics.

The resulting changes in sepsis care were substantial. For example, improved early recognition of infected patients meeting the criteria for sepsis resulted in large changes in the standardized diagnostic criteria used to label patients (Figure 3A). Implementing screening strategies using serum lactate testing for any patient receiving blood cultures resulted in a roughly 10‐fold increase in the use of lactate testing in the emergency department (Figure 3B). Earlier recognition of sepsis also increased the number of patients receiving early antibiotics and receiving central venous catheters for quantitative resuscitation.[23]

Figure 3

CLOSING THE LOOP TOWARD CONTINUOUS LEARNING IN SEPSIS

Leveraging timely and actionable data steeped within an aligned organizational milieu resulted in large‐scale changes across a heterogeneous set of hospitals. However, to realize the true vision of a learning hospital system, a looming question remained: Could the data generated as the byproduct of routine care now be used to complete the virtuous cycle and drive new scientific discovery (Figure 2, step 3)?

PRINCIPLES FOR LEVERAGING DATA IN THE LEARNING HOSPITAL SYSTEM

The object lesson of using data to drive improved sepsis care and further new scientific discovery offers some important insights for continuous learning.

CONCLUSION

Hospitalizations in the United States are costly and associated with high mortality and morbidity; the toll of hospitalization also extends well beyond hospital discharge. The promise of the learning hospital system has marked improvements in the quality of hospital care, especially where healthcare systems can steep critical investments in data and digital infrastructure within the right culture, incentives, and leadership. Where continuous learning is achieved, data generated during routine care offer the potential to yield new scientific discovery and drive further improvements in hospital care.

In the landmark Best Care at Lower Cost report, the Institute of Medicine presents a compelling vision of a US healthcare system where science, information technology, incentives, and care culture are brought together seamlessly to produce high‐quality healthcare.[1] At the center of this transformation is the learning healthcare system, a system characterized by its ability to leverage data arising from care provision to drive rapid improvements in care delivery.[2] When steeped within the right organizational milieu, these data help to close the virtuous cycle of continuous learning moving from science to evidence to care and back to new science. The anticipated end result is a healthcare system that can provide Americans with superior care at lower cost.

Hospital‐based practitioners will recognize the inpatient setting as an ideal demonstration opportunity for continuous learning. Hospital care is costly, accounting for more than 30% of all US healthcare costs[3]; intensive care alone accounts for a notable proportion of the US gross domestic product.[4] Inpatient care is associated with significant mortality and morbidity, and its use is often greatly increased in patients' last days.[5, 6] Fortunately, the inpatient setting also offers an ideal opportunity to leverage high‐quality data to help inform and improve care. The digitization of medicine means that far more data are now available through electronic health records, medical devices, and tests.[7] This is particularly true for inpatients, for whom a large volume of data are produced even over relatively short hospital stays.

Whereas the challenge to improve hospital care is daunting, there is an incredible opportunity to advance the quality of inpatient care through realizing the vision of the learning hospital system. In the sections that follow, we use an object lessonsepsis care within hospitals of the Kaiser Permanente Northern California (KPNC) integrated healthcare delivery systemto evaluate the challenges and insights gleaned from working toward building a learning hospital system. Then, we describe further steps that could enhance the use of inpatient data to drive improved care.

THE FRAMEWORK OF A LEARNING HEALTHCARE SYSTEM

Best Care at Lower Cost notes a fundamental paradox in US healthcare: although we have witnessed a dramatic expansion in biomedical knowledge, innovative therapies and surgical procedures, and clinical treatments to extend survival, US healthcare persistently falls short on the basic dimensions of quality, outcomes, cost, and equity.[1] The proposed path forward lies in building the learning healthcare system, a system characterized by continuous knowledge development, improvement, and application. Figure 1 shows the critical nodes in the framework for continuous learning, which include: (1) the development of new scientific knowledge (science), (2) the translation of science into clinical evidence of efficacy (evidence), and (3) the application of efficacious interventions through effective care delivery (care). In healthcare today, transitions between these nodes are rife with missed or wasted opportunities like delays in applying high‐quality evidence or poorly managed insights arising from scientific discovery. If such opportunities could be recovered, however, the quality of healthcare could be improved dramatically.[8]

Figure 1

The pursuit of continuous learning is aided by rapid changes in the quality and quantity of biomedical data available over the past decade, especially through the use of electronic health records, novel biomolecular tools, and digital sensors.[2, 7, 9] The Internet has ushered in a new era of data connectivity, for example, allowing for highly engaged communication between patients and providers as well as collaboration between professional or citizen scientists on data of unprecedented scale.[10] New methodologic approaches, including data mining and machine learning, increasingly leverage commodity hardware to conduct previously computationally intractable analyses.[9] Moreover, the development of domain ontologies fosters the discovery of meaningful insights from data of heterogeneous types.[11]

Ultimately, however, improvements in data alone are inadequate to achieve continuous learning. As shown in Figure 1, whereas data form the channels that allow for transitions from science to evidence to care, novel insights need to be steeped within the right culture, motivated by the right incentives, and supported by the right leaders.[1, 12] Within the sustainable learning healthcare system, knowledge generation feeds practice change with the support and guidance of system leadership; improved practice, in turn, generates new knowledge and completes the virtuous cycle of learning.

THE PROMISE OF CONTINUOUS LEARNING IN HOSPITAL SETTINGS

The hospital is an ideal setting in which to foster continuous learning because advances in inpatient care have the potential to substantially improve healthcare quality and value.[8] Americans were hospitalized roughly 37 million times in 2012; in total, these episodes cost $378 billion.[3] Over 700,000 patients die in US hospitals annually, with reports showing that many patients utilize greatly increased inpatient and critical care services near the end of their lives in a manner that appears misaligned with their preferences.[11, 13] Hospital care is also highly variable in quality and cost; this heterogeneity is not closely associated with improved outcomes.[14, 15] Preventable harm and medical injury occur commonly in hospitals and are now recognized to be a leading cause of inpatient death.[16] Finally, emerging research illuminates the substantial toll that acute care has on patients and families resulting in new comorbidity, functional or neuropsychiatric impairment, rehospitalization, and financial burden that persist long after patients are discharged.[17]

Fortunately, inpatient care also exhibits several qualities that improve the likelihood that continuous learning can be achieved. Although it is clear that hospitalizations occur within the arc of a patient's larger health trajectory, these distinct episodes offer the potential to observe patient trajectories and treatments evolving within relatively compressed time intervals; over that same interval, a large volume of data are produced. Stored within comprehensive electronic health records, these granular data now allow inpatient episodes to be digitally recapitulated with high fidelity, bolstering their use in driving care improvements.[18]

AN OBJECT LESSON IN THE LEARNING FRAMEWORK: SEPSIS CARE

Translating Science to Evidence in Sepsis

Although sepsis has attracted great attention in modern hospital care, sepsis was described long ago by Hippocrates to describe the process by which wounds fester.[19] Recast after the confirmation of germ theory, sepsis came to be known primarily as the blood poisoning resulting from pathogenic organisms.[20] However, with the advent of antibiotics, numerous scientific studies now recognize that sepsis actually results from the dysregulated host immune response to systemic infection, which can also cause organ dysfunction.[21] Based on this knowledge, landmark translational and clinical studies in the 2000s provided strong evidence that early identification of sepsis patients and aggressive infection control and resuscitation were associated with improved mortality (Figure 2, step 1).[22]

Figure 2

Translating Evidence to Care in Sepsis at KPNC

In 2007, the leadership of KPNC initiated a regional effort to improve the quality of care and reduce the variability in performance at its medical centers (Table 1).[23] Reviewing data from nearly 1000 inpatientsthe last 50 consecutive hospital deaths from each of 19 medical centersa mortality diagnostic based on Institute for Healthcare Improvement recommendations[24] revealed that sepsis had a major impact on hospital outcomes. For example, even though sepsis patients were still relatively under‐recognized at the time, accounting for fewer than 3% of hospitalizations, they contributed to one‐quarter of hospital deaths. In light of these compelling data, senior regional leadership identified reducing sepsis mortality as a key performance improvement goal (Figure 2, step 2).

Table 1.Timeline of Elements in the Kaiser Permanente Northern California Sepsis Performance Improvement Effort

Time Period	Event Summary
NOTE: Specific elements related to data infrastructure and analysis are indicated in italics. Abbreviations: ARISE, Australasian Resuscitation in Sepsis Evaluation; CMS SEP‐1, Centers for Medicare and Medicaide Services Severe Sepsis and Septic Shock: Management Bundle; ED, emergency department; EGDT, early goal‐directed therapy; KP, Kaiser Permanente; ProCESS, Protocolized Care for Early Septic Shock; ProMISe, Protocolized Care for Early Septic Shock.
2007	Variability in hospital standardized mortality ratio observed, indicating an opportunity to drive improved outcomes.
	Initiation of staggered implementation of unified electronic medical record across all KP sites (starting in 2006 and ending in 2009).
Spring 2008	Mortality diagnostic chart review completed identifying sepsis and infection‐related causes as key factors in hospital outcomes.
May 2008	Regional Mortality Summit held with a focus on patient safety and mortality reduction efforts through performance improvement. Executive regional and local leadership alignment to focus on sepsis performance improvement.
Summer 2008	Sepsis Steering Committee evaluates best available evidence, develops treatment algorithms, and plans for medical center pilots.
Fall 2008	Pilot intervention deployed at 2 medical centers.
November 2008	First Regional Sepsis Summit: development of sepsis performance improvement playbook, training materials, implementation plans, and measurement strategy.
November 2008	All medical centers begin to form multidisciplinary sepsis teams and performance improvement committees, obtain equipment and supplies including assembly of a sepsis cart. Multidisciplinary teams included ED physician champion, ED nurse champion, improvement advisor, hospitalists, intensivists, quality improvement personnel, nurse educators, and even resident physicians.
January 2009	Performance data collection begins on EGDT processes and outcomes. Initiation of 2 key elements to enhance screening for and detection of sepsis: (1) concomitant ordering of serum lactic acid along with blood cultures, and (2) definition of lactate >2.0 as a critical lab value.
	Use of manual chart review for case finding and central database entry because of ongoing implementation of electronic medical record and limited sepsis‐specific data infrastructure.
March 2009	Regional train the trainer sessions occur and local educational spread efforts begin including: collaborative calls, in‐person training events, and medical center site visits.
August 2009	Grant funding from the Gordon and Betty Moore Foundation begins with a planned 2‐year duration providing funding for improvement advisors with performance improvement expertise and data infrastructure development.
November 2009	Second Regional Sepsis Summit. Identification of intermediate lactate sepsis patients having significant mortality.
January 2010	Initiate measurement of performance for intermediate lactate sepsis patients with a focus on lactate clearance as an outcome measure of interest.
2010	Development of an intranet Web‐based data abstraction tool to identify cases and auto‐populate specific fields for review. Facilities were responsible for review of cases at the local level to foster rapid feedback cycles for local performance improvement. Standardized data query tools were deployed to foster local medical center engagement and system‐level evaluation.
	Accompanying development of a sepsis performance improvement scorecard allowing for comparison of longitudinal performance metrics across all facilities. Scorecard elements included: proportion of lactates drawn following ED blood culture, EGDT‐specific bundle elements (ie, number of EGDT cases, antibiotics within 1 hour, first central venous pressure within 2 hours of EGDT start, target mean arterial pressure achievement), repeat lactate elements, balancing measures for central line placement (ie, pneumothorax, central line infection), and overall sepsis statistics.
April 2011	Third Regional Sepsis Summit. Refinement of EGDT bundle and further development of intermediate lactate bundle approach, including piloting specific treatment bundles targeting this population. Collaborative performance improvement environment in which successful strategies at 1 site were rapidly disseminated to other sites including the Sepsis Alert and the Sepsis Clock.
May 2012	Research analysis of fluid volume and lactate clearance in intermediate lactate sepsis population begins.
February 2013	Fourth Regional Sepsis Summit. Regional spread of intermediate lactate bundle including the use of fluids, antibiotics, and repeat lactate measurements.
May 2013	Research analysis of the contribution of sepsis to hospital deaths (within KP and in national sample) as well as post‐sepsis resource utilization and mortality
March 2014	Publication of ProCESS randomized clinical trial, requiring systemic reevaluation of EGDT‐based sepsis strategy. Subsequent publications of ARISE and ProMISe trials confirming findings from ProCESS. Updated approach under consideration and informally disseminated to practitioners.
October 2014	Updated sepsis treatment guidelines and data capture strategy fully implemented moving away from a catheter‐based strategy for all EGDT‐eligible patients.
October 2015	Sixth Regional Sepsis Summit held to adjust sepsis treatment and data measurement strategy to align more closely with CMS SEP‐1 guidelines.

Based on the principles of performance improvement methodology, clinical and operational leaders established an environment with aligned culture, incentives, and leadership around sepsis care. The effort was launched in late 2008 at a Sepsis Summit, bringing together a multidisciplinary group of stakeholders (eg, hospitalist, emergency department, and intensive care chiefs of staff and nursing managers; medical center and nursing executive and operational leadership) and providing sepsis care pathways based on the best available evidence.[23] Regional investments in the digital infrastructure to support implementation resulted in the provision of granular data within monthly sepsis scorecards quantifying each medical center's performance and trends for a diverse set of sepsis bundle metrics.

The resulting changes in sepsis care were substantial. For example, improved early recognition of infected patients meeting the criteria for sepsis resulted in large changes in the standardized diagnostic criteria used to label patients (Figure 3A). Implementing screening strategies using serum lactate testing for any patient receiving blood cultures resulted in a roughly 10‐fold increase in the use of lactate testing in the emergency department (Figure 3B). Earlier recognition of sepsis also increased the number of patients receiving early antibiotics and receiving central venous catheters for quantitative resuscitation.[23]

Figure 3

CLOSING THE LOOP TOWARD CONTINUOUS LEARNING IN SEPSIS

Leveraging timely and actionable data steeped within an aligned organizational milieu resulted in large‐scale changes across a heterogeneous set of hospitals. However, to realize the true vision of a learning hospital system, a looming question remained: Could the data generated as the byproduct of routine care now be used to complete the virtuous cycle and drive new scientific discovery (Figure 2, step 3)?

PRINCIPLES FOR LEVERAGING DATA IN THE LEARNING HOSPITAL SYSTEM

The object lesson of using data to drive improved sepsis care and further new scientific discovery offers some important insights for continuous learning.

CONCLUSION

Hospitalizations in the United States are costly and associated with high mortality and morbidity; the toll of hospitalization also extends well beyond hospital discharge. The promise of the learning hospital system has marked improvements in the quality of hospital care, especially where healthcare systems can steep critical investments in data and digital infrastructure within the right culture, incentives, and leadership. Where continuous learning is achieved, data generated during routine care offer the potential to yield new scientific discovery and drive further improvements in hospital care.

In the landmark Best Care at Lower Cost report, the Institute of Medicine presents a compelling vision of a US healthcare system where science, information technology, incentives, and care culture are brought together seamlessly to produce high‐quality healthcare.[1] At the center of this transformation is the learning healthcare system, a system characterized by its ability to leverage data arising from care provision to drive rapid improvements in care delivery.[2] When steeped within the right organizational milieu, these data help to close the virtuous cycle of continuous learning moving from science to evidence to care and back to new science. The anticipated end result is a healthcare system that can provide Americans with superior care at lower cost.

Hospital‐based practitioners will recognize the inpatient setting as an ideal demonstration opportunity for continuous learning. Hospital care is costly, accounting for more than 30% of all US healthcare costs[3]; intensive care alone accounts for a notable proportion of the US gross domestic product.[4] Inpatient care is associated with significant mortality and morbidity, and its use is often greatly increased in patients' last days.[5, 6] Fortunately, the inpatient setting also offers an ideal opportunity to leverage high‐quality data to help inform and improve care. The digitization of medicine means that far more data are now available through electronic health records, medical devices, and tests.[7] This is particularly true for inpatients, for whom a large volume of data are produced even over relatively short hospital stays.

Whereas the challenge to improve hospital care is daunting, there is an incredible opportunity to advance the quality of inpatient care through realizing the vision of the learning hospital system. In the sections that follow, we use an object lessonsepsis care within hospitals of the Kaiser Permanente Northern California (KPNC) integrated healthcare delivery systemto evaluate the challenges and insights gleaned from working toward building a learning hospital system. Then, we describe further steps that could enhance the use of inpatient data to drive improved care.

THE FRAMEWORK OF A LEARNING HEALTHCARE SYSTEM

Best Care at Lower Cost notes a fundamental paradox in US healthcare: although we have witnessed a dramatic expansion in biomedical knowledge, innovative therapies and surgical procedures, and clinical treatments to extend survival, US healthcare persistently falls short on the basic dimensions of quality, outcomes, cost, and equity.[1] The proposed path forward lies in building the learning healthcare system, a system characterized by continuous knowledge development, improvement, and application. Figure 1 shows the critical nodes in the framework for continuous learning, which include: (1) the development of new scientific knowledge (science), (2) the translation of science into clinical evidence of efficacy (evidence), and (3) the application of efficacious interventions through effective care delivery (care). In healthcare today, transitions between these nodes are rife with missed or wasted opportunities like delays in applying high‐quality evidence or poorly managed insights arising from scientific discovery. If such opportunities could be recovered, however, the quality of healthcare could be improved dramatically.[8]

Figure 1

The pursuit of continuous learning is aided by rapid changes in the quality and quantity of biomedical data available over the past decade, especially through the use of electronic health records, novel biomolecular tools, and digital sensors.[2, 7, 9] The Internet has ushered in a new era of data connectivity, for example, allowing for highly engaged communication between patients and providers as well as collaboration between professional or citizen scientists on data of unprecedented scale.[10] New methodologic approaches, including data mining and machine learning, increasingly leverage commodity hardware to conduct previously computationally intractable analyses.[9] Moreover, the development of domain ontologies fosters the discovery of meaningful insights from data of heterogeneous types.[11]

Ultimately, however, improvements in data alone are inadequate to achieve continuous learning. As shown in Figure 1, whereas data form the channels that allow for transitions from science to evidence to care, novel insights need to be steeped within the right culture, motivated by the right incentives, and supported by the right leaders.[1, 12] Within the sustainable learning healthcare system, knowledge generation feeds practice change with the support and guidance of system leadership; improved practice, in turn, generates new knowledge and completes the virtuous cycle of learning.

THE PROMISE OF CONTINUOUS LEARNING IN HOSPITAL SETTINGS

The hospital is an ideal setting in which to foster continuous learning because advances in inpatient care have the potential to substantially improve healthcare quality and value.[8] Americans were hospitalized roughly 37 million times in 2012; in total, these episodes cost $378 billion.[3] Over 700,000 patients die in US hospitals annually, with reports showing that many patients utilize greatly increased inpatient and critical care services near the end of their lives in a manner that appears misaligned with their preferences.[11, 13] Hospital care is also highly variable in quality and cost; this heterogeneity is not closely associated with improved outcomes.[14, 15] Preventable harm and medical injury occur commonly in hospitals and are now recognized to be a leading cause of inpatient death.[16] Finally, emerging research illuminates the substantial toll that acute care has on patients and families resulting in new comorbidity, functional or neuropsychiatric impairment, rehospitalization, and financial burden that persist long after patients are discharged.[17]

Fortunately, inpatient care also exhibits several qualities that improve the likelihood that continuous learning can be achieved. Although it is clear that hospitalizations occur within the arc of a patient's larger health trajectory, these distinct episodes offer the potential to observe patient trajectories and treatments evolving within relatively compressed time intervals; over that same interval, a large volume of data are produced. Stored within comprehensive electronic health records, these granular data now allow inpatient episodes to be digitally recapitulated with high fidelity, bolstering their use in driving care improvements.[18]

AN OBJECT LESSON IN THE LEARNING FRAMEWORK: SEPSIS CARE

Translating Science to Evidence in Sepsis

Although sepsis has attracted great attention in modern hospital care, sepsis was described long ago by Hippocrates to describe the process by which wounds fester.[19] Recast after the confirmation of germ theory, sepsis came to be known primarily as the blood poisoning resulting from pathogenic organisms.[20] However, with the advent of antibiotics, numerous scientific studies now recognize that sepsis actually results from the dysregulated host immune response to systemic infection, which can also cause organ dysfunction.[21] Based on this knowledge, landmark translational and clinical studies in the 2000s provided strong evidence that early identification of sepsis patients and aggressive infection control and resuscitation were associated with improved mortality (Figure 2, step 1).[22]

Figure 2

Translating Evidence to Care in Sepsis at KPNC

In 2007, the leadership of KPNC initiated a regional effort to improve the quality of care and reduce the variability in performance at its medical centers (Table 1).[23] Reviewing data from nearly 1000 inpatientsthe last 50 consecutive hospital deaths from each of 19 medical centersa mortality diagnostic based on Institute for Healthcare Improvement recommendations[24] revealed that sepsis had a major impact on hospital outcomes. For example, even though sepsis patients were still relatively under‐recognized at the time, accounting for fewer than 3% of hospitalizations, they contributed to one‐quarter of hospital deaths. In light of these compelling data, senior regional leadership identified reducing sepsis mortality as a key performance improvement goal (Figure 2, step 2).

Table 1.Timeline of Elements in the Kaiser Permanente Northern California Sepsis Performance Improvement Effort

Time Period	Event Summary
NOTE: Specific elements related to data infrastructure and analysis are indicated in italics. Abbreviations: ARISE, Australasian Resuscitation in Sepsis Evaluation; CMS SEP‐1, Centers for Medicare and Medicaide Services Severe Sepsis and Septic Shock: Management Bundle; ED, emergency department; EGDT, early goal‐directed therapy; KP, Kaiser Permanente; ProCESS, Protocolized Care for Early Septic Shock; ProMISe, Protocolized Care for Early Septic Shock.
2007	Variability in hospital standardized mortality ratio observed, indicating an opportunity to drive improved outcomes.
	Initiation of staggered implementation of unified electronic medical record across all KP sites (starting in 2006 and ending in 2009).
Spring 2008	Mortality diagnostic chart review completed identifying sepsis and infection‐related causes as key factors in hospital outcomes.
May 2008	Regional Mortality Summit held with a focus on patient safety and mortality reduction efforts through performance improvement. Executive regional and local leadership alignment to focus on sepsis performance improvement.
Summer 2008	Sepsis Steering Committee evaluates best available evidence, develops treatment algorithms, and plans for medical center pilots.
Fall 2008	Pilot intervention deployed at 2 medical centers.
November 2008	First Regional Sepsis Summit: development of sepsis performance improvement playbook, training materials, implementation plans, and measurement strategy.
November 2008	All medical centers begin to form multidisciplinary sepsis teams and performance improvement committees, obtain equipment and supplies including assembly of a sepsis cart. Multidisciplinary teams included ED physician champion, ED nurse champion, improvement advisor, hospitalists, intensivists, quality improvement personnel, nurse educators, and even resident physicians.
January 2009	Performance data collection begins on EGDT processes and outcomes. Initiation of 2 key elements to enhance screening for and detection of sepsis: (1) concomitant ordering of serum lactic acid along with blood cultures, and (2) definition of lactate >2.0 as a critical lab value.
	Use of manual chart review for case finding and central database entry because of ongoing implementation of electronic medical record and limited sepsis‐specific data infrastructure.
March 2009	Regional train the trainer sessions occur and local educational spread efforts begin including: collaborative calls, in‐person training events, and medical center site visits.
August 2009	Grant funding from the Gordon and Betty Moore Foundation begins with a planned 2‐year duration providing funding for improvement advisors with performance improvement expertise and data infrastructure development.
November 2009	Second Regional Sepsis Summit. Identification of intermediate lactate sepsis patients having significant mortality.
January 2010	Initiate measurement of performance for intermediate lactate sepsis patients with a focus on lactate clearance as an outcome measure of interest.
2010	Development of an intranet Web‐based data abstraction tool to identify cases and auto‐populate specific fields for review. Facilities were responsible for review of cases at the local level to foster rapid feedback cycles for local performance improvement. Standardized data query tools were deployed to foster local medical center engagement and system‐level evaluation.
	Accompanying development of a sepsis performance improvement scorecard allowing for comparison of longitudinal performance metrics across all facilities. Scorecard elements included: proportion of lactates drawn following ED blood culture, EGDT‐specific bundle elements (ie, number of EGDT cases, antibiotics within 1 hour, first central venous pressure within 2 hours of EGDT start, target mean arterial pressure achievement), repeat lactate elements, balancing measures for central line placement (ie, pneumothorax, central line infection), and overall sepsis statistics.
April 2011	Third Regional Sepsis Summit. Refinement of EGDT bundle and further development of intermediate lactate bundle approach, including piloting specific treatment bundles targeting this population. Collaborative performance improvement environment in which successful strategies at 1 site were rapidly disseminated to other sites including the Sepsis Alert and the Sepsis Clock.
May 2012	Research analysis of fluid volume and lactate clearance in intermediate lactate sepsis population begins.
February 2013	Fourth Regional Sepsis Summit. Regional spread of intermediate lactate bundle including the use of fluids, antibiotics, and repeat lactate measurements.
May 2013	Research analysis of the contribution of sepsis to hospital deaths (within KP and in national sample) as well as post‐sepsis resource utilization and mortality
March 2014	Publication of ProCESS randomized clinical trial, requiring systemic reevaluation of EGDT‐based sepsis strategy. Subsequent publications of ARISE and ProMISe trials confirming findings from ProCESS. Updated approach under consideration and informally disseminated to practitioners.
October 2014	Updated sepsis treatment guidelines and data capture strategy fully implemented moving away from a catheter‐based strategy for all EGDT‐eligible patients.
October 2015	Sixth Regional Sepsis Summit held to adjust sepsis treatment and data measurement strategy to align more closely with CMS SEP‐1 guidelines.

Based on the principles of performance improvement methodology, clinical and operational leaders established an environment with aligned culture, incentives, and leadership around sepsis care. The effort was launched in late 2008 at a Sepsis Summit, bringing together a multidisciplinary group of stakeholders (eg, hospitalist, emergency department, and intensive care chiefs of staff and nursing managers; medical center and nursing executive and operational leadership) and providing sepsis care pathways based on the best available evidence.[23] Regional investments in the digital infrastructure to support implementation resulted in the provision of granular data within monthly sepsis scorecards quantifying each medical center's performance and trends for a diverse set of sepsis bundle metrics.

The resulting changes in sepsis care were substantial. For example, improved early recognition of infected patients meeting the criteria for sepsis resulted in large changes in the standardized diagnostic criteria used to label patients (Figure 3A). Implementing screening strategies using serum lactate testing for any patient receiving blood cultures resulted in a roughly 10‐fold increase in the use of lactate testing in the emergency department (Figure 3B). Earlier recognition of sepsis also increased the number of patients receiving early antibiotics and receiving central venous catheters for quantitative resuscitation.[23]

Figure 3

CLOSING THE LOOP TOWARD CONTINUOUS LEARNING IN SEPSIS

Leveraging timely and actionable data steeped within an aligned organizational milieu resulted in large‐scale changes across a heterogeneous set of hospitals. However, to realize the true vision of a learning hospital system, a looming question remained: Could the data generated as the byproduct of routine care now be used to complete the virtuous cycle and drive new scientific discovery (Figure 2, step 3)?

PRINCIPLES FOR LEVERAGING DATA IN THE LEARNING HOSPITAL SYSTEM

The object lesson of using data to drive improved sepsis care and further new scientific discovery offers some important insights for continuous learning.

CONCLUSION

Hospitalizations in the United States are costly and associated with high mortality and morbidity; the toll of hospitalization also extends well beyond hospital discharge. The promise of the learning hospital system has marked improvements in the quality of hospital care, especially where healthcare systems can steep critical investments in data and digital infrastructure within the right culture, incentives, and leadership. Where continuous learning is achieved, data generated during routine care offer the potential to yield new scientific discovery and drive further improvements in hospital care.