Brief Reports

There is a growing appreciation of gender disparities in career advancement in medicine. By 2004, approximately 50% of medical school graduates were women, yet considerable differences persist between genders in compensation, faculty rank, and leadership positions.^1-3 According to the Association of American Medical Colleges (AAMC), women account for only 25% of full professors, 18% of department chairs, and 18% of medical school deans.¹ Women are also underrepresented in other areas of leadership such as division directors, professional society leadership, and hospital executives.^4-6

Specialties that are predominantly women, including pediatrics, are not immune to gender disparities. Women represent 71% of pediatric residents¹ and currently constitute two-thirds of active pediatricians in the United States.⁷ However, there is a disproportionately low number of women ascending the pediatric academic ladder, with only 35% of full professors² and 28% of department chairs being women.¹ Pediatrics also was noted to have the fifth-largest gender pay gap across 40 specialties.³ These disparities can contribute to burnout, poorer patient outcomes, and decreased advancement of women known as the “leaky pipeline.”^1,8,9

There is some evidence that gender disparities may be improving among younger professionals with increasing percentages of women as leaders and decreasing pay gaps.^10,11 These potential positive trends provide hope that fields in medicine early in their development may demonstrate fewer gender disparities. One of the youngest fields of medicine is pediatric hospital medicine (PHM), which officially became a recognized pediatric subspecialty in 2017.¹² There is no literature to date describing gender disparities in PHM. We aimed to explore the gender distribution of university-based PHM program leadership and to compare this gender distribution with that seen in the broader field of PHM.

This study was Institutional Review Board–approved as non–human subjects research through University of Chicago, Chicago, Illinois. From January to March 2020, the authors performed web-based searches for PHM division directors or program leaders in the United States. Because there is no single database of PHM programs in the United States, we used the AAMC list of Liaison Committee on Medical Education (LCME)–accredited US medical schools; medical schools in Puerto Rico were not included, nor were pending and provisional institutions. If an institution had multiple practice sites for its students, the primary site for third-year medical student clerkship rotations was included. If a medical school had multiple branches, each with its own primary inpatient pediatrics site, these sites were included. If there was no PHM division director, a program leader (lead hospitalist) was substituted and counted as long as the role was formally designated. This leadership role is herein referred to under the umbrella term of “division director.”

We searched medical school web pages, affiliated hospital web pages, and Google. All program leadership information (divisional and fellowship, if present) was confirmed through direct communication with the program, most commonly with division directors, and included name, gender, title, and presence of associate/assistant leader, gender, and title. Associate division directors were only included if it was a formal leadership position. Associate directors of research, quality, etc, were not included due to the limited number of formal positions noted on further review. Of note, the terms “associate” and “assistant” are referring to leadership positions and not academic ranks.

Fellowship leadership was included if affiliated with a US medical school in the primary list. Medical schools with multiple PHM fellowships were included as separate observations. The leadership was confirmed using the methods described above and cross-referenced through the PHM Fellowship Program website. PHM fellowship programs starting in 2020 were included if leadership was determined.

All leadership positions were verified by two authors, and all authors reviewed the master list to identify errors.

To determine the overall gender breakdown in the specialty, we used three estimates: 2019 American Board of Pediatrics (ABP) PHM Board Certification Exam applicants, the 2019 American Academy of Pediatrics Section on Hospital Medicine membership, and a random sample of all PHM faculty in 25% of the programs included in this study.⁴

Descriptive statistics using 95% confidence intervals for proportions were used. Differences between proportions were evaluated using a two-proportion z test with the null hypothesis that the two proportions are the same and significance set at P < .05. 

Of the 150 AAMC LCME–accredited medical school departments of pediatrics evaluated, a total of 142 programs were included; eight programs were excluded due to not providing inpatient pediatric services.

Division Leadership

The proportion of women PHM division directors was 55% (95% CI, 47%-63%) in this sample of 146 leaders from 142 programs (4 programs had coleaders). In the 113 programs with standalone PHM divisions or sections, the proportion of women division directors was 56% (95% CI, 47%-64%). In the 29 hospitalist groups that were not standalone (ie, embedded in another division), the proportion of women leaders was similar at 52% (95% CI, 34%-69%). In 24 programs with 27 formally designated associate directors (1 program had 3 associate directors and 1 program had 2), 81% of associate directors were women (95% CI, 63%-92%).

Fellowship Leadership

A total of 51 PHM fellowship programs had 53 directors (2 had codirectors), and 66% of the fellowship directors were women (95% CI, 53%-77%). A total of 31 programs had 34 assistant directors (3 programs had 2 assistants), and 82% of the assistant fellowship directors were women (95% CI, 66%-92%).

Comparison With the Field at Large

The inaugural ABP PHM board certification exam in 2019 had 1,627 applicants with 70% women (95% CI, 68%-73%) (Suzanne Woods, MD, email communication, December 4, 2019). The American Academy of Pediatrics Section on Hospital Medicine, the largest PHM-specific organization, has 2,299 practicing physician members with 71% women (95% CI, 69%-73%) (Niccole Alexander, email communication, November 25, 2019). Our random sample of 25% of university-based PHM programs contained 1,063 faculty members with 72% women (95% CI, 69%-75%).

The Table provides P values for comparisons of the proportion of women in each of the above-described leadership roles compared to the most conservative estimate of women in the field from the estimates given above (ie, 70%). Compared with the field at large, women appear to be underrepresented as division directors (70% vs 55%; P < .001) but not as fellowship directors (70% vs 66%; P = .5). There is a higher proportion of women in all associate/assistant director roles, compared with the population (82% vs 70%; P = .04).

We found a significant difference between the proportion of women as PHM division directors (55%) when compared with the proportion of women physicians in PHM (70%), which suggests that women are underrepresented in clinical leadership at university-based pediatric hospitalist programs. Similar findings are described in other specialties, including notably adult hospital medicine.⁴ Burden et al found that only 16% of hospital medicine program leaders were women despite an equal number of women and men in the field. PHM has a much larger proportion of women, compared with that of hospital medicine, and yet women are still underrepresented as program leaders.

We found no disparities between the proportion of women as PHM fellowship directors and the field at large. These results are similar to those of other studies, which showed a higher number of women in educational leadership roles and lower representation in roles with influence over policy and allocation of resources.^13,14 Although the proportion of women in educational roles itself is not a concern, there is evidence that these positions may be undervalued by some institutions, which provide these positions with lower salaries and fewer opportunities for career advancement.^13,14

Interestingly, women are well-represented in associate/assistant director roles at both the division and fellowship leader level when comparing the distribution in those roles with that of the PHM field at large. This finding suggests that the pipeline of women is robust and potentially may indicate positive change. Alternatively, this finding may reflect a previously described phenomenon of the “sticky floor” in which women are “stuck” in these supportive roles and do not necessarily advance to higher-impact positions.¹⁵ We found a statistically significant higher proportion of women in the combined group of all associate/assistant directors compared with the overall population, which raises the concern that supportive leadership roles may represent “women’s work.”¹⁶ Future studies are needed to track whether these women truly advance or whether women are overrepresented in supportive leadership positions at the expense of primary leadership positions.

Adequate representation of women alone is not sufficient to achieve gender equity in medicine. We need to understand why there is a lower representation of women in leadership positions. Some barriers have already been described, including gender bias in promotions,¹⁷ higher demands outside of work,¹⁸ and lower pay,³ though none are specific to PHM. A further qualitative exploration of PHM leadership would help describe any barriers women in PHM specifically may be facing in their career trajectory. In addition, more information is needed to explore the experience of women with intersectional identities in PHM, especially since they may experience increased bias and discrimination.¹⁹

Limitations of this study include the lack of a centralized list of PHM programs and data on PHM workforce. Our three estimates for the proportion of women in PHM were similar at 70%-71%; however, these are only proxies for the true gender distribution of PHM physicians, which is unknown. PHM leadership targets of close to 70% women would be reflective of the field at large; however, institutional variation may exist, and ideally leadership should be diverse and reflective of its faculty members. Our study only describes university-based PHM programs and, therefore, is not necessarily generalizable to nonuniversity programs. Further studies are needed to evaluate any potential differences based on program type. In our study, gender was used in binary terms; however, we acknowledge that gender exists on a spectrum.

As a specialty early in development with a robust pipeline of women, PHM is in a unique position to lead the way in gender equity. However, women appear to be underrepresented as division directors at university-based PHM programs. Achieving proportional representation of women leaders is imperative for tapping into the full potential of the community and ensuring that the goals of the field are representative of the population.

Special thanks to Lucille Lester, MD, who asked the question that started this road to discovery.

There is a growing appreciation of gender disparities in career advancement in medicine. By 2004, approximately 50% of medical school graduates were women, yet considerable differences persist between genders in compensation, faculty rank, and leadership positions.^1-3 According to the Association of American Medical Colleges (AAMC), women account for only 25% of full professors, 18% of department chairs, and 18% of medical school deans.¹ Women are also underrepresented in other areas of leadership such as division directors, professional society leadership, and hospital executives.^4-6

Specialties that are predominantly women, including pediatrics, are not immune to gender disparities. Women represent 71% of pediatric residents¹ and currently constitute two-thirds of active pediatricians in the United States.⁷ However, there is a disproportionately low number of women ascending the pediatric academic ladder, with only 35% of full professors² and 28% of department chairs being women.¹ Pediatrics also was noted to have the fifth-largest gender pay gap across 40 specialties.³ These disparities can contribute to burnout, poorer patient outcomes, and decreased advancement of women known as the “leaky pipeline.”^1,8,9

There is some evidence that gender disparities may be improving among younger professionals with increasing percentages of women as leaders and decreasing pay gaps.^10,11 These potential positive trends provide hope that fields in medicine early in their development may demonstrate fewer gender disparities. One of the youngest fields of medicine is pediatric hospital medicine (PHM), which officially became a recognized pediatric subspecialty in 2017.¹² There is no literature to date describing gender disparities in PHM. We aimed to explore the gender distribution of university-based PHM program leadership and to compare this gender distribution with that seen in the broader field of PHM.

This study was Institutional Review Board–approved as non–human subjects research through University of Chicago, Chicago, Illinois. From January to March 2020, the authors performed web-based searches for PHM division directors or program leaders in the United States. Because there is no single database of PHM programs in the United States, we used the AAMC list of Liaison Committee on Medical Education (LCME)–accredited US medical schools; medical schools in Puerto Rico were not included, nor were pending and provisional institutions. If an institution had multiple practice sites for its students, the primary site for third-year medical student clerkship rotations was included. If a medical school had multiple branches, each with its own primary inpatient pediatrics site, these sites were included. If there was no PHM division director, a program leader (lead hospitalist) was substituted and counted as long as the role was formally designated. This leadership role is herein referred to under the umbrella term of “division director.”

We searched medical school web pages, affiliated hospital web pages, and Google. All program leadership information (divisional and fellowship, if present) was confirmed through direct communication with the program, most commonly with division directors, and included name, gender, title, and presence of associate/assistant leader, gender, and title. Associate division directors were only included if it was a formal leadership position. Associate directors of research, quality, etc, were not included due to the limited number of formal positions noted on further review. Of note, the terms “associate” and “assistant” are referring to leadership positions and not academic ranks.

Fellowship leadership was included if affiliated with a US medical school in the primary list. Medical schools with multiple PHM fellowships were included as separate observations. The leadership was confirmed using the methods described above and cross-referenced through the PHM Fellowship Program website. PHM fellowship programs starting in 2020 were included if leadership was determined.

All leadership positions were verified by two authors, and all authors reviewed the master list to identify errors.

To determine the overall gender breakdown in the specialty, we used three estimates: 2019 American Board of Pediatrics (ABP) PHM Board Certification Exam applicants, the 2019 American Academy of Pediatrics Section on Hospital Medicine membership, and a random sample of all PHM faculty in 25% of the programs included in this study.⁴

Descriptive statistics using 95% confidence intervals for proportions were used. Differences between proportions were evaluated using a two-proportion z test with the null hypothesis that the two proportions are the same and significance set at P < .05. 

Of the 150 AAMC LCME–accredited medical school departments of pediatrics evaluated, a total of 142 programs were included; eight programs were excluded due to not providing inpatient pediatric services.

Division Leadership

The proportion of women PHM division directors was 55% (95% CI, 47%-63%) in this sample of 146 leaders from 142 programs (4 programs had coleaders). In the 113 programs with standalone PHM divisions or sections, the proportion of women division directors was 56% (95% CI, 47%-64%). In the 29 hospitalist groups that were not standalone (ie, embedded in another division), the proportion of women leaders was similar at 52% (95% CI, 34%-69%). In 24 programs with 27 formally designated associate directors (1 program had 3 associate directors and 1 program had 2), 81% of associate directors were women (95% CI, 63%-92%).

Fellowship Leadership

A total of 51 PHM fellowship programs had 53 directors (2 had codirectors), and 66% of the fellowship directors were women (95% CI, 53%-77%). A total of 31 programs had 34 assistant directors (3 programs had 2 assistants), and 82% of the assistant fellowship directors were women (95% CI, 66%-92%).

Comparison With the Field at Large

The inaugural ABP PHM board certification exam in 2019 had 1,627 applicants with 70% women (95% CI, 68%-73%) (Suzanne Woods, MD, email communication, December 4, 2019). The American Academy of Pediatrics Section on Hospital Medicine, the largest PHM-specific organization, has 2,299 practicing physician members with 71% women (95% CI, 69%-73%) (Niccole Alexander, email communication, November 25, 2019). Our random sample of 25% of university-based PHM programs contained 1,063 faculty members with 72% women (95% CI, 69%-75%).

The Table provides P values for comparisons of the proportion of women in each of the above-described leadership roles compared to the most conservative estimate of women in the field from the estimates given above (ie, 70%). Compared with the field at large, women appear to be underrepresented as division directors (70% vs 55%; P < .001) but not as fellowship directors (70% vs 66%; P = .5). There is a higher proportion of women in all associate/assistant director roles, compared with the population (82% vs 70%; P = .04).

We found a significant difference between the proportion of women as PHM division directors (55%) when compared with the proportion of women physicians in PHM (70%), which suggests that women are underrepresented in clinical leadership at university-based pediatric hospitalist programs. Similar findings are described in other specialties, including notably adult hospital medicine.⁴ Burden et al found that only 16% of hospital medicine program leaders were women despite an equal number of women and men in the field. PHM has a much larger proportion of women, compared with that of hospital medicine, and yet women are still underrepresented as program leaders.

We found no disparities between the proportion of women as PHM fellowship directors and the field at large. These results are similar to those of other studies, which showed a higher number of women in educational leadership roles and lower representation in roles with influence over policy and allocation of resources.^13,14 Although the proportion of women in educational roles itself is not a concern, there is evidence that these positions may be undervalued by some institutions, which provide these positions with lower salaries and fewer opportunities for career advancement.^13,14

Interestingly, women are well-represented in associate/assistant director roles at both the division and fellowship leader level when comparing the distribution in those roles with that of the PHM field at large. This finding suggests that the pipeline of women is robust and potentially may indicate positive change. Alternatively, this finding may reflect a previously described phenomenon of the “sticky floor” in which women are “stuck” in these supportive roles and do not necessarily advance to higher-impact positions.¹⁵ We found a statistically significant higher proportion of women in the combined group of all associate/assistant directors compared with the overall population, which raises the concern that supportive leadership roles may represent “women’s work.”¹⁶ Future studies are needed to track whether these women truly advance or whether women are overrepresented in supportive leadership positions at the expense of primary leadership positions.

Adequate representation of women alone is not sufficient to achieve gender equity in medicine. We need to understand why there is a lower representation of women in leadership positions. Some barriers have already been described, including gender bias in promotions,¹⁷ higher demands outside of work,¹⁸ and lower pay,³ though none are specific to PHM. A further qualitative exploration of PHM leadership would help describe any barriers women in PHM specifically may be facing in their career trajectory. In addition, more information is needed to explore the experience of women with intersectional identities in PHM, especially since they may experience increased bias and discrimination.¹⁹

Limitations of this study include the lack of a centralized list of PHM programs and data on PHM workforce. Our three estimates for the proportion of women in PHM were similar at 70%-71%; however, these are only proxies for the true gender distribution of PHM physicians, which is unknown. PHM leadership targets of close to 70% women would be reflective of the field at large; however, institutional variation may exist, and ideally leadership should be diverse and reflective of its faculty members. Our study only describes university-based PHM programs and, therefore, is not necessarily generalizable to nonuniversity programs. Further studies are needed to evaluate any potential differences based on program type. In our study, gender was used in binary terms; however, we acknowledge that gender exists on a spectrum.

As a specialty early in development with a robust pipeline of women, PHM is in a unique position to lead the way in gender equity. However, women appear to be underrepresented as division directors at university-based PHM programs. Achieving proportional representation of women leaders is imperative for tapping into the full potential of the community and ensuring that the goals of the field are representative of the population.

Special thanks to Lucille Lester, MD, who asked the question that started this road to discovery.

There is a growing appreciation of gender disparities in career advancement in medicine. By 2004, approximately 50% of medical school graduates were women, yet considerable differences persist between genders in compensation, faculty rank, and leadership positions.^1-3 According to the Association of American Medical Colleges (AAMC), women account for only 25% of full professors, 18% of department chairs, and 18% of medical school deans.¹ Women are also underrepresented in other areas of leadership such as division directors, professional society leadership, and hospital executives.^4-6

Specialties that are predominantly women, including pediatrics, are not immune to gender disparities. Women represent 71% of pediatric residents¹ and currently constitute two-thirds of active pediatricians in the United States.⁷ However, there is a disproportionately low number of women ascending the pediatric academic ladder, with only 35% of full professors² and 28% of department chairs being women.¹ Pediatrics also was noted to have the fifth-largest gender pay gap across 40 specialties.³ These disparities can contribute to burnout, poorer patient outcomes, and decreased advancement of women known as the “leaky pipeline.”^1,8,9

There is some evidence that gender disparities may be improving among younger professionals with increasing percentages of women as leaders and decreasing pay gaps.^10,11 These potential positive trends provide hope that fields in medicine early in their development may demonstrate fewer gender disparities. One of the youngest fields of medicine is pediatric hospital medicine (PHM), which officially became a recognized pediatric subspecialty in 2017.¹² There is no literature to date describing gender disparities in PHM. We aimed to explore the gender distribution of university-based PHM program leadership and to compare this gender distribution with that seen in the broader field of PHM.

This study was Institutional Review Board–approved as non–human subjects research through University of Chicago, Chicago, Illinois. From January to March 2020, the authors performed web-based searches for PHM division directors or program leaders in the United States. Because there is no single database of PHM programs in the United States, we used the AAMC list of Liaison Committee on Medical Education (LCME)–accredited US medical schools; medical schools in Puerto Rico were not included, nor were pending and provisional institutions. If an institution had multiple practice sites for its students, the primary site for third-year medical student clerkship rotations was included. If a medical school had multiple branches, each with its own primary inpatient pediatrics site, these sites were included. If there was no PHM division director, a program leader (lead hospitalist) was substituted and counted as long as the role was formally designated. This leadership role is herein referred to under the umbrella term of “division director.”

We searched medical school web pages, affiliated hospital web pages, and Google. All program leadership information (divisional and fellowship, if present) was confirmed through direct communication with the program, most commonly with division directors, and included name, gender, title, and presence of associate/assistant leader, gender, and title. Associate division directors were only included if it was a formal leadership position. Associate directors of research, quality, etc, were not included due to the limited number of formal positions noted on further review. Of note, the terms “associate” and “assistant” are referring to leadership positions and not academic ranks.

Fellowship leadership was included if affiliated with a US medical school in the primary list. Medical schools with multiple PHM fellowships were included as separate observations. The leadership was confirmed using the methods described above and cross-referenced through the PHM Fellowship Program website. PHM fellowship programs starting in 2020 were included if leadership was determined.

All leadership positions were verified by two authors, and all authors reviewed the master list to identify errors.

To determine the overall gender breakdown in the specialty, we used three estimates: 2019 American Board of Pediatrics (ABP) PHM Board Certification Exam applicants, the 2019 American Academy of Pediatrics Section on Hospital Medicine membership, and a random sample of all PHM faculty in 25% of the programs included in this study.⁴

Descriptive statistics using 95% confidence intervals for proportions were used. Differences between proportions were evaluated using a two-proportion z test with the null hypothesis that the two proportions are the same and significance set at P < .05. 

Of the 150 AAMC LCME–accredited medical school departments of pediatrics evaluated, a total of 142 programs were included; eight programs were excluded due to not providing inpatient pediatric services.

Division Leadership

The proportion of women PHM division directors was 55% (95% CI, 47%-63%) in this sample of 146 leaders from 142 programs (4 programs had coleaders). In the 113 programs with standalone PHM divisions or sections, the proportion of women division directors was 56% (95% CI, 47%-64%). In the 29 hospitalist groups that were not standalone (ie, embedded in another division), the proportion of women leaders was similar at 52% (95% CI, 34%-69%). In 24 programs with 27 formally designated associate directors (1 program had 3 associate directors and 1 program had 2), 81% of associate directors were women (95% CI, 63%-92%).

Fellowship Leadership

A total of 51 PHM fellowship programs had 53 directors (2 had codirectors), and 66% of the fellowship directors were women (95% CI, 53%-77%). A total of 31 programs had 34 assistant directors (3 programs had 2 assistants), and 82% of the assistant fellowship directors were women (95% CI, 66%-92%).

Comparison With the Field at Large

The inaugural ABP PHM board certification exam in 2019 had 1,627 applicants with 70% women (95% CI, 68%-73%) (Suzanne Woods, MD, email communication, December 4, 2019). The American Academy of Pediatrics Section on Hospital Medicine, the largest PHM-specific organization, has 2,299 practicing physician members with 71% women (95% CI, 69%-73%) (Niccole Alexander, email communication, November 25, 2019). Our random sample of 25% of university-based PHM programs contained 1,063 faculty members with 72% women (95% CI, 69%-75%).

The Table provides P values for comparisons of the proportion of women in each of the above-described leadership roles compared to the most conservative estimate of women in the field from the estimates given above (ie, 70%). Compared with the field at large, women appear to be underrepresented as division directors (70% vs 55%; P < .001) but not as fellowship directors (70% vs 66%; P = .5). There is a higher proportion of women in all associate/assistant director roles, compared with the population (82% vs 70%; P = .04).

We found a significant difference between the proportion of women as PHM division directors (55%) when compared with the proportion of women physicians in PHM (70%), which suggests that women are underrepresented in clinical leadership at university-based pediatric hospitalist programs. Similar findings are described in other specialties, including notably adult hospital medicine.⁴ Burden et al found that only 16% of hospital medicine program leaders were women despite an equal number of women and men in the field. PHM has a much larger proportion of women, compared with that of hospital medicine, and yet women are still underrepresented as program leaders.

We found no disparities between the proportion of women as PHM fellowship directors and the field at large. These results are similar to those of other studies, which showed a higher number of women in educational leadership roles and lower representation in roles with influence over policy and allocation of resources.^13,14 Although the proportion of women in educational roles itself is not a concern, there is evidence that these positions may be undervalued by some institutions, which provide these positions with lower salaries and fewer opportunities for career advancement.^13,14

Interestingly, women are well-represented in associate/assistant director roles at both the division and fellowship leader level when comparing the distribution in those roles with that of the PHM field at large. This finding suggests that the pipeline of women is robust and potentially may indicate positive change. Alternatively, this finding may reflect a previously described phenomenon of the “sticky floor” in which women are “stuck” in these supportive roles and do not necessarily advance to higher-impact positions.¹⁵ We found a statistically significant higher proportion of women in the combined group of all associate/assistant directors compared with the overall population, which raises the concern that supportive leadership roles may represent “women’s work.”¹⁶ Future studies are needed to track whether these women truly advance or whether women are overrepresented in supportive leadership positions at the expense of primary leadership positions.

Adequate representation of women alone is not sufficient to achieve gender equity in medicine. We need to understand why there is a lower representation of women in leadership positions. Some barriers have already been described, including gender bias in promotions,¹⁷ higher demands outside of work,¹⁸ and lower pay,³ though none are specific to PHM. A further qualitative exploration of PHM leadership would help describe any barriers women in PHM specifically may be facing in their career trajectory. In addition, more information is needed to explore the experience of women with intersectional identities in PHM, especially since they may experience increased bias and discrimination.¹⁹

Limitations of this study include the lack of a centralized list of PHM programs and data on PHM workforce. Our three estimates for the proportion of women in PHM were similar at 70%-71%; however, these are only proxies for the true gender distribution of PHM physicians, which is unknown. PHM leadership targets of close to 70% women would be reflective of the field at large; however, institutional variation may exist, and ideally leadership should be diverse and reflective of its faculty members. Our study only describes university-based PHM programs and, therefore, is not necessarily generalizable to nonuniversity programs. Further studies are needed to evaluate any potential differences based on program type. In our study, gender was used in binary terms; however, we acknowledge that gender exists on a spectrum.

As a specialty early in development with a robust pipeline of women, PHM is in a unique position to lead the way in gender equity. However, women appear to be underrepresented as division directors at university-based PHM programs. Achieving proportional representation of women leaders is imperative for tapping into the full potential of the community and ensuring that the goals of the field are representative of the population.

Special thanks to Lucille Lester, MD, who asked the question that started this road to discovery.

Hypoxemic respiratory failure is a hallmark of severe coronavirus disease 2019 (COVID-19). Initial guidelines favored early mechanical ventilation (MV) over traditional noninvasive strategies, such as high-flow nasal cannula (HFNC) and noninvasive positive pressure ventilation (NIV), based on perceived ineffectiveness and dangers extrapolated from severe acute respiratory syndrome coronavirus (SARS-CoV) and Middle East respiratory syndrome coronavirus (MERS-CoV) patients.^1,2 As COVID-19 progressed, early MV became associated with prolonged ventilator courses and high mortality.^3-6 Simultaneously, data emerged that HFNC/NIV and self-proning, could successfully stabilize some COVID-19 patients.^7-10 Based on evolving evidence, we implemented a noninvasive COVID-19 respiratory protocol (NCRP) that promoted the early use of HFNC, NIV, and self-proning for hypoxemia in patients with COVID-19, with the intention of avoiding MV in some patients. The protocol was implemented throughout our hospital system, from the Emergency Departments (EDs) to the medical floors and critical care units.

Although preliminary evidence supported the use of HFNC, NIV, and self-proning, the impact of a system-wide noninvasive COVID-19 respiratory protocol on safety has not been well described. The objective of this study was to evaluate patient safety outcomes after implementation of the NCRP, including intubation rate and mortality.

Study Design and Setting

We performed a retrospective chart review, adhering to SQUIRE (Standards for Quality Improvement Reporting Excellence) Guidelines, to assess safety outcomes after implementation of the NCRP.¹¹ Baystate Health is a not-for-profit, integrated healthcare system in western Massachusetts composed of four hospitals and one free-standing ED with 980 beds serving over 800,000 people. The Baystate Health IRB determined that this project did not meet criteria for Human Subjects Research.

Selection of Participants

A consecutive sample of adults (≥18 years old) admitted to the hospital with a positive nucleic acid test for SARS-CoV-2 (reverse transcriptase–polymerase chain reaction [RT-PCR]) test via nasopharyngeal swab (Cepheid or Roche Cobas 6800) between March 15, 2020, and April 15, 2020, were included. Participants were identified by either an order for the COVID-19 test with a positive result or a discharge diagnosis of COVID-19. Daily rapid response team (RRT), intensive care unit (ICU), and COVID-19 unit logs were reviewed to ensure all COVID-19 patients were included. Patients with positive tests admitted for reasons unrelated to COVID-19 infections, such as patients in labor, were excluded.

Interventions

At the start of the COVID-19 pandemic, the Baystate Health system adopted a conservative approach to the respiratory management of patients with COVID-19. This approach started with nasal cannula up to 6 L/min or nonrebreather up to 15 L/min. If the patient remained in respiratory distress, intubation was recommended.

Based on emerging evidence, the NCRP was created. The details of the NCRP implementation have been previously described.¹² Briefly, over a 4-day period (April 3, 2020, to April 7, 2020), a multidisciplinary team developed, refined, and rapidly implemented a COVID-19 respiratory protocol that encouraged the early use of HFNC, NIV, and self-proning in clinically appropriate patients with hypoxemia and respiratory distress due to COVID-19 prior to intubation across all departments of the Baystate Health system (Appendix 1).

Measurements

A chart review was performed using a structured data collection form (Appendix 2). The data collection form was piloted by three physician-researchers. Data abstraction was performed by 16 clinicians. Abstractors were practicing emergency providers and hospitalists and were blinded to the study outcomes. Abstractors received a 1-hour training and abstracted data from at least five charts in parallel with investigators. An additional 10% of charts were double abstracted to calculate interrater reliability for five variables determined a priori.

To validate the capture of outcomes of interest, we triangulated data sources by cross-referencing the monthly RRT log, the ICU list, all orders for HFNC, and RRT activations. Data abstraction occurred from April 21, 2020, to April 30, 2020. Patients who were still hospitalized after April 30,2020, were followed until hospital discharge, ending July 1, 2020.

Outcomes and Analysis

The primary outcome was mortality, defined as the proportion of deaths by admissions during the post–NCRP implementation period (April 3, 2020, to April 15, 2020), compared with the preimplementation period (March 15, 2020, to April 2, 2020). Deaths were stratified by patient code status (do not resuscitate/do not intubate [DNR/DNI] established prior to admission vs Full Code or presumed Full Code). Mortality outcomes were evaluated using one-sided Fisher exact tests.

To assess whether the protocol led to an increase in the use of the interventions and a decrease in intubations, we compared the use of proning, HFNC, NIV, and intubation before the protocol was implemented and with use after. Intubation rates were analyzed using interrupted time series (piecemeal regression), without adjustments, using a cut point of April 2, 2020.

Secondary outcomes included unexpected cardiac arrests, ICU transfers and consultations, and RRT activations during the postimplementation period, compared with the preimplementation period. Secondary outcomes were evaluated using standard chi-square tests (χ²). Additional descriptive outcomes included use of the NCRP, overall and by components, and in-hospital rates of MV.

From March 15, 2020, through April 15, 2020, there were 469 patients with COVID-19 admitted to the four hospitals of the Baystate Health system. Patients had an average age of 70 years (SD, 16.4), 241 (52%) were female, and 336 (72%) spoke English as their primary language. Most patients, 405 (86.4%), required supplemental oxygen upon being admitted to the hospital (Table 1).

Postimplementation Mortality

Overall, 123 (26.2%) patients died during the study period. In the preimplementation cohort, 24% (61 of 254) of patients died, compared with 28.8% (62 of 215) in the postimplementation cohort (one-sided Fisher exact, P = .14). Excluding patients with an established DNR/DNI prior to admission, 21.8% (48 of 220) patients died in the preimplementation period vs 21.9% (35 of 160) patients after implementation of the NCRP (Table 2).

Secondary Safety Outcomes

There was no increase in RRT activations (preimplementation, 16.5% [42 of 254], vs postimplementation, 11.6% [25 of 215]; χ²P = 0.17) or ICU consultations (preimplementation, 18.1% [47 of 254], vs postimplementation, 16.3% [35 of 215]; χ²P = 0.52). ICU transfers decreased in the postimplementation period (preimplementation, 26.8% [68 of 254], vs postimplementation, 13.5% [29 of 215], χ²P < .001). There was one unexpected cardiac arrest documented in the postimplementation period, compared to none before implementation.

NCRP Protocol Implementation

After implementation, the proportion of patients using HFNC increased from 5.5% (14 of 254) to 24.7% (53 of 215), and self-proning increased from 7.5% (19 of 254) to 22.8% (49 of 215). The proportion of patients who were intubated (MV) decreased from 25.2% (64 of 254) to 10.7% (23 of 215) (χ²P < .01). Interrupted time series analysis demonstrated an immediate reduction in the proportion of patients intubated after the intervention (incident rate ratio, 0.44; 95% CI, 0.23-0.83; P = .012) (Figure). The median time from admission to MV was longer in the postimplementation period patients (postimplementation, 1.4 days; interquartile range, 0.21-2.9; vs preimplementation, 0.66 days; IQR 0.23-1.69).

Interrater Reliability

Interrater reliability for variables chosen a priori was k = 1.0 for self-proning, k = 1.0 for intubation, k = 0.95 for discharge disposition, k = 0.94 for nasal cannula, and k = 0.74 for HFNC.

The rapid spread of SARS-CoV-2 led to early recommendations based on minimal data. As evidence emerged, hospitals were forced to adapt to protect patients and medical providers. As a healthcare system, we incorporated emerging evidence to rapidly implement a noninvasive respiratory treatment protocol. Aware of the methodological problems in evaluating the NCRP itself, we integrated best practices of quality improvement to examine multiple patient safety outcomes after NCRP implementation. We found the rate of intubation decreased with no significant increase in mortality, ICU transfers, RRT activations, or unexpected deaths after the implementation of the NCRP.

Although we were unable to measure all confounders and changes that co-occurred during the study period, initial vital signs, age, BMI, past medical history, and use of oxygen were similar between the pre- and postimplementation cohorts. Further, there were many constants worth noting. First, COVID-19 respiratory protocols were highly regulated to ensure patient safety and minimize COVID-19 transmission. Second, there were no new nonrespiratory treatments or medications during the study. Third, although the COVID-19 hospital census rose during the study, it never overwhelmed resources; there was no rationing of clinical care.

The nonsignificant increase in mortality in the postimplementation period was limited to patients with an established DNR/DNI prior to admission. Established DNR/DNI patients were largely from skilled nursing facilities that were disproportionally impacted in the postimplementation period through clustered outbreaks of COVID-19 in our region, which likely contributed to the increased mortality.¹³

Additionally, despite decreased MV rates in the postimplementation period, we did not find a concurrent decrease in mortality. We do not believe this is a failure of noninvasive treatments. Rather, the increased proportion of DNR/DNI patients, combined with increased nursing home outbreaks in the postimplementation period likely influenced mortality. The postimplementation decreases in ICU transfers and RRT activations supports this hypothesis.

Finally, it is worth nothing that, although the goal of decreasing intubations was to improve patient care and decrease mortality, a decrease in intubations alone, without a change in mortality, may be important because mechanical ventilation has been associated with increased morbidity, such as posttraumatic stress disorder.¹⁴

Taken together, the post–NCRP implementation period appears to have been safe for patients, compared to the preimplementation period’s protocol. Future research may help understand the impact of specific noninvasive interventions on COVID-19–related MV and mortality.

Limitations

Given the urgency of COVID-19 treatment, the NCRP was designed as a quality improvement initiative rather than a prospective trial. Issues of selection bias and confounding limit our ability to evaluate the effect of the NCRP itself. Additionally, unmeasured patient and provider factors may have influenced outcomes. For example, increased provider knowledge and experience treating COVID-19 may have improved outcomes over time, and unmeasured patient characteristics may have been different in the pre- and postimplementation groups. Finally, our study was limited to a single healthcare system, which may limit generalizability

That said, the objective of our study was to evaluate patient safety outcomes of the NCRP, an important first step while other hospital systems continue to confront increasing rates of COVID-19 and must decide on appropriate respiratory management. To that end, our enrollment captured 469 COVID-19 admissions across four diverse hospitals without obvious differences in initial measured covariates. Further, the strict protocolization of respiratory treatments, the evaluation of multiple safety outcomes, and the complete patient follow-up all support the conclusion that NCRP in the postimplementation period did not increase adverse patient outcomes. Further studies are needed to determine the efficacy of the NCRP protocol itself.

In our health system, patients with COVID-19 did not experience a significant increase in mortality, RRT activations, or ICU admissions despite decreased rates of MV after implementation of a respiratory protocol that encouraged early noninvasive management of COVID-19 respiratory distress.

The authors would like to acknowledge Elizabeth Coray, Joseph Lahey, Richard Gabor, Cheryl Greenstein, Sarah Badach, Marie Boutin, Adrienne Wurl, Anthony Kitchen, Michelle Holton, Matthew Shapiro, Eleanor Ragone, Nageshwar Jonnalagadda, Ryan Flynn, Raghuveer Rakasi, and Jasmine Paadam.

Hypoxemic respiratory failure is a hallmark of severe coronavirus disease 2019 (COVID-19). Initial guidelines favored early mechanical ventilation (MV) over traditional noninvasive strategies, such as high-flow nasal cannula (HFNC) and noninvasive positive pressure ventilation (NIV), based on perceived ineffectiveness and dangers extrapolated from severe acute respiratory syndrome coronavirus (SARS-CoV) and Middle East respiratory syndrome coronavirus (MERS-CoV) patients.^1,2 As COVID-19 progressed, early MV became associated with prolonged ventilator courses and high mortality.^3-6 Simultaneously, data emerged that HFNC/NIV and self-proning, could successfully stabilize some COVID-19 patients.^7-10 Based on evolving evidence, we implemented a noninvasive COVID-19 respiratory protocol (NCRP) that promoted the early use of HFNC, NIV, and self-proning for hypoxemia in patients with COVID-19, with the intention of avoiding MV in some patients. The protocol was implemented throughout our hospital system, from the Emergency Departments (EDs) to the medical floors and critical care units.

Although preliminary evidence supported the use of HFNC, NIV, and self-proning, the impact of a system-wide noninvasive COVID-19 respiratory protocol on safety has not been well described. The objective of this study was to evaluate patient safety outcomes after implementation of the NCRP, including intubation rate and mortality.

Study Design and Setting

We performed a retrospective chart review, adhering to SQUIRE (Standards for Quality Improvement Reporting Excellence) Guidelines, to assess safety outcomes after implementation of the NCRP.¹¹ Baystate Health is a not-for-profit, integrated healthcare system in western Massachusetts composed of four hospitals and one free-standing ED with 980 beds serving over 800,000 people. The Baystate Health IRB determined that this project did not meet criteria for Human Subjects Research.

Selection of Participants

A consecutive sample of adults (≥18 years old) admitted to the hospital with a positive nucleic acid test for SARS-CoV-2 (reverse transcriptase–polymerase chain reaction [RT-PCR]) test via nasopharyngeal swab (Cepheid or Roche Cobas 6800) between March 15, 2020, and April 15, 2020, were included. Participants were identified by either an order for the COVID-19 test with a positive result or a discharge diagnosis of COVID-19. Daily rapid response team (RRT), intensive care unit (ICU), and COVID-19 unit logs were reviewed to ensure all COVID-19 patients were included. Patients with positive tests admitted for reasons unrelated to COVID-19 infections, such as patients in labor, were excluded.

Interventions

At the start of the COVID-19 pandemic, the Baystate Health system adopted a conservative approach to the respiratory management of patients with COVID-19. This approach started with nasal cannula up to 6 L/min or nonrebreather up to 15 L/min. If the patient remained in respiratory distress, intubation was recommended.

Based on emerging evidence, the NCRP was created. The details of the NCRP implementation have been previously described.¹² Briefly, over a 4-day period (April 3, 2020, to April 7, 2020), a multidisciplinary team developed, refined, and rapidly implemented a COVID-19 respiratory protocol that encouraged the early use of HFNC, NIV, and self-proning in clinically appropriate patients with hypoxemia and respiratory distress due to COVID-19 prior to intubation across all departments of the Baystate Health system (Appendix 1).

Measurements

A chart review was performed using a structured data collection form (Appendix 2). The data collection form was piloted by three physician-researchers. Data abstraction was performed by 16 clinicians. Abstractors were practicing emergency providers and hospitalists and were blinded to the study outcomes. Abstractors received a 1-hour training and abstracted data from at least five charts in parallel with investigators. An additional 10% of charts were double abstracted to calculate interrater reliability for five variables determined a priori.

To validate the capture of outcomes of interest, we triangulated data sources by cross-referencing the monthly RRT log, the ICU list, all orders for HFNC, and RRT activations. Data abstraction occurred from April 21, 2020, to April 30, 2020. Patients who were still hospitalized after April 30,2020, were followed until hospital discharge, ending July 1, 2020.

Outcomes and Analysis

The primary outcome was mortality, defined as the proportion of deaths by admissions during the post–NCRP implementation period (April 3, 2020, to April 15, 2020), compared with the preimplementation period (March 15, 2020, to April 2, 2020). Deaths were stratified by patient code status (do not resuscitate/do not intubate [DNR/DNI] established prior to admission vs Full Code or presumed Full Code). Mortality outcomes were evaluated using one-sided Fisher exact tests.

To assess whether the protocol led to an increase in the use of the interventions and a decrease in intubations, we compared the use of proning, HFNC, NIV, and intubation before the protocol was implemented and with use after. Intubation rates were analyzed using interrupted time series (piecemeal regression), without adjustments, using a cut point of April 2, 2020.

Secondary outcomes included unexpected cardiac arrests, ICU transfers and consultations, and RRT activations during the postimplementation period, compared with the preimplementation period. Secondary outcomes were evaluated using standard chi-square tests (χ²). Additional descriptive outcomes included use of the NCRP, overall and by components, and in-hospital rates of MV.

From March 15, 2020, through April 15, 2020, there were 469 patients with COVID-19 admitted to the four hospitals of the Baystate Health system. Patients had an average age of 70 years (SD, 16.4), 241 (52%) were female, and 336 (72%) spoke English as their primary language. Most patients, 405 (86.4%), required supplemental oxygen upon being admitted to the hospital (Table 1).

Postimplementation Mortality

Overall, 123 (26.2%) patients died during the study period. In the preimplementation cohort, 24% (61 of 254) of patients died, compared with 28.8% (62 of 215) in the postimplementation cohort (one-sided Fisher exact, P = .14). Excluding patients with an established DNR/DNI prior to admission, 21.8% (48 of 220) patients died in the preimplementation period vs 21.9% (35 of 160) patients after implementation of the NCRP (Table 2).

Secondary Safety Outcomes

There was no increase in RRT activations (preimplementation, 16.5% [42 of 254], vs postimplementation, 11.6% [25 of 215]; χ²P = 0.17) or ICU consultations (preimplementation, 18.1% [47 of 254], vs postimplementation, 16.3% [35 of 215]; χ²P = 0.52). ICU transfers decreased in the postimplementation period (preimplementation, 26.8% [68 of 254], vs postimplementation, 13.5% [29 of 215], χ²P < .001). There was one unexpected cardiac arrest documented in the postimplementation period, compared to none before implementation.

NCRP Protocol Implementation

After implementation, the proportion of patients using HFNC increased from 5.5% (14 of 254) to 24.7% (53 of 215), and self-proning increased from 7.5% (19 of 254) to 22.8% (49 of 215). The proportion of patients who were intubated (MV) decreased from 25.2% (64 of 254) to 10.7% (23 of 215) (χ²P < .01). Interrupted time series analysis demonstrated an immediate reduction in the proportion of patients intubated after the intervention (incident rate ratio, 0.44; 95% CI, 0.23-0.83; P = .012) (Figure). The median time from admission to MV was longer in the postimplementation period patients (postimplementation, 1.4 days; interquartile range, 0.21-2.9; vs preimplementation, 0.66 days; IQR 0.23-1.69).

Interrater Reliability

Interrater reliability for variables chosen a priori was k = 1.0 for self-proning, k = 1.0 for intubation, k = 0.95 for discharge disposition, k = 0.94 for nasal cannula, and k = 0.74 for HFNC.

The rapid spread of SARS-CoV-2 led to early recommendations based on minimal data. As evidence emerged, hospitals were forced to adapt to protect patients and medical providers. As a healthcare system, we incorporated emerging evidence to rapidly implement a noninvasive respiratory treatment protocol. Aware of the methodological problems in evaluating the NCRP itself, we integrated best practices of quality improvement to examine multiple patient safety outcomes after NCRP implementation. We found the rate of intubation decreased with no significant increase in mortality, ICU transfers, RRT activations, or unexpected deaths after the implementation of the NCRP.

Although we were unable to measure all confounders and changes that co-occurred during the study period, initial vital signs, age, BMI, past medical history, and use of oxygen were similar between the pre- and postimplementation cohorts. Further, there were many constants worth noting. First, COVID-19 respiratory protocols were highly regulated to ensure patient safety and minimize COVID-19 transmission. Second, there were no new nonrespiratory treatments or medications during the study. Third, although the COVID-19 hospital census rose during the study, it never overwhelmed resources; there was no rationing of clinical care.

The nonsignificant increase in mortality in the postimplementation period was limited to patients with an established DNR/DNI prior to admission. Established DNR/DNI patients were largely from skilled nursing facilities that were disproportionally impacted in the postimplementation period through clustered outbreaks of COVID-19 in our region, which likely contributed to the increased mortality.¹³

Additionally, despite decreased MV rates in the postimplementation period, we did not find a concurrent decrease in mortality. We do not believe this is a failure of noninvasive treatments. Rather, the increased proportion of DNR/DNI patients, combined with increased nursing home outbreaks in the postimplementation period likely influenced mortality. The postimplementation decreases in ICU transfers and RRT activations supports this hypothesis.

Finally, it is worth nothing that, although the goal of decreasing intubations was to improve patient care and decrease mortality, a decrease in intubations alone, without a change in mortality, may be important because mechanical ventilation has been associated with increased morbidity, such as posttraumatic stress disorder.¹⁴

Taken together, the post–NCRP implementation period appears to have been safe for patients, compared to the preimplementation period’s protocol. Future research may help understand the impact of specific noninvasive interventions on COVID-19–related MV and mortality.

Limitations

Given the urgency of COVID-19 treatment, the NCRP was designed as a quality improvement initiative rather than a prospective trial. Issues of selection bias and confounding limit our ability to evaluate the effect of the NCRP itself. Additionally, unmeasured patient and provider factors may have influenced outcomes. For example, increased provider knowledge and experience treating COVID-19 may have improved outcomes over time, and unmeasured patient characteristics may have been different in the pre- and postimplementation groups. Finally, our study was limited to a single healthcare system, which may limit generalizability

That said, the objective of our study was to evaluate patient safety outcomes of the NCRP, an important first step while other hospital systems continue to confront increasing rates of COVID-19 and must decide on appropriate respiratory management. To that end, our enrollment captured 469 COVID-19 admissions across four diverse hospitals without obvious differences in initial measured covariates. Further, the strict protocolization of respiratory treatments, the evaluation of multiple safety outcomes, and the complete patient follow-up all support the conclusion that NCRP in the postimplementation period did not increase adverse patient outcomes. Further studies are needed to determine the efficacy of the NCRP protocol itself.

In our health system, patients with COVID-19 did not experience a significant increase in mortality, RRT activations, or ICU admissions despite decreased rates of MV after implementation of a respiratory protocol that encouraged early noninvasive management of COVID-19 respiratory distress.

The authors would like to acknowledge Elizabeth Coray, Joseph Lahey, Richard Gabor, Cheryl Greenstein, Sarah Badach, Marie Boutin, Adrienne Wurl, Anthony Kitchen, Michelle Holton, Matthew Shapiro, Eleanor Ragone, Nageshwar Jonnalagadda, Ryan Flynn, Raghuveer Rakasi, and Jasmine Paadam.

Hypoxemic respiratory failure is a hallmark of severe coronavirus disease 2019 (COVID-19). Initial guidelines favored early mechanical ventilation (MV) over traditional noninvasive strategies, such as high-flow nasal cannula (HFNC) and noninvasive positive pressure ventilation (NIV), based on perceived ineffectiveness and dangers extrapolated from severe acute respiratory syndrome coronavirus (SARS-CoV) and Middle East respiratory syndrome coronavirus (MERS-CoV) patients.^1,2 As COVID-19 progressed, early MV became associated with prolonged ventilator courses and high mortality.^3-6 Simultaneously, data emerged that HFNC/NIV and self-proning, could successfully stabilize some COVID-19 patients.^7-10 Based on evolving evidence, we implemented a noninvasive COVID-19 respiratory protocol (NCRP) that promoted the early use of HFNC, NIV, and self-proning for hypoxemia in patients with COVID-19, with the intention of avoiding MV in some patients. The protocol was implemented throughout our hospital system, from the Emergency Departments (EDs) to the medical floors and critical care units.

Although preliminary evidence supported the use of HFNC, NIV, and self-proning, the impact of a system-wide noninvasive COVID-19 respiratory protocol on safety has not been well described. The objective of this study was to evaluate patient safety outcomes after implementation of the NCRP, including intubation rate and mortality.

Study Design and Setting

We performed a retrospective chart review, adhering to SQUIRE (Standards for Quality Improvement Reporting Excellence) Guidelines, to assess safety outcomes after implementation of the NCRP.¹¹ Baystate Health is a not-for-profit, integrated healthcare system in western Massachusetts composed of four hospitals and one free-standing ED with 980 beds serving over 800,000 people. The Baystate Health IRB determined that this project did not meet criteria for Human Subjects Research.

Selection of Participants

A consecutive sample of adults (≥18 years old) admitted to the hospital with a positive nucleic acid test for SARS-CoV-2 (reverse transcriptase–polymerase chain reaction [RT-PCR]) test via nasopharyngeal swab (Cepheid or Roche Cobas 6800) between March 15, 2020, and April 15, 2020, were included. Participants were identified by either an order for the COVID-19 test with a positive result or a discharge diagnosis of COVID-19. Daily rapid response team (RRT), intensive care unit (ICU), and COVID-19 unit logs were reviewed to ensure all COVID-19 patients were included. Patients with positive tests admitted for reasons unrelated to COVID-19 infections, such as patients in labor, were excluded.

Interventions

At the start of the COVID-19 pandemic, the Baystate Health system adopted a conservative approach to the respiratory management of patients with COVID-19. This approach started with nasal cannula up to 6 L/min or nonrebreather up to 15 L/min. If the patient remained in respiratory distress, intubation was recommended.

Based on emerging evidence, the NCRP was created. The details of the NCRP implementation have been previously described.¹² Briefly, over a 4-day period (April 3, 2020, to April 7, 2020), a multidisciplinary team developed, refined, and rapidly implemented a COVID-19 respiratory protocol that encouraged the early use of HFNC, NIV, and self-proning in clinically appropriate patients with hypoxemia and respiratory distress due to COVID-19 prior to intubation across all departments of the Baystate Health system (Appendix 1).

Measurements

A chart review was performed using a structured data collection form (Appendix 2). The data collection form was piloted by three physician-researchers. Data abstraction was performed by 16 clinicians. Abstractors were practicing emergency providers and hospitalists and were blinded to the study outcomes. Abstractors received a 1-hour training and abstracted data from at least five charts in parallel with investigators. An additional 10% of charts were double abstracted to calculate interrater reliability for five variables determined a priori.

To validate the capture of outcomes of interest, we triangulated data sources by cross-referencing the monthly RRT log, the ICU list, all orders for HFNC, and RRT activations. Data abstraction occurred from April 21, 2020, to April 30, 2020. Patients who were still hospitalized after April 30,2020, were followed until hospital discharge, ending July 1, 2020.

Outcomes and Analysis

The primary outcome was mortality, defined as the proportion of deaths by admissions during the post–NCRP implementation period (April 3, 2020, to April 15, 2020), compared with the preimplementation period (March 15, 2020, to April 2, 2020). Deaths were stratified by patient code status (do not resuscitate/do not intubate [DNR/DNI] established prior to admission vs Full Code or presumed Full Code). Mortality outcomes were evaluated using one-sided Fisher exact tests.

To assess whether the protocol led to an increase in the use of the interventions and a decrease in intubations, we compared the use of proning, HFNC, NIV, and intubation before the protocol was implemented and with use after. Intubation rates were analyzed using interrupted time series (piecemeal regression), without adjustments, using a cut point of April 2, 2020.

Secondary outcomes included unexpected cardiac arrests, ICU transfers and consultations, and RRT activations during the postimplementation period, compared with the preimplementation period. Secondary outcomes were evaluated using standard chi-square tests (χ²). Additional descriptive outcomes included use of the NCRP, overall and by components, and in-hospital rates of MV.

From March 15, 2020, through April 15, 2020, there were 469 patients with COVID-19 admitted to the four hospitals of the Baystate Health system. Patients had an average age of 70 years (SD, 16.4), 241 (52%) were female, and 336 (72%) spoke English as their primary language. Most patients, 405 (86.4%), required supplemental oxygen upon being admitted to the hospital (Table 1).

Postimplementation Mortality

Overall, 123 (26.2%) patients died during the study period. In the preimplementation cohort, 24% (61 of 254) of patients died, compared with 28.8% (62 of 215) in the postimplementation cohort (one-sided Fisher exact, P = .14). Excluding patients with an established DNR/DNI prior to admission, 21.8% (48 of 220) patients died in the preimplementation period vs 21.9% (35 of 160) patients after implementation of the NCRP (Table 2).

Secondary Safety Outcomes

There was no increase in RRT activations (preimplementation, 16.5% [42 of 254], vs postimplementation, 11.6% [25 of 215]; χ²P = 0.17) or ICU consultations (preimplementation, 18.1% [47 of 254], vs postimplementation, 16.3% [35 of 215]; χ²P = 0.52). ICU transfers decreased in the postimplementation period (preimplementation, 26.8% [68 of 254], vs postimplementation, 13.5% [29 of 215], χ²P < .001). There was one unexpected cardiac arrest documented in the postimplementation period, compared to none before implementation.

NCRP Protocol Implementation

After implementation, the proportion of patients using HFNC increased from 5.5% (14 of 254) to 24.7% (53 of 215), and self-proning increased from 7.5% (19 of 254) to 22.8% (49 of 215). The proportion of patients who were intubated (MV) decreased from 25.2% (64 of 254) to 10.7% (23 of 215) (χ²P < .01). Interrupted time series analysis demonstrated an immediate reduction in the proportion of patients intubated after the intervention (incident rate ratio, 0.44; 95% CI, 0.23-0.83; P = .012) (Figure). The median time from admission to MV was longer in the postimplementation period patients (postimplementation, 1.4 days; interquartile range, 0.21-2.9; vs preimplementation, 0.66 days; IQR 0.23-1.69).

Interrater Reliability

Interrater reliability for variables chosen a priori was k = 1.0 for self-proning, k = 1.0 for intubation, k = 0.95 for discharge disposition, k = 0.94 for nasal cannula, and k = 0.74 for HFNC.

The rapid spread of SARS-CoV-2 led to early recommendations based on minimal data. As evidence emerged, hospitals were forced to adapt to protect patients and medical providers. As a healthcare system, we incorporated emerging evidence to rapidly implement a noninvasive respiratory treatment protocol. Aware of the methodological problems in evaluating the NCRP itself, we integrated best practices of quality improvement to examine multiple patient safety outcomes after NCRP implementation. We found the rate of intubation decreased with no significant increase in mortality, ICU transfers, RRT activations, or unexpected deaths after the implementation of the NCRP.

Although we were unable to measure all confounders and changes that co-occurred during the study period, initial vital signs, age, BMI, past medical history, and use of oxygen were similar between the pre- and postimplementation cohorts. Further, there were many constants worth noting. First, COVID-19 respiratory protocols were highly regulated to ensure patient safety and minimize COVID-19 transmission. Second, there were no new nonrespiratory treatments or medications during the study. Third, although the COVID-19 hospital census rose during the study, it never overwhelmed resources; there was no rationing of clinical care.

The nonsignificant increase in mortality in the postimplementation period was limited to patients with an established DNR/DNI prior to admission. Established DNR/DNI patients were largely from skilled nursing facilities that were disproportionally impacted in the postimplementation period through clustered outbreaks of COVID-19 in our region, which likely contributed to the increased mortality.¹³

Additionally, despite decreased MV rates in the postimplementation period, we did not find a concurrent decrease in mortality. We do not believe this is a failure of noninvasive treatments. Rather, the increased proportion of DNR/DNI patients, combined with increased nursing home outbreaks in the postimplementation period likely influenced mortality. The postimplementation decreases in ICU transfers and RRT activations supports this hypothesis.

Finally, it is worth nothing that, although the goal of decreasing intubations was to improve patient care and decrease mortality, a decrease in intubations alone, without a change in mortality, may be important because mechanical ventilation has been associated with increased morbidity, such as posttraumatic stress disorder.¹⁴

Taken together, the post–NCRP implementation period appears to have been safe for patients, compared to the preimplementation period’s protocol. Future research may help understand the impact of specific noninvasive interventions on COVID-19–related MV and mortality.

Limitations

Given the urgency of COVID-19 treatment, the NCRP was designed as a quality improvement initiative rather than a prospective trial. Issues of selection bias and confounding limit our ability to evaluate the effect of the NCRP itself. Additionally, unmeasured patient and provider factors may have influenced outcomes. For example, increased provider knowledge and experience treating COVID-19 may have improved outcomes over time, and unmeasured patient characteristics may have been different in the pre- and postimplementation groups. Finally, our study was limited to a single healthcare system, which may limit generalizability

That said, the objective of our study was to evaluate patient safety outcomes of the NCRP, an important first step while other hospital systems continue to confront increasing rates of COVID-19 and must decide on appropriate respiratory management. To that end, our enrollment captured 469 COVID-19 admissions across four diverse hospitals without obvious differences in initial measured covariates. Further, the strict protocolization of respiratory treatments, the evaluation of multiple safety outcomes, and the complete patient follow-up all support the conclusion that NCRP in the postimplementation period did not increase adverse patient outcomes. Further studies are needed to determine the efficacy of the NCRP protocol itself.

In our health system, patients with COVID-19 did not experience a significant increase in mortality, RRT activations, or ICU admissions despite decreased rates of MV after implementation of a respiratory protocol that encouraged early noninvasive management of COVID-19 respiratory distress.

The authors would like to acknowledge Elizabeth Coray, Joseph Lahey, Richard Gabor, Cheryl Greenstein, Sarah Badach, Marie Boutin, Adrienne Wurl, Anthony Kitchen, Michelle Holton, Matthew Shapiro, Eleanor Ragone, Nageshwar Jonnalagadda, Ryan Flynn, Raghuveer Rakasi, and Jasmine Paadam.

Early reports showed high mortality from coronavirus disease 2019 (COVID-19), while current United States data mortality rates are lower, raising hope that new treatments and management strategies have improved outcomes. For instance, Centers for Disease Control and Prevention data show that 6.7% of cases resulted in death in April, compared with 1.9% in September.¹ However, the demographics of those infected have also changed, and more available testing may mean more comprehensive identification and earlier treatment. Nationally, for instance, the median age of confirmed cases was 38 years at the end of August, down from 46 years at the start of May.² Therefore, whether decreasing COVID-19 mortality rates simply reflect changing demographics or represent actual improvements in clinical care is unknown. The objective of this analysis was to assess outcomes over time in a single health system, accounting for changes in demographics, clinical factors, and severity of disease at presentation.

We analyzed monthly mortality rates for admissions between March 1 and August 31, 2020, in a single health system in New York City. Outcomes were obtained as of October 8, 2020. We included all hospitalizations of people 18 years and older with laboratory-confirmed severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection identified during the hospitalization or in the prior 2 weeks, excluding those admitted to hospice care. Patients with multiple hospitalizations (N=208 patients, 229 hospitalizations, 4.4%) were included repeatedly if they continued to have laboratory-confirmed disease. Patients without admission vital signs (N=28) were excluded. Mortality was defined as in-hospital death or discharge to hospice care. In-house laboratory testing began March 16 and all inpatients were tested for SARS-CoV-2 by April 1; elective surgeries resumed May 4-11 and were only conducted on confirmed SARS-CoV-2–negative patients.

All data were obtained from the electronic health record (Epic Systems, Verona, Wisconsin). Diagnosis codes were obtained from the problem list, past medical history, and billing codes. In addition, we used objective data such as hemoglobin A1c, ejection fraction, outpatient creatinine, and outpatient blood pressure to augment problem list diagnoses where relevant.

Based on prior literature, we constructed multivariable logistic regression models for mortality adjusting for age; sex; self-reported race and ethnicity; body mass index; smoking history; presence of hypertension, heart failure, hyperlipidemia, coronary artery disease, diabetes, cancer, chronic kidney disease, dementia, or pulmonary disease individually as dummy variables; and admission oxygen saturation, D-dimer, ferritin, and C-reactive protein.^3-6 In the first model (C statistic 0.82), we did not include month of admission as a covariate and calculated the ratio of the sum of observed and expected deaths (obtained from the model) in each month to obtain the standardized mortality ratio (SMR) for each month. We then multiplied each period’s SMR by the overall average crude mortality to generate monthly adjusted mortality rates. We calculated Poisson control limits and indicated points outside the control limits as significantly different.

In a second model (C statistic 0.84), we included month as a covariate and calculated average marginal effects (AME) for each time period by using the margins library in R,⁷ which uses a discrete first-difference in predicted outcomes to obtain the AME. The average marginal effect represents the percentage point difference between the reference period (March) and a subsequent time period in probability of death or discharge to hospice, for equivalent patients. We obtained lower and upper confidence intervals for the AME using a bootstrapping approach described in Green.⁸ Finally, we conducted two sensitivity analyses: one, restricting the analysis to only those patients with principal diagnosis of COVID-19, sepsis, or respiratory disease (see Appendix A for complete list of codes) and one restricting the analysis to only those with length of stay of at least 3 days.

All statistical analyses were conducted with R, version 4.0.2. All analyses used 2-sided statistical tests, and we considered a P value < .05 to be statistically significant without adjustment for multiple testing. The NYU institutional review board approved the study and granted a waiver of consent and a waiver of the Health Information Portability and Accountability Act.

We included 5,121 hospitalizations, of which 5,118 (99.94%) had known outcomes (death or hospital discharge). Peak hospitalizations occurred in late March to mid-April, which accounted for 53% of the hospitalizations. Median length of stay for patients who died or were discharged to hospice was 8 days (interquartile range, 4-15; max 140 days). The median age and the proportion male or with any comorbidity decreased over time (Table). For instance, the proportion with any chronic condition decreased from 81% in March to 72% in August.

Adjusted mortality dropped each month, from 25.6% in March to 7.6% in August (Table and Figure). The SMR declined progressively over time, from 1.26 (95% CI, 1.15-1.39) in March to 0.38 (95% CI, 0.12-0.88) in August (Table). The adjusted average marginal effect was also significantly lower than in March in every subsequent month, reaching a maximum of an average 18.2 (95% CI, 12.0-24.4) percentage point decrease in probability of death in August, accounting for changes in demographics and clinical severity (Table and Appendix B). The decrease in unadjusted mortality over time was observed across age groups (Appendix C).

Results of the two sensitivity analyses were similar (Appendices D and E), though attenuated in the case of the sepsis/respiratory cohort, with adjusted mortality falling from 31.4% to 14.4%, SMR decreasing from 1.28 (95% CI, 1.16-1.41) to 0.59 (95% CI, 0.16-1.50), and AME in August 17.0 percentage points (95% CI, 6.0-28.1).

In this study of COVID-19 mortality over 6 months at a single health system, we found that changes in demographics and severity of illness at presentation did not fully explain decreases in mortality seen over time. Even after risk adjustment for a variety of clinical and demographic factors, including severity of illness at presentation, mortality was significantly and progressively lower over the course of the study period.

Similar risk-adjusted results have been preliminarily reported among intensive care unit patients in a preprint from the United Kingdom.⁹ Incremental improvements in outcomes are likely a combination of increasing clinical experience, decreasing hospital volume, growing use of new pharmacologic treatments (such as systemic corticosteroids,¹⁰ remdesivir,¹¹ and anticytokine treatments), nonpharmacologic treatments (such as placing the patient in the prone position, or proning, rather than on their back), earlier intervention, community awareness, and, potentially, lower viral load exposure from increased mask wearing and social distancing.¹²

Strengths of this study include highly detailed electronic health record data on hospitalizations at three different hospitals, a diverse patient population,⁶ near-complete study outcomes, and a lengthy period of investigation of 6 months. However, this study does have limitations. All patients were from a single geographic region and treated within a single health system, though restricting data to one system reduces institution-level variability and allows us to assess how care may have evolved with growing experience. Aggregating data from numerous health systems that might be at different stages of local outbreaks, provide different quality of care, and contribute different numbers of patients in each period introduces its own biases. We were also unable to disentangle different potential explanatory factors given the observational nature of the study. Residual confounding, such as a higher proportion of particularly frail patients admitted in earlier periods, is also a possibility, though the fact that we observed declines across all age groups mitigates this concern. Thresholds for hospital admission may also have changed over time with less severely ill patients being admitted in the later time periods. While changing admission thresholds could have contributed to higher survival rates in the latter portions of the study, our inclusion of several highly predictive clinical and laboratory results likely captured many aspects of disease severity.

In summary, data from one health system suggest that COVID-19 remains a serious disease for high-risk patients, but that mortality rates are improving over time.

Early reports showed high mortality from coronavirus disease 2019 (COVID-19), while current United States data mortality rates are lower, raising hope that new treatments and management strategies have improved outcomes. For instance, Centers for Disease Control and Prevention data show that 6.7% of cases resulted in death in April, compared with 1.9% in September.¹ However, the demographics of those infected have also changed, and more available testing may mean more comprehensive identification and earlier treatment. Nationally, for instance, the median age of confirmed cases was 38 years at the end of August, down from 46 years at the start of May.² Therefore, whether decreasing COVID-19 mortality rates simply reflect changing demographics or represent actual improvements in clinical care is unknown. The objective of this analysis was to assess outcomes over time in a single health system, accounting for changes in demographics, clinical factors, and severity of disease at presentation.

We analyzed monthly mortality rates for admissions between March 1 and August 31, 2020, in a single health system in New York City. Outcomes were obtained as of October 8, 2020. We included all hospitalizations of people 18 years and older with laboratory-confirmed severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection identified during the hospitalization or in the prior 2 weeks, excluding those admitted to hospice care. Patients with multiple hospitalizations (N=208 patients, 229 hospitalizations, 4.4%) were included repeatedly if they continued to have laboratory-confirmed disease. Patients without admission vital signs (N=28) were excluded. Mortality was defined as in-hospital death or discharge to hospice care. In-house laboratory testing began March 16 and all inpatients were tested for SARS-CoV-2 by April 1; elective surgeries resumed May 4-11 and were only conducted on confirmed SARS-CoV-2–negative patients.

All data were obtained from the electronic health record (Epic Systems, Verona, Wisconsin). Diagnosis codes were obtained from the problem list, past medical history, and billing codes. In addition, we used objective data such as hemoglobin A1c, ejection fraction, outpatient creatinine, and outpatient blood pressure to augment problem list diagnoses where relevant.

Based on prior literature, we constructed multivariable logistic regression models for mortality adjusting for age; sex; self-reported race and ethnicity; body mass index; smoking history; presence of hypertension, heart failure, hyperlipidemia, coronary artery disease, diabetes, cancer, chronic kidney disease, dementia, or pulmonary disease individually as dummy variables; and admission oxygen saturation, D-dimer, ferritin, and C-reactive protein.^3-6 In the first model (C statistic 0.82), we did not include month of admission as a covariate and calculated the ratio of the sum of observed and expected deaths (obtained from the model) in each month to obtain the standardized mortality ratio (SMR) for each month. We then multiplied each period’s SMR by the overall average crude mortality to generate monthly adjusted mortality rates. We calculated Poisson control limits and indicated points outside the control limits as significantly different.

In a second model (C statistic 0.84), we included month as a covariate and calculated average marginal effects (AME) for each time period by using the margins library in R,⁷ which uses a discrete first-difference in predicted outcomes to obtain the AME. The average marginal effect represents the percentage point difference between the reference period (March) and a subsequent time period in probability of death or discharge to hospice, for equivalent patients. We obtained lower and upper confidence intervals for the AME using a bootstrapping approach described in Green.⁸ Finally, we conducted two sensitivity analyses: one, restricting the analysis to only those patients with principal diagnosis of COVID-19, sepsis, or respiratory disease (see Appendix A for complete list of codes) and one restricting the analysis to only those with length of stay of at least 3 days.

All statistical analyses were conducted with R, version 4.0.2. All analyses used 2-sided statistical tests, and we considered a P value < .05 to be statistically significant without adjustment for multiple testing. The NYU institutional review board approved the study and granted a waiver of consent and a waiver of the Health Information Portability and Accountability Act.

We included 5,121 hospitalizations, of which 5,118 (99.94%) had known outcomes (death or hospital discharge). Peak hospitalizations occurred in late March to mid-April, which accounted for 53% of the hospitalizations. Median length of stay for patients who died or were discharged to hospice was 8 days (interquartile range, 4-15; max 140 days). The median age and the proportion male or with any comorbidity decreased over time (Table). For instance, the proportion with any chronic condition decreased from 81% in March to 72% in August.

Adjusted mortality dropped each month, from 25.6% in March to 7.6% in August (Table and Figure). The SMR declined progressively over time, from 1.26 (95% CI, 1.15-1.39) in March to 0.38 (95% CI, 0.12-0.88) in August (Table). The adjusted average marginal effect was also significantly lower than in March in every subsequent month, reaching a maximum of an average 18.2 (95% CI, 12.0-24.4) percentage point decrease in probability of death in August, accounting for changes in demographics and clinical severity (Table and Appendix B). The decrease in unadjusted mortality over time was observed across age groups (Appendix C).

Results of the two sensitivity analyses were similar (Appendices D and E), though attenuated in the case of the sepsis/respiratory cohort, with adjusted mortality falling from 31.4% to 14.4%, SMR decreasing from 1.28 (95% CI, 1.16-1.41) to 0.59 (95% CI, 0.16-1.50), and AME in August 17.0 percentage points (95% CI, 6.0-28.1).

In this study of COVID-19 mortality over 6 months at a single health system, we found that changes in demographics and severity of illness at presentation did not fully explain decreases in mortality seen over time. Even after risk adjustment for a variety of clinical and demographic factors, including severity of illness at presentation, mortality was significantly and progressively lower over the course of the study period.

Similar risk-adjusted results have been preliminarily reported among intensive care unit patients in a preprint from the United Kingdom.⁹ Incremental improvements in outcomes are likely a combination of increasing clinical experience, decreasing hospital volume, growing use of new pharmacologic treatments (such as systemic corticosteroids,¹⁰ remdesivir,¹¹ and anticytokine treatments), nonpharmacologic treatments (such as placing the patient in the prone position, or proning, rather than on their back), earlier intervention, community awareness, and, potentially, lower viral load exposure from increased mask wearing and social distancing.¹²

Strengths of this study include highly detailed electronic health record data on hospitalizations at three different hospitals, a diverse patient population,⁶ near-complete study outcomes, and a lengthy period of investigation of 6 months. However, this study does have limitations. All patients were from a single geographic region and treated within a single health system, though restricting data to one system reduces institution-level variability and allows us to assess how care may have evolved with growing experience. Aggregating data from numerous health systems that might be at different stages of local outbreaks, provide different quality of care, and contribute different numbers of patients in each period introduces its own biases. We were also unable to disentangle different potential explanatory factors given the observational nature of the study. Residual confounding, such as a higher proportion of particularly frail patients admitted in earlier periods, is also a possibility, though the fact that we observed declines across all age groups mitigates this concern. Thresholds for hospital admission may also have changed over time with less severely ill patients being admitted in the later time periods. While changing admission thresholds could have contributed to higher survival rates in the latter portions of the study, our inclusion of several highly predictive clinical and laboratory results likely captured many aspects of disease severity.

In summary, data from one health system suggest that COVID-19 remains a serious disease for high-risk patients, but that mortality rates are improving over time.

Early reports showed high mortality from coronavirus disease 2019 (COVID-19), while current United States data mortality rates are lower, raising hope that new treatments and management strategies have improved outcomes. For instance, Centers for Disease Control and Prevention data show that 6.7% of cases resulted in death in April, compared with 1.9% in September.¹ However, the demographics of those infected have also changed, and more available testing may mean more comprehensive identification and earlier treatment. Nationally, for instance, the median age of confirmed cases was 38 years at the end of August, down from 46 years at the start of May.² Therefore, whether decreasing COVID-19 mortality rates simply reflect changing demographics or represent actual improvements in clinical care is unknown. The objective of this analysis was to assess outcomes over time in a single health system, accounting for changes in demographics, clinical factors, and severity of disease at presentation.

We analyzed monthly mortality rates for admissions between March 1 and August 31, 2020, in a single health system in New York City. Outcomes were obtained as of October 8, 2020. We included all hospitalizations of people 18 years and older with laboratory-confirmed severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection identified during the hospitalization or in the prior 2 weeks, excluding those admitted to hospice care. Patients with multiple hospitalizations (N=208 patients, 229 hospitalizations, 4.4%) were included repeatedly if they continued to have laboratory-confirmed disease. Patients without admission vital signs (N=28) were excluded. Mortality was defined as in-hospital death or discharge to hospice care. In-house laboratory testing began March 16 and all inpatients were tested for SARS-CoV-2 by April 1; elective surgeries resumed May 4-11 and were only conducted on confirmed SARS-CoV-2–negative patients.

All data were obtained from the electronic health record (Epic Systems, Verona, Wisconsin). Diagnosis codes were obtained from the problem list, past medical history, and billing codes. In addition, we used objective data such as hemoglobin A1c, ejection fraction, outpatient creatinine, and outpatient blood pressure to augment problem list diagnoses where relevant.

Based on prior literature, we constructed multivariable logistic regression models for mortality adjusting for age; sex; self-reported race and ethnicity; body mass index; smoking history; presence of hypertension, heart failure, hyperlipidemia, coronary artery disease, diabetes, cancer, chronic kidney disease, dementia, or pulmonary disease individually as dummy variables; and admission oxygen saturation, D-dimer, ferritin, and C-reactive protein.^3-6 In the first model (C statistic 0.82), we did not include month of admission as a covariate and calculated the ratio of the sum of observed and expected deaths (obtained from the model) in each month to obtain the standardized mortality ratio (SMR) for each month. We then multiplied each period’s SMR by the overall average crude mortality to generate monthly adjusted mortality rates. We calculated Poisson control limits and indicated points outside the control limits as significantly different.

In a second model (C statistic 0.84), we included month as a covariate and calculated average marginal effects (AME) for each time period by using the margins library in R,⁷ which uses a discrete first-difference in predicted outcomes to obtain the AME. The average marginal effect represents the percentage point difference between the reference period (March) and a subsequent time period in probability of death or discharge to hospice, for equivalent patients. We obtained lower and upper confidence intervals for the AME using a bootstrapping approach described in Green.⁸ Finally, we conducted two sensitivity analyses: one, restricting the analysis to only those patients with principal diagnosis of COVID-19, sepsis, or respiratory disease (see Appendix A for complete list of codes) and one restricting the analysis to only those with length of stay of at least 3 days.

All statistical analyses were conducted with R, version 4.0.2. All analyses used 2-sided statistical tests, and we considered a P value < .05 to be statistically significant without adjustment for multiple testing. The NYU institutional review board approved the study and granted a waiver of consent and a waiver of the Health Information Portability and Accountability Act.

We included 5,121 hospitalizations, of which 5,118 (99.94%) had known outcomes (death or hospital discharge). Peak hospitalizations occurred in late March to mid-April, which accounted for 53% of the hospitalizations. Median length of stay for patients who died or were discharged to hospice was 8 days (interquartile range, 4-15; max 140 days). The median age and the proportion male or with any comorbidity decreased over time (Table). For instance, the proportion with any chronic condition decreased from 81% in March to 72% in August.

Adjusted mortality dropped each month, from 25.6% in March to 7.6% in August (Table and Figure). The SMR declined progressively over time, from 1.26 (95% CI, 1.15-1.39) in March to 0.38 (95% CI, 0.12-0.88) in August (Table). The adjusted average marginal effect was also significantly lower than in March in every subsequent month, reaching a maximum of an average 18.2 (95% CI, 12.0-24.4) percentage point decrease in probability of death in August, accounting for changes in demographics and clinical severity (Table and Appendix B). The decrease in unadjusted mortality over time was observed across age groups (Appendix C).

Results of the two sensitivity analyses were similar (Appendices D and E), though attenuated in the case of the sepsis/respiratory cohort, with adjusted mortality falling from 31.4% to 14.4%, SMR decreasing from 1.28 (95% CI, 1.16-1.41) to 0.59 (95% CI, 0.16-1.50), and AME in August 17.0 percentage points (95% CI, 6.0-28.1).

In this study of COVID-19 mortality over 6 months at a single health system, we found that changes in demographics and severity of illness at presentation did not fully explain decreases in mortality seen over time. Even after risk adjustment for a variety of clinical and demographic factors, including severity of illness at presentation, mortality was significantly and progressively lower over the course of the study period.

Similar risk-adjusted results have been preliminarily reported among intensive care unit patients in a preprint from the United Kingdom.⁹ Incremental improvements in outcomes are likely a combination of increasing clinical experience, decreasing hospital volume, growing use of new pharmacologic treatments (such as systemic corticosteroids,¹⁰ remdesivir,¹¹ and anticytokine treatments), nonpharmacologic treatments (such as placing the patient in the prone position, or proning, rather than on their back), earlier intervention, community awareness, and, potentially, lower viral load exposure from increased mask wearing and social distancing.¹²

Strengths of this study include highly detailed electronic health record data on hospitalizations at three different hospitals, a diverse patient population,⁶ near-complete study outcomes, and a lengthy period of investigation of 6 months. However, this study does have limitations. All patients were from a single geographic region and treated within a single health system, though restricting data to one system reduces institution-level variability and allows us to assess how care may have evolved with growing experience. Aggregating data from numerous health systems that might be at different stages of local outbreaks, provide different quality of care, and contribute different numbers of patients in each period introduces its own biases. We were also unable to disentangle different potential explanatory factors given the observational nature of the study. Residual confounding, such as a higher proportion of particularly frail patients admitted in earlier periods, is also a possibility, though the fact that we observed declines across all age groups mitigates this concern. Thresholds for hospital admission may also have changed over time with less severely ill patients being admitted in the later time periods. While changing admission thresholds could have contributed to higher survival rates in the latter portions of the study, our inclusion of several highly predictive clinical and laboratory results likely captured many aspects of disease severity.

In summary, data from one health system suggest that COVID-19 remains a serious disease for high-risk patients, but that mortality rates are improving over time.

Continuous pulse oximetry monitoring in children with bronchiolitis who don’t require supplemental oxygen is discouraged by practice guidelines and is recognized as a form of medical overuse.^1-3 This practice can be associated with negative outcomes, including prolonged length of stay,^4-6 increased cost of hospitalization,⁷ and alarm fatigue among nurses.⁸ Despite initiatives to reduce continuous pulse oximetry monitoring in stable patients with bronchiolitis,^1,2 wide practice variation exists between hospitals.^9,10 Previous studies have shown that higher prevalence of inpatient bronchiolitis admissions is associated with decreased utilization of unnecessary interventions.¹¹ However, the relationship between pulse oximetry use and bronchiolitis prevalence has not been studied. The objective of this study is to test the hypothesis that hospital units with lower proportions of patients admitted for bronchiolitis and those with fewer general pediatrics patients relative to subspecialty patients would have higher rates of pulse oximetry overuse.

Study Design

We conducted a substudy of the Pediatric Research in Inpatient Settings (PRIS) Network’s Eliminating Monitoring Overuse (EMO) pulse oximetry study,^10,12 a 56-hospital cross-sectional study that used direct observation to measure the prevalence of continuous pulse oximetry monitoring in hospitalized infants with bronchiolitis who did not require supplemental oxygen between December 1, 2018, through March 31, 2019. This substudy was not included as part of the original aims of the project and was proposed as a separate analysis during data collection. For US sites, the Institutional Review Board (IRB) at Children’s Hospital of Philadelphia approved the study and served as the central IRB. The Research Ethics Board at University of Calgary also approved the study.

Site Selection

Hospitals with at least 60 observations were eligible for inclusion. Of the 32 hospitals that conducted the minimum observations, 25 agreed to participate (21 free-standing children’s hospitals, 3 children’s hospitals within general hospitals, and 1 community hospital).

Patient Population

The parent study included patients aged 8 weeks through 23 months with a primary diagnosis of bronchiolitis. Patients were included only if they were not receiving supplemental oxygen or nasal cannula flow at the time of data collection. The inclusion and exclusion criteria were for both the parent study and the substudy. Further inclusion and exclusion criteria have been described previously.^10,12

Data Collection

In order to ascertain continuous pulse oximetry monitoring status, staff at each hospital performed observational rounds by walking to the bedside of each patient who met inclusion criteria. Additional methodology for the parent study has been published elsewhere.^10,12

Bronchiolitis Admission Volume by Unit

Collaborators at each hospital gathered bronchiolitis census data from each unit that admitted patients with bronchiolitis. Units were identified prior to data collection and were characterized at the institution level based on previous local definitions. Each site was responsible for using institution-specific data collection methods for determining bronchiolitis and total admissions on each unit (eg, departmental reports or directly querying admissions data using International Classification of Diseases, Tenth Revision, diagnosis codes for bronchiolitis) over the same period as the parent study. Following data analysis, bronchiolitis admission burden was classified into five categories, based on less than 10%, 10% to less than 20%, 20% to less than 30%, 30% to less than 40%, or 40% or more of total admissions having a primary discharge diagnosis of bronchiolitis during the study period. This categorization allowed investigators to determine whether there was a dose-dependent response among categories.

Unit Composition

Site investigators also completed a survey identifying which patients were admitted to each unit (eg, general pediatrics only, medical subspecialty, surgical). Based on these results, units were further classified into seven types (Appendix Table). For the final analysis, units caring exclusively for general pediatrics patients were compared to all other unit types.

Analysis

Bronchiolitis admission burden and unit composition data were combined with observations of pulse oximetry monitoring use of patients not requiring supplemental oxygen from the parent study. We determined unadjusted observed monitoring proportions for each unit’s bronchiolitis admission burden category across all 25 hospitals. This was calculated as a simple proportion of the total number of observations during which patients were continuously monitored divided by the total number of observations performed within each unit’s admission category. We then calculated unadjusted odds ratios using the 40% and higher bronchiolitis admission burden category as a reference. We calculated similar proportions and odds ratios for the dichotomous unit composition variable. Next, we used mixed-effects logistic regression with a random intercept for each hospital to allow for differences in baseline monitoring rates, which varied widely between hospitals (2% to 92%),¹⁰ to calculate adjusted odds ratios for the unit’s admission category and unit’s composition. We also adjusted for the same covariates used in the primary study’s analysis (Table).¹⁰

We analyzed 2,366 observations of bronchiolitis patients from 25 hospitals. Most observations were concentrated in freestanding children’s hospitals (89%), and 50% were from hospitals with more than 250 pediatric beds. Observations were well distributed among the five categories of admit burden (Table).

In unadjusted regression, the relationship between admission burden and rate of pulse oximetry use did not appear to be dose-dependent, and 95% CIs were wide. We then analyzed the data accounting for baseline differences in hospital monitoring rates and adjusted for the covariates significantly associated with continuous pulse oximetry monitoring in the primary study’s analysis with use of a mixed-effects model. As shown in the Table, low-burden units in which bronchiolitis constituted less than 10% of total admissions had a 2.16-fold increased odds of unnecessary pulse oximetry monitoring compared to high-burden units in which bronchiolitis constituted 40% or more of total admissions (95% CI, 1.27-3.69; P = .01).

In examining the subspecialty unit composition, 596 observations (25.2%) were conducted on units exclusively caring for general pediatrics patients. In the mixed-effects model adjusted for bronchiolitis admission burden and the covariates used in the study’s primary analysis, units exclusively caring for general pediatrics patients did not have significantly different independent odds of pulse oximetry monitoring use compared to units with a mixed patient population (OR 1.01; 95% CI, 0.71-1.45; P = .95) (Appendix Table).

In this multicenter observational study of children hospitalized with bronchiolitis not concurrently receiving supplemental oxygen, units that only occasionally cared for bronchiolitis patients appeared to be more likely to overuse continuous pulse oximetry during bronchiolitis hospitalizations.

This finding was not immediately apparent when examining the raw data because of wide hospital-level variation in continuous pulse oximetry monitoring use. However, when the high degree of hospital-level variation in baseline overuse was accounted for with use of a random intercept for each hospital in the mixed-effects model, units that cared for higher proportions of bronchiolitis patients had significantly lower odds of continuous pulse oximetry monitoring use compared to units that cared for these infants infrequently.

As many institutions have subspecialized units to cultivate nursing expertise for care of certain diseases and patient populations, we hypothesized that units caring primarily for children on general pediatrics units would also have lower rates of monitoring overuse compared to mixed units. Interestingly, these units did not perform better, likely because potential cultural factors that might contribute to differences in monitoring are accounted for by bronchiolitis admission burden.

Our findings build on prior literature by demonstrating that unit-level, as well as hospital-level, factors appear to drive overuse in healthcare. A prior single-site retrospective cohort study demonstrated an association between higher prevalence of inpatient bronchiolitis and decreased use of unnecessary interventions such as laboratory and radiographic testing, as well as steroid and antibiotic administration.¹¹ Although study of the relationship between volume and quality is not new to healthcare, to our knowledge, this study is the first to examine the relationship between pulse oximetry overuse in bronchiolitis and unit-level factors like admission burden and subspecialty composition.

There are several limitations. First, because the study population included only children not receiving supplemental oxygen, both the parent study and this substudy assumed that all observed use of pulse oximetry monitoring was overuse. In some cases, however, there may have been other compelling clinical reasons, institutional policies, or differences in pulse oximetry availability that were not captured during data collection or in our adjusted model. Second, hospitals used convenience sampling. It is possible this resulted in samples that were not representative of each unit’s underlying patient population or monitoring practice. In addition, not all of the 32 eligible sites were able to provide data related to hospital admissions at the unit level and thus are not included in our analysis. This remains a potential source of hospital-level selection bias.

These findings demonstrate that high bronchiolitis admission burden correlates with lower rates of unnecessary pulse oximetry monitoring in bronchiolitis. We speculate that these outcomes might reflect differing degrees of nursing comfort, expertise, and unit-level norms in caring for bronchiolitis patients, although our study was not designed to establish underlying causes. Identification of operating principles that underpin low pulse oximetry monitoring on high-burden units will provide guidance for decreasing unnecessary monitoring and will inform future studies seeking ways to discourage continuous pulse oximetry monitoring in low-risk infants. Given the institutional variation in monitoring rates, future studies examining both institution-wide and unit-level interventions will be necessary to decrease unnecessary pulse oximetry monitoring in bronchiolitis. Furthermore, these findings may be relevant to studying care quality in other disease processes, with bronchiolitis serving as a model illness for overuse.

The authors acknowledge the National Heart, Lung, and Blood Institute of the National Institutes of Health scientists who contributed their expertise to this project as part of the U01 Cooperative Agreement funding mechanism as federal employees conducting their official job duties: Lora Reineck, MD, MS, Karen Bienstock, MS, and Cheryl Boyce, PhD. The authors thank the executive council of the Pediatric Research in Inpatient Settings Network for their contributions to the early scientific development of this project. The network assessed a Collaborative Support Fee for access to the hospitals and support of this project.

The authors thank the PRIS Network collaborators for their major contributions to data collection (see Appendix).

Continuous pulse oximetry monitoring in children with bronchiolitis who don’t require supplemental oxygen is discouraged by practice guidelines and is recognized as a form of medical overuse.^1-3 This practice can be associated with negative outcomes, including prolonged length of stay,^4-6 increased cost of hospitalization,⁷ and alarm fatigue among nurses.⁸ Despite initiatives to reduce continuous pulse oximetry monitoring in stable patients with bronchiolitis,^1,2 wide practice variation exists between hospitals.^9,10 Previous studies have shown that higher prevalence of inpatient bronchiolitis admissions is associated with decreased utilization of unnecessary interventions.¹¹ However, the relationship between pulse oximetry use and bronchiolitis prevalence has not been studied. The objective of this study is to test the hypothesis that hospital units with lower proportions of patients admitted for bronchiolitis and those with fewer general pediatrics patients relative to subspecialty patients would have higher rates of pulse oximetry overuse.

Study Design

We conducted a substudy of the Pediatric Research in Inpatient Settings (PRIS) Network’s Eliminating Monitoring Overuse (EMO) pulse oximetry study,^10,12 a 56-hospital cross-sectional study that used direct observation to measure the prevalence of continuous pulse oximetry monitoring in hospitalized infants with bronchiolitis who did not require supplemental oxygen between December 1, 2018, through March 31, 2019. This substudy was not included as part of the original aims of the project and was proposed as a separate analysis during data collection. For US sites, the Institutional Review Board (IRB) at Children’s Hospital of Philadelphia approved the study and served as the central IRB. The Research Ethics Board at University of Calgary also approved the study.

Site Selection

Hospitals with at least 60 observations were eligible for inclusion. Of the 32 hospitals that conducted the minimum observations, 25 agreed to participate (21 free-standing children’s hospitals, 3 children’s hospitals within general hospitals, and 1 community hospital).

Patient Population

The parent study included patients aged 8 weeks through 23 months with a primary diagnosis of bronchiolitis. Patients were included only if they were not receiving supplemental oxygen or nasal cannula flow at the time of data collection. The inclusion and exclusion criteria were for both the parent study and the substudy. Further inclusion and exclusion criteria have been described previously.^10,12

Data Collection

In order to ascertain continuous pulse oximetry monitoring status, staff at each hospital performed observational rounds by walking to the bedside of each patient who met inclusion criteria. Additional methodology for the parent study has been published elsewhere.^10,12

Bronchiolitis Admission Volume by Unit

Collaborators at each hospital gathered bronchiolitis census data from each unit that admitted patients with bronchiolitis. Units were identified prior to data collection and were characterized at the institution level based on previous local definitions. Each site was responsible for using institution-specific data collection methods for determining bronchiolitis and total admissions on each unit (eg, departmental reports or directly querying admissions data using International Classification of Diseases, Tenth Revision, diagnosis codes for bronchiolitis) over the same period as the parent study. Following data analysis, bronchiolitis admission burden was classified into five categories, based on less than 10%, 10% to less than 20%, 20% to less than 30%, 30% to less than 40%, or 40% or more of total admissions having a primary discharge diagnosis of bronchiolitis during the study period. This categorization allowed investigators to determine whether there was a dose-dependent response among categories.

Unit Composition

Site investigators also completed a survey identifying which patients were admitted to each unit (eg, general pediatrics only, medical subspecialty, surgical). Based on these results, units were further classified into seven types (Appendix Table). For the final analysis, units caring exclusively for general pediatrics patients were compared to all other unit types.

Analysis

Bronchiolitis admission burden and unit composition data were combined with observations of pulse oximetry monitoring use of patients not requiring supplemental oxygen from the parent study. We determined unadjusted observed monitoring proportions for each unit’s bronchiolitis admission burden category across all 25 hospitals. This was calculated as a simple proportion of the total number of observations during which patients were continuously monitored divided by the total number of observations performed within each unit’s admission category. We then calculated unadjusted odds ratios using the 40% and higher bronchiolitis admission burden category as a reference. We calculated similar proportions and odds ratios for the dichotomous unit composition variable. Next, we used mixed-effects logistic regression with a random intercept for each hospital to allow for differences in baseline monitoring rates, which varied widely between hospitals (2% to 92%),¹⁰ to calculate adjusted odds ratios for the unit’s admission category and unit’s composition. We also adjusted for the same covariates used in the primary study’s analysis (Table).¹⁰

We analyzed 2,366 observations of bronchiolitis patients from 25 hospitals. Most observations were concentrated in freestanding children’s hospitals (89%), and 50% were from hospitals with more than 250 pediatric beds. Observations were well distributed among the five categories of admit burden (Table).

In unadjusted regression, the relationship between admission burden and rate of pulse oximetry use did not appear to be dose-dependent, and 95% CIs were wide. We then analyzed the data accounting for baseline differences in hospital monitoring rates and adjusted for the covariates significantly associated with continuous pulse oximetry monitoring in the primary study’s analysis with use of a mixed-effects model. As shown in the Table, low-burden units in which bronchiolitis constituted less than 10% of total admissions had a 2.16-fold increased odds of unnecessary pulse oximetry monitoring compared to high-burden units in which bronchiolitis constituted 40% or more of total admissions (95% CI, 1.27-3.69; P = .01).

In examining the subspecialty unit composition, 596 observations (25.2%) were conducted on units exclusively caring for general pediatrics patients. In the mixed-effects model adjusted for bronchiolitis admission burden and the covariates used in the study’s primary analysis, units exclusively caring for general pediatrics patients did not have significantly different independent odds of pulse oximetry monitoring use compared to units with a mixed patient population (OR 1.01; 95% CI, 0.71-1.45; P = .95) (Appendix Table).

In this multicenter observational study of children hospitalized with bronchiolitis not concurrently receiving supplemental oxygen, units that only occasionally cared for bronchiolitis patients appeared to be more likely to overuse continuous pulse oximetry during bronchiolitis hospitalizations.

This finding was not immediately apparent when examining the raw data because of wide hospital-level variation in continuous pulse oximetry monitoring use. However, when the high degree of hospital-level variation in baseline overuse was accounted for with use of a random intercept for each hospital in the mixed-effects model, units that cared for higher proportions of bronchiolitis patients had significantly lower odds of continuous pulse oximetry monitoring use compared to units that cared for these infants infrequently.

As many institutions have subspecialized units to cultivate nursing expertise for care of certain diseases and patient populations, we hypothesized that units caring primarily for children on general pediatrics units would also have lower rates of monitoring overuse compared to mixed units. Interestingly, these units did not perform better, likely because potential cultural factors that might contribute to differences in monitoring are accounted for by bronchiolitis admission burden.

Our findings build on prior literature by demonstrating that unit-level, as well as hospital-level, factors appear to drive overuse in healthcare. A prior single-site retrospective cohort study demonstrated an association between higher prevalence of inpatient bronchiolitis and decreased use of unnecessary interventions such as laboratory and radiographic testing, as well as steroid and antibiotic administration.¹¹ Although study of the relationship between volume and quality is not new to healthcare, to our knowledge, this study is the first to examine the relationship between pulse oximetry overuse in bronchiolitis and unit-level factors like admission burden and subspecialty composition.

There are several limitations. First, because the study population included only children not receiving supplemental oxygen, both the parent study and this substudy assumed that all observed use of pulse oximetry monitoring was overuse. In some cases, however, there may have been other compelling clinical reasons, institutional policies, or differences in pulse oximetry availability that were not captured during data collection or in our adjusted model. Second, hospitals used convenience sampling. It is possible this resulted in samples that were not representative of each unit’s underlying patient population or monitoring practice. In addition, not all of the 32 eligible sites were able to provide data related to hospital admissions at the unit level and thus are not included in our analysis. This remains a potential source of hospital-level selection bias.

These findings demonstrate that high bronchiolitis admission burden correlates with lower rates of unnecessary pulse oximetry monitoring in bronchiolitis. We speculate that these outcomes might reflect differing degrees of nursing comfort, expertise, and unit-level norms in caring for bronchiolitis patients, although our study was not designed to establish underlying causes. Identification of operating principles that underpin low pulse oximetry monitoring on high-burden units will provide guidance for decreasing unnecessary monitoring and will inform future studies seeking ways to discourage continuous pulse oximetry monitoring in low-risk infants. Given the institutional variation in monitoring rates, future studies examining both institution-wide and unit-level interventions will be necessary to decrease unnecessary pulse oximetry monitoring in bronchiolitis. Furthermore, these findings may be relevant to studying care quality in other disease processes, with bronchiolitis serving as a model illness for overuse.

The authors acknowledge the National Heart, Lung, and Blood Institute of the National Institutes of Health scientists who contributed their expertise to this project as part of the U01 Cooperative Agreement funding mechanism as federal employees conducting their official job duties: Lora Reineck, MD, MS, Karen Bienstock, MS, and Cheryl Boyce, PhD. The authors thank the executive council of the Pediatric Research in Inpatient Settings Network for their contributions to the early scientific development of this project. The network assessed a Collaborative Support Fee for access to the hospitals and support of this project.

The authors thank the PRIS Network collaborators for their major contributions to data collection (see Appendix).

Continuous pulse oximetry monitoring in children with bronchiolitis who don’t require supplemental oxygen is discouraged by practice guidelines and is recognized as a form of medical overuse.^1-3 This practice can be associated with negative outcomes, including prolonged length of stay,^4-6 increased cost of hospitalization,⁷ and alarm fatigue among nurses.⁸ Despite initiatives to reduce continuous pulse oximetry monitoring in stable patients with bronchiolitis,^1,2 wide practice variation exists between hospitals.^9,10 Previous studies have shown that higher prevalence of inpatient bronchiolitis admissions is associated with decreased utilization of unnecessary interventions.¹¹ However, the relationship between pulse oximetry use and bronchiolitis prevalence has not been studied. The objective of this study is to test the hypothesis that hospital units with lower proportions of patients admitted for bronchiolitis and those with fewer general pediatrics patients relative to subspecialty patients would have higher rates of pulse oximetry overuse.

Study Design

We conducted a substudy of the Pediatric Research in Inpatient Settings (PRIS) Network’s Eliminating Monitoring Overuse (EMO) pulse oximetry study,^10,12 a 56-hospital cross-sectional study that used direct observation to measure the prevalence of continuous pulse oximetry monitoring in hospitalized infants with bronchiolitis who did not require supplemental oxygen between December 1, 2018, through March 31, 2019. This substudy was not included as part of the original aims of the project and was proposed as a separate analysis during data collection. For US sites, the Institutional Review Board (IRB) at Children’s Hospital of Philadelphia approved the study and served as the central IRB. The Research Ethics Board at University of Calgary also approved the study.

Site Selection

Hospitals with at least 60 observations were eligible for inclusion. Of the 32 hospitals that conducted the minimum observations, 25 agreed to participate (21 free-standing children’s hospitals, 3 children’s hospitals within general hospitals, and 1 community hospital).

Patient Population

The parent study included patients aged 8 weeks through 23 months with a primary diagnosis of bronchiolitis. Patients were included only if they were not receiving supplemental oxygen or nasal cannula flow at the time of data collection. The inclusion and exclusion criteria were for both the parent study and the substudy. Further inclusion and exclusion criteria have been described previously.^10,12

Data Collection

In order to ascertain continuous pulse oximetry monitoring status, staff at each hospital performed observational rounds by walking to the bedside of each patient who met inclusion criteria. Additional methodology for the parent study has been published elsewhere.^10,12

Bronchiolitis Admission Volume by Unit

Collaborators at each hospital gathered bronchiolitis census data from each unit that admitted patients with bronchiolitis. Units were identified prior to data collection and were characterized at the institution level based on previous local definitions. Each site was responsible for using institution-specific data collection methods for determining bronchiolitis and total admissions on each unit (eg, departmental reports or directly querying admissions data using International Classification of Diseases, Tenth Revision, diagnosis codes for bronchiolitis) over the same period as the parent study. Following data analysis, bronchiolitis admission burden was classified into five categories, based on less than 10%, 10% to less than 20%, 20% to less than 30%, 30% to less than 40%, or 40% or more of total admissions having a primary discharge diagnosis of bronchiolitis during the study period. This categorization allowed investigators to determine whether there was a dose-dependent response among categories.

Unit Composition

Site investigators also completed a survey identifying which patients were admitted to each unit (eg, general pediatrics only, medical subspecialty, surgical). Based on these results, units were further classified into seven types (Appendix Table). For the final analysis, units caring exclusively for general pediatrics patients were compared to all other unit types.

Analysis

Bronchiolitis admission burden and unit composition data were combined with observations of pulse oximetry monitoring use of patients not requiring supplemental oxygen from the parent study. We determined unadjusted observed monitoring proportions for each unit’s bronchiolitis admission burden category across all 25 hospitals. This was calculated as a simple proportion of the total number of observations during which patients were continuously monitored divided by the total number of observations performed within each unit’s admission category. We then calculated unadjusted odds ratios using the 40% and higher bronchiolitis admission burden category as a reference. We calculated similar proportions and odds ratios for the dichotomous unit composition variable. Next, we used mixed-effects logistic regression with a random intercept for each hospital to allow for differences in baseline monitoring rates, which varied widely between hospitals (2% to 92%),¹⁰ to calculate adjusted odds ratios for the unit’s admission category and unit’s composition. We also adjusted for the same covariates used in the primary study’s analysis (Table).¹⁰

We analyzed 2,366 observations of bronchiolitis patients from 25 hospitals. Most observations were concentrated in freestanding children’s hospitals (89%), and 50% were from hospitals with more than 250 pediatric beds. Observations were well distributed among the five categories of admit burden (Table).

In unadjusted regression, the relationship between admission burden and rate of pulse oximetry use did not appear to be dose-dependent, and 95% CIs were wide. We then analyzed the data accounting for baseline differences in hospital monitoring rates and adjusted for the covariates significantly associated with continuous pulse oximetry monitoring in the primary study’s analysis with use of a mixed-effects model. As shown in the Table, low-burden units in which bronchiolitis constituted less than 10% of total admissions had a 2.16-fold increased odds of unnecessary pulse oximetry monitoring compared to high-burden units in which bronchiolitis constituted 40% or more of total admissions (95% CI, 1.27-3.69; P = .01).

In examining the subspecialty unit composition, 596 observations (25.2%) were conducted on units exclusively caring for general pediatrics patients. In the mixed-effects model adjusted for bronchiolitis admission burden and the covariates used in the study’s primary analysis, units exclusively caring for general pediatrics patients did not have significantly different independent odds of pulse oximetry monitoring use compared to units with a mixed patient population (OR 1.01; 95% CI, 0.71-1.45; P = .95) (Appendix Table).

In this multicenter observational study of children hospitalized with bronchiolitis not concurrently receiving supplemental oxygen, units that only occasionally cared for bronchiolitis patients appeared to be more likely to overuse continuous pulse oximetry during bronchiolitis hospitalizations.

This finding was not immediately apparent when examining the raw data because of wide hospital-level variation in continuous pulse oximetry monitoring use. However, when the high degree of hospital-level variation in baseline overuse was accounted for with use of a random intercept for each hospital in the mixed-effects model, units that cared for higher proportions of bronchiolitis patients had significantly lower odds of continuous pulse oximetry monitoring use compared to units that cared for these infants infrequently.

As many institutions have subspecialized units to cultivate nursing expertise for care of certain diseases and patient populations, we hypothesized that units caring primarily for children on general pediatrics units would also have lower rates of monitoring overuse compared to mixed units. Interestingly, these units did not perform better, likely because potential cultural factors that might contribute to differences in monitoring are accounted for by bronchiolitis admission burden.

Our findings build on prior literature by demonstrating that unit-level, as well as hospital-level, factors appear to drive overuse in healthcare. A prior single-site retrospective cohort study demonstrated an association between higher prevalence of inpatient bronchiolitis and decreased use of unnecessary interventions such as laboratory and radiographic testing, as well as steroid and antibiotic administration.¹¹ Although study of the relationship between volume and quality is not new to healthcare, to our knowledge, this study is the first to examine the relationship between pulse oximetry overuse in bronchiolitis and unit-level factors like admission burden and subspecialty composition.

There are several limitations. First, because the study population included only children not receiving supplemental oxygen, both the parent study and this substudy assumed that all observed use of pulse oximetry monitoring was overuse. In some cases, however, there may have been other compelling clinical reasons, institutional policies, or differences in pulse oximetry availability that were not captured during data collection or in our adjusted model. Second, hospitals used convenience sampling. It is possible this resulted in samples that were not representative of each unit’s underlying patient population or monitoring practice. In addition, not all of the 32 eligible sites were able to provide data related to hospital admissions at the unit level and thus are not included in our analysis. This remains a potential source of hospital-level selection bias.

These findings demonstrate that high bronchiolitis admission burden correlates with lower rates of unnecessary pulse oximetry monitoring in bronchiolitis. We speculate that these outcomes might reflect differing degrees of nursing comfort, expertise, and unit-level norms in caring for bronchiolitis patients, although our study was not designed to establish underlying causes. Identification of operating principles that underpin low pulse oximetry monitoring on high-burden units will provide guidance for decreasing unnecessary monitoring and will inform future studies seeking ways to discourage continuous pulse oximetry monitoring in low-risk infants. Given the institutional variation in monitoring rates, future studies examining both institution-wide and unit-level interventions will be necessary to decrease unnecessary pulse oximetry monitoring in bronchiolitis. Furthermore, these findings may be relevant to studying care quality in other disease processes, with bronchiolitis serving as a model illness for overuse.

The authors acknowledge the National Heart, Lung, and Blood Institute of the National Institutes of Health scientists who contributed their expertise to this project as part of the U01 Cooperative Agreement funding mechanism as federal employees conducting their official job duties: Lora Reineck, MD, MS, Karen Bienstock, MS, and Cheryl Boyce, PhD. The authors thank the executive council of the Pediatric Research in Inpatient Settings Network for their contributions to the early scientific development of this project. The network assessed a Collaborative Support Fee for access to the hospitals and support of this project.

The authors thank the PRIS Network collaborators for their major contributions to data collection (see Appendix).

Unrecognized in-hospital clinical deterioration can lead to substantial morbidity and mortality.¹ As a result, hospitals have implemented systems to identify and mitigate this form of potentially preventable harm.^2-4 Cardiopulmonary arrest rates are useful metrics to evaluate the effectiveness of systems designed to identify and respond to deteriorating adult patients.⁵ Pediatric arrests outside of the intensive care unit (ICU) are rare; therefore, the identification of valid and more frequent proximal measures of deterioration is critical to the assessment of current systems and to guide future improvement efforts.⁶

Bonafide et al developed and validated the critical deterioration event (CDE) metric, demonstrating that children who were transferred to the ICU and who received noninvasive ventilation, intubation, or vasopressor initiation within 12 hours of transfer had an over 13-fold increased risk of in-hospital mortality.⁷ Implementation of a rapid response system was subsequently associated with a decrease in the trajectory of CDEs.² At Cincinnati Children’s Hospital Medical Center (CCHMC), an additional proximal outcome measure was developed for unrecognized clinical deterioration: emergency transfers (ETs).^8,9 An event meets criteria for an ET when the patient undergoes intubation, inotropic support, or three or more fluid boluses in the first hour after arrival or prior to ICU transfer.⁹ Recently, ETs were associated with an increased in-hospital mortality, ICU length of stay, and post-transfer hospital length of stay when compared with nonemergent transfers.^10,11

While both CDEs and ETs were associated with adverse outcomes in children and may be modifiable through better rapid response systems, researchers have not previously compared the extent to which CDEs and ETs capture similar versus distinct events. Furthermore, the ability of focused situation awareness interventions to identify high-risk patients has not previously been assessed. Situation awareness is defined as the perception of elements in the environment, the comprehension of their meaning, and the projection of their status in the near future.¹² Clinically, improved situation awareness can lead to earlier recognition of deterioration and a reduction in failure to rescue events.⁹ The objectives of this study were to (1) describe CDEs and ETs and assess for similarities, differences, and trends, and (2) evaluate the utility of situation awareness interventions to detect patients who experience these events.

Setting and Inclusion Criteria

We conducted a retrospective cross-sectional study at CCHMC, a free-standing tertiary care children’s hospital. We included all patients cared for outside of the ICU during their hospitalization from January 2016 to July 2018. Transfer to the ICU included the pediatric and the cardiac ICUs.

Study Definitions

CDEs were events in which a patient received noninvasive ventilation, intubation, or vasopressor initiation within 12 hours of ICU transfer (Figure).⁷ ETs were events in which a patient underwent intubation, inotropes, or three or more fluid boluses in the first hour after arrival or before transfer (Figure).⁹ We examined two distinct situation awareness interventions: watcher identification and the pediatric early warning score (PEWS). A watcher is a situation awareness concern based on clinician perception, or “gut feeling,” that the patient is at high risk for deterioration.^9,13 When clinicians designate a patient as a watcher in the electronic medical record, they establish an action plan, reassessment timeline, and objective criteria for activation of the rapid response team to assess the patient. Watcher patients are discussed at institution-wide safety huddles three times daily. The PEWS is a reproducible assessment of the patient’s status based on physiologic parameters, including behavior, cardiovascular, and respiratory assessments.^3,4 At CCHMC, a Monaghan PEWS score is calculated with each assessment of vital signs.¹⁴ The bedside nurse calls the physician or advanced practice provider to assess the patient for a score of 4 or greater.

Event Identification and Classification

Two trained research nurses (C.F. and D.H.) manually reviewed all ICU transfers during the study period to determine if CDE criteria were met. Events meeting CDE criteria were classified as respiratory (requiring noninvasive or invasive ventilation), cardiac (requiring inotropes), or cardiopulmonary resuscitation (CPR) in which cardiac and respiratory interventions were initiated simultaneously. Additional information obtained included the time the patient met CDE criteria relative to the time of ICU transfer, watcher identification prior to the event, and the highest PEWS documented within 12 hours of the event. A physician (T.S.) performed manual chart review of each CDE as an additional validation step. ETs during the study period were obtained from an existing institutional database. ICU transfers meeting ET criteria are entered into this database in nearly real time by the inpatient nurse manager; this nurse attends all rapid response team calls and is aware of the disposition for each event. A physician (T.S.) performed manual chart review of each ET to determine event classification by intervention type, watcher identification, and the highest PEWS documented within 12 hours of the event. All CDEs and ETs were cross-referenced to determine overlap.

Outcome Measures and Statistical Analysis

The primary outcomes were CDEs and ETs, calculated as absolute counts and number of events per 10,000 non-ICU patient days. Events were classified by (1) category of intervention, (2) watcher identification prior to the event, and (3) PEWS of 4 or greater documented in the 12 hours prior to the event.

Incidence and Overlap of CDEs and ETs

There were 1,828 ICU transfers during the study period, of which 365 (20%) met criteria for a CDE, ET, or both. Among events captured, 359 (98.4%) met criteria for a CDE, occurring at a rate of 16.7 per 10,000 non-ICU patient days, and 88 (24.1%) met criteria for an ET, occurring at a rate of 4.1 per 10,000 non-ICU patient days (Table). Of the 88 ETs, 82 also met criteria for a CDE.

Timing and Categorization of CDEs and ETs

Despite the 12-hour time horizon, most CDEs (62.1%) met criteria within 1 hour of ICU transfer, and 79.9% met criteria within 3 hours (Figure). Respiratory events were most common for both CDEs (80.5%) and ETs (47.7%) (Table). Of respiratory CDEs, 67.4% required noninvasive ventilation, and 32.5% required invasive ventilation. Fluid or inotrope support were responsible for 11.7% of CDEs and nearly one-third of ETs; of note, the CDE definition does not include fluid boluses. Less than 10% of CDEs were characterized by CPR, whereas this accounted for 22.7% of ETs.

Identification of Events by Situation Awareness Interventions

The Table depicts the identification of events by watcher status and PEWS. All events were included for watcher identification, and events with a documented score in the 12 hours prior to transfer were included for PEWS. While half or less of the events were captured by watcher or PEWS separately, over 85% of events were captured by either one or both of the situation awareness interventions. The situation awareness interventions identified CDEs and ETs similarly.

This study is the first to classify and compare two proximal measures of clinical deterioration in children. Given that children with escalating respiratory symptoms are often treated successfully outside of the ICU, the findings that most events are respiratory in nature and occur within 1 hour of transfer are not unexpected. The analysis of situation awareness interventions suggests that neither watcher identification nor PEWS is independently sufficient to predict future deterioration. These findings support the necessity of both a clinician “gut feeling” and objective vital sign and physical exam findings to indicate a patient’s clinical status.⁹ Initiatives to improve the early recognition and mitigation of patient deterioration should focus on both tools to initiate an escalation of care, and work to understand gaps in these identification systems, which currently miss approximately 15% of acutely deteriorating patients. Although most patients had watcher identification or elevated PEWS prior to the event, they still required emergent life-sustaining care, which suggests that opportunities exist to improve mitigation and escalation pathways as a critical prevention effort.^7,10

It is likely that CDEs and ETs are important outcome metrics in the evaluation of pediatric escalation systems, including rapid response systems.¹⁵ ETs are less common and more specific for unrecognized deterioration, which makes them a more feasible early metric for assessment. CDEs, which are likely more sensitive, may be useful in settings in which deterioration is rare or a more common outcome enhances power to detect the effect of interventions.¹⁰

This study has limitations and lends itself to future work. While CDEs and ETs are more common than cardiopulmonary arrest, they remain relatively uncommon. This was a single-site study at a large, tertiary care, free-standing children’s hospital, so generalizability to centers with different characteristics and patient populations may be limited. Future work should focus on comparing patient-level outcomes of CDEs and ETs, including length of stay and mortality. The determination of specific diagnoses and conditions associated with CDEs and ETs may inform targeted preventive improvement science interventions.

CDEs were roughly fourfold more common than ETs, with most CDEs occurring within 1 hour of ICU transfer. Most patients were identified by either watcher status or elevated PEWS, suggesting that these tools, when utilized as complementary situation awareness interventions, are important for identifying patients at risk for deterioration. Opportunities exist for improved escalation plans for patients identified as high-risk to prevent the need for emergent life-sustaining intervention.

Unrecognized in-hospital clinical deterioration can lead to substantial morbidity and mortality.¹ As a result, hospitals have implemented systems to identify and mitigate this form of potentially preventable harm.^2-4 Cardiopulmonary arrest rates are useful metrics to evaluate the effectiveness of systems designed to identify and respond to deteriorating adult patients.⁵ Pediatric arrests outside of the intensive care unit (ICU) are rare; therefore, the identification of valid and more frequent proximal measures of deterioration is critical to the assessment of current systems and to guide future improvement efforts.⁶

Bonafide et al developed and validated the critical deterioration event (CDE) metric, demonstrating that children who were transferred to the ICU and who received noninvasive ventilation, intubation, or vasopressor initiation within 12 hours of transfer had an over 13-fold increased risk of in-hospital mortality.⁷ Implementation of a rapid response system was subsequently associated with a decrease in the trajectory of CDEs.² At Cincinnati Children’s Hospital Medical Center (CCHMC), an additional proximal outcome measure was developed for unrecognized clinical deterioration: emergency transfers (ETs).^8,9 An event meets criteria for an ET when the patient undergoes intubation, inotropic support, or three or more fluid boluses in the first hour after arrival or prior to ICU transfer.⁹ Recently, ETs were associated with an increased in-hospital mortality, ICU length of stay, and post-transfer hospital length of stay when compared with nonemergent transfers.^10,11

While both CDEs and ETs were associated with adverse outcomes in children and may be modifiable through better rapid response systems, researchers have not previously compared the extent to which CDEs and ETs capture similar versus distinct events. Furthermore, the ability of focused situation awareness interventions to identify high-risk patients has not previously been assessed. Situation awareness is defined as the perception of elements in the environment, the comprehension of their meaning, and the projection of their status in the near future.¹² Clinically, improved situation awareness can lead to earlier recognition of deterioration and a reduction in failure to rescue events.⁹ The objectives of this study were to (1) describe CDEs and ETs and assess for similarities, differences, and trends, and (2) evaluate the utility of situation awareness interventions to detect patients who experience these events.

Setting and Inclusion Criteria

We conducted a retrospective cross-sectional study at CCHMC, a free-standing tertiary care children’s hospital. We included all patients cared for outside of the ICU during their hospitalization from January 2016 to July 2018. Transfer to the ICU included the pediatric and the cardiac ICUs.

Study Definitions

CDEs were events in which a patient received noninvasive ventilation, intubation, or vasopressor initiation within 12 hours of ICU transfer (Figure).⁷ ETs were events in which a patient underwent intubation, inotropes, or three or more fluid boluses in the first hour after arrival or before transfer (Figure).⁹ We examined two distinct situation awareness interventions: watcher identification and the pediatric early warning score (PEWS). A watcher is a situation awareness concern based on clinician perception, or “gut feeling,” that the patient is at high risk for deterioration.^9,13 When clinicians designate a patient as a watcher in the electronic medical record, they establish an action plan, reassessment timeline, and objective criteria for activation of the rapid response team to assess the patient. Watcher patients are discussed at institution-wide safety huddles three times daily. The PEWS is a reproducible assessment of the patient’s status based on physiologic parameters, including behavior, cardiovascular, and respiratory assessments.^3,4 At CCHMC, a Monaghan PEWS score is calculated with each assessment of vital signs.¹⁴ The bedside nurse calls the physician or advanced practice provider to assess the patient for a score of 4 or greater.

Event Identification and Classification

Two trained research nurses (C.F. and D.H.) manually reviewed all ICU transfers during the study period to determine if CDE criteria were met. Events meeting CDE criteria were classified as respiratory (requiring noninvasive or invasive ventilation), cardiac (requiring inotropes), or cardiopulmonary resuscitation (CPR) in which cardiac and respiratory interventions were initiated simultaneously. Additional information obtained included the time the patient met CDE criteria relative to the time of ICU transfer, watcher identification prior to the event, and the highest PEWS documented within 12 hours of the event. A physician (T.S.) performed manual chart review of each CDE as an additional validation step. ETs during the study period were obtained from an existing institutional database. ICU transfers meeting ET criteria are entered into this database in nearly real time by the inpatient nurse manager; this nurse attends all rapid response team calls and is aware of the disposition for each event. A physician (T.S.) performed manual chart review of each ET to determine event classification by intervention type, watcher identification, and the highest PEWS documented within 12 hours of the event. All CDEs and ETs were cross-referenced to determine overlap.

Outcome Measures and Statistical Analysis

The primary outcomes were CDEs and ETs, calculated as absolute counts and number of events per 10,000 non-ICU patient days. Events were classified by (1) category of intervention, (2) watcher identification prior to the event, and (3) PEWS of 4 or greater documented in the 12 hours prior to the event.

Incidence and Overlap of CDEs and ETs

There were 1,828 ICU transfers during the study period, of which 365 (20%) met criteria for a CDE, ET, or both. Among events captured, 359 (98.4%) met criteria for a CDE, occurring at a rate of 16.7 per 10,000 non-ICU patient days, and 88 (24.1%) met criteria for an ET, occurring at a rate of 4.1 per 10,000 non-ICU patient days (Table). Of the 88 ETs, 82 also met criteria for a CDE.

Timing and Categorization of CDEs and ETs

Despite the 12-hour time horizon, most CDEs (62.1%) met criteria within 1 hour of ICU transfer, and 79.9% met criteria within 3 hours (Figure). Respiratory events were most common for both CDEs (80.5%) and ETs (47.7%) (Table). Of respiratory CDEs, 67.4% required noninvasive ventilation, and 32.5% required invasive ventilation. Fluid or inotrope support were responsible for 11.7% of CDEs and nearly one-third of ETs; of note, the CDE definition does not include fluid boluses. Less than 10% of CDEs were characterized by CPR, whereas this accounted for 22.7% of ETs.

Identification of Events by Situation Awareness Interventions

The Table depicts the identification of events by watcher status and PEWS. All events were included for watcher identification, and events with a documented score in the 12 hours prior to transfer were included for PEWS. While half or less of the events were captured by watcher or PEWS separately, over 85% of events were captured by either one or both of the situation awareness interventions. The situation awareness interventions identified CDEs and ETs similarly.

This study is the first to classify and compare two proximal measures of clinical deterioration in children. Given that children with escalating respiratory symptoms are often treated successfully outside of the ICU, the findings that most events are respiratory in nature and occur within 1 hour of transfer are not unexpected. The analysis of situation awareness interventions suggests that neither watcher identification nor PEWS is independently sufficient to predict future deterioration. These findings support the necessity of both a clinician “gut feeling” and objective vital sign and physical exam findings to indicate a patient’s clinical status.⁹ Initiatives to improve the early recognition and mitigation of patient deterioration should focus on both tools to initiate an escalation of care, and work to understand gaps in these identification systems, which currently miss approximately 15% of acutely deteriorating patients. Although most patients had watcher identification or elevated PEWS prior to the event, they still required emergent life-sustaining care, which suggests that opportunities exist to improve mitigation and escalation pathways as a critical prevention effort.^7,10

It is likely that CDEs and ETs are important outcome metrics in the evaluation of pediatric escalation systems, including rapid response systems.¹⁵ ETs are less common and more specific for unrecognized deterioration, which makes them a more feasible early metric for assessment. CDEs, which are likely more sensitive, may be useful in settings in which deterioration is rare or a more common outcome enhances power to detect the effect of interventions.¹⁰

This study has limitations and lends itself to future work. While CDEs and ETs are more common than cardiopulmonary arrest, they remain relatively uncommon. This was a single-site study at a large, tertiary care, free-standing children’s hospital, so generalizability to centers with different characteristics and patient populations may be limited. Future work should focus on comparing patient-level outcomes of CDEs and ETs, including length of stay and mortality. The determination of specific diagnoses and conditions associated with CDEs and ETs may inform targeted preventive improvement science interventions.

CDEs were roughly fourfold more common than ETs, with most CDEs occurring within 1 hour of ICU transfer. Most patients were identified by either watcher status or elevated PEWS, suggesting that these tools, when utilized as complementary situation awareness interventions, are important for identifying patients at risk for deterioration. Opportunities exist for improved escalation plans for patients identified as high-risk to prevent the need for emergent life-sustaining intervention.

Unrecognized in-hospital clinical deterioration can lead to substantial morbidity and mortality.¹ As a result, hospitals have implemented systems to identify and mitigate this form of potentially preventable harm.^2-4 Cardiopulmonary arrest rates are useful metrics to evaluate the effectiveness of systems designed to identify and respond to deteriorating adult patients.⁵ Pediatric arrests outside of the intensive care unit (ICU) are rare; therefore, the identification of valid and more frequent proximal measures of deterioration is critical to the assessment of current systems and to guide future improvement efforts.⁶

Bonafide et al developed and validated the critical deterioration event (CDE) metric, demonstrating that children who were transferred to the ICU and who received noninvasive ventilation, intubation, or vasopressor initiation within 12 hours of transfer had an over 13-fold increased risk of in-hospital mortality.⁷ Implementation of a rapid response system was subsequently associated with a decrease in the trajectory of CDEs.² At Cincinnati Children’s Hospital Medical Center (CCHMC), an additional proximal outcome measure was developed for unrecognized clinical deterioration: emergency transfers (ETs).^8,9 An event meets criteria for an ET when the patient undergoes intubation, inotropic support, or three or more fluid boluses in the first hour after arrival or prior to ICU transfer.⁹ Recently, ETs were associated with an increased in-hospital mortality, ICU length of stay, and post-transfer hospital length of stay when compared with nonemergent transfers.^10,11

While both CDEs and ETs were associated with adverse outcomes in children and may be modifiable through better rapid response systems, researchers have not previously compared the extent to which CDEs and ETs capture similar versus distinct events. Furthermore, the ability of focused situation awareness interventions to identify high-risk patients has not previously been assessed. Situation awareness is defined as the perception of elements in the environment, the comprehension of their meaning, and the projection of their status in the near future.¹² Clinically, improved situation awareness can lead to earlier recognition of deterioration and a reduction in failure to rescue events.⁹ The objectives of this study were to (1) describe CDEs and ETs and assess for similarities, differences, and trends, and (2) evaluate the utility of situation awareness interventions to detect patients who experience these events.

Setting and Inclusion Criteria

We conducted a retrospective cross-sectional study at CCHMC, a free-standing tertiary care children’s hospital. We included all patients cared for outside of the ICU during their hospitalization from January 2016 to July 2018. Transfer to the ICU included the pediatric and the cardiac ICUs.

Study Definitions

CDEs were events in which a patient received noninvasive ventilation, intubation, or vasopressor initiation within 12 hours of ICU transfer (Figure).⁷ ETs were events in which a patient underwent intubation, inotropes, or three or more fluid boluses in the first hour after arrival or before transfer (Figure).⁹ We examined two distinct situation awareness interventions: watcher identification and the pediatric early warning score (PEWS). A watcher is a situation awareness concern based on clinician perception, or “gut feeling,” that the patient is at high risk for deterioration.^9,13 When clinicians designate a patient as a watcher in the electronic medical record, they establish an action plan, reassessment timeline, and objective criteria for activation of the rapid response team to assess the patient. Watcher patients are discussed at institution-wide safety huddles three times daily. The PEWS is a reproducible assessment of the patient’s status based on physiologic parameters, including behavior, cardiovascular, and respiratory assessments.^3,4 At CCHMC, a Monaghan PEWS score is calculated with each assessment of vital signs.¹⁴ The bedside nurse calls the physician or advanced practice provider to assess the patient for a score of 4 or greater.

Event Identification and Classification

Two trained research nurses (C.F. and D.H.) manually reviewed all ICU transfers during the study period to determine if CDE criteria were met. Events meeting CDE criteria were classified as respiratory (requiring noninvasive or invasive ventilation), cardiac (requiring inotropes), or cardiopulmonary resuscitation (CPR) in which cardiac and respiratory interventions were initiated simultaneously. Additional information obtained included the time the patient met CDE criteria relative to the time of ICU transfer, watcher identification prior to the event, and the highest PEWS documented within 12 hours of the event. A physician (T.S.) performed manual chart review of each CDE as an additional validation step. ETs during the study period were obtained from an existing institutional database. ICU transfers meeting ET criteria are entered into this database in nearly real time by the inpatient nurse manager; this nurse attends all rapid response team calls and is aware of the disposition for each event. A physician (T.S.) performed manual chart review of each ET to determine event classification by intervention type, watcher identification, and the highest PEWS documented within 12 hours of the event. All CDEs and ETs were cross-referenced to determine overlap.

Outcome Measures and Statistical Analysis

The primary outcomes were CDEs and ETs, calculated as absolute counts and number of events per 10,000 non-ICU patient days. Events were classified by (1) category of intervention, (2) watcher identification prior to the event, and (3) PEWS of 4 or greater documented in the 12 hours prior to the event.

Incidence and Overlap of CDEs and ETs

There were 1,828 ICU transfers during the study period, of which 365 (20%) met criteria for a CDE, ET, or both. Among events captured, 359 (98.4%) met criteria for a CDE, occurring at a rate of 16.7 per 10,000 non-ICU patient days, and 88 (24.1%) met criteria for an ET, occurring at a rate of 4.1 per 10,000 non-ICU patient days (Table). Of the 88 ETs, 82 also met criteria for a CDE.

Timing and Categorization of CDEs and ETs

Despite the 12-hour time horizon, most CDEs (62.1%) met criteria within 1 hour of ICU transfer, and 79.9% met criteria within 3 hours (Figure). Respiratory events were most common for both CDEs (80.5%) and ETs (47.7%) (Table). Of respiratory CDEs, 67.4% required noninvasive ventilation, and 32.5% required invasive ventilation. Fluid or inotrope support were responsible for 11.7% of CDEs and nearly one-third of ETs; of note, the CDE definition does not include fluid boluses. Less than 10% of CDEs were characterized by CPR, whereas this accounted for 22.7% of ETs.

Identification of Events by Situation Awareness Interventions

The Table depicts the identification of events by watcher status and PEWS. All events were included for watcher identification, and events with a documented score in the 12 hours prior to transfer were included for PEWS. While half or less of the events were captured by watcher or PEWS separately, over 85% of events were captured by either one or both of the situation awareness interventions. The situation awareness interventions identified CDEs and ETs similarly.

This study is the first to classify and compare two proximal measures of clinical deterioration in children. Given that children with escalating respiratory symptoms are often treated successfully outside of the ICU, the findings that most events are respiratory in nature and occur within 1 hour of transfer are not unexpected. The analysis of situation awareness interventions suggests that neither watcher identification nor PEWS is independently sufficient to predict future deterioration. These findings support the necessity of both a clinician “gut feeling” and objective vital sign and physical exam findings to indicate a patient’s clinical status.⁹ Initiatives to improve the early recognition and mitigation of patient deterioration should focus on both tools to initiate an escalation of care, and work to understand gaps in these identification systems, which currently miss approximately 15% of acutely deteriorating patients. Although most patients had watcher identification or elevated PEWS prior to the event, they still required emergent life-sustaining care, which suggests that opportunities exist to improve mitigation and escalation pathways as a critical prevention effort.^7,10

It is likely that CDEs and ETs are important outcome metrics in the evaluation of pediatric escalation systems, including rapid response systems.¹⁵ ETs are less common and more specific for unrecognized deterioration, which makes them a more feasible early metric for assessment. CDEs, which are likely more sensitive, may be useful in settings in which deterioration is rare or a more common outcome enhances power to detect the effect of interventions.¹⁰

This study has limitations and lends itself to future work. While CDEs and ETs are more common than cardiopulmonary arrest, they remain relatively uncommon. This was a single-site study at a large, tertiary care, free-standing children’s hospital, so generalizability to centers with different characteristics and patient populations may be limited. Future work should focus on comparing patient-level outcomes of CDEs and ETs, including length of stay and mortality. The determination of specific diagnoses and conditions associated with CDEs and ETs may inform targeted preventive improvement science interventions.

CDEs were roughly fourfold more common than ETs, with most CDEs occurring within 1 hour of ICU transfer. Most patients were identified by either watcher status or elevated PEWS, suggesting that these tools, when utilized as complementary situation awareness interventions, are important for identifying patients at risk for deterioration. Opportunities exist for improved escalation plans for patients identified as high-risk to prevent the need for emergent life-sustaining intervention.

As part of improvement collaboratives that aimed to reduce overuse of continuous pulse oximetry in children hospitalized with bronchiolitis, researchers used the presence of an active order for it as a proxy for the actual use of such monitoring.^1,2 With use of this proxy, investigators on a national study documented a high burden of continuous oximetry overuse (86.5% before quality improvement interventions and 45.5% after),¹ but the validity of orders in representing actual monitoring practice is unknown. If the presence of an active pulse oximetry order accurately identifies infants on monitors, electronic health record data could inform epidemiologic estimates of monitoring overuse and measure the success of quality improvement and deimplementation interventions. Alternatively, if nurses commonly begin and/or discontinue pulse oximetry without updated orders, a pulse oximetry order would not be an accurate proxy, and additional data capture methods (eg, bedside observation or data capture from bedside monitors) would be needed.

Understanding the validity of orders for detection of actual use is critical because continuous pulse oximetry monitoring is considered an overused practice in pediatric acute viral bronchiolitis,³ and national guidelines recommend against its use in low-risk hospitalized children.^4,5 Continuous monitoring may identify trivial, self-resolving oxygen desaturation and its use is not associated with improved outcomes.^6-9 When self-resolving desaturations are treated with additional supplemental oxygen, hospital stays may be unnecessarily prolonged.¹⁰ In order to reduce unnecessary continuous pulse oximetry use, measurement of the extent of the overused practice is necessary. In this 56-hospital study,¹¹ we aimed to determine the validity of using active continuous pulse oximetry orders instead of bedside observation of actual monitor use.

Design

In this multicenter, repeated cross-sectional study, investigators used direct bedside observation to determine continuous pulse oximetry monitor use and then assessed whether an active continuous monitoring order was present in the electronic health record. The study took place during one bronchiolitis season, December 1, 2018, through March 31, 2019.

Setting and Patients

Investigators at 56 freestanding children’s hospitals, children’s hospitals within general hospitals, and community hospitals in the Pediatric Research in Inpatient Settings (PRIS) Network collected data on infants aged 8 weeks to 23 months who were hospitalized with bronchiolitis. As this work was a substudy of the larger Eliminating Monitor Overuse study, only infants not currently receiving supplemental oxygen were included.¹¹ Investigators observed eligible infants outside of the intensive care unit on general hospital medicine units. We excluded infants born premature (documented prematurity of <28 weeks’ gestation or documented “premature” without a gestational age listed), as well as those with a home oxygen requirement, cyanotic congenital heart disease, pulmonary hypertension, tracheostomy, primary neuromuscular disease, immunodeficiency, or cancer.

Data Collection

Investigators used the electronic health record to identify eligible infants. Investigators entered patient rooms to confirm the infant was not on supplemental oxygen (hence confirming eligibility for the study) and determine if continuous pulse oximetry was actively in use by examining the monitor display for a pulse oximetry waveform. Investigators then confirmed if active orders for pulse oximetry were present in the patient’s chart. Per study design, site investigators aimed to observe approximately half of eligible infants during the day (10 am to 5 pm) and the other half during the night (11 pm to 7 am).

Analysis

We excluded patients with conditional orders (eg, monitored only when certain conditions exist, such as when asleep) because of the time-varying and wide range of conditions that could be specified. Furthermore, conditional orders would not be useful as proxies to measure oximetry use because investigators would still need additional data (eg, bedside observation) to determine current monitoring status.

We calculated the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of active orders using the reference standard of direct bedside observation, as well as corresponding 95% CIs that accounted for within-hospital clustering. We calculated these test characteristics overall and as stratified across four age groups: 8 weeks to 5 months, 6 months to 11 months, 12 months to 17 months, and 18 months to 23 months. We also calculated the test characteristics for each hospital. We decided a priori that a PPV and NPV of 80% would represent a reasonable threshold to use active orders as a proxy in multicenter research. For hospital-level analyses we included only hospitals with 60 or more total observations and more than 15 observations with active orders for PPV and more than 15 observations without active orders for NPV. We used Stata (StataCorp LLC, College Station, Texas) version 15.1 for analysis.

For US sites, the Institutional Review Board (IRB) at Children’s Hospital of Philadelphia approved the study as the single reviewing IRB, and the remaining US sites established reliance agreements with the reviewing IRB. Research Ethics Boards at the Canadian sites (University of Calgary and The Hospital for Sick Children) also reviewed and approved the study. All sites granted waivers of consent, assent, parental permission, and HIPAA authorization.

Investigators completed 3,612 observations in 56 hospitals. This included 33 freestanding children’s hospitals, 14 hospitals within large general hospitals, and 9 community hospitals. Of 3,612 completed observations, on 631 occasions (17%) patients had conditional orders (eg, continuous monitoring only when sleeping) and were excluded from further analysis.

Most pulse oximetry–monitored infants did not have an active monitoring order (670 out of 1,309; sensitivity of 49%). Test characteristics, stratified by age group, are presented in the Table. Across all observations, the overall PPV was 77% (95% CI, 72-82), and the overall NPV was 69% (95% CI, 61-77). Variation of all test characteristics across age group was small (eg, the sensitivity ranged from 43% to 51%).

With inclusion of only those hospitals with sufficient observations, hospital-level variation in the PPV and NPV of using active orders was substantial (PPV range of 48% to 96% and NPV range of 30% to 98%). Only two hospitals had both a PPV and NPV for using monitor orders that exceeded the 80% threshold.

Active continuous pulse oximetry orders did not accurately represent actual monitoring status in this study. Monitoring orders alone frequently misrepresent true monitoring status and, as such, should be interpreted with caution in research or quality improvement activities. If more valid estimates of monitoring use and overuse are needed, potential measurement options include direct observation, as used in our study, as well as the use of more complex data streams such as the output of monitoring devices or pulse oximetry data in the electronic health record. In only two of the hospitals, using active continuous monitoring orders was a reasonable proxy for detecting actual monitor use. Monitoring orders could potentially be validly used for deimplementation efforts at those centers; other hospitals could consider targeted improvement efforts (eg, morning huddles examining the discordance between monitoring orders and monitoring status) to improve the accuracy of using continuous pulse oximetry orders.

We acknowledge several limitations of this study. Site investigators employed a convenience sampling approach, so it is possible that some investigators observed sicker or less sick infants. Although the PRIS network includes a geographically diverse group of North American hospitals, community hospitals were underrepresented in this study. Our results hence generalize more precisely to freestanding children’s hospitals than to community hospitals. We did not observe infants currently on supplemental oxygen, so we do not know to what degree using orders is valid in that context. We did not collect data on why actual monitoring status differed from monitoring orders and hence cannot quantify to what extent different factors (eg, nurse belief that monitors are a safety net or infants inadvertently left on monitors after a spot check pulse oximetry reading) contributed to this discordance. Finally, our study only examined one electronic health record variable—the presence of an active order. It may be that other variables in the health record (eg, minute-by-minute pulse oximetry values in a vital sign flowsheet) are much better proxies of actual continuous monitor use.

Using an active order for continuous pulse oximetry has poor sensitivity, PPV, and NPV for detecting true monitoring status at the bedside. Teams intending to measure the actual use of pulse oximetry should be aware of the limitations of using active orders alone as an accurate measure of pulse oximetry monitoring.

We thank the NHLBI scientists who contributed to this project as part of the U01 Cooperative Agreement funding mechanism: Lora Reineck, MD, MS, Karen Bienstock, MS, and Cheryl Boyce, PhD.

We thank the Executive Council of the PRIS Network for their contributions to the early scientific development of this project. We thank the PRIS site investigators for their major contributions to the Eliminating Monitor Overuse (EMO) Study data collection. Each listed collaborator is a group author for the PRIS Network in this manuscript. Their names can be found in the online supplemental information.

As part of improvement collaboratives that aimed to reduce overuse of continuous pulse oximetry in children hospitalized with bronchiolitis, researchers used the presence of an active order for it as a proxy for the actual use of such monitoring.^1,2 With use of this proxy, investigators on a national study documented a high burden of continuous oximetry overuse (86.5% before quality improvement interventions and 45.5% after),¹ but the validity of orders in representing actual monitoring practice is unknown. If the presence of an active pulse oximetry order accurately identifies infants on monitors, electronic health record data could inform epidemiologic estimates of monitoring overuse and measure the success of quality improvement and deimplementation interventions. Alternatively, if nurses commonly begin and/or discontinue pulse oximetry without updated orders, a pulse oximetry order would not be an accurate proxy, and additional data capture methods (eg, bedside observation or data capture from bedside monitors) would be needed.

Understanding the validity of orders for detection of actual use is critical because continuous pulse oximetry monitoring is considered an overused practice in pediatric acute viral bronchiolitis,³ and national guidelines recommend against its use in low-risk hospitalized children.^4,5 Continuous monitoring may identify trivial, self-resolving oxygen desaturation and its use is not associated with improved outcomes.^6-9 When self-resolving desaturations are treated with additional supplemental oxygen, hospital stays may be unnecessarily prolonged.¹⁰ In order to reduce unnecessary continuous pulse oximetry use, measurement of the extent of the overused practice is necessary. In this 56-hospital study,¹¹ we aimed to determine the validity of using active continuous pulse oximetry orders instead of bedside observation of actual monitor use.

Design

In this multicenter, repeated cross-sectional study, investigators used direct bedside observation to determine continuous pulse oximetry monitor use and then assessed whether an active continuous monitoring order was present in the electronic health record. The study took place during one bronchiolitis season, December 1, 2018, through March 31, 2019.

Setting and Patients

Investigators at 56 freestanding children’s hospitals, children’s hospitals within general hospitals, and community hospitals in the Pediatric Research in Inpatient Settings (PRIS) Network collected data on infants aged 8 weeks to 23 months who were hospitalized with bronchiolitis. As this work was a substudy of the larger Eliminating Monitor Overuse study, only infants not currently receiving supplemental oxygen were included.¹¹ Investigators observed eligible infants outside of the intensive care unit on general hospital medicine units. We excluded infants born premature (documented prematurity of <28 weeks’ gestation or documented “premature” without a gestational age listed), as well as those with a home oxygen requirement, cyanotic congenital heart disease, pulmonary hypertension, tracheostomy, primary neuromuscular disease, immunodeficiency, or cancer.

Data Collection

Investigators used the electronic health record to identify eligible infants. Investigators entered patient rooms to confirm the infant was not on supplemental oxygen (hence confirming eligibility for the study) and determine if continuous pulse oximetry was actively in use by examining the monitor display for a pulse oximetry waveform. Investigators then confirmed if active orders for pulse oximetry were present in the patient’s chart. Per study design, site investigators aimed to observe approximately half of eligible infants during the day (10 am to 5 pm) and the other half during the night (11 pm to 7 am).

Analysis

We excluded patients with conditional orders (eg, monitored only when certain conditions exist, such as when asleep) because of the time-varying and wide range of conditions that could be specified. Furthermore, conditional orders would not be useful as proxies to measure oximetry use because investigators would still need additional data (eg, bedside observation) to determine current monitoring status.

We calculated the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of active orders using the reference standard of direct bedside observation, as well as corresponding 95% CIs that accounted for within-hospital clustering. We calculated these test characteristics overall and as stratified across four age groups: 8 weeks to 5 months, 6 months to 11 months, 12 months to 17 months, and 18 months to 23 months. We also calculated the test characteristics for each hospital. We decided a priori that a PPV and NPV of 80% would represent a reasonable threshold to use active orders as a proxy in multicenter research. For hospital-level analyses we included only hospitals with 60 or more total observations and more than 15 observations with active orders for PPV and more than 15 observations without active orders for NPV. We used Stata (StataCorp LLC, College Station, Texas) version 15.1 for analysis.

For US sites, the Institutional Review Board (IRB) at Children’s Hospital of Philadelphia approved the study as the single reviewing IRB, and the remaining US sites established reliance agreements with the reviewing IRB. Research Ethics Boards at the Canadian sites (University of Calgary and The Hospital for Sick Children) also reviewed and approved the study. All sites granted waivers of consent, assent, parental permission, and HIPAA authorization.

Investigators completed 3,612 observations in 56 hospitals. This included 33 freestanding children’s hospitals, 14 hospitals within large general hospitals, and 9 community hospitals. Of 3,612 completed observations, on 631 occasions (17%) patients had conditional orders (eg, continuous monitoring only when sleeping) and were excluded from further analysis.

Most pulse oximetry–monitored infants did not have an active monitoring order (670 out of 1,309; sensitivity of 49%). Test characteristics, stratified by age group, are presented in the Table. Across all observations, the overall PPV was 77% (95% CI, 72-82), and the overall NPV was 69% (95% CI, 61-77). Variation of all test characteristics across age group was small (eg, the sensitivity ranged from 43% to 51%).

With inclusion of only those hospitals with sufficient observations, hospital-level variation in the PPV and NPV of using active orders was substantial (PPV range of 48% to 96% and NPV range of 30% to 98%). Only two hospitals had both a PPV and NPV for using monitor orders that exceeded the 80% threshold.

Active continuous pulse oximetry orders did not accurately represent actual monitoring status in this study. Monitoring orders alone frequently misrepresent true monitoring status and, as such, should be interpreted with caution in research or quality improvement activities. If more valid estimates of monitoring use and overuse are needed, potential measurement options include direct observation, as used in our study, as well as the use of more complex data streams such as the output of monitoring devices or pulse oximetry data in the electronic health record. In only two of the hospitals, using active continuous monitoring orders was a reasonable proxy for detecting actual monitor use. Monitoring orders could potentially be validly used for deimplementation efforts at those centers; other hospitals could consider targeted improvement efforts (eg, morning huddles examining the discordance between monitoring orders and monitoring status) to improve the accuracy of using continuous pulse oximetry orders.

We acknowledge several limitations of this study. Site investigators employed a convenience sampling approach, so it is possible that some investigators observed sicker or less sick infants. Although the PRIS network includes a geographically diverse group of North American hospitals, community hospitals were underrepresented in this study. Our results hence generalize more precisely to freestanding children’s hospitals than to community hospitals. We did not observe infants currently on supplemental oxygen, so we do not know to what degree using orders is valid in that context. We did not collect data on why actual monitoring status differed from monitoring orders and hence cannot quantify to what extent different factors (eg, nurse belief that monitors are a safety net or infants inadvertently left on monitors after a spot check pulse oximetry reading) contributed to this discordance. Finally, our study only examined one electronic health record variable—the presence of an active order. It may be that other variables in the health record (eg, minute-by-minute pulse oximetry values in a vital sign flowsheet) are much better proxies of actual continuous monitor use.

Using an active order for continuous pulse oximetry has poor sensitivity, PPV, and NPV for detecting true monitoring status at the bedside. Teams intending to measure the actual use of pulse oximetry should be aware of the limitations of using active orders alone as an accurate measure of pulse oximetry monitoring.

We thank the NHLBI scientists who contributed to this project as part of the U01 Cooperative Agreement funding mechanism: Lora Reineck, MD, MS, Karen Bienstock, MS, and Cheryl Boyce, PhD.

We thank the Executive Council of the PRIS Network for their contributions to the early scientific development of this project. We thank the PRIS site investigators for their major contributions to the Eliminating Monitor Overuse (EMO) Study data collection. Each listed collaborator is a group author for the PRIS Network in this manuscript. Their names can be found in the online supplemental information.

As part of improvement collaboratives that aimed to reduce overuse of continuous pulse oximetry in children hospitalized with bronchiolitis, researchers used the presence of an active order for it as a proxy for the actual use of such monitoring.^1,2 With use of this proxy, investigators on a national study documented a high burden of continuous oximetry overuse (86.5% before quality improvement interventions and 45.5% after),¹ but the validity of orders in representing actual monitoring practice is unknown. If the presence of an active pulse oximetry order accurately identifies infants on monitors, electronic health record data could inform epidemiologic estimates of monitoring overuse and measure the success of quality improvement and deimplementation interventions. Alternatively, if nurses commonly begin and/or discontinue pulse oximetry without updated orders, a pulse oximetry order would not be an accurate proxy, and additional data capture methods (eg, bedside observation or data capture from bedside monitors) would be needed.

Understanding the validity of orders for detection of actual use is critical because continuous pulse oximetry monitoring is considered an overused practice in pediatric acute viral bronchiolitis,³ and national guidelines recommend against its use in low-risk hospitalized children.^4,5 Continuous monitoring may identify trivial, self-resolving oxygen desaturation and its use is not associated with improved outcomes.^6-9 When self-resolving desaturations are treated with additional supplemental oxygen, hospital stays may be unnecessarily prolonged.¹⁰ In order to reduce unnecessary continuous pulse oximetry use, measurement of the extent of the overused practice is necessary. In this 56-hospital study,¹¹ we aimed to determine the validity of using active continuous pulse oximetry orders instead of bedside observation of actual monitor use.

Design

In this multicenter, repeated cross-sectional study, investigators used direct bedside observation to determine continuous pulse oximetry monitor use and then assessed whether an active continuous monitoring order was present in the electronic health record. The study took place during one bronchiolitis season, December 1, 2018, through March 31, 2019.

Setting and Patients

Investigators at 56 freestanding children’s hospitals, children’s hospitals within general hospitals, and community hospitals in the Pediatric Research in Inpatient Settings (PRIS) Network collected data on infants aged 8 weeks to 23 months who were hospitalized with bronchiolitis. As this work was a substudy of the larger Eliminating Monitor Overuse study, only infants not currently receiving supplemental oxygen were included.¹¹ Investigators observed eligible infants outside of the intensive care unit on general hospital medicine units. We excluded infants born premature (documented prematurity of <28 weeks’ gestation or documented “premature” without a gestational age listed), as well as those with a home oxygen requirement, cyanotic congenital heart disease, pulmonary hypertension, tracheostomy, primary neuromuscular disease, immunodeficiency, or cancer.

Data Collection

Investigators used the electronic health record to identify eligible infants. Investigators entered patient rooms to confirm the infant was not on supplemental oxygen (hence confirming eligibility for the study) and determine if continuous pulse oximetry was actively in use by examining the monitor display for a pulse oximetry waveform. Investigators then confirmed if active orders for pulse oximetry were present in the patient’s chart. Per study design, site investigators aimed to observe approximately half of eligible infants during the day (10 am to 5 pm) and the other half during the night (11 pm to 7 am).

Analysis

We excluded patients with conditional orders (eg, monitored only when certain conditions exist, such as when asleep) because of the time-varying and wide range of conditions that could be specified. Furthermore, conditional orders would not be useful as proxies to measure oximetry use because investigators would still need additional data (eg, bedside observation) to determine current monitoring status.

We calculated the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of active orders using the reference standard of direct bedside observation, as well as corresponding 95% CIs that accounted for within-hospital clustering. We calculated these test characteristics overall and as stratified across four age groups: 8 weeks to 5 months, 6 months to 11 months, 12 months to 17 months, and 18 months to 23 months. We also calculated the test characteristics for each hospital. We decided a priori that a PPV and NPV of 80% would represent a reasonable threshold to use active orders as a proxy in multicenter research. For hospital-level analyses we included only hospitals with 60 or more total observations and more than 15 observations with active orders for PPV and more than 15 observations without active orders for NPV. We used Stata (StataCorp LLC, College Station, Texas) version 15.1 for analysis.

For US sites, the Institutional Review Board (IRB) at Children’s Hospital of Philadelphia approved the study as the single reviewing IRB, and the remaining US sites established reliance agreements with the reviewing IRB. Research Ethics Boards at the Canadian sites (University of Calgary and The Hospital for Sick Children) also reviewed and approved the study. All sites granted waivers of consent, assent, parental permission, and HIPAA authorization.

Investigators completed 3,612 observations in 56 hospitals. This included 33 freestanding children’s hospitals, 14 hospitals within large general hospitals, and 9 community hospitals. Of 3,612 completed observations, on 631 occasions (17%) patients had conditional orders (eg, continuous monitoring only when sleeping) and were excluded from further analysis.

Most pulse oximetry–monitored infants did not have an active monitoring order (670 out of 1,309; sensitivity of 49%). Test characteristics, stratified by age group, are presented in the Table. Across all observations, the overall PPV was 77% (95% CI, 72-82), and the overall NPV was 69% (95% CI, 61-77). Variation of all test characteristics across age group was small (eg, the sensitivity ranged from 43% to 51%).

With inclusion of only those hospitals with sufficient observations, hospital-level variation in the PPV and NPV of using active orders was substantial (PPV range of 48% to 96% and NPV range of 30% to 98%). Only two hospitals had both a PPV and NPV for using monitor orders that exceeded the 80% threshold.

Active continuous pulse oximetry orders did not accurately represent actual monitoring status in this study. Monitoring orders alone frequently misrepresent true monitoring status and, as such, should be interpreted with caution in research or quality improvement activities. If more valid estimates of monitoring use and overuse are needed, potential measurement options include direct observation, as used in our study, as well as the use of more complex data streams such as the output of monitoring devices or pulse oximetry data in the electronic health record. In only two of the hospitals, using active continuous monitoring orders was a reasonable proxy for detecting actual monitor use. Monitoring orders could potentially be validly used for deimplementation efforts at those centers; other hospitals could consider targeted improvement efforts (eg, morning huddles examining the discordance between monitoring orders and monitoring status) to improve the accuracy of using continuous pulse oximetry orders.

We acknowledge several limitations of this study. Site investigators employed a convenience sampling approach, so it is possible that some investigators observed sicker or less sick infants. Although the PRIS network includes a geographically diverse group of North American hospitals, community hospitals were underrepresented in this study. Our results hence generalize more precisely to freestanding children’s hospitals than to community hospitals. We did not observe infants currently on supplemental oxygen, so we do not know to what degree using orders is valid in that context. We did not collect data on why actual monitoring status differed from monitoring orders and hence cannot quantify to what extent different factors (eg, nurse belief that monitors are a safety net or infants inadvertently left on monitors after a spot check pulse oximetry reading) contributed to this discordance. Finally, our study only examined one electronic health record variable—the presence of an active order. It may be that other variables in the health record (eg, minute-by-minute pulse oximetry values in a vital sign flowsheet) are much better proxies of actual continuous monitor use.

Using an active order for continuous pulse oximetry has poor sensitivity, PPV, and NPV for detecting true monitoring status at the bedside. Teams intending to measure the actual use of pulse oximetry should be aware of the limitations of using active orders alone as an accurate measure of pulse oximetry monitoring.

We thank the NHLBI scientists who contributed to this project as part of the U01 Cooperative Agreement funding mechanism: Lora Reineck, MD, MS, Karen Bienstock, MS, and Cheryl Boyce, PhD.

We thank the Executive Council of the PRIS Network for their contributions to the early scientific development of this project. We thank the PRIS site investigators for their major contributions to the Eliminating Monitor Overuse (EMO) Study data collection. Each listed collaborator is a group author for the PRIS Network in this manuscript. Their names can be found in the online supplemental information.