Original Research

Social determinants of health (SDoH) are social, economic, environmental, and occupational factors that are known to influence an individual’s health care utilization and clinical outcomes.^1,2 Because the Veterans Health Administration (VHA) is charged to address both the medical and nonmedical needs of the veteran population, it is increasingly interested in the impact SDoH have on veteran care.^3,4 To combat the adverse impact of such factors, the VHA has implemented several large-scale programs across the US that focus on prevalent SDoH, such as homelessness, substance abuse, and alcohol use disorders.^5,6 While such risk factors are generally universal in their distribution, variation across regions, between urban and rural spaces, and even within cities has been shown to exist in private settings.⁷ Understanding such variability potentially could be helpful to US Department of Veterans Affairs (VA) policymakers and leaders to better allocate funding and resources to address such issues.

Although previous work has highlighted regional and neighborhood-level variability of SDoH, no study has examined the facility-level variability of commonly encountered social risk factors within the VHA.^4,8 The aim of this study was to describe the interfacility variation of 5 common SDoH known to influence health and health outcomes among a national cohort of veterans hospitalized for common medical issues by using administrative data.

Methods

We used a national cohort of veterans aged ≥ 65 years who were hospitalized at a VHA acute care facility with a primary discharge diagnosis of acute myocardial infarction (AMI), heart failure (HF), or pneumonia in 2012. These conditions were chosen because they are publicly reported and frequently used for interfacility comparison.

Using the International Classification of Diseases–9th Revision (ICD-9) and VHA clinical stop codes, we calculated the median documented proportion of patients with any of the following 5 SDoH: lived alone, marginal housing, alcohol use disorder, substance use disorder, and use of substance use services for patients presenting with HF, MI, and pneumonia (Table). These SDoH were chosen because they are intervenable risk factors for which the VHA has several programs (eg, homeless outreach, substance abuse, and tobacco cessation). To examine the variability of these SDoH across VHA facilities, we determined the number of hospitals that had a sufficient number of admissions (≥ 50) to be included in the analyses. We then examined the administratively documented, facility-level variation in the proportion of individuals with any of the 5 SDoH administrative codes and examined the distribution of their use across all qualifying facilities.

Because variability may be due to regional coding differences, we examined the difference in the estimated prevalence of the risk factor lives alone by using a previously developed natural language processing (NLP) program.⁹ The NLP program is a rule-based system designed to automatically extract information that requires inferencing from clinical notes (eg, discharge summaries and nursing, social work, emergency department physician, primary care, and hospital admission notes). For instance, the program identifies whether there was direct or indirect evidence that the patient did or did not live alone. In addition to extracting data on lives alone, the NLP program has the capacity to extract information on lack of social support and living alone—2 characteristics without VHA interventions, which were not examined here. The NLP program was developed and evaluated using at least 1 year of notes prior to index hospitalization. Because this program was developed and validated on a 2012 data set, we were limited to using a cohort from this year as well.

All analyses were conducted using SAS Version 9.4. The San Francisco VA Medical Center Institutional Review Board approved this study.

 

Results

In total, 21,991 patients with either HF (9,853), pneumonia (9,362), or AMI (2,776) were identified across 91 VHA facilities. The majority were male (98%) and had a median (SD) age of 77.0 (9.0) years. The median facility-level proportion of veterans who had any of the SDoH risk factors extracted through administrative codes was low across all conditions, ranging from 0.5 to 2.2%. The most prevalent factors among patients admitted for HF, AMI, and pneumonia were lives alone (2.0% [Interquartile range (IQR), 1.0-5.2], 1.4% [IQR, 0-3.4], and 1.9% [IQR, 0.7-5.4]), substance use disorder (1.2% [IQR, 0-2.2], 1.6% [IQR: 0-3.0], and 1.3% [IQR, 0-2.2] and use of substance use services (0.9% [IQR, 0-1.6%], 1.0% [IQR, 0-1.7%], and 1.6% [IQR, 0-2.2%], respectively [Table]).

When utilizing the NLP algorithm, the documented prevalence of lives alone in the free text of the medical record was higher than administrative coding across all conditions (12.3% vs. 2.2%; P < .01). Among each of the 3 assessed conditions, HF (14.4% vs 2.0%, P < .01) had higher levels of lives alone compared with pneumonia (11% vs 1.9%, P < .01), and AMI (10.2% vs 1.4%, P < .01) when using the NLP algorithm. When we examined the documented facility-level variation in the proportion of individuals with any of the 5 SDoH administrative codes or NLP, we found large variability across all facilities—regardless of extraction method (Figure).

Discussion

While SDoH are known to impact health outcomes, the presence of these risk factors in administrative data among individuals hospitalized for common medical issues is low and variable across VHA facilities. Understanding the documented, facility-level variability of these measures may assist the VHA in determining how it invests time and resources—as different facilities may disproportionately serve a higher number of vulnerable individuals. Beyond the VHA, these findings have generalizable lessons for the US health care system, which has come to recognize how these risk factors impact patients’ health.¹⁰

Although the proportion of individuals with any of the assessed SDoH identified by administrative data was low, our findings are in line with recent studies that showed other risk factors such as social isolation (0.65%), housing issues (0.19%), and financial strain (0.07%) had similarly low prevalence.^8,11 Although the exact prevalence of such factors remains unclear, these findings highlight that SDoH do not appear to be well documented in administrative data. Low coding rates are likely due to the fact that SDoH administrative codes are not tied to financial reimbursement—thus not incentivizing their use by clinicians or hospital systems.

In 2014, an Institute of Medicine report suggested that collection of SDoH in electronic health data as a means to better empower clinicians and health care systems to address social disparities and further support research in SDoH.¹² Since then, data collection using SDoH screening tools has become more common across settings, but is not consistently translated to standardized data due to lack of industry consensus and technical barriers.¹³ To improve this process, the Centers for Medicare and Medicaid Services created “z-codes” for the ICD-10 classification system—a subset of codes that are meant to better capture patients’ underlying social risk.¹⁴ It remains to be seen if such administrative codes have improved the documentation of SDoH.

As health care systems have grown to understand the impact of SDoH on health outcomes,other means of collecting these data have evolved.^1,10 For example, NLP-based extraction methods and electronic screening tools have been proposed and utilized as alternative for obtaining this information. Our findings suggest that some of these measures (eg, lives alone) often may be documented as part of routine care in the electronic health record, thus highlighting NLP as a tool to obtain such data. However, other studies using NLP technology to extract SDoH have shown this technology is often complicated by quality issues (ie, missing data), complex methods, and poor integration with current information technology infrastructures—thus limiting its use in health care delivery.^15-18

While variance among SDoH across a national health care system is natural, it remains an important systems-level characteristic that health care leaders and policymakers should appreciate. As health care systems disperse financial resources and initiate quality improvement initiatives to address SDoH, knowing that not all facilities are equally affected by SDoH should impact allocation of such resources and energies. Although previous work has highlighted regional and neighborhood levels of variation within the VHA and other health care systems, to our knowledge, this is the first study to examine variability at the facility-level within the VHA.^2,4,13,19

 

Limitations

There are several limitations to this study. First, though our findings are in line with previous data in other health care systems, generalizability beyond the VA, which primarily cares for older, male patients, may be limited.⁸ Though, as the nation’s largest health care system, lessons from the VHA can still be useful for other health care systems as they consider SDoH variation. Second, among the many SDoH previously identified to impact health, our analysis only focused on 5 such variables. Administrative and medical record documentation of other SDoH may be more common and less variable across institutions. Third, while our data suggests facility-level variation in these measures, this may be in part related to variation in coding across facilities. However, the single SDoH variable extracted using NLP also varied at the facility-level, suggesting that coding may not entirely drive the variation observed.

Conclusions

As US health care systems continue to address SDoH, our findings highlight the various challenges in obtaining accurate data on a patient’s social risk. Moreover, these findings highlight the large variability that exists among institutions in a national integrated health care system. Future work should explore the prevalence and variance of other SDoH as a means to help guide resource allocation and prioritize spending to better address SDoH where it is most needed.

Acknowledgments

This work was supported by NHLBI R01 RO1 HL116522-01A1. Support for VA/CMS data is provided by the US Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development, Health Services Research and Development, VA Information Resource Center (Project Numbers SDR 02-237 and 98-004).

Social determinants of health (SDoH) are social, economic, environmental, and occupational factors that are known to influence an individual’s health care utilization and clinical outcomes.^1,2 Because the Veterans Health Administration (VHA) is charged to address both the medical and nonmedical needs of the veteran population, it is increasingly interested in the impact SDoH have on veteran care.^3,4 To combat the adverse impact of such factors, the VHA has implemented several large-scale programs across the US that focus on prevalent SDoH, such as homelessness, substance abuse, and alcohol use disorders.^5,6 While such risk factors are generally universal in their distribution, variation across regions, between urban and rural spaces, and even within cities has been shown to exist in private settings.⁷ Understanding such variability potentially could be helpful to US Department of Veterans Affairs (VA) policymakers and leaders to better allocate funding and resources to address such issues.

Although previous work has highlighted regional and neighborhood-level variability of SDoH, no study has examined the facility-level variability of commonly encountered social risk factors within the VHA.^4,8 The aim of this study was to describe the interfacility variation of 5 common SDoH known to influence health and health outcomes among a national cohort of veterans hospitalized for common medical issues by using administrative data.

Methods

We used a national cohort of veterans aged ≥ 65 years who were hospitalized at a VHA acute care facility with a primary discharge diagnosis of acute myocardial infarction (AMI), heart failure (HF), or pneumonia in 2012. These conditions were chosen because they are publicly reported and frequently used for interfacility comparison.

Using the International Classification of Diseases–9th Revision (ICD-9) and VHA clinical stop codes, we calculated the median documented proportion of patients with any of the following 5 SDoH: lived alone, marginal housing, alcohol use disorder, substance use disorder, and use of substance use services for patients presenting with HF, MI, and pneumonia (Table). These SDoH were chosen because they are intervenable risk factors for which the VHA has several programs (eg, homeless outreach, substance abuse, and tobacco cessation). To examine the variability of these SDoH across VHA facilities, we determined the number of hospitals that had a sufficient number of admissions (≥ 50) to be included in the analyses. We then examined the administratively documented, facility-level variation in the proportion of individuals with any of the 5 SDoH administrative codes and examined the distribution of their use across all qualifying facilities.

Because variability may be due to regional coding differences, we examined the difference in the estimated prevalence of the risk factor lives alone by using a previously developed natural language processing (NLP) program.⁹ The NLP program is a rule-based system designed to automatically extract information that requires inferencing from clinical notes (eg, discharge summaries and nursing, social work, emergency department physician, primary care, and hospital admission notes). For instance, the program identifies whether there was direct or indirect evidence that the patient did or did not live alone. In addition to extracting data on lives alone, the NLP program has the capacity to extract information on lack of social support and living alone—2 characteristics without VHA interventions, which were not examined here. The NLP program was developed and evaluated using at least 1 year of notes prior to index hospitalization. Because this program was developed and validated on a 2012 data set, we were limited to using a cohort from this year as well.

All analyses were conducted using SAS Version 9.4. The San Francisco VA Medical Center Institutional Review Board approved this study.

 

Results

In total, 21,991 patients with either HF (9,853), pneumonia (9,362), or AMI (2,776) were identified across 91 VHA facilities. The majority were male (98%) and had a median (SD) age of 77.0 (9.0) years. The median facility-level proportion of veterans who had any of the SDoH risk factors extracted through administrative codes was low across all conditions, ranging from 0.5 to 2.2%. The most prevalent factors among patients admitted for HF, AMI, and pneumonia were lives alone (2.0% [Interquartile range (IQR), 1.0-5.2], 1.4% [IQR, 0-3.4], and 1.9% [IQR, 0.7-5.4]), substance use disorder (1.2% [IQR, 0-2.2], 1.6% [IQR: 0-3.0], and 1.3% [IQR, 0-2.2] and use of substance use services (0.9% [IQR, 0-1.6%], 1.0% [IQR, 0-1.7%], and 1.6% [IQR, 0-2.2%], respectively [Table]).

When utilizing the NLP algorithm, the documented prevalence of lives alone in the free text of the medical record was higher than administrative coding across all conditions (12.3% vs. 2.2%; P < .01). Among each of the 3 assessed conditions, HF (14.4% vs 2.0%, P < .01) had higher levels of lives alone compared with pneumonia (11% vs 1.9%, P < .01), and AMI (10.2% vs 1.4%, P < .01) when using the NLP algorithm. When we examined the documented facility-level variation in the proportion of individuals with any of the 5 SDoH administrative codes or NLP, we found large variability across all facilities—regardless of extraction method (Figure).

Discussion

While SDoH are known to impact health outcomes, the presence of these risk factors in administrative data among individuals hospitalized for common medical issues is low and variable across VHA facilities. Understanding the documented, facility-level variability of these measures may assist the VHA in determining how it invests time and resources—as different facilities may disproportionately serve a higher number of vulnerable individuals. Beyond the VHA, these findings have generalizable lessons for the US health care system, which has come to recognize how these risk factors impact patients’ health.¹⁰

Although the proportion of individuals with any of the assessed SDoH identified by administrative data was low, our findings are in line with recent studies that showed other risk factors such as social isolation (0.65%), housing issues (0.19%), and financial strain (0.07%) had similarly low prevalence.^8,11 Although the exact prevalence of such factors remains unclear, these findings highlight that SDoH do not appear to be well documented in administrative data. Low coding rates are likely due to the fact that SDoH administrative codes are not tied to financial reimbursement—thus not incentivizing their use by clinicians or hospital systems.

In 2014, an Institute of Medicine report suggested that collection of SDoH in electronic health data as a means to better empower clinicians and health care systems to address social disparities and further support research in SDoH.¹² Since then, data collection using SDoH screening tools has become more common across settings, but is not consistently translated to standardized data due to lack of industry consensus and technical barriers.¹³ To improve this process, the Centers for Medicare and Medicaid Services created “z-codes” for the ICD-10 classification system—a subset of codes that are meant to better capture patients’ underlying social risk.¹⁴ It remains to be seen if such administrative codes have improved the documentation of SDoH.

As health care systems have grown to understand the impact of SDoH on health outcomes,other means of collecting these data have evolved.^1,10 For example, NLP-based extraction methods and electronic screening tools have been proposed and utilized as alternative for obtaining this information. Our findings suggest that some of these measures (eg, lives alone) often may be documented as part of routine care in the electronic health record, thus highlighting NLP as a tool to obtain such data. However, other studies using NLP technology to extract SDoH have shown this technology is often complicated by quality issues (ie, missing data), complex methods, and poor integration with current information technology infrastructures—thus limiting its use in health care delivery.^15-18

While variance among SDoH across a national health care system is natural, it remains an important systems-level characteristic that health care leaders and policymakers should appreciate. As health care systems disperse financial resources and initiate quality improvement initiatives to address SDoH, knowing that not all facilities are equally affected by SDoH should impact allocation of such resources and energies. Although previous work has highlighted regional and neighborhood levels of variation within the VHA and other health care systems, to our knowledge, this is the first study to examine variability at the facility-level within the VHA.^2,4,13,19

 

Limitations

There are several limitations to this study. First, though our findings are in line with previous data in other health care systems, generalizability beyond the VA, which primarily cares for older, male patients, may be limited.⁸ Though, as the nation’s largest health care system, lessons from the VHA can still be useful for other health care systems as they consider SDoH variation. Second, among the many SDoH previously identified to impact health, our analysis only focused on 5 such variables. Administrative and medical record documentation of other SDoH may be more common and less variable across institutions. Third, while our data suggests facility-level variation in these measures, this may be in part related to variation in coding across facilities. However, the single SDoH variable extracted using NLP also varied at the facility-level, suggesting that coding may not entirely drive the variation observed.

Conclusions

As US health care systems continue to address SDoH, our findings highlight the various challenges in obtaining accurate data on a patient’s social risk. Moreover, these findings highlight the large variability that exists among institutions in a national integrated health care system. Future work should explore the prevalence and variance of other SDoH as a means to help guide resource allocation and prioritize spending to better address SDoH where it is most needed.

Acknowledgments

This work was supported by NHLBI R01 RO1 HL116522-01A1. Support for VA/CMS data is provided by the US Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development, Health Services Research and Development, VA Information Resource Center (Project Numbers SDR 02-237 and 98-004).

Social determinants of health (SDoH) are social, economic, environmental, and occupational factors that are known to influence an individual’s health care utilization and clinical outcomes.^1,2 Because the Veterans Health Administration (VHA) is charged to address both the medical and nonmedical needs of the veteran population, it is increasingly interested in the impact SDoH have on veteran care.^3,4 To combat the adverse impact of such factors, the VHA has implemented several large-scale programs across the US that focus on prevalent SDoH, such as homelessness, substance abuse, and alcohol use disorders.^5,6 While such risk factors are generally universal in their distribution, variation across regions, between urban and rural spaces, and even within cities has been shown to exist in private settings.⁷ Understanding such variability potentially could be helpful to US Department of Veterans Affairs (VA) policymakers and leaders to better allocate funding and resources to address such issues.

Although previous work has highlighted regional and neighborhood-level variability of SDoH, no study has examined the facility-level variability of commonly encountered social risk factors within the VHA.^4,8 The aim of this study was to describe the interfacility variation of 5 common SDoH known to influence health and health outcomes among a national cohort of veterans hospitalized for common medical issues by using administrative data.

Methods

We used a national cohort of veterans aged ≥ 65 years who were hospitalized at a VHA acute care facility with a primary discharge diagnosis of acute myocardial infarction (AMI), heart failure (HF), or pneumonia in 2012. These conditions were chosen because they are publicly reported and frequently used for interfacility comparison.

Using the International Classification of Diseases–9th Revision (ICD-9) and VHA clinical stop codes, we calculated the median documented proportion of patients with any of the following 5 SDoH: lived alone, marginal housing, alcohol use disorder, substance use disorder, and use of substance use services for patients presenting with HF, MI, and pneumonia (Table). These SDoH were chosen because they are intervenable risk factors for which the VHA has several programs (eg, homeless outreach, substance abuse, and tobacco cessation). To examine the variability of these SDoH across VHA facilities, we determined the number of hospitals that had a sufficient number of admissions (≥ 50) to be included in the analyses. We then examined the administratively documented, facility-level variation in the proportion of individuals with any of the 5 SDoH administrative codes and examined the distribution of their use across all qualifying facilities.

Because variability may be due to regional coding differences, we examined the difference in the estimated prevalence of the risk factor lives alone by using a previously developed natural language processing (NLP) program.⁹ The NLP program is a rule-based system designed to automatically extract information that requires inferencing from clinical notes (eg, discharge summaries and nursing, social work, emergency department physician, primary care, and hospital admission notes). For instance, the program identifies whether there was direct or indirect evidence that the patient did or did not live alone. In addition to extracting data on lives alone, the NLP program has the capacity to extract information on lack of social support and living alone—2 characteristics without VHA interventions, which were not examined here. The NLP program was developed and evaluated using at least 1 year of notes prior to index hospitalization. Because this program was developed and validated on a 2012 data set, we were limited to using a cohort from this year as well.

All analyses were conducted using SAS Version 9.4. The San Francisco VA Medical Center Institutional Review Board approved this study.

 

Results

In total, 21,991 patients with either HF (9,853), pneumonia (9,362), or AMI (2,776) were identified across 91 VHA facilities. The majority were male (98%) and had a median (SD) age of 77.0 (9.0) years. The median facility-level proportion of veterans who had any of the SDoH risk factors extracted through administrative codes was low across all conditions, ranging from 0.5 to 2.2%. The most prevalent factors among patients admitted for HF, AMI, and pneumonia were lives alone (2.0% [Interquartile range (IQR), 1.0-5.2], 1.4% [IQR, 0-3.4], and 1.9% [IQR, 0.7-5.4]), substance use disorder (1.2% [IQR, 0-2.2], 1.6% [IQR: 0-3.0], and 1.3% [IQR, 0-2.2] and use of substance use services (0.9% [IQR, 0-1.6%], 1.0% [IQR, 0-1.7%], and 1.6% [IQR, 0-2.2%], respectively [Table]).

When utilizing the NLP algorithm, the documented prevalence of lives alone in the free text of the medical record was higher than administrative coding across all conditions (12.3% vs. 2.2%; P < .01). Among each of the 3 assessed conditions, HF (14.4% vs 2.0%, P < .01) had higher levels of lives alone compared with pneumonia (11% vs 1.9%, P < .01), and AMI (10.2% vs 1.4%, P < .01) when using the NLP algorithm. When we examined the documented facility-level variation in the proportion of individuals with any of the 5 SDoH administrative codes or NLP, we found large variability across all facilities—regardless of extraction method (Figure).

Discussion

While SDoH are known to impact health outcomes, the presence of these risk factors in administrative data among individuals hospitalized for common medical issues is low and variable across VHA facilities. Understanding the documented, facility-level variability of these measures may assist the VHA in determining how it invests time and resources—as different facilities may disproportionately serve a higher number of vulnerable individuals. Beyond the VHA, these findings have generalizable lessons for the US health care system, which has come to recognize how these risk factors impact patients’ health.¹⁰

Although the proportion of individuals with any of the assessed SDoH identified by administrative data was low, our findings are in line with recent studies that showed other risk factors such as social isolation (0.65%), housing issues (0.19%), and financial strain (0.07%) had similarly low prevalence.^8,11 Although the exact prevalence of such factors remains unclear, these findings highlight that SDoH do not appear to be well documented in administrative data. Low coding rates are likely due to the fact that SDoH administrative codes are not tied to financial reimbursement—thus not incentivizing their use by clinicians or hospital systems.

In 2014, an Institute of Medicine report suggested that collection of SDoH in electronic health data as a means to better empower clinicians and health care systems to address social disparities and further support research in SDoH.¹² Since then, data collection using SDoH screening tools has become more common across settings, but is not consistently translated to standardized data due to lack of industry consensus and technical barriers.¹³ To improve this process, the Centers for Medicare and Medicaid Services created “z-codes” for the ICD-10 classification system—a subset of codes that are meant to better capture patients’ underlying social risk.¹⁴ It remains to be seen if such administrative codes have improved the documentation of SDoH.

As health care systems have grown to understand the impact of SDoH on health outcomes,other means of collecting these data have evolved.^1,10 For example, NLP-based extraction methods and electronic screening tools have been proposed and utilized as alternative for obtaining this information. Our findings suggest that some of these measures (eg, lives alone) often may be documented as part of routine care in the electronic health record, thus highlighting NLP as a tool to obtain such data. However, other studies using NLP technology to extract SDoH have shown this technology is often complicated by quality issues (ie, missing data), complex methods, and poor integration with current information technology infrastructures—thus limiting its use in health care delivery.^15-18

While variance among SDoH across a national health care system is natural, it remains an important systems-level characteristic that health care leaders and policymakers should appreciate. As health care systems disperse financial resources and initiate quality improvement initiatives to address SDoH, knowing that not all facilities are equally affected by SDoH should impact allocation of such resources and energies. Although previous work has highlighted regional and neighborhood levels of variation within the VHA and other health care systems, to our knowledge, this is the first study to examine variability at the facility-level within the VHA.^2,4,13,19

 

Limitations

There are several limitations to this study. First, though our findings are in line with previous data in other health care systems, generalizability beyond the VA, which primarily cares for older, male patients, may be limited.⁸ Though, as the nation’s largest health care system, lessons from the VHA can still be useful for other health care systems as they consider SDoH variation. Second, among the many SDoH previously identified to impact health, our analysis only focused on 5 such variables. Administrative and medical record documentation of other SDoH may be more common and less variable across institutions. Third, while our data suggests facility-level variation in these measures, this may be in part related to variation in coding across facilities. However, the single SDoH variable extracted using NLP also varied at the facility-level, suggesting that coding may not entirely drive the variation observed.

Conclusions

As US health care systems continue to address SDoH, our findings highlight the various challenges in obtaining accurate data on a patient’s social risk. Moreover, these findings highlight the large variability that exists among institutions in a national integrated health care system. Future work should explore the prevalence and variance of other SDoH as a means to help guide resource allocation and prioritize spending to better address SDoH where it is most needed.

Acknowledgments

This work was supported by NHLBI R01 RO1 HL116522-01A1. Support for VA/CMS data is provided by the US Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development, Health Services Research and Development, VA Information Resource Center (Project Numbers SDR 02-237 and 98-004).

Posttraumatic stress disorder (PTSD) affects about 10 to 25% of veterans in the US and is associated with reductions in quality of life and poor occupational functioning.^1,2 PTSD is often associated with multiple cognitive deficits that play a role in a number of clinical symptoms and impair cognition beyond what can be solely attributed to the effects of physical or psychological trauma.^3-5 Although the literature on the pattern and magnitude of cognitive deficits associated with PTSD is mixed, dysfunction in attention, verbal memory, speed of information processing, working memory, and executive functioning are the most consistent findings.^6-11Verbal memory and attention seem to be particularly negatively impacted by PTSD and especially so in combat-exposed war veterans.^7,12 Verbal memory difficulties in returning war veterans also may mediate quality of life and be particularly disruptive to everyday functioning.¹³ Further, evidence exists that a diagnosis of PTSD is associated with increased risk for dementia and deficits in episodic memory in older adults.^14,15

The PTSD-associated cognitive deficits are routinely assessed through neuropsychological measures within the US Department of Veteran Affairs (VA). The Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) is a commonly used cognitive screening measure in medical settings, and prior research has reinforced its clinical utility across a variety of populations, including Alzheimer disease, schizophrenia, Parkinson disease, Huntington disease, stroke, and traumatic brain injury (TBI).^16-24

McKay and colleagues previously examined the use of the RBANS within a sample of individuals who had a history of moderate-to-severe TBIs, with findings suggesting the RBANS is a valid and reliable screening measure in this population.²⁵However, McKay and colleagues used a carefully defined sample in a cognitive neurorehabilitation setting, many of whom experienced a TBI significant enough to require ongoing medical monitoring, attendant care, or substantial support services.

The influence of PTSD-associated cognitive deficits on the RBANS performance is unclear, and which subtests of the measure, if any, are differentially impacted in individuals with and those without a diagnosis of PTSD is uncertain. Further, less is known about the influence of PTSD in outpatient clinical settings when PTSD and TBI are not necessarily the primary presenting problem. The purpose of the current study was to determine the influence of a PTSD diagnosis on performance on the RBANS in an outpatient VA setting.

Methods

Participants included 153 veterans who were 90% male with a mean (SD) age of 46.8 (11.3) years and a mean (SD) education of 14.2 (2.3) years from a catchment area ranging from Montana south through western Texas, and all states west of that line, sequentially evaluated as part of a clinic workup at the California War Related Illness and Injury Study Center (WRIISC-CA). WRIISC-CA is a second-level evaluation clinic under patient primary care in the VA system dedicated to providing comprehensive medical evaluations on postdeployment veterans with complex medical concerns, including possible TBI and PTSD. Participants included 23 Vietnam-era, 72 Operation Desert Storm/Desert Shield-era, and 58 Operation Iraqi Freedom/Enduring Freedom-era veterans. We have previously published a more thorough analysis of medical characteristics for a WRIISC-CA sample.²⁶

A Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM IV) diagnosis of current PTSD was determined by the Clinician-Administered PTSD Scale (CAPS-IV), as administered or supervised by a licensed clinical psychologist during the course of the larger medical evaluation.²⁷ Given the co-occurring nature of TBI and PTSD and their complicated relationship with regard to cognitive functioning, all veterans also underwent a comprehensive examination by a board-certified neurologist to assess for a possible history of TBI, based on the presence of at least 1 past event according to the guidelines recommended by the American Congress of Rehabilitation Medicine.^28,29Veterans were categorized as having a history of no TBI, mild TBI, or moderate TBI. No veterans met criteria for history of severe TBI.Veterans were excluded from the analysis if unable to complete the mental health, neurological, or cognitive evaluations. Informed consent was obtained consistent with the Declaration of Helsinki and institutional guidelines established by the VA Palo Alto Human Subjects Review Committee. The study was approved by the VA Palo Alto and Stanford School of Medicine institutional review boards.

 

Cognitive Measures

All veterans completed a targeted cognitive battery that included the following: a reading recognition measure designed to estimate premorbid intellectual functioning (Wechsler Test of Adult Reading [WTAR]); a measure assessing auditory attention and working memory ability (Wechsler Adults Intelligence Scale-IV [WAIS-IV] Digit Span subtest); a measure assessing processing speed, attention, and cognitive flexibility (Trails A and B); and the RBANS.^16,30-32The focus of the current study was on the RBANS, a brief cognitive screening measure that contains 12 subtests examining a variety of cognitive functions. Given that all participants were veterans receiving outpatient services, there was no nonpatient control group for comparison. To address this, all raw data were converted to standardized scores based on healthy normative data provided within the test manual. Specifically, the 12 RBANS subtest scores were converted to age-corrected standardized z scores, which in turn created a total summary score and 5 composite summary indexes: immediate memory, visuospatial/constructional, attention, language, and delayed memory. All veterans completed the Form A version of the measure.

Statistical Analyses

Group level differences on selective demographic and cognitive measures between veterans with a diagnosis of PTSD and those without were examined using t tests. Cognitive variables included standardized scores for the RBANS, including age-adjusted total summary score, index scores, and subtest scores.¹⁶ Estimated full-scale IQ and standardized summary scores from the WTAR, demographically adjusted standardized scores for the total time to complete Trails A and time to complete Trails B, and age-adjusted standardized scores for the WAIS-IV Digit Span subtest (forward, backward, and sequencing trials, as well as the summary total score) were examined for group differences.^30,31,33 To further examine the association between PTSD and RBANS performance, multivariate multiple regressions were conducted using measures of episodic memory and processing speed from the RBANS (ie, story tasks, list learning tasks, and coding subtests). These specific measures were selected ad hoc based on extant literature.^6,10The dependent variable for each analysis was the standardized score from the selected subtest; PTSD status, a diagnosis of TBI, a diagnosis of co-occurring TBI and PTSD, gender, and years of education were predictor variables.

Results

Of the 153 study participants, 98 (64%) met DSM-4 criteria for current PTSD, whereas 55 (36%) did not (Table). There was no group statistical difference between veterans with or without a diagnosis of PTSD for age, education, or gender (P < .05). A diagnosis of PTSD tended to be more frequent in participants with a history of head injury (χ² = 7.72; P < .05). Veterans with a diagnosis of PTSD performed significantly worse on the RBANS Story Recall subtest compared with the results of those without PTSD (t[138] = 3.10; P < .01); performance on other cognitive measures was not significantly different between the PTSD groups. A diagnosis of PTSD was also significantly associated with self-reported depressive symptoms (Beck Depression Inventory-II; t[123] = -2.81; P < .01). Depressive symptoms were not associated with a history of TBI, and group differences were not significant.

Given the high co-occurrence of PTSD and TBI (68%) in our PTSD sample, secondary analyses examined the association of select diagnoses with performance on the RBANS, specifically veterans with a historical diagnosis of TBI (n = 92) from those without a diagnosis of TBI (n = 61), as well as those with co-occurring PTSD and TBI (n = 71) from those without (n = 82). The majority of the sample met criteria for a history of mild TBI (n = 79) when compared with moderate TBI (n = 13); none met criteria for a past history of severe TBI. PTSD status (β = .63, P = .04) and years of education (β = .16, P < .01) were associated with performance on the RBANS Story Recall subtest (R²= .23, F[5,139] = 8.11, P < .01). Education was the only significant predictor for the rest of the multivariate multiple regressions (all P < .05). A diagnosis of TBI or co-occurring PTSD and TBI was not significantly associated with performance on the Story Memory, Story Recall, List Learning, List Recall, or Coding subtests. multivariate analysis of variance tests for the hypothesis of an overall main effect of PTSD (F(5,130) = 1.08, P = .34), TBI (F[5,130] = .91, P = .48), or PTSD+TBI (F[5,130] =.47, P = .80) on the 4 selected tests were not significant.

 

Discussion

The findings of the present study suggest that veterans with PTSD perform worse on specific RBANS subtests compared with veterans without PTSD. Specifically, worse performance on the Story Recall subtest of the RBANS memory index was a significant predictor of a diagnosis of PTSD within the statistical model. This association with PTSD was not seen in other demographic (excluding education) or cognitive measures, including other memory tasks, such as List Recall and Figure Recall, and attentional measures, such as WAIS-IV Digit Span, and the Trail Making Test. Overall RBANS index scores were not significantly different between groups, though this is not surprising given that recent research suggests the RBANS composite scores have questionable validity and reliability.³⁴

The finding that a measure of episodic memory is most influenced by PTSD status is consistent with prior research.³⁵ However, there are several possible reasons why Story Recall in particular showed the greatest association, even more than other episodic memory measures. A review by Isaac and colleagues found a diagnosis of PTSD correlated with frontal lobe-associated memory deficits.⁶ As Story Recall provides only 2 rehearsal trials compared with the 4 trials provided in the RBANS List Learning subtest, it is possible that Story Recall relies more on attentional processes than on learning with repetition.

Research has indicated attention and verbal episodic memory dysfunction are associated with a diagnosis of PTSD in combat veterans, and individuals with a diagnosis of PTSD show deficits in executive functioning, including attention difficulties beyond what is seen in trauma-exposed controls.^4,7,8,11,35Furthermore, a diagnosis of PTSD has been shown to be associated with impaired performance on the Logical Memory subtest of the Wechsler Memory Scale-Revised, a very similar measure to the RBANS Story Recall.³⁶

The present finding that performance on a RBANS subtest was associated with a diagnosis of PTSD but not a history of TBI is not surprising. The majority of the present sample who reported a history of TBI met criteria for a remote head injury of mild severity (86%). Cognitive symptoms related to mild TBI are thought to generally resolve over time, and recent research suggests that PTSD symptoms may account for a substantial portion of reported postconcussive symptoms.^37,38Similarly, recent research suggests a diagnosis of mild TBI does not necessarily result in additive cognitive impairment in combat veterans with a diagnosis of PTSD, and that a diagnosis of PTSD is more strongly associated with cognitive symptoms than is mild TBI.^5,39,40

The lack of association with RBANS performance and co-occurring PTSD and TBI is less clear. Although both conditions are heterogenous, it may be that individuals with a diagnosis solely of PTSD are quantitively different from those with a concurrent diagnosis of PTSD and TBI (ie, PTSD presumed due to a mild TBI). Specifically, the impact of PTSD on cognition may be related to symptom severity and indexed trauma. A published systematic review on the PTSD-related cognitive impairment showed a medium-to-strong effect size for severity of PTSD symptoms on cognitive performance, with war trauma showing the strongest effect.⁴In particular, individuals who experience repeated or complex trauma are prone to experience PTSD symptoms with concurrent cognitive deficits, again suggesting the possibility of qualitative differences between outpatient veterans with PTSD and those with mild TBI associated PTSD.⁴¹While disentangling PTSD and mild TBI symptoms are notoriously difficult, future research aiming to examine these factors may be beneficial in the ability to draw larger conclusions on the relationship between cognition and PTSD.

 

Limitations

Several limitations may affect the generalizability of the findings. The present study used a veteran sample referred to a specialty clinic for complicated postdeployment health concerns. Although findings may not be representative of an inpatient population or clinics that focus solely on TBI, they may more adequately reflect veterans using clinical services at VA medical centers. We also did not include measures of PTSD symptom severity (eg, Posttraumatic Stress Disorder Checklist), instead using diagnosis based on the gold standard CAPS. In addition, the likelihood of the presence of a remote TBI was based on a clinical interview with a neurologist and not on acute neurologic findings. TBI is a heterogenous diagnosis, with multiple factors that likely influence cognitive performance, including location of the injury, type of injury, and time since injury, which may be lost during group analysis. Further, the RBANS is not intended to serve as a method for a differential diagnosis of PTSD or TBI. Concordant with this, the intention of the current study was to capture the quality of cognitive function on the RBANS within individuals with PTSD.

Conslusions

The ability for veterans to remember a short story following a delay (ie, RBANS Story Recall subtest) was negatively associated with a diagnosis of PTSD. Further, the RBANS best captured cognitive deficits associated with PTSD compared with those with a history of mild TBI, or co-occurring mild TBI and PTSD. These findings may provide insight into the interpretation and attribution of cognitive deficits in the veteran population and holds potential to guide future research examining focused cognitive phenotypes to provide precision targets in individual treatment.

Posttraumatic stress disorder (PTSD) affects about 10 to 25% of veterans in the US and is associated with reductions in quality of life and poor occupational functioning.^1,2 PTSD is often associated with multiple cognitive deficits that play a role in a number of clinical symptoms and impair cognition beyond what can be solely attributed to the effects of physical or psychological trauma.^3-5 Although the literature on the pattern and magnitude of cognitive deficits associated with PTSD is mixed, dysfunction in attention, verbal memory, speed of information processing, working memory, and executive functioning are the most consistent findings.^6-11Verbal memory and attention seem to be particularly negatively impacted by PTSD and especially so in combat-exposed war veterans.^7,12 Verbal memory difficulties in returning war veterans also may mediate quality of life and be particularly disruptive to everyday functioning.¹³ Further, evidence exists that a diagnosis of PTSD is associated with increased risk for dementia and deficits in episodic memory in older adults.^14,15

The PTSD-associated cognitive deficits are routinely assessed through neuropsychological measures within the US Department of Veteran Affairs (VA). The Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) is a commonly used cognitive screening measure in medical settings, and prior research has reinforced its clinical utility across a variety of populations, including Alzheimer disease, schizophrenia, Parkinson disease, Huntington disease, stroke, and traumatic brain injury (TBI).^16-24

McKay and colleagues previously examined the use of the RBANS within a sample of individuals who had a history of moderate-to-severe TBIs, with findings suggesting the RBANS is a valid and reliable screening measure in this population.²⁵However, McKay and colleagues used a carefully defined sample in a cognitive neurorehabilitation setting, many of whom experienced a TBI significant enough to require ongoing medical monitoring, attendant care, or substantial support services.

The influence of PTSD-associated cognitive deficits on the RBANS performance is unclear, and which subtests of the measure, if any, are differentially impacted in individuals with and those without a diagnosis of PTSD is uncertain. Further, less is known about the influence of PTSD in outpatient clinical settings when PTSD and TBI are not necessarily the primary presenting problem. The purpose of the current study was to determine the influence of a PTSD diagnosis on performance on the RBANS in an outpatient VA setting.

Methods

Participants included 153 veterans who were 90% male with a mean (SD) age of 46.8 (11.3) years and a mean (SD) education of 14.2 (2.3) years from a catchment area ranging from Montana south through western Texas, and all states west of that line, sequentially evaluated as part of a clinic workup at the California War Related Illness and Injury Study Center (WRIISC-CA). WRIISC-CA is a second-level evaluation clinic under patient primary care in the VA system dedicated to providing comprehensive medical evaluations on postdeployment veterans with complex medical concerns, including possible TBI and PTSD. Participants included 23 Vietnam-era, 72 Operation Desert Storm/Desert Shield-era, and 58 Operation Iraqi Freedom/Enduring Freedom-era veterans. We have previously published a more thorough analysis of medical characteristics for a WRIISC-CA sample.²⁶

A Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM IV) diagnosis of current PTSD was determined by the Clinician-Administered PTSD Scale (CAPS-IV), as administered or supervised by a licensed clinical psychologist during the course of the larger medical evaluation.²⁷ Given the co-occurring nature of TBI and PTSD and their complicated relationship with regard to cognitive functioning, all veterans also underwent a comprehensive examination by a board-certified neurologist to assess for a possible history of TBI, based on the presence of at least 1 past event according to the guidelines recommended by the American Congress of Rehabilitation Medicine.^28,29Veterans were categorized as having a history of no TBI, mild TBI, or moderate TBI. No veterans met criteria for history of severe TBI.Veterans were excluded from the analysis if unable to complete the mental health, neurological, or cognitive evaluations. Informed consent was obtained consistent with the Declaration of Helsinki and institutional guidelines established by the VA Palo Alto Human Subjects Review Committee. The study was approved by the VA Palo Alto and Stanford School of Medicine institutional review boards.

 

Cognitive Measures

All veterans completed a targeted cognitive battery that included the following: a reading recognition measure designed to estimate premorbid intellectual functioning (Wechsler Test of Adult Reading [WTAR]); a measure assessing auditory attention and working memory ability (Wechsler Adults Intelligence Scale-IV [WAIS-IV] Digit Span subtest); a measure assessing processing speed, attention, and cognitive flexibility (Trails A and B); and the RBANS.^16,30-32The focus of the current study was on the RBANS, a brief cognitive screening measure that contains 12 subtests examining a variety of cognitive functions. Given that all participants were veterans receiving outpatient services, there was no nonpatient control group for comparison. To address this, all raw data were converted to standardized scores based on healthy normative data provided within the test manual. Specifically, the 12 RBANS subtest scores were converted to age-corrected standardized z scores, which in turn created a total summary score and 5 composite summary indexes: immediate memory, visuospatial/constructional, attention, language, and delayed memory. All veterans completed the Form A version of the measure.

Statistical Analyses

Group level differences on selective demographic and cognitive measures between veterans with a diagnosis of PTSD and those without were examined using t tests. Cognitive variables included standardized scores for the RBANS, including age-adjusted total summary score, index scores, and subtest scores.¹⁶ Estimated full-scale IQ and standardized summary scores from the WTAR, demographically adjusted standardized scores for the total time to complete Trails A and time to complete Trails B, and age-adjusted standardized scores for the WAIS-IV Digit Span subtest (forward, backward, and sequencing trials, as well as the summary total score) were examined for group differences.^30,31,33 To further examine the association between PTSD and RBANS performance, multivariate multiple regressions were conducted using measures of episodic memory and processing speed from the RBANS (ie, story tasks, list learning tasks, and coding subtests). These specific measures were selected ad hoc based on extant literature.^6,10The dependent variable for each analysis was the standardized score from the selected subtest; PTSD status, a diagnosis of TBI, a diagnosis of co-occurring TBI and PTSD, gender, and years of education were predictor variables.

Results

Of the 153 study participants, 98 (64%) met DSM-4 criteria for current PTSD, whereas 55 (36%) did not (Table). There was no group statistical difference between veterans with or without a diagnosis of PTSD for age, education, or gender (P < .05). A diagnosis of PTSD tended to be more frequent in participants with a history of head injury (χ² = 7.72; P < .05). Veterans with a diagnosis of PTSD performed significantly worse on the RBANS Story Recall subtest compared with the results of those without PTSD (t[138] = 3.10; P < .01); performance on other cognitive measures was not significantly different between the PTSD groups. A diagnosis of PTSD was also significantly associated with self-reported depressive symptoms (Beck Depression Inventory-II; t[123] = -2.81; P < .01). Depressive symptoms were not associated with a history of TBI, and group differences were not significant.

Given the high co-occurrence of PTSD and TBI (68%) in our PTSD sample, secondary analyses examined the association of select diagnoses with performance on the RBANS, specifically veterans with a historical diagnosis of TBI (n = 92) from those without a diagnosis of TBI (n = 61), as well as those with co-occurring PTSD and TBI (n = 71) from those without (n = 82). The majority of the sample met criteria for a history of mild TBI (n = 79) when compared with moderate TBI (n = 13); none met criteria for a past history of severe TBI. PTSD status (β = .63, P = .04) and years of education (β = .16, P < .01) were associated with performance on the RBANS Story Recall subtest (R²= .23, F[5,139] = 8.11, P < .01). Education was the only significant predictor for the rest of the multivariate multiple regressions (all P < .05). A diagnosis of TBI or co-occurring PTSD and TBI was not significantly associated with performance on the Story Memory, Story Recall, List Learning, List Recall, or Coding subtests. multivariate analysis of variance tests for the hypothesis of an overall main effect of PTSD (F(5,130) = 1.08, P = .34), TBI (F[5,130] = .91, P = .48), or PTSD+TBI (F[5,130] =.47, P = .80) on the 4 selected tests were not significant.

 

Discussion

The findings of the present study suggest that veterans with PTSD perform worse on specific RBANS subtests compared with veterans without PTSD. Specifically, worse performance on the Story Recall subtest of the RBANS memory index was a significant predictor of a diagnosis of PTSD within the statistical model. This association with PTSD was not seen in other demographic (excluding education) or cognitive measures, including other memory tasks, such as List Recall and Figure Recall, and attentional measures, such as WAIS-IV Digit Span, and the Trail Making Test. Overall RBANS index scores were not significantly different between groups, though this is not surprising given that recent research suggests the RBANS composite scores have questionable validity and reliability.³⁴

The finding that a measure of episodic memory is most influenced by PTSD status is consistent with prior research.³⁵ However, there are several possible reasons why Story Recall in particular showed the greatest association, even more than other episodic memory measures. A review by Isaac and colleagues found a diagnosis of PTSD correlated with frontal lobe-associated memory deficits.⁶ As Story Recall provides only 2 rehearsal trials compared with the 4 trials provided in the RBANS List Learning subtest, it is possible that Story Recall relies more on attentional processes than on learning with repetition.

Research has indicated attention and verbal episodic memory dysfunction are associated with a diagnosis of PTSD in combat veterans, and individuals with a diagnosis of PTSD show deficits in executive functioning, including attention difficulties beyond what is seen in trauma-exposed controls.^4,7,8,11,35Furthermore, a diagnosis of PTSD has been shown to be associated with impaired performance on the Logical Memory subtest of the Wechsler Memory Scale-Revised, a very similar measure to the RBANS Story Recall.³⁶

The present finding that performance on a RBANS subtest was associated with a diagnosis of PTSD but not a history of TBI is not surprising. The majority of the present sample who reported a history of TBI met criteria for a remote head injury of mild severity (86%). Cognitive symptoms related to mild TBI are thought to generally resolve over time, and recent research suggests that PTSD symptoms may account for a substantial portion of reported postconcussive symptoms.^37,38Similarly, recent research suggests a diagnosis of mild TBI does not necessarily result in additive cognitive impairment in combat veterans with a diagnosis of PTSD, and that a diagnosis of PTSD is more strongly associated with cognitive symptoms than is mild TBI.^5,39,40

The lack of association with RBANS performance and co-occurring PTSD and TBI is less clear. Although both conditions are heterogenous, it may be that individuals with a diagnosis solely of PTSD are quantitively different from those with a concurrent diagnosis of PTSD and TBI (ie, PTSD presumed due to a mild TBI). Specifically, the impact of PTSD on cognition may be related to symptom severity and indexed trauma. A published systematic review on the PTSD-related cognitive impairment showed a medium-to-strong effect size for severity of PTSD symptoms on cognitive performance, with war trauma showing the strongest effect.⁴In particular, individuals who experience repeated or complex trauma are prone to experience PTSD symptoms with concurrent cognitive deficits, again suggesting the possibility of qualitative differences between outpatient veterans with PTSD and those with mild TBI associated PTSD.⁴¹While disentangling PTSD and mild TBI symptoms are notoriously difficult, future research aiming to examine these factors may be beneficial in the ability to draw larger conclusions on the relationship between cognition and PTSD.

 

Limitations

Several limitations may affect the generalizability of the findings. The present study used a veteran sample referred to a specialty clinic for complicated postdeployment health concerns. Although findings may not be representative of an inpatient population or clinics that focus solely on TBI, they may more adequately reflect veterans using clinical services at VA medical centers. We also did not include measures of PTSD symptom severity (eg, Posttraumatic Stress Disorder Checklist), instead using diagnosis based on the gold standard CAPS. In addition, the likelihood of the presence of a remote TBI was based on a clinical interview with a neurologist and not on acute neurologic findings. TBI is a heterogenous diagnosis, with multiple factors that likely influence cognitive performance, including location of the injury, type of injury, and time since injury, which may be lost during group analysis. Further, the RBANS is not intended to serve as a method for a differential diagnosis of PTSD or TBI. Concordant with this, the intention of the current study was to capture the quality of cognitive function on the RBANS within individuals with PTSD.

Conslusions

The ability for veterans to remember a short story following a delay (ie, RBANS Story Recall subtest) was negatively associated with a diagnosis of PTSD. Further, the RBANS best captured cognitive deficits associated with PTSD compared with those with a history of mild TBI, or co-occurring mild TBI and PTSD. These findings may provide insight into the interpretation and attribution of cognitive deficits in the veteran population and holds potential to guide future research examining focused cognitive phenotypes to provide precision targets in individual treatment.

Posttraumatic stress disorder (PTSD) affects about 10 to 25% of veterans in the US and is associated with reductions in quality of life and poor occupational functioning.^1,2 PTSD is often associated with multiple cognitive deficits that play a role in a number of clinical symptoms and impair cognition beyond what can be solely attributed to the effects of physical or psychological trauma.^3-5 Although the literature on the pattern and magnitude of cognitive deficits associated with PTSD is mixed, dysfunction in attention, verbal memory, speed of information processing, working memory, and executive functioning are the most consistent findings.^6-11Verbal memory and attention seem to be particularly negatively impacted by PTSD and especially so in combat-exposed war veterans.^7,12 Verbal memory difficulties in returning war veterans also may mediate quality of life and be particularly disruptive to everyday functioning.¹³ Further, evidence exists that a diagnosis of PTSD is associated with increased risk for dementia and deficits in episodic memory in older adults.^14,15

The PTSD-associated cognitive deficits are routinely assessed through neuropsychological measures within the US Department of Veteran Affairs (VA). The Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) is a commonly used cognitive screening measure in medical settings, and prior research has reinforced its clinical utility across a variety of populations, including Alzheimer disease, schizophrenia, Parkinson disease, Huntington disease, stroke, and traumatic brain injury (TBI).^16-24

McKay and colleagues previously examined the use of the RBANS within a sample of individuals who had a history of moderate-to-severe TBIs, with findings suggesting the RBANS is a valid and reliable screening measure in this population.²⁵However, McKay and colleagues used a carefully defined sample in a cognitive neurorehabilitation setting, many of whom experienced a TBI significant enough to require ongoing medical monitoring, attendant care, or substantial support services.

The influence of PTSD-associated cognitive deficits on the RBANS performance is unclear, and which subtests of the measure, if any, are differentially impacted in individuals with and those without a diagnosis of PTSD is uncertain. Further, less is known about the influence of PTSD in outpatient clinical settings when PTSD and TBI are not necessarily the primary presenting problem. The purpose of the current study was to determine the influence of a PTSD diagnosis on performance on the RBANS in an outpatient VA setting.

Methods

Participants included 153 veterans who were 90% male with a mean (SD) age of 46.8 (11.3) years and a mean (SD) education of 14.2 (2.3) years from a catchment area ranging from Montana south through western Texas, and all states west of that line, sequentially evaluated as part of a clinic workup at the California War Related Illness and Injury Study Center (WRIISC-CA). WRIISC-CA is a second-level evaluation clinic under patient primary care in the VA system dedicated to providing comprehensive medical evaluations on postdeployment veterans with complex medical concerns, including possible TBI and PTSD. Participants included 23 Vietnam-era, 72 Operation Desert Storm/Desert Shield-era, and 58 Operation Iraqi Freedom/Enduring Freedom-era veterans. We have previously published a more thorough analysis of medical characteristics for a WRIISC-CA sample.²⁶

A Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM IV) diagnosis of current PTSD was determined by the Clinician-Administered PTSD Scale (CAPS-IV), as administered or supervised by a licensed clinical psychologist during the course of the larger medical evaluation.²⁷ Given the co-occurring nature of TBI and PTSD and their complicated relationship with regard to cognitive functioning, all veterans also underwent a comprehensive examination by a board-certified neurologist to assess for a possible history of TBI, based on the presence of at least 1 past event according to the guidelines recommended by the American Congress of Rehabilitation Medicine.^28,29Veterans were categorized as having a history of no TBI, mild TBI, or moderate TBI. No veterans met criteria for history of severe TBI.Veterans were excluded from the analysis if unable to complete the mental health, neurological, or cognitive evaluations. Informed consent was obtained consistent with the Declaration of Helsinki and institutional guidelines established by the VA Palo Alto Human Subjects Review Committee. The study was approved by the VA Palo Alto and Stanford School of Medicine institutional review boards.

 

Cognitive Measures

All veterans completed a targeted cognitive battery that included the following: a reading recognition measure designed to estimate premorbid intellectual functioning (Wechsler Test of Adult Reading [WTAR]); a measure assessing auditory attention and working memory ability (Wechsler Adults Intelligence Scale-IV [WAIS-IV] Digit Span subtest); a measure assessing processing speed, attention, and cognitive flexibility (Trails A and B); and the RBANS.^16,30-32The focus of the current study was on the RBANS, a brief cognitive screening measure that contains 12 subtests examining a variety of cognitive functions. Given that all participants were veterans receiving outpatient services, there was no nonpatient control group for comparison. To address this, all raw data were converted to standardized scores based on healthy normative data provided within the test manual. Specifically, the 12 RBANS subtest scores were converted to age-corrected standardized z scores, which in turn created a total summary score and 5 composite summary indexes: immediate memory, visuospatial/constructional, attention, language, and delayed memory. All veterans completed the Form A version of the measure.

Statistical Analyses

Group level differences on selective demographic and cognitive measures between veterans with a diagnosis of PTSD and those without were examined using t tests. Cognitive variables included standardized scores for the RBANS, including age-adjusted total summary score, index scores, and subtest scores.¹⁶ Estimated full-scale IQ and standardized summary scores from the WTAR, demographically adjusted standardized scores for the total time to complete Trails A and time to complete Trails B, and age-adjusted standardized scores for the WAIS-IV Digit Span subtest (forward, backward, and sequencing trials, as well as the summary total score) were examined for group differences.^30,31,33 To further examine the association between PTSD and RBANS performance, multivariate multiple regressions were conducted using measures of episodic memory and processing speed from the RBANS (ie, story tasks, list learning tasks, and coding subtests). These specific measures were selected ad hoc based on extant literature.^6,10The dependent variable for each analysis was the standardized score from the selected subtest; PTSD status, a diagnosis of TBI, a diagnosis of co-occurring TBI and PTSD, gender, and years of education were predictor variables.

Results

Of the 153 study participants, 98 (64%) met DSM-4 criteria for current PTSD, whereas 55 (36%) did not (Table). There was no group statistical difference between veterans with or without a diagnosis of PTSD for age, education, or gender (P < .05). A diagnosis of PTSD tended to be more frequent in participants with a history of head injury (χ² = 7.72; P < .05). Veterans with a diagnosis of PTSD performed significantly worse on the RBANS Story Recall subtest compared with the results of those without PTSD (t[138] = 3.10; P < .01); performance on other cognitive measures was not significantly different between the PTSD groups. A diagnosis of PTSD was also significantly associated with self-reported depressive symptoms (Beck Depression Inventory-II; t[123] = -2.81; P < .01). Depressive symptoms were not associated with a history of TBI, and group differences were not significant.

Given the high co-occurrence of PTSD and TBI (68%) in our PTSD sample, secondary analyses examined the association of select diagnoses with performance on the RBANS, specifically veterans with a historical diagnosis of TBI (n = 92) from those without a diagnosis of TBI (n = 61), as well as those with co-occurring PTSD and TBI (n = 71) from those without (n = 82). The majority of the sample met criteria for a history of mild TBI (n = 79) when compared with moderate TBI (n = 13); none met criteria for a past history of severe TBI. PTSD status (β = .63, P = .04) and years of education (β = .16, P < .01) were associated with performance on the RBANS Story Recall subtest (R²= .23, F[5,139] = 8.11, P < .01). Education was the only significant predictor for the rest of the multivariate multiple regressions (all P < .05). A diagnosis of TBI or co-occurring PTSD and TBI was not significantly associated with performance on the Story Memory, Story Recall, List Learning, List Recall, or Coding subtests. multivariate analysis of variance tests for the hypothesis of an overall main effect of PTSD (F(5,130) = 1.08, P = .34), TBI (F[5,130] = .91, P = .48), or PTSD+TBI (F[5,130] =.47, P = .80) on the 4 selected tests were not significant.

 

Discussion

The findings of the present study suggest that veterans with PTSD perform worse on specific RBANS subtests compared with veterans without PTSD. Specifically, worse performance on the Story Recall subtest of the RBANS memory index was a significant predictor of a diagnosis of PTSD within the statistical model. This association with PTSD was not seen in other demographic (excluding education) or cognitive measures, including other memory tasks, such as List Recall and Figure Recall, and attentional measures, such as WAIS-IV Digit Span, and the Trail Making Test. Overall RBANS index scores were not significantly different between groups, though this is not surprising given that recent research suggests the RBANS composite scores have questionable validity and reliability.³⁴

The finding that a measure of episodic memory is most influenced by PTSD status is consistent with prior research.³⁵ However, there are several possible reasons why Story Recall in particular showed the greatest association, even more than other episodic memory measures. A review by Isaac and colleagues found a diagnosis of PTSD correlated with frontal lobe-associated memory deficits.⁶ As Story Recall provides only 2 rehearsal trials compared with the 4 trials provided in the RBANS List Learning subtest, it is possible that Story Recall relies more on attentional processes than on learning with repetition.

Research has indicated attention and verbal episodic memory dysfunction are associated with a diagnosis of PTSD in combat veterans, and individuals with a diagnosis of PTSD show deficits in executive functioning, including attention difficulties beyond what is seen in trauma-exposed controls.^4,7,8,11,35Furthermore, a diagnosis of PTSD has been shown to be associated with impaired performance on the Logical Memory subtest of the Wechsler Memory Scale-Revised, a very similar measure to the RBANS Story Recall.³⁶

The present finding that performance on a RBANS subtest was associated with a diagnosis of PTSD but not a history of TBI is not surprising. The majority of the present sample who reported a history of TBI met criteria for a remote head injury of mild severity (86%). Cognitive symptoms related to mild TBI are thought to generally resolve over time, and recent research suggests that PTSD symptoms may account for a substantial portion of reported postconcussive symptoms.^37,38Similarly, recent research suggests a diagnosis of mild TBI does not necessarily result in additive cognitive impairment in combat veterans with a diagnosis of PTSD, and that a diagnosis of PTSD is more strongly associated with cognitive symptoms than is mild TBI.^5,39,40

The lack of association with RBANS performance and co-occurring PTSD and TBI is less clear. Although both conditions are heterogenous, it may be that individuals with a diagnosis solely of PTSD are quantitively different from those with a concurrent diagnosis of PTSD and TBI (ie, PTSD presumed due to a mild TBI). Specifically, the impact of PTSD on cognition may be related to symptom severity and indexed trauma. A published systematic review on the PTSD-related cognitive impairment showed a medium-to-strong effect size for severity of PTSD symptoms on cognitive performance, with war trauma showing the strongest effect.⁴In particular, individuals who experience repeated or complex trauma are prone to experience PTSD symptoms with concurrent cognitive deficits, again suggesting the possibility of qualitative differences between outpatient veterans with PTSD and those with mild TBI associated PTSD.⁴¹While disentangling PTSD and mild TBI symptoms are notoriously difficult, future research aiming to examine these factors may be beneficial in the ability to draw larger conclusions on the relationship between cognition and PTSD.

 

Limitations

Several limitations may affect the generalizability of the findings. The present study used a veteran sample referred to a specialty clinic for complicated postdeployment health concerns. Although findings may not be representative of an inpatient population or clinics that focus solely on TBI, they may more adequately reflect veterans using clinical services at VA medical centers. We also did not include measures of PTSD symptom severity (eg, Posttraumatic Stress Disorder Checklist), instead using diagnosis based on the gold standard CAPS. In addition, the likelihood of the presence of a remote TBI was based on a clinical interview with a neurologist and not on acute neurologic findings. TBI is a heterogenous diagnosis, with multiple factors that likely influence cognitive performance, including location of the injury, type of injury, and time since injury, which may be lost during group analysis. Further, the RBANS is not intended to serve as a method for a differential diagnosis of PTSD or TBI. Concordant with this, the intention of the current study was to capture the quality of cognitive function on the RBANS within individuals with PTSD.

Conslusions

The ability for veterans to remember a short story following a delay (ie, RBANS Story Recall subtest) was negatively associated with a diagnosis of PTSD. Further, the RBANS best captured cognitive deficits associated with PTSD compared with those with a history of mild TBI, or co-occurring mild TBI and PTSD. These findings may provide insight into the interpretation and attribution of cognitive deficits in the veteran population and holds potential to guide future research examining focused cognitive phenotypes to provide precision targets in individual treatment.

Since 2001, nearly 2,000 US military service members have sustained traumatically acquired limb loss while serving in conflict zones primarily in Afghanistan and Iraq.¹ Although most of these patients receive acute and long-term care in a military health facility, polytrauma programs within the Veterans Health Administration (VHA) treat other military patients with traumatic injuries while others receive specialized care in civilian medical programs. The Military Advanced Training Center (MATC) at Walter Reed National Military Medical Center (WRNMMC) provides a comprehensive rehabilitation program for patients with acquired traumatic limb loss.²

In this paper, we argue that receiving long-term care in military settings provides unique value for military patients because of the background therapeutic work such settings can provide. Currently, there are policy discussions that center on consolidating military health care under the oversight of the Defense Health Agency. This approach would develop a more centralized administration while also pursuing other measures to improve efficiency. When evaluating the current system, one key question remains: Would military service members and dependents seeking specific care or long-term rehabilitation programs be more effectively treated in nonmilitary settings?

Based on qualitative research, we argue that keeping a diverse range of military health programs has a positive and therapeutic impact. We also argue that the emergent literature about the importance of military culture to patients and the need for military cultural competence training for nonmilitary clinicians coupled with the results of a qualitative study of former patients at WRNMMC demonstrate that the social context at military treatment facilities offers a positive therapeutic impact.³

 

Program Description

This article is grounded in research conducted in the US Armed Forces Amputee Patient Care Program at WRNMMC. The study received WRNMMC Institutional Review Board approval in February 2012 and again for the continuation study in January 2015. The lead investigator for the research project was a medical anthropologist who worked with a research unit in the WRNMMC Department of Rehabilitation.

The main period of data collection occurred in 2 waves, the first between 2012 and 2014 and the second between 2015 and 2019. Patients arrived at WRNMMC within several days from the site of their injuries (nearly all were from Iraq and Afghanistan) via military medical facilities in Germany. After a period of recovery from the acute phase of their injuries, patients transitioned to outpatient housing and began their longer phase of care in the outpatient MATC.

On MATC admission, patients were assigned an occupational therapist, physical therapist, and prosthetist. In addition, rehabilitation physicians and orthopedic surgeons oversaw patient care. Social work and other programs provided additional services as needed.² Patients were treated primarily for their orthopedic and extremity trauma and for neuropsychiatric injuries, such as mild traumatic brain injury. Other behavioral health services were available to support patients who reported symptoms of posttraumatic stress disorder, anxiety, depression, or other neuropsychological issues.

Patients had multiple daily appointments that shifted throughout the duration of their care. Initially a patient might have 2 physical therapy and 2 occupational therapy appointments daily, with each session lasting about an hour. Appointments with the orthotics and prosthetics service, which could be considerably longer were added as needed. These appointments required multiple castings, fittings, adjustments, and other activities. This also was the case with wound care, behavioral health, and other services and departments.

Cultural Competency

A recently published special issue of Academic Psychiatry described the important role that basic knowledge of military culture plays in effective care delivery to active-duty service members, guard and reserve, and veteran patients and families.⁴ Reger and colleagues also emphasized the importance of awareness of military culture to civilian clinicians particularly those providing care to service members.⁵

This concern with gaps in knowledge about recognizing the realities of military culture has given rise to an emergent literature on military cultural competence training for clinical providers.⁶ Cultural competence in health care settings is understood to be the practice of providing care within a social framework that acknowledges the social and cultural background of patients.⁷ In the military context, as in others, these discussions often are limited to behavioral health settings.⁸ This emergent literature provides researchers with important insights into understanding the scope and scale of military culture and the importance of delivering culturally competent care.

Beyond the concept of cultural competence, recognizing the importance of culture can be used to understand positive therapeutic impacts. In discussions of culture, service-members, veterans, and family members are shown to have adopted a set of ideas, values, roles, and behaviors. Mastering an awareness of those attributes is part of the process of delivering culturally competent care. At WRNMMC and other military treatment facilities, those attributes are “baked in” to the delivery of service—even when that service is provided by civilians. How that process operates is important to understanding the impact of the organization of clinical care.

 

Methods

Data were extracted from research conducted between 2012 and 2014 that investigated how former patients evaluated their posttreatment lives considering the care received in the MATC at WRNMMC. We used a lightly structured set of interview questions and categories in each interview that focused on 3 themes in the individual’s pathway to injury: education, joining their branch of service, injury experience. The focus was on developing an understanding of how events antecedent to the injury experience could influence the rehabilitation experience and postcare life. The second focus was on the experience of rehabilitation and to learn how the individual navigated community living after leaving care.

Results

Thirty-five participants with lower extremity amputations were recruited who had been discharged from the Amputee Patient Care Program ≥ 12 months prior to the study (Table 1). Participants were interviewed either over the telephone, or when possible, in person. Interviews were based on a lightly structured schedule designed to elicit accounts of community integration, which attended to reports of belongingness supported by accounts of social engagement in work, school, family, and social events. Interviews were analyzed using a modified content analysis approach. The study did not rely on a structured interview, but as is the case with many qualitative and ethnographic interviews, each session shared themes in common, such as questions about injury experience, rehabilitation experience, life after care (work, school, relationships), and so forth. Interviews were conducted by the lead author who was a medical anthropologist with training in health services research.

Participants generally described their post-care lives as “successful” that had been built on “good outcomes.” We left these concepts loosely defined to grant participants latitude in developing their own definitions for these ideas. That said, there is reason to view participants' lives as meeting specific criteria of success (Table 2). For example:

• 16 participants attended higher education postrehabilitation;
• 18 participants were working, or had worked, at the time of the interview;
• There was overlap between these groups and the total that had worked or attended school was 33 of 35; and
• 2 participants who had neither worked nor attended school were still recovering from injury complications at the time of the interview.

Family and relationships were other areas of success (Table 3). Twenty-three participants were currently in long-term relationships, including a mix of marriage and cohabitation households, while 3 were recently divorced and 2 were divorced for a longer term. Of the 5 participants who had been divorced, 3 were interested in pursuing new relationships. All 5 of these participants had children and were actively involved in their lives. Seven participants were not in relationships. Two participants did not have or seek relationships because of complications associated with their ongoing recovery.

Whether considering the claims of participants, or how the literature conceptualizes successful community living, the evidence of success is supported by the accounts of work, school, and relationships. The attribution of these successes, in part, to the MATC rehabilitation program is important to understand because of the implications that this has on program value. Three features of the program continually emerged in interviews: recovering alongside peers, routine access to the entire treatment team, and ongoing relationships with key health care providers (HCPs).

 

Working Alongside Peers

Peers are an important element in how former patients remember their time in the rehabilitation program at WRNMMC. One benefit of recovering alongside peers is that it changes patients’ experiences of time. Being with other military patients creates a transitional time that participants said they valued as they shifted from the immediacy of their deployment experiences to the longer term demands of recovery and community reintegration.⁹ Additionally, sharing the clinical space with patients who had come before allowed participants to visualize a living timeline of their proposed recovery.

For most former patients, remembering the social intensity of their rehabilitation program is an important element in their narratives of recovery. The participants in our study do not necessarily maintain ties with their former peers, but nearly all of them point to support from other patients as being key in their own recovery from both the physical and psychological consequences of their injury.

Routine Access to the Treatment Team

The weekly amputee clinics that put surgeons, physicians, occupational and physical therapists, social workers, and prosthetists in a room with each patient worked to alleviate stress and anxiety in participants’ minds around the complexities of their injuries and care. One of the benefits of a group meeting is that it reduces the risk of miscommunication among HCPs and between HCPs and patients.^7,10

These weekly sessions with HCPs and patients led to a second advantage; they promoted patient autonomy and participation in clinical decisions. Patients were able to negotiate clinical goals with their HCPs and then act on them almost immediately. In addition, patients with complex physical injuries, often with neurological or psychiatric comorbidities, were able to describe the full range of their challenges, and HCPs had an opportunity to check in with patients about the problems they faced and level of severity.

The ability to marshal clinical HCPs to attend weekly meetings of this nature with patients may be key for distinguishing military health care from VHA and civilian counterparts. More research on how clinical team/ patient meetings occur in other settings is needed. But one of the hallmark features of these clinic meetings at WRNMMC was their open-endedness. Patients typically were not bound by 15 minute or other temporally delimited meeting intervals. This research indicates that in military health care, the patient is the leader.

Continuity of Care

Continuity of care is a well-understood benefit to working with the same HCP. There were additional unanticipated benefits to assigning patients to HCPs with whom they had worked. The long-term period of care (5-24 months) gave patients the opportunity to develop multifaceted relationships with their HCPs and empowered them to advocate and negotiate for their outcome goals. In addition, the majority of frontline HCPs in physical and occupational therapy were civil ians (all prosthetic providers were civilians). These ongoing relationships had the impact of socializing clinicians into the expectations of military culture (around physical training, endurance and resilience, and disregard of pain).

Physical and occupational therapists occupied multiple roles for their patients, including being teachers, coaches, and sounding boards. Participants frequently described the way that their physical or occupational therapist could, on one hand, push them to achieve more in terms of physical functioning. But on the other hand, participants also talked about the emotional and psychological support they could receive based both on the long duration of their work with their care providers.

 

Conclusions

The ability to recover alongside peers, have access to the whole treatment team and develop long-term relationships with key care HCPs served as drivers for positive recovery. The impact of these 3 drivers of the social organization of the Amputee Patient Care Program represent an opportunity to highlight the role that the social context of military health care to use in achieving positive therapeutic outcomes. Former patients of the WRNMMC program could rely on a familiar and dependable social context for their care. This social context draws heavily on elements of military culture that structure the preinjury worlds of work and life that patients occupied. Based on these results we argue that the presence of rehabilitation and other clinical units in military medical settings offers an important value to patients and HCPs.

Acknowledgments

This work was supported by the Center for Rehabilitation Science Research, Department of Physical Medicine & Rehabilitation, Uniformed Services University, Bethesda, MD (awards HU0001-11-1-0004 and HU0001-15-2-0003).

Since 2001, nearly 2,000 US military service members have sustained traumatically acquired limb loss while serving in conflict zones primarily in Afghanistan and Iraq.¹ Although most of these patients receive acute and long-term care in a military health facility, polytrauma programs within the Veterans Health Administration (VHA) treat other military patients with traumatic injuries while others receive specialized care in civilian medical programs. The Military Advanced Training Center (MATC) at Walter Reed National Military Medical Center (WRNMMC) provides a comprehensive rehabilitation program for patients with acquired traumatic limb loss.²

In this paper, we argue that receiving long-term care in military settings provides unique value for military patients because of the background therapeutic work such settings can provide. Currently, there are policy discussions that center on consolidating military health care under the oversight of the Defense Health Agency. This approach would develop a more centralized administration while also pursuing other measures to improve efficiency. When evaluating the current system, one key question remains: Would military service members and dependents seeking specific care or long-term rehabilitation programs be more effectively treated in nonmilitary settings?

Based on qualitative research, we argue that keeping a diverse range of military health programs has a positive and therapeutic impact. We also argue that the emergent literature about the importance of military culture to patients and the need for military cultural competence training for nonmilitary clinicians coupled with the results of a qualitative study of former patients at WRNMMC demonstrate that the social context at military treatment facilities offers a positive therapeutic impact.³

 

Program Description

This article is grounded in research conducted in the US Armed Forces Amputee Patient Care Program at WRNMMC. The study received WRNMMC Institutional Review Board approval in February 2012 and again for the continuation study in January 2015. The lead investigator for the research project was a medical anthropologist who worked with a research unit in the WRNMMC Department of Rehabilitation.

The main period of data collection occurred in 2 waves, the first between 2012 and 2014 and the second between 2015 and 2019. Patients arrived at WRNMMC within several days from the site of their injuries (nearly all were from Iraq and Afghanistan) via military medical facilities in Germany. After a period of recovery from the acute phase of their injuries, patients transitioned to outpatient housing and began their longer phase of care in the outpatient MATC.

On MATC admission, patients were assigned an occupational therapist, physical therapist, and prosthetist. In addition, rehabilitation physicians and orthopedic surgeons oversaw patient care. Social work and other programs provided additional services as needed.² Patients were treated primarily for their orthopedic and extremity trauma and for neuropsychiatric injuries, such as mild traumatic brain injury. Other behavioral health services were available to support patients who reported symptoms of posttraumatic stress disorder, anxiety, depression, or other neuropsychological issues.

Patients had multiple daily appointments that shifted throughout the duration of their care. Initially a patient might have 2 physical therapy and 2 occupational therapy appointments daily, with each session lasting about an hour. Appointments with the orthotics and prosthetics service, which could be considerably longer were added as needed. These appointments required multiple castings, fittings, adjustments, and other activities. This also was the case with wound care, behavioral health, and other services and departments.

Cultural Competency

A recently published special issue of Academic Psychiatry described the important role that basic knowledge of military culture plays in effective care delivery to active-duty service members, guard and reserve, and veteran patients and families.⁴ Reger and colleagues also emphasized the importance of awareness of military culture to civilian clinicians particularly those providing care to service members.⁵

This concern with gaps in knowledge about recognizing the realities of military culture has given rise to an emergent literature on military cultural competence training for clinical providers.⁶ Cultural competence in health care settings is understood to be the practice of providing care within a social framework that acknowledges the social and cultural background of patients.⁷ In the military context, as in others, these discussions often are limited to behavioral health settings.⁸ This emergent literature provides researchers with important insights into understanding the scope and scale of military culture and the importance of delivering culturally competent care.

Beyond the concept of cultural competence, recognizing the importance of culture can be used to understand positive therapeutic impacts. In discussions of culture, service-members, veterans, and family members are shown to have adopted a set of ideas, values, roles, and behaviors. Mastering an awareness of those attributes is part of the process of delivering culturally competent care. At WRNMMC and other military treatment facilities, those attributes are “baked in” to the delivery of service—even when that service is provided by civilians. How that process operates is important to understanding the impact of the organization of clinical care.

 

Methods

Data were extracted from research conducted between 2012 and 2014 that investigated how former patients evaluated their posttreatment lives considering the care received in the MATC at WRNMMC. We used a lightly structured set of interview questions and categories in each interview that focused on 3 themes in the individual’s pathway to injury: education, joining their branch of service, injury experience. The focus was on developing an understanding of how events antecedent to the injury experience could influence the rehabilitation experience and postcare life. The second focus was on the experience of rehabilitation and to learn how the individual navigated community living after leaving care.

Results

Thirty-five participants with lower extremity amputations were recruited who had been discharged from the Amputee Patient Care Program ≥ 12 months prior to the study (Table 1). Participants were interviewed either over the telephone, or when possible, in person. Interviews were based on a lightly structured schedule designed to elicit accounts of community integration, which attended to reports of belongingness supported by accounts of social engagement in work, school, family, and social events. Interviews were analyzed using a modified content analysis approach. The study did not rely on a structured interview, but as is the case with many qualitative and ethnographic interviews, each session shared themes in common, such as questions about injury experience, rehabilitation experience, life after care (work, school, relationships), and so forth. Interviews were conducted by the lead author who was a medical anthropologist with training in health services research.

Participants generally described their post-care lives as “successful” that had been built on “good outcomes.” We left these concepts loosely defined to grant participants latitude in developing their own definitions for these ideas. That said, there is reason to view participants' lives as meeting specific criteria of success (Table 2). For example:

• 16 participants attended higher education postrehabilitation;
• 18 participants were working, or had worked, at the time of the interview;
• There was overlap between these groups and the total that had worked or attended school was 33 of 35; and
• 2 participants who had neither worked nor attended school were still recovering from injury complications at the time of the interview.

Family and relationships were other areas of success (Table 3). Twenty-three participants were currently in long-term relationships, including a mix of marriage and cohabitation households, while 3 were recently divorced and 2 were divorced for a longer term. Of the 5 participants who had been divorced, 3 were interested in pursuing new relationships. All 5 of these participants had children and were actively involved in their lives. Seven participants were not in relationships. Two participants did not have or seek relationships because of complications associated with their ongoing recovery.

Whether considering the claims of participants, or how the literature conceptualizes successful community living, the evidence of success is supported by the accounts of work, school, and relationships. The attribution of these successes, in part, to the MATC rehabilitation program is important to understand because of the implications that this has on program value. Three features of the program continually emerged in interviews: recovering alongside peers, routine access to the entire treatment team, and ongoing relationships with key health care providers (HCPs).

 

Working Alongside Peers

Peers are an important element in how former patients remember their time in the rehabilitation program at WRNMMC. One benefit of recovering alongside peers is that it changes patients’ experiences of time. Being with other military patients creates a transitional time that participants said they valued as they shifted from the immediacy of their deployment experiences to the longer term demands of recovery and community reintegration.⁹ Additionally, sharing the clinical space with patients who had come before allowed participants to visualize a living timeline of their proposed recovery.

For most former patients, remembering the social intensity of their rehabilitation program is an important element in their narratives of recovery. The participants in our study do not necessarily maintain ties with their former peers, but nearly all of them point to support from other patients as being key in their own recovery from both the physical and psychological consequences of their injury.

Routine Access to the Treatment Team

The weekly amputee clinics that put surgeons, physicians, occupational and physical therapists, social workers, and prosthetists in a room with each patient worked to alleviate stress and anxiety in participants’ minds around the complexities of their injuries and care. One of the benefits of a group meeting is that it reduces the risk of miscommunication among HCPs and between HCPs and patients.^7,10

These weekly sessions with HCPs and patients led to a second advantage; they promoted patient autonomy and participation in clinical decisions. Patients were able to negotiate clinical goals with their HCPs and then act on them almost immediately. In addition, patients with complex physical injuries, often with neurological or psychiatric comorbidities, were able to describe the full range of their challenges, and HCPs had an opportunity to check in with patients about the problems they faced and level of severity.

The ability to marshal clinical HCPs to attend weekly meetings of this nature with patients may be key for distinguishing military health care from VHA and civilian counterparts. More research on how clinical team/ patient meetings occur in other settings is needed. But one of the hallmark features of these clinic meetings at WRNMMC was their open-endedness. Patients typically were not bound by 15 minute or other temporally delimited meeting intervals. This research indicates that in military health care, the patient is the leader.

Continuity of Care

Continuity of care is a well-understood benefit to working with the same HCP. There were additional unanticipated benefits to assigning patients to HCPs with whom they had worked. The long-term period of care (5-24 months) gave patients the opportunity to develop multifaceted relationships with their HCPs and empowered them to advocate and negotiate for their outcome goals. In addition, the majority of frontline HCPs in physical and occupational therapy were civil ians (all prosthetic providers were civilians). These ongoing relationships had the impact of socializing clinicians into the expectations of military culture (around physical training, endurance and resilience, and disregard of pain).

Physical and occupational therapists occupied multiple roles for their patients, including being teachers, coaches, and sounding boards. Participants frequently described the way that their physical or occupational therapist could, on one hand, push them to achieve more in terms of physical functioning. But on the other hand, participants also talked about the emotional and psychological support they could receive based both on the long duration of their work with their care providers.

 

Conclusions

The ability to recover alongside peers, have access to the whole treatment team and develop long-term relationships with key care HCPs served as drivers for positive recovery. The impact of these 3 drivers of the social organization of the Amputee Patient Care Program represent an opportunity to highlight the role that the social context of military health care to use in achieving positive therapeutic outcomes. Former patients of the WRNMMC program could rely on a familiar and dependable social context for their care. This social context draws heavily on elements of military culture that structure the preinjury worlds of work and life that patients occupied. Based on these results we argue that the presence of rehabilitation and other clinical units in military medical settings offers an important value to patients and HCPs.

Acknowledgments

This work was supported by the Center for Rehabilitation Science Research, Department of Physical Medicine & Rehabilitation, Uniformed Services University, Bethesda, MD (awards HU0001-11-1-0004 and HU0001-15-2-0003).

Since 2001, nearly 2,000 US military service members have sustained traumatically acquired limb loss while serving in conflict zones primarily in Afghanistan and Iraq.¹ Although most of these patients receive acute and long-term care in a military health facility, polytrauma programs within the Veterans Health Administration (VHA) treat other military patients with traumatic injuries while others receive specialized care in civilian medical programs. The Military Advanced Training Center (MATC) at Walter Reed National Military Medical Center (WRNMMC) provides a comprehensive rehabilitation program for patients with acquired traumatic limb loss.²

In this paper, we argue that receiving long-term care in military settings provides unique value for military patients because of the background therapeutic work such settings can provide. Currently, there are policy discussions that center on consolidating military health care under the oversight of the Defense Health Agency. This approach would develop a more centralized administration while also pursuing other measures to improve efficiency. When evaluating the current system, one key question remains: Would military service members and dependents seeking specific care or long-term rehabilitation programs be more effectively treated in nonmilitary settings?

Based on qualitative research, we argue that keeping a diverse range of military health programs has a positive and therapeutic impact. We also argue that the emergent literature about the importance of military culture to patients and the need for military cultural competence training for nonmilitary clinicians coupled with the results of a qualitative study of former patients at WRNMMC demonstrate that the social context at military treatment facilities offers a positive therapeutic impact.³

 

Program Description

This article is grounded in research conducted in the US Armed Forces Amputee Patient Care Program at WRNMMC. The study received WRNMMC Institutional Review Board approval in February 2012 and again for the continuation study in January 2015. The lead investigator for the research project was a medical anthropologist who worked with a research unit in the WRNMMC Department of Rehabilitation.

The main period of data collection occurred in 2 waves, the first between 2012 and 2014 and the second between 2015 and 2019. Patients arrived at WRNMMC within several days from the site of their injuries (nearly all were from Iraq and Afghanistan) via military medical facilities in Germany. After a period of recovery from the acute phase of their injuries, patients transitioned to outpatient housing and began their longer phase of care in the outpatient MATC.

On MATC admission, patients were assigned an occupational therapist, physical therapist, and prosthetist. In addition, rehabilitation physicians and orthopedic surgeons oversaw patient care. Social work and other programs provided additional services as needed.² Patients were treated primarily for their orthopedic and extremity trauma and for neuropsychiatric injuries, such as mild traumatic brain injury. Other behavioral health services were available to support patients who reported symptoms of posttraumatic stress disorder, anxiety, depression, or other neuropsychological issues.

Patients had multiple daily appointments that shifted throughout the duration of their care. Initially a patient might have 2 physical therapy and 2 occupational therapy appointments daily, with each session lasting about an hour. Appointments with the orthotics and prosthetics service, which could be considerably longer were added as needed. These appointments required multiple castings, fittings, adjustments, and other activities. This also was the case with wound care, behavioral health, and other services and departments.

Cultural Competency

A recently published special issue of Academic Psychiatry described the important role that basic knowledge of military culture plays in effective care delivery to active-duty service members, guard and reserve, and veteran patients and families.⁴ Reger and colleagues also emphasized the importance of awareness of military culture to civilian clinicians particularly those providing care to service members.⁵

This concern with gaps in knowledge about recognizing the realities of military culture has given rise to an emergent literature on military cultural competence training for clinical providers.⁶ Cultural competence in health care settings is understood to be the practice of providing care within a social framework that acknowledges the social and cultural background of patients.⁷ In the military context, as in others, these discussions often are limited to behavioral health settings.⁸ This emergent literature provides researchers with important insights into understanding the scope and scale of military culture and the importance of delivering culturally competent care.

Beyond the concept of cultural competence, recognizing the importance of culture can be used to understand positive therapeutic impacts. In discussions of culture, service-members, veterans, and family members are shown to have adopted a set of ideas, values, roles, and behaviors. Mastering an awareness of those attributes is part of the process of delivering culturally competent care. At WRNMMC and other military treatment facilities, those attributes are “baked in” to the delivery of service—even when that service is provided by civilians. How that process operates is important to understanding the impact of the organization of clinical care.

 

Methods

Data were extracted from research conducted between 2012 and 2014 that investigated how former patients evaluated their posttreatment lives considering the care received in the MATC at WRNMMC. We used a lightly structured set of interview questions and categories in each interview that focused on 3 themes in the individual’s pathway to injury: education, joining their branch of service, injury experience. The focus was on developing an understanding of how events antecedent to the injury experience could influence the rehabilitation experience and postcare life. The second focus was on the experience of rehabilitation and to learn how the individual navigated community living after leaving care.

Results

Thirty-five participants with lower extremity amputations were recruited who had been discharged from the Amputee Patient Care Program ≥ 12 months prior to the study (Table 1). Participants were interviewed either over the telephone, or when possible, in person. Interviews were based on a lightly structured schedule designed to elicit accounts of community integration, which attended to reports of belongingness supported by accounts of social engagement in work, school, family, and social events. Interviews were analyzed using a modified content analysis approach. The study did not rely on a structured interview, but as is the case with many qualitative and ethnographic interviews, each session shared themes in common, such as questions about injury experience, rehabilitation experience, life after care (work, school, relationships), and so forth. Interviews were conducted by the lead author who was a medical anthropologist with training in health services research.

Participants generally described their post-care lives as “successful” that had been built on “good outcomes.” We left these concepts loosely defined to grant participants latitude in developing their own definitions for these ideas. That said, there is reason to view participants' lives as meeting specific criteria of success (Table 2). For example:

• 16 participants attended higher education postrehabilitation;
• 18 participants were working, or had worked, at the time of the interview;
• There was overlap between these groups and the total that had worked or attended school was 33 of 35; and
• 2 participants who had neither worked nor attended school were still recovering from injury complications at the time of the interview.

Family and relationships were other areas of success (Table 3). Twenty-three participants were currently in long-term relationships, including a mix of marriage and cohabitation households, while 3 were recently divorced and 2 were divorced for a longer term. Of the 5 participants who had been divorced, 3 were interested in pursuing new relationships. All 5 of these participants had children and were actively involved in their lives. Seven participants were not in relationships. Two participants did not have or seek relationships because of complications associated with their ongoing recovery.

Whether considering the claims of participants, or how the literature conceptualizes successful community living, the evidence of success is supported by the accounts of work, school, and relationships. The attribution of these successes, in part, to the MATC rehabilitation program is important to understand because of the implications that this has on program value. Three features of the program continually emerged in interviews: recovering alongside peers, routine access to the entire treatment team, and ongoing relationships with key health care providers (HCPs).

 

Working Alongside Peers

Peers are an important element in how former patients remember their time in the rehabilitation program at WRNMMC. One benefit of recovering alongside peers is that it changes patients’ experiences of time. Being with other military patients creates a transitional time that participants said they valued as they shifted from the immediacy of their deployment experiences to the longer term demands of recovery and community reintegration.⁹ Additionally, sharing the clinical space with patients who had come before allowed participants to visualize a living timeline of their proposed recovery.

For most former patients, remembering the social intensity of their rehabilitation program is an important element in their narratives of recovery. The participants in our study do not necessarily maintain ties with their former peers, but nearly all of them point to support from other patients as being key in their own recovery from both the physical and psychological consequences of their injury.

Routine Access to the Treatment Team

The weekly amputee clinics that put surgeons, physicians, occupational and physical therapists, social workers, and prosthetists in a room with each patient worked to alleviate stress and anxiety in participants’ minds around the complexities of their injuries and care. One of the benefits of a group meeting is that it reduces the risk of miscommunication among HCPs and between HCPs and patients.^7,10

These weekly sessions with HCPs and patients led to a second advantage; they promoted patient autonomy and participation in clinical decisions. Patients were able to negotiate clinical goals with their HCPs and then act on them almost immediately. In addition, patients with complex physical injuries, often with neurological or psychiatric comorbidities, were able to describe the full range of their challenges, and HCPs had an opportunity to check in with patients about the problems they faced and level of severity.

The ability to marshal clinical HCPs to attend weekly meetings of this nature with patients may be key for distinguishing military health care from VHA and civilian counterparts. More research on how clinical team/ patient meetings occur in other settings is needed. But one of the hallmark features of these clinic meetings at WRNMMC was their open-endedness. Patients typically were not bound by 15 minute or other temporally delimited meeting intervals. This research indicates that in military health care, the patient is the leader.

Continuity of Care

Continuity of care is a well-understood benefit to working with the same HCP. There were additional unanticipated benefits to assigning patients to HCPs with whom they had worked. The long-term period of care (5-24 months) gave patients the opportunity to develop multifaceted relationships with their HCPs and empowered them to advocate and negotiate for their outcome goals. In addition, the majority of frontline HCPs in physical and occupational therapy were civil ians (all prosthetic providers were civilians). These ongoing relationships had the impact of socializing clinicians into the expectations of military culture (around physical training, endurance and resilience, and disregard of pain).

Physical and occupational therapists occupied multiple roles for their patients, including being teachers, coaches, and sounding boards. Participants frequently described the way that their physical or occupational therapist could, on one hand, push them to achieve more in terms of physical functioning. But on the other hand, participants also talked about the emotional and psychological support they could receive based both on the long duration of their work with their care providers.

 

Conclusions

The ability to recover alongside peers, have access to the whole treatment team and develop long-term relationships with key care HCPs served as drivers for positive recovery. The impact of these 3 drivers of the social organization of the Amputee Patient Care Program represent an opportunity to highlight the role that the social context of military health care to use in achieving positive therapeutic outcomes. Former patients of the WRNMMC program could rely on a familiar and dependable social context for their care. This social context draws heavily on elements of military culture that structure the preinjury worlds of work and life that patients occupied. Based on these results we argue that the presence of rehabilitation and other clinical units in military medical settings offers an important value to patients and HCPs.

Acknowledgments

This work was supported by the Center for Rehabilitation Science Research, Department of Physical Medicine & Rehabilitation, Uniformed Services University, Bethesda, MD (awards HU0001-11-1-0004 and HU0001-15-2-0003).

The ability of advanced practice providers (APPs) to practice independently has been a recent topic of discussion among both the medical community and legislatures. Advanced practice provider is an umbrella term that includes physician assistants (PAs) and advanced practice registered nurses, including nurse practitioners (NPs), clinical nurse specialists, certified nurse-midwives, and certified registered nurse anesthetists. Since Congress passed the Balanced Budget Act of 1997, APPs can bill and be paid independently if they are not practicing incident to a physician or in a facility.¹ Currently, NPs can practice independently in 27 states and Washington, DC. Physician assistants are required to practice under the supervision of a physician; however, the extent of supervision varies by state.² Advocates for broadening the scope of practice for APPs argue that NPs and PAs will help to fill the physician deficit, particularly in primary care and rural regions. It has been projected that by 2025, the United States will require an additional 46,000 primary care providers to meet growing medical needs.³

On October 3, 2019, President Donald Trump issued the Executive Order on Protecting and Improving Medicare for Our Nation’s Seniors, in which he proposed an alternative to “Medicare for all.”⁴ This order instructed the Secretary of Health and Human Services to prepare a regulation that would “eliminate burdensome regulatory billing requirements, conditions of participation, supervision requirements, benefit definitions and all other licensure requirements . . . that are more stringent than applicable Federal or State laws require and that limit professionals from practicing at the top of their field.” Furthermore, President Trump proposed that “services provided by clinicians, including physicians, physician assistants, and nurse practitioners, are appropriately reimbursed in accordance with the work performed rather than the clinician’s occupation.”⁴

In response to the executive order, members of the medical community utilized Reddit, an online public forum, and Medscape, a medical news website, to vocalize opinions on the executive order.^5,6 Our goal was to analyze the characteristics of those who participated in the discussion and their points of view on the plan to broaden the scope of practice and change the Medicare reimbursement plans for APPs.

Methods

All comments on the October 3, 2019, Medscape article, “Trump Executive Order Seeks Proposals on Medicare Pay for NPs, PAs,”⁵ and the corresponding Reddit discussion on this article⁶ were reviewed and characterized by the type of commenter—doctor of medicine (MD)/doctor of osteopathic medicine (DO), NP/RN/certified registered nurse anesthetist, PA, medical student, PA student, NP student, pharmacist, dietician, emergency medical technician, scribe, or unknown—as identified in their username, title, or in the text of the comment. Gender of the commenter was recorded when provided. Commenters were further grouped by their support or lack of support for the executive order based on their comments. Patients’ comments underwent further qualitative analysis to identify general themes.

All analyses were conducted with RStudio statistical software. Analyses were reported as proportions. Variables were compared by χ² and Fisher exact tests. Odds ratios with 95% CIs were calculated. P<.05 was considered statistically significant.

Results

A total of 352 comments (130 on Medscape and 222 on Reddit) posted by 155 unique users (57 on Medscape and 98 on Reddit) were included in the analysis (Table 1). Of the 51 Medscape commenters who identified a gender, 60.7% were male and 39.2% were female. Reddit commenters did not identify a gender. Commenters included MD and DO physicians (43.2%), NPs/RNs/certified registered nurse anesthetists (13.5%), medical students (11.0%), PAs (9.7%), pharmacists (3.2%), NP students (1.9%), PA students (1.3%), emergency medical technicians (1.3%), dieticians (0.6%), and scribes (0.6%). Physicians (54.5% vs 36.73%; P=.032) and NPs (22.8% vs 8.2%; P=.009) made up a larger percentage of all comments on Medscape compared to Reddit, where medical students were more prevalent (16.3% vs 1.8%; P=.005). Nursing students and PA students more commonly posted on Reddit (4.08% of Reddit commenters vs 1.75% of Medscape commenters), though this difference did not achieve statistical significance.

A majority of commenters did not support the executive order, with only 20.6% approving of the plan, 54.8% disapproving, and 24.5% remaining neutral (Figure). Advanced practice providers—NPs, PAs, NP/PA students, and APPs not otherwise specified—were more likely to support the executive order, with 52.3% voicing their support compared to only 4.8% of physicians and medical students expressing support (P<.0001). Similarly, physicians and medical students were more likely to disapprove of the order, with 75.0% voicing concerns compared to only 27.3% of APPs dissenting (P<.0001). A similar percentage of both physicians/medical students and APPs remained neutral (20.2% vs 18.2%). Commenters on Medscape were more likely to voice support for the executive order than those on Reddit (36.8% vs 11.2%; P=.0002), likely due to the higher percentage of NP and PA comments on the former.

 

Overall, the most commonly discussed topic was provider reimbursement (22.6% of all comments)(Table 2). Physicians and medical students were more likely to discuss physician expertise compared to APPs (32.1% vs 4.5%; P<.001). They also were more likely to raise concerns that the executive order would discourage future generations of physicians from pursuing medicine (15.5% vs 0%; P=.01). Advanced practice providers were more likely than physicians/medical students to comment on the breadth of NP and/or PA training (38.6% vs 19.0%; P=.02). The eTable shows representative comments for each theme encountered.

A subgroup analysis of the comments written by physicians supporting the executive order (n=4) and APPs disapproving of the order (n=12) was performed to identify the dissenting opinions. Physicians who supported the order discussed the need for improved pay for equal work (n=3), the competency of NP and PA training (n=2), the ability of a practice to generate more profit from APPs (n=1), and possible benefits of APPs providing primary care while MDs perform more specialized care (n=1). Of the APPs who did not support the order, there were 4 PAs, 2 registered nurses, 2 NPs, 2 NP students, and 2 PA students. The most common themes discussed were the differences in APP education and training (n=6), lack of desire for further responsibilities (n=4), and the adequacy of the current scope of practice (n=3).

Comment

President Trump’s executive order follows a trend of decreasing required oversight of APPs; however, this study indicates that these policies would face pushback from many physicians. These results are consistent with a prior study that analyzed 309 comments on an article in The New York Times made by physicians, APPs, patients, and laypeople, in which 24.7% had mistrust of APPs and 14.9% had concerns over APP supervision compared to 9% who supported APP independent practice.⁷ It is clear that there is a serious divide in opinion that threatens to harm the existing collaborations between physicians and APPs.

Primary Care Coverage With APPs
In the comments analyzed in our study, supporters of the executive order argued that an increase in APPs practicing independently would provide much-needed primary care coverage to patients in underserved regions. However, APPs are instead well represented across most specialties, with a majority in dermatology. Of the 4 million procedures billed independently by APPs in 2012, 54.8% were in the field of dermatology.⁸ The employment of APPs by dermatologists has grown from 28% of practices in 2005 to 46% in 2014, making this issue of particular importance to our field.^9,10

Education and Training of APPs
In our analysis, many physicians cited concerns about the education and training of APPs. Dermatologists receive approximately 10,000 hours of training over the course of residency. Per the American Academy of Physician Assistants, PAs spend more than 2000 hours over a 26-month period on various clinical rotations, “with an emphasis on primary care.”¹¹ There are multiple routes to become an advanced practice RN with varying classroom and clinical requirements, with one pathway requiring a bachelor of science in nursing, followed by a master’s degree requiring 500 to 700 hours of supervised clinical work. Although the Dermatology Nurses’ Association and Society of Dermatology Physician Assistants (http://www.dermpa.org) provide online modules, annual conventions with training workshops, and short fellowship programs, neither have formal guidelines on minimum requirements to diagnose and treat dermatologic conditions.² Despite the lack of formalized dermatologic training, APPs billed for 13.4% of all dermatology procedures submitted to Medicare in 2015.¹²

Quality of Patient Care
In our study, physicians also voiced concern over reduced quality of patient care. In a review of 33,647 skin cancer screening examinations, PAs biopsied an average of 39.4 skin lesions, while dermatologists biopsied an average of 25.4 skin lesions to diagnose 1 case of melanoma.¹³ In addition, nonphysician providers accounted for 37.9% of defendants in 174 legal cases related to injury from cutaneous laser surgery.¹⁴ Before further laws are enacted regarding the independent practice and billing by NPs and PAs in the field of dermatology, further research is needed to address patient outcomes and safety.

Limitations
This study was subject to several limitations. Because of a lack of other sources offering discussions on the topic, our sample size was limited. Self-identification of users presents a challenge, as an individual can pose as a physician or APP without validation of credentials. Although great care was taken to minimize bias, grouping comments into broad categories may misinterpret a poster’s intentions. Furthermore, the data collected represent only a small proportion of the medical community—readers of Medscape and Reddit who have the motivation to create a user profile and post a comment rather than put their efforts into lobbying or contacting legislators. Those posting may have stronger political opinions or more poignant experiences than the general public. Although selection bias impacts the generalizability of our findings, this analysis allows for deeper insight into the beliefs of a vocal subset of the medical community who may not have the opportunity to present their opinions elsewhere.

Conclusion

Our analysis of the response to President Trump’s executive order reveals that a rollout of these regulations would be met with strong opposition. On October 29, 2019, more than 100 professional organizations, including the American Medical Association and the American Academy of Dermatology, wrote a letter to the Secretary of Health and Human Services that eloquently echoed the sentiments of the physician commenters in this study: “Scope of practice of health care professionals should be based on standardized, adequate training and demonstrated competence in patient care, not politics. While all health care professionals share an important role in providing care to patients, their skillset is not interchangeable with that of a fully trained physician.”¹⁵ The executive order would lead to a major shift in the current medical landscape, and as such, it is prudent that these concerns are addressed.

The ability of advanced practice providers (APPs) to practice independently has been a recent topic of discussion among both the medical community and legislatures. Advanced practice provider is an umbrella term that includes physician assistants (PAs) and advanced practice registered nurses, including nurse practitioners (NPs), clinical nurse specialists, certified nurse-midwives, and certified registered nurse anesthetists. Since Congress passed the Balanced Budget Act of 1997, APPs can bill and be paid independently if they are not practicing incident to a physician or in a facility.¹ Currently, NPs can practice independently in 27 states and Washington, DC. Physician assistants are required to practice under the supervision of a physician; however, the extent of supervision varies by state.² Advocates for broadening the scope of practice for APPs argue that NPs and PAs will help to fill the physician deficit, particularly in primary care and rural regions. It has been projected that by 2025, the United States will require an additional 46,000 primary care providers to meet growing medical needs.³

On October 3, 2019, President Donald Trump issued the Executive Order on Protecting and Improving Medicare for Our Nation’s Seniors, in which he proposed an alternative to “Medicare for all.”⁴ This order instructed the Secretary of Health and Human Services to prepare a regulation that would “eliminate burdensome regulatory billing requirements, conditions of participation, supervision requirements, benefit definitions and all other licensure requirements . . . that are more stringent than applicable Federal or State laws require and that limit professionals from practicing at the top of their field.” Furthermore, President Trump proposed that “services provided by clinicians, including physicians, physician assistants, and nurse practitioners, are appropriately reimbursed in accordance with the work performed rather than the clinician’s occupation.”⁴

In response to the executive order, members of the medical community utilized Reddit, an online public forum, and Medscape, a medical news website, to vocalize opinions on the executive order.^5,6 Our goal was to analyze the characteristics of those who participated in the discussion and their points of view on the plan to broaden the scope of practice and change the Medicare reimbursement plans for APPs.

Methods

All comments on the October 3, 2019, Medscape article, “Trump Executive Order Seeks Proposals on Medicare Pay for NPs, PAs,”⁵ and the corresponding Reddit discussion on this article⁶ were reviewed and characterized by the type of commenter—doctor of medicine (MD)/doctor of osteopathic medicine (DO), NP/RN/certified registered nurse anesthetist, PA, medical student, PA student, NP student, pharmacist, dietician, emergency medical technician, scribe, or unknown—as identified in their username, title, or in the text of the comment. Gender of the commenter was recorded when provided. Commenters were further grouped by their support or lack of support for the executive order based on their comments. Patients’ comments underwent further qualitative analysis to identify general themes.

All analyses were conducted with RStudio statistical software. Analyses were reported as proportions. Variables were compared by χ² and Fisher exact tests. Odds ratios with 95% CIs were calculated. P<.05 was considered statistically significant.

Results

A total of 352 comments (130 on Medscape and 222 on Reddit) posted by 155 unique users (57 on Medscape and 98 on Reddit) were included in the analysis (Table 1). Of the 51 Medscape commenters who identified a gender, 60.7% were male and 39.2% were female. Reddit commenters did not identify a gender. Commenters included MD and DO physicians (43.2%), NPs/RNs/certified registered nurse anesthetists (13.5%), medical students (11.0%), PAs (9.7%), pharmacists (3.2%), NP students (1.9%), PA students (1.3%), emergency medical technicians (1.3%), dieticians (0.6%), and scribes (0.6%). Physicians (54.5% vs 36.73%; P=.032) and NPs (22.8% vs 8.2%; P=.009) made up a larger percentage of all comments on Medscape compared to Reddit, where medical students were more prevalent (16.3% vs 1.8%; P=.005). Nursing students and PA students more commonly posted on Reddit (4.08% of Reddit commenters vs 1.75% of Medscape commenters), though this difference did not achieve statistical significance.

A majority of commenters did not support the executive order, with only 20.6% approving of the plan, 54.8% disapproving, and 24.5% remaining neutral (Figure). Advanced practice providers—NPs, PAs, NP/PA students, and APPs not otherwise specified—were more likely to support the executive order, with 52.3% voicing their support compared to only 4.8% of physicians and medical students expressing support (P<.0001). Similarly, physicians and medical students were more likely to disapprove of the order, with 75.0% voicing concerns compared to only 27.3% of APPs dissenting (P<.0001). A similar percentage of both physicians/medical students and APPs remained neutral (20.2% vs 18.2%). Commenters on Medscape were more likely to voice support for the executive order than those on Reddit (36.8% vs 11.2%; P=.0002), likely due to the higher percentage of NP and PA comments on the former.

 

Overall, the most commonly discussed topic was provider reimbursement (22.6% of all comments)(Table 2). Physicians and medical students were more likely to discuss physician expertise compared to APPs (32.1% vs 4.5%; P<.001). They also were more likely to raise concerns that the executive order would discourage future generations of physicians from pursuing medicine (15.5% vs 0%; P=.01). Advanced practice providers were more likely than physicians/medical students to comment on the breadth of NP and/or PA training (38.6% vs 19.0%; P=.02). The eTable shows representative comments for each theme encountered.

A subgroup analysis of the comments written by physicians supporting the executive order (n=4) and APPs disapproving of the order (n=12) was performed to identify the dissenting opinions. Physicians who supported the order discussed the need for improved pay for equal work (n=3), the competency of NP and PA training (n=2), the ability of a practice to generate more profit from APPs (n=1), and possible benefits of APPs providing primary care while MDs perform more specialized care (n=1). Of the APPs who did not support the order, there were 4 PAs, 2 registered nurses, 2 NPs, 2 NP students, and 2 PA students. The most common themes discussed were the differences in APP education and training (n=6), lack of desire for further responsibilities (n=4), and the adequacy of the current scope of practice (n=3).

Comment

President Trump’s executive order follows a trend of decreasing required oversight of APPs; however, this study indicates that these policies would face pushback from many physicians. These results are consistent with a prior study that analyzed 309 comments on an article in The New York Times made by physicians, APPs, patients, and laypeople, in which 24.7% had mistrust of APPs and 14.9% had concerns over APP supervision compared to 9% who supported APP independent practice.⁷ It is clear that there is a serious divide in opinion that threatens to harm the existing collaborations between physicians and APPs.

Primary Care Coverage With APPs
In the comments analyzed in our study, supporters of the executive order argued that an increase in APPs practicing independently would provide much-needed primary care coverage to patients in underserved regions. However, APPs are instead well represented across most specialties, with a majority in dermatology. Of the 4 million procedures billed independently by APPs in 2012, 54.8% were in the field of dermatology.⁸ The employment of APPs by dermatologists has grown from 28% of practices in 2005 to 46% in 2014, making this issue of particular importance to our field.^9,10

Education and Training of APPs
In our analysis, many physicians cited concerns about the education and training of APPs. Dermatologists receive approximately 10,000 hours of training over the course of residency. Per the American Academy of Physician Assistants, PAs spend more than 2000 hours over a 26-month period on various clinical rotations, “with an emphasis on primary care.”¹¹ There are multiple routes to become an advanced practice RN with varying classroom and clinical requirements, with one pathway requiring a bachelor of science in nursing, followed by a master’s degree requiring 500 to 700 hours of supervised clinical work. Although the Dermatology Nurses’ Association and Society of Dermatology Physician Assistants (http://www.dermpa.org) provide online modules, annual conventions with training workshops, and short fellowship programs, neither have formal guidelines on minimum requirements to diagnose and treat dermatologic conditions.² Despite the lack of formalized dermatologic training, APPs billed for 13.4% of all dermatology procedures submitted to Medicare in 2015.¹²

Quality of Patient Care
In our study, physicians also voiced concern over reduced quality of patient care. In a review of 33,647 skin cancer screening examinations, PAs biopsied an average of 39.4 skin lesions, while dermatologists biopsied an average of 25.4 skin lesions to diagnose 1 case of melanoma.¹³ In addition, nonphysician providers accounted for 37.9% of defendants in 174 legal cases related to injury from cutaneous laser surgery.¹⁴ Before further laws are enacted regarding the independent practice and billing by NPs and PAs in the field of dermatology, further research is needed to address patient outcomes and safety.

Limitations
This study was subject to several limitations. Because of a lack of other sources offering discussions on the topic, our sample size was limited. Self-identification of users presents a challenge, as an individual can pose as a physician or APP without validation of credentials. Although great care was taken to minimize bias, grouping comments into broad categories may misinterpret a poster’s intentions. Furthermore, the data collected represent only a small proportion of the medical community—readers of Medscape and Reddit who have the motivation to create a user profile and post a comment rather than put their efforts into lobbying or contacting legislators. Those posting may have stronger political opinions or more poignant experiences than the general public. Although selection bias impacts the generalizability of our findings, this analysis allows for deeper insight into the beliefs of a vocal subset of the medical community who may not have the opportunity to present their opinions elsewhere.

Conclusion

Our analysis of the response to President Trump’s executive order reveals that a rollout of these regulations would be met with strong opposition. On October 29, 2019, more than 100 professional organizations, including the American Medical Association and the American Academy of Dermatology, wrote a letter to the Secretary of Health and Human Services that eloquently echoed the sentiments of the physician commenters in this study: “Scope of practice of health care professionals should be based on standardized, adequate training and demonstrated competence in patient care, not politics. While all health care professionals share an important role in providing care to patients, their skillset is not interchangeable with that of a fully trained physician.”¹⁵ The executive order would lead to a major shift in the current medical landscape, and as such, it is prudent that these concerns are addressed.

The ability of advanced practice providers (APPs) to practice independently has been a recent topic of discussion among both the medical community and legislatures. Advanced practice provider is an umbrella term that includes physician assistants (PAs) and advanced practice registered nurses, including nurse practitioners (NPs), clinical nurse specialists, certified nurse-midwives, and certified registered nurse anesthetists. Since Congress passed the Balanced Budget Act of 1997, APPs can bill and be paid independently if they are not practicing incident to a physician or in a facility.¹ Currently, NPs can practice independently in 27 states and Washington, DC. Physician assistants are required to practice under the supervision of a physician; however, the extent of supervision varies by state.² Advocates for broadening the scope of practice for APPs argue that NPs and PAs will help to fill the physician deficit, particularly in primary care and rural regions. It has been projected that by 2025, the United States will require an additional 46,000 primary care providers to meet growing medical needs.³

On October 3, 2019, President Donald Trump issued the Executive Order on Protecting and Improving Medicare for Our Nation’s Seniors, in which he proposed an alternative to “Medicare for all.”⁴ This order instructed the Secretary of Health and Human Services to prepare a regulation that would “eliminate burdensome regulatory billing requirements, conditions of participation, supervision requirements, benefit definitions and all other licensure requirements . . . that are more stringent than applicable Federal or State laws require and that limit professionals from practicing at the top of their field.” Furthermore, President Trump proposed that “services provided by clinicians, including physicians, physician assistants, and nurse practitioners, are appropriately reimbursed in accordance with the work performed rather than the clinician’s occupation.”⁴

In response to the executive order, members of the medical community utilized Reddit, an online public forum, and Medscape, a medical news website, to vocalize opinions on the executive order.^5,6 Our goal was to analyze the characteristics of those who participated in the discussion and their points of view on the plan to broaden the scope of practice and change the Medicare reimbursement plans for APPs.

Methods

All comments on the October 3, 2019, Medscape article, “Trump Executive Order Seeks Proposals on Medicare Pay for NPs, PAs,”⁵ and the corresponding Reddit discussion on this article⁶ were reviewed and characterized by the type of commenter—doctor of medicine (MD)/doctor of osteopathic medicine (DO), NP/RN/certified registered nurse anesthetist, PA, medical student, PA student, NP student, pharmacist, dietician, emergency medical technician, scribe, or unknown—as identified in their username, title, or in the text of the comment. Gender of the commenter was recorded when provided. Commenters were further grouped by their support or lack of support for the executive order based on their comments. Patients’ comments underwent further qualitative analysis to identify general themes.

All analyses were conducted with RStudio statistical software. Analyses were reported as proportions. Variables were compared by χ² and Fisher exact tests. Odds ratios with 95% CIs were calculated. P<.05 was considered statistically significant.

Results

A total of 352 comments (130 on Medscape and 222 on Reddit) posted by 155 unique users (57 on Medscape and 98 on Reddit) were included in the analysis (Table 1). Of the 51 Medscape commenters who identified a gender, 60.7% were male and 39.2% were female. Reddit commenters did not identify a gender. Commenters included MD and DO physicians (43.2%), NPs/RNs/certified registered nurse anesthetists (13.5%), medical students (11.0%), PAs (9.7%), pharmacists (3.2%), NP students (1.9%), PA students (1.3%), emergency medical technicians (1.3%), dieticians (0.6%), and scribes (0.6%). Physicians (54.5% vs 36.73%; P=.032) and NPs (22.8% vs 8.2%; P=.009) made up a larger percentage of all comments on Medscape compared to Reddit, where medical students were more prevalent (16.3% vs 1.8%; P=.005). Nursing students and PA students more commonly posted on Reddit (4.08% of Reddit commenters vs 1.75% of Medscape commenters), though this difference did not achieve statistical significance.

A majority of commenters did not support the executive order, with only 20.6% approving of the plan, 54.8% disapproving, and 24.5% remaining neutral (Figure). Advanced practice providers—NPs, PAs, NP/PA students, and APPs not otherwise specified—were more likely to support the executive order, with 52.3% voicing their support compared to only 4.8% of physicians and medical students expressing support (P<.0001). Similarly, physicians and medical students were more likely to disapprove of the order, with 75.0% voicing concerns compared to only 27.3% of APPs dissenting (P<.0001). A similar percentage of both physicians/medical students and APPs remained neutral (20.2% vs 18.2%). Commenters on Medscape were more likely to voice support for the executive order than those on Reddit (36.8% vs 11.2%; P=.0002), likely due to the higher percentage of NP and PA comments on the former.

 

Overall, the most commonly discussed topic was provider reimbursement (22.6% of all comments)(Table 2). Physicians and medical students were more likely to discuss physician expertise compared to APPs (32.1% vs 4.5%; P<.001). They also were more likely to raise concerns that the executive order would discourage future generations of physicians from pursuing medicine (15.5% vs 0%; P=.01). Advanced practice providers were more likely than physicians/medical students to comment on the breadth of NP and/or PA training (38.6% vs 19.0%; P=.02). The eTable shows representative comments for each theme encountered.

A subgroup analysis of the comments written by physicians supporting the executive order (n=4) and APPs disapproving of the order (n=12) was performed to identify the dissenting opinions. Physicians who supported the order discussed the need for improved pay for equal work (n=3), the competency of NP and PA training (n=2), the ability of a practice to generate more profit from APPs (n=1), and possible benefits of APPs providing primary care while MDs perform more specialized care (n=1). Of the APPs who did not support the order, there were 4 PAs, 2 registered nurses, 2 NPs, 2 NP students, and 2 PA students. The most common themes discussed were the differences in APP education and training (n=6), lack of desire for further responsibilities (n=4), and the adequacy of the current scope of practice (n=3).

Comment

President Trump’s executive order follows a trend of decreasing required oversight of APPs; however, this study indicates that these policies would face pushback from many physicians. These results are consistent with a prior study that analyzed 309 comments on an article in The New York Times made by physicians, APPs, patients, and laypeople, in which 24.7% had mistrust of APPs and 14.9% had concerns over APP supervision compared to 9% who supported APP independent practice.⁷ It is clear that there is a serious divide in opinion that threatens to harm the existing collaborations between physicians and APPs.

Primary Care Coverage With APPs
In the comments analyzed in our study, supporters of the executive order argued that an increase in APPs practicing independently would provide much-needed primary care coverage to patients in underserved regions. However, APPs are instead well represented across most specialties, with a majority in dermatology. Of the 4 million procedures billed independently by APPs in 2012, 54.8% were in the field of dermatology.⁸ The employment of APPs by dermatologists has grown from 28% of practices in 2005 to 46% in 2014, making this issue of particular importance to our field.^9,10

Education and Training of APPs
In our analysis, many physicians cited concerns about the education and training of APPs. Dermatologists receive approximately 10,000 hours of training over the course of residency. Per the American Academy of Physician Assistants, PAs spend more than 2000 hours over a 26-month period on various clinical rotations, “with an emphasis on primary care.”¹¹ There are multiple routes to become an advanced practice RN with varying classroom and clinical requirements, with one pathway requiring a bachelor of science in nursing, followed by a master’s degree requiring 500 to 700 hours of supervised clinical work. Although the Dermatology Nurses’ Association and Society of Dermatology Physician Assistants (http://www.dermpa.org) provide online modules, annual conventions with training workshops, and short fellowship programs, neither have formal guidelines on minimum requirements to diagnose and treat dermatologic conditions.² Despite the lack of formalized dermatologic training, APPs billed for 13.4% of all dermatology procedures submitted to Medicare in 2015.¹²

Quality of Patient Care
In our study, physicians also voiced concern over reduced quality of patient care. In a review of 33,647 skin cancer screening examinations, PAs biopsied an average of 39.4 skin lesions, while dermatologists biopsied an average of 25.4 skin lesions to diagnose 1 case of melanoma.¹³ In addition, nonphysician providers accounted for 37.9% of defendants in 174 legal cases related to injury from cutaneous laser surgery.¹⁴ Before further laws are enacted regarding the independent practice and billing by NPs and PAs in the field of dermatology, further research is needed to address patient outcomes and safety.

Limitations
This study was subject to several limitations. Because of a lack of other sources offering discussions on the topic, our sample size was limited. Self-identification of users presents a challenge, as an individual can pose as a physician or APP without validation of credentials. Although great care was taken to minimize bias, grouping comments into broad categories may misinterpret a poster’s intentions. Furthermore, the data collected represent only a small proportion of the medical community—readers of Medscape and Reddit who have the motivation to create a user profile and post a comment rather than put their efforts into lobbying or contacting legislators. Those posting may have stronger political opinions or more poignant experiences than the general public. Although selection bias impacts the generalizability of our findings, this analysis allows for deeper insight into the beliefs of a vocal subset of the medical community who may not have the opportunity to present their opinions elsewhere.

Conclusion

Our analysis of the response to President Trump’s executive order reveals that a rollout of these regulations would be met with strong opposition. On October 29, 2019, more than 100 professional organizations, including the American Medical Association and the American Academy of Dermatology, wrote a letter to the Secretary of Health and Human Services that eloquently echoed the sentiments of the physician commenters in this study: “Scope of practice of health care professionals should be based on standardized, adequate training and demonstrated competence in patient care, not politics. While all health care professionals share an important role in providing care to patients, their skillset is not interchangeable with that of a fully trained physician.”¹⁵ The executive order would lead to a major shift in the current medical landscape, and as such, it is prudent that these concerns are addressed.

Preparing patients for hospital discharge requires multiple tasks that cross professional boundaries. Clinician’s roles may be ambiguous, and responsibility for a safe high-quality discharge is often diffused among the team rather than being defined as the core responsibility of a single member.^1-8 Without a shared understanding of patient resources and tasks involved in anticipatory planning, lapses in discharge preparation can occur, which places patients at increased risk for harm after hospitalization.^3-7 As a result, organizations like the Centers for Medicare & Medicaid Services (CMS) have called for team-based patient-centered discharge planning.⁸ Yet to develop more effective team-based discharge planning interventions, a more nuanced understanding of how healthcare teams work together is needed.^2,3,9

Shared mental models (SMMs) provide a useful theoretical framework and measurement approach for examining how interprofessional teams coordinate complex tasks like hospital discharge.^10-13 SMMs represent the team members’ collective understanding and organized knowledge of key elements needed for teams to perform effectively.^9-11 Validated questionnaires can be used to measure two key properties of SMMs: the degree to which team members have a similar understanding of the situation at hand (team SMM convergence) and to what extent this understanding is aligned with the patient (team-­patient SMM convergence).^10,11 Researchers have found that teams with higher-quality SMMs have a better understanding of what is happening and why, have clearer expectations of their roles and tasks, and can better predict what might happen next.^10,12 As a result, these teams more effectively coordinate actions and adapt to task demands even in cases of high complexity, uncertainty, and stress.^10-13 Prior studies examining healthcare teams in emergency departments,^14-16 critical care units,^17,18 and operating rooms¹⁹ suggest high-quality SMMs are needed to safely care for patients.¹³ Yet there has been limited evaluation of SMMs in general internal medicine, much less during hospital discharge.^9,13 The purpose of this study was to examine SMMs for a critical task of the inpatient team: developing a shared understanding of the patient’s readiness for hospital discharge.^20,21

 

METHODS

Design

We used a cross-sectional survey design at a single Midwestern community hospital to determine inpatient care teams’ SMMs of patient hospital discharge readiness. This study is part of a larger mixed-methods evaluation of interprofessional hospital discharge teamwork in older adult patients at risk for a poor transition to home.⁹ Data were collected using questionnaires from patients and their team (nurse, coordinator, and physician) within 4 hours of the patient leaving the hospital. First, we measured the teams’ assessment, team convergence, and team-­patient convergence on patient readiness for discharge from the hospital. Then, after identifying relevant potential predictors from the literature, we developed regression models to predict the teams’ assessment, team convergence, and team-patient convergence of discharge readiness based on these variables. Our local institutional review board approved this study.

Sample and Participants

We used a convenience sampling approach to identify eligible discharge events consisting of the patient and care team.⁹ We focused on patients at high-risk for poor hospital-to-home transitions.^3,22 Eligible events included older patients (≥65 years) who were discharged home without home health or hospice services and admitted with a primary diagnosis of heart failure, acute myocardial infarction, hip replacement, knee replacement, pneumonia, or chronic obstructive pulmonary disease. Patient exclusion criteria included inability to complete study forms because of mental incapacity or a language barrier. Discharge team member inclusion criteria included the bedside nurse, attending physician, and coordinator (a unit-dedicated discharge nurse or social worker) caring for the patient participant at the time of hospital discharge. Each discharge team was unique: The same three individuals could not be included as a “team” for more than one discharge event, although individual members could be included as a part of other teams with a different set of individuals. Appendix A provides an enrollment flowchart.

Conceptual Framework

We applied the SMM conceptual framework to the context of hospital discharge. As shown in the Figure, SMMs are examined at the team level and contain the critical knowledge held by the team to be effective.^15,16 From a patient-centered perspective, patients are considered the expert on how ready they feel to be discharged home.^20,23,24 In this case, the SMM content is the discharge team members’ shared assessment of how ready the patient is for hospital discharge (Figure, B).¹⁰ Convergence is the degree of agreement among individual mental models.^10-13 In this study we examined two types of convergence: (1) team convergence, or the team members degree of agreement on the patient’s readiness for discharge (Figure, C), and (2) team-patient convergence, or the degree to which the team’s SMM aligns with the patient’s mental model (Figure, D).^10-13

Measures and Variables

Readiness for Hospital Discharge Scales/Short Form

We used parallel clinician and patient versions of the Readiness for Hospital Discharge Scale/Short Form (RHDS/SF)^25-28 to determine the teams’ assessment of discharge readiness, team SMM convergence, and team-patient SMM convergence.

The RHDS/SF scales are 8-item validated instruments that use a Likert scale (0 for not ready to 10 for totally ready) to assess the individual clinician’s or patient’s perceptions of how ready the patient is to be discharged.^20,25,27 The RHDS/SF instruments include four dimensions conceptualized as crucial to patient readiness for discharge and important to anticipatory planning: (1) Personal Status, physical-emotional state of the patient before discharge; (2) Knowledge, perceived adequacy of information needed to respond to common posthospitalization concerns/problems; (3) Coping Ability, perceived ability to self-manage health care needs; and (4) Expected Support, emotional-physical assistance available (Appendix B).^20,25,27 The RHDS/SF instruments’ results are calculated as a mean of item scores, with higher individual scores indicating the rater assessed the patient as being more ready for hospital discharge.²⁰ The RHDS/SF scales have undergone rigorous psychometric testing and are linked to patient outcomes (eg, readmissions, emergency room visits, patient coping difficulties after discharge, and patient-rated quality of preparation for posthospital care).^20,25-28 For example, predictive validity assessments for adult medical-surgical patients found lower Nurse-RHDS/SF scores are associated with a six- to ninefold increase in 30-day readmission risk.²⁰

 

Contextual Variables

We reviewed the literature to identify potential patient and system factors associated with adverse transitional care outcomes^1-8 and/or higher quality SMMs in other settings.^10-19 For example, patient characteristics included age, principal diagnosis, length of stay, number of comorbidities, and cognition impairment (using the Short Portable Mini Mental Status Questionnaire²⁹).^2,22,30 Examples of system factor include teamwork and communication quality^1-6 on day of discharge, as well as educational background and experience of clinicians on the team.^31-33 We adapted a validated survey using 7-point Likert scale questions to determine teamwork quality and communication quality during individual patent discharges.³³ Appendix C provides descriptions of all variables.

Data Collection

Patient recruitment occurred from February to October 2017 in a single community hospital in Iowa.⁹ We identified potentially eligible events in collaboration with the unit charge nurses. Patients were screened 24 to 48 hours prior to anticipated day of discharge; those interested/eligible underwent informed consent procedures.⁹ We collected data from the patient and their corresponding bedside nurse, coordinator, and attending physician on the day of discharge. After the discharge order was placed and care instructions were provided, the patient completed a demographic survey, Short Portable Mini Mental Status Questionnaire, and the Patient-RHDS/SF. Individual team members completed a survey with the demographic information, their respective versions of RHDS/SF, and day-of-discharge teamwork-related questions. On average, the survey took clinicians less than 5 minutes to complete. We performed a chart review to determine additional patient characteristics such as principal diagnosis, length of stay, and number of comorbidities.

Data Analysis

Team Assessment of Patient Discharge Readiness

The teams’ shared assessment (SMM content) was determined by averaging the members’ individual scores on the Clinician-RHDS/SF.³⁴ Discharge events with higher team assessments indicated the team perceived the patient as being readier for hospital discharge. Guided by prior research, we examined the RHDS/SF scores as a continuous variable and as a four-level categorical variable of readiness: low (<7), moderate (7-7.9), high (8-8.9), and very high (9-10).²⁰

Team SMM Convergence

To determine the teams’ convergence on patient discharge readiness, we calculated an adjusted interrater agreement index (r*_wg(j))^35,36 for each team using the individual clinicians’ scores on the RHDS/SF. These convergence values were categorized into four agreement levels: low agreement (<0.7), moderate agreement (0.7-0.79), high agreement (0.8-0.89), and very high agreement (0.9-1). See Appendix D for the r*_wg(j) equation.^35,36

Team-Patient SMM Convergence

To determine the team-patient SMM convergence, we subtracted the team’s assessment of patient discharge readiness from the Patient-RHDS/SF score. We used a one-unit change on the RHDS/SF (1 point on the 0-10 scale) as a meaningful difference between the patient’s self-assessment and teams’ assessment on readiness for hospital discharge. This definition for divergence aligns with prior RHDS psychometric testing studies^20,27 and research examining convergence between patient and nurse assessments.²⁸ For example, Weiss and colleagues²⁷ found a 1-point decrease in the RN-RHDS/SF item mean was associated with a 45% increase in likelihood of postdischarge utilization (hospital readmission and emergency room department visits). Therefore, we defined convergence of team-patient SMMs (or similar patient and team scores) as those with an absolute difference score less than 1 point, whereas teams with low team-patient SMM convergence (or divergent patient and team scores) were defined as having an absolute difference score greater than 1 point.

 

Prediction Models

For the exploratory aim, we first examined the bivariate relationship between the outcome variables (discharge teams’ assessment of patient readiness, team convergence, and team-patient convergence) and the identified contextual variables. We also checked for potential collinearity among the explanatory variables. Then we used a linear stepwise regression procedure to identify factors associated with each continuous outcome variable. Due to the small sample size, we performed separate backward stepwise regression selection analyses for the three outcomes of interest. The candidate explanatory variables were evaluated using P < .20 for model entry. Final models were evaluated using leave-one-out cross validation. STATA (v.15.1, StataCorp; 2017) was used for analysis.

RESULTS

Sample

A total of 64 discharge events were included in this study. All discharge teams had a unique composition including 64 patients and varying combinations of 56 individual nurses (n = 27), physicians (n = 23), and coordinators (n = 6). Each event had three team members (ie, a nurse, a coordinator, and a physician) with no missing data. The majority of the 64 patient participants were White, retired, had a high school education, and lived in their own home with only one other person (Table 1).

Interprofessional Teams’ SMM of Readiness for Hospital Discharge

While the majority of teams perceived patients had high readiness for hospital discharge (mean, 8.5 out of 10; SD, 0.91), patients scores were nearly a full point lower (mean, 7.7; SD, 1.6; Table 2). The largest difference across categories was in the low-readiness category with 27% of patient scores falling into this category vs only 9.4% of discharge team mean scores. The mean SMM convergence of team perception of patients’ readiness for discharge was 0.90 (SD, 0.10); however, scores ranged from 0.66 (low agreement) to 1 (complete agreement). The average SMM team-patient convergence, or the discrepancy between the discharge team mean scores and the patient total scores across domains, was 1.16 (SD, 0.82). Of the 64 discharge events, 42.2% had similar team-patient perceptions of readiness for discharge, 9.4% had the patient reporting higher readiness for discharge than the team, and 48.4% had a team assessment rating of higher readiness for discharge than the patient’s self-assessment.

Prediction Models

In the exploratory analysis, we created individual linear regression models to predict the teams’ assessment, team convergence, and team-patient convergence for readiness of hospital discharge (Table 3; Appendix E). Factors associated with the teams’ assessment of discharge readiness included whether the patient was married and had less cognitive impairment, both of which were positively related to a higher-­rated readiness among teams. An important system factor was higher quality of communication among team members, which was positively associated with teams’ assessment of patient discharge readiness. In contrast, only patient factors—married patients and those with a principal diagnosis of heart failure—were associated with more convergent team SMMs. Team-patient convergence was positively associated with two patient factors: marital status (married) and fewer comorbidities. However, team-patient convergence was also associated with two system factors: teams with a bachelor’s level–trained nurse (compared with a nurse with an associate degree ) and teams reporting a higher quality of teamwork on day of discharge.

 

DISCUSSION

Our study applied novel approaches to explore the interprofessional teams’ understanding of discharge readiness, a concept known to be an important predictor of patient outcomes after discharge, including readmission.^20,28 We found that discharge teams frequently had poor quality SMMs of hospital discharge readiness. Despite having a discharge order and receiving home care instructions, one in four patients reported low readiness for hospital discharge. Additionally, discharge teams frequently overestimated patient’s readiness for hospital discharge. Misalignment on patient readiness for discharge occurred both within the discharge team (ie, low team convergence) and between patients and their care teams (ie, low team-patient convergence). The potential importance of this disagreement is substantiated by prior work suggesting that divergence in readiness ratings between nurses and patients are associated with postdischarge coping difficulties.²⁸

Previous readiness for discharge has been measured from the perspective of the patient,^20,21,27,28 nurse,^20,25-28 and physician,³⁷ yet rarely has the teams’ perspective been examined. We add to this literature by measuring the team’s perspective, as well as agreement between team and patient, on the individual patient’s readiness for discharge. Notably, we found that higher-quality communication is positively related to teams’ assessment of discharge readiness, with teams that reported higher quality teamwork having more convergent team-patient SMMs. Our results support many qualitative studies identifying communication and teamwork as major factors in teams’ effectiveness in discharge planning.^1-7,9 However, given the small sample size in this study, additional research is needed to further understand these relationships, as well as link SMMs to patient outcomes such as hospital readmission.

In an attempt to improve discharge planning, hospitals are increasingly assessing readiness for discharge as a low-intensity, low-cost intervention.^26,27 Yet, recent evidence suggests that readiness assessments alone have minimal impact on reducing hospital readmissions.²⁶ To be successful, these assessments likely depend on quality interprofessional communication and ensuring the patient’s voice is incorporated into the discharge decision process.²⁶ However, there have been few ways to effectively evaluate these types of team interventions.⁹ Measuring SMM properties holds promise for identifying specific team mechanisms that may influence the effectiveness and fidelity of interventions for team-based discharge planning. As our findings indicate, SMMs provide a theoretical and methodological basis for evaluating if readiness for discharge was team based (convergence among team members) and patient centered (convergence among team assessment and patient self-assessment). Researchers and improvement scientists can use the approach outlined to evaluate team-based patient-centered interventions for hospital discharge planning.⁹

This study provides a unique contribution to the growing work in the team science of SMMs.^9,10 We rigorously evaluated SMMs of key stakeholders (patients and their interprofessional team) in “real-time” clinical practice using a patient-centered assessment linked with postdischarge outcomes.^20,27,28 However, it is still unknown how much convergence is needed (and with whom) to safely discharge patients.¹³ Prior studies suggest highly convergent SMMs increase team performance when they are also accurate.^10-13 Convergence alone should not be sought because this may reflect groupthink or clinical inertia.^10,15 To improve discharge team performance over time,^10‑13 it is important to assess not only patient’s readiness on the day of discharge but also how prepared the patient actually was for the recovery period following acute care. In the larger mixed-methods study, we found that teams’ with more convergent SMMs on teamwork quality were associated with patient’s reported quality of transition 30-days after discharge.⁹ Together, these findings further highlight the importance of aligning patient and interprofessional team members perspectives during the discharge planning, as well as providing clinicians with regular feedback about patient’s postdischarge experiences and outcomes.

To optimize team performance, the discharge planning process must be considered from an interprofessional team perspective as it functions in real-world practice settings. There are increasing pressures to discharge patients “quicker and sicker,” to simplify and standardize clinical process, and to provide patient-centered care.^3,5-8 Without thoughtful interventions to facilitate communication during discharge planning, these pressures likely reinforce inaccurate assumptions regarding the work of fellow team members and force teams to think “fast” instead of “slow.”^38-40 One approach to overcome such barriers is to focus on building a high-quality interprofessional SMM around discharge readiness. For example, the RHDS/SF questions could be integrated into the electronic medical records, displayed on dashboards, and discussed regularly during discharge rounds. In particular, to strengthen the team’s SMM and quality of teamwork, together the staff can ask three practical questions (Appendix F): (1) Do we think the patient is ready for discharge? (2) To what extent do we all agree the patient is ready for discharge? (3) Does our assessment of discharge readiness match the patient’s? During this high-risk transition point, asking these questions might allow the team to move from thinking fast to thinking slowly so they can more effectively identify heuristics they may be using inaccurately, prevent blind spots, and move toward high reliability.^{10,13,18,38-40}

This study has limitations. First, events were recruited from patients with any of only six conditions at a single hospital. Other settings, patient condition types, or team compositions of other clinicians may differ in results. Second, in this study the SMM content was focused on readiness for hospital discharge among four key stakeholders. It is possible other SMM content needs to be shared among the interprofessional discharge team (eg, caregivers’ perspectives,^2,6-8 resource availability,^3-6 clinicians’ roles^4,9) or additional members should be included (eg, physical therapists, nursing assistants, home health consultants, or primary care clinicians). Although this study focused on a patient-centered outcome (Patient-RHDS/SF), we did not examine other important outcomes such as hospital readmission. Additionally, due to the small sample size, these results have limited generalizability and should be interpreted with caution. Last, we limited data collection to the day of hospital discharge; future studies might consider assessing discharge readiness throughout hospitalization.

 

CONCLUSION

Readying patients for hospital discharge is a time-sensitivehigh-risk task requiring multiple healthcare professionals to concurrently assess patient needs, formulate an anticipatory care plan, provide education, and arrange for postdischarge needs.^20,21 Despite this, few studies have analyzed teamwork aspects to understand how these transitions could be improved.⁹ By piloting SMM measurement and describing factors that affect SMMs, we provide a step toward identifying and evaluating strategies to assist interprofessional care teams in preparing patients for a safe, high-quality, patient-centered hospital discharge.

Presentations

This work was presented at the Midwest Nursing Research Society’s 2018 Annual Research Conference in Cleveland, Ohio, as well as at AcademyHealth’s 2019 Annual Research Meeting in Washington, District of Columbia.

Preparing patients for hospital discharge requires multiple tasks that cross professional boundaries. Clinician’s roles may be ambiguous, and responsibility for a safe high-quality discharge is often diffused among the team rather than being defined as the core responsibility of a single member.^1-8 Without a shared understanding of patient resources and tasks involved in anticipatory planning, lapses in discharge preparation can occur, which places patients at increased risk for harm after hospitalization.^3-7 As a result, organizations like the Centers for Medicare & Medicaid Services (CMS) have called for team-based patient-centered discharge planning.⁸ Yet to develop more effective team-based discharge planning interventions, a more nuanced understanding of how healthcare teams work together is needed.^2,3,9

Shared mental models (SMMs) provide a useful theoretical framework and measurement approach for examining how interprofessional teams coordinate complex tasks like hospital discharge.^10-13 SMMs represent the team members’ collective understanding and organized knowledge of key elements needed for teams to perform effectively.^9-11 Validated questionnaires can be used to measure two key properties of SMMs: the degree to which team members have a similar understanding of the situation at hand (team SMM convergence) and to what extent this understanding is aligned with the patient (team-­patient SMM convergence).^10,11 Researchers have found that teams with higher-quality SMMs have a better understanding of what is happening and why, have clearer expectations of their roles and tasks, and can better predict what might happen next.^10,12 As a result, these teams more effectively coordinate actions and adapt to task demands even in cases of high complexity, uncertainty, and stress.^10-13 Prior studies examining healthcare teams in emergency departments,^14-16 critical care units,^17,18 and operating rooms¹⁹ suggest high-quality SMMs are needed to safely care for patients.¹³ Yet there has been limited evaluation of SMMs in general internal medicine, much less during hospital discharge.^9,13 The purpose of this study was to examine SMMs for a critical task of the inpatient team: developing a shared understanding of the patient’s readiness for hospital discharge.^20,21

 

METHODS

Design

We used a cross-sectional survey design at a single Midwestern community hospital to determine inpatient care teams’ SMMs of patient hospital discharge readiness. This study is part of a larger mixed-methods evaluation of interprofessional hospital discharge teamwork in older adult patients at risk for a poor transition to home.⁹ Data were collected using questionnaires from patients and their team (nurse, coordinator, and physician) within 4 hours of the patient leaving the hospital. First, we measured the teams’ assessment, team convergence, and team-­patient convergence on patient readiness for discharge from the hospital. Then, after identifying relevant potential predictors from the literature, we developed regression models to predict the teams’ assessment, team convergence, and team-patient convergence of discharge readiness based on these variables. Our local institutional review board approved this study.

Sample and Participants

We used a convenience sampling approach to identify eligible discharge events consisting of the patient and care team.⁹ We focused on patients at high-risk for poor hospital-to-home transitions.^3,22 Eligible events included older patients (≥65 years) who were discharged home without home health or hospice services and admitted with a primary diagnosis of heart failure, acute myocardial infarction, hip replacement, knee replacement, pneumonia, or chronic obstructive pulmonary disease. Patient exclusion criteria included inability to complete study forms because of mental incapacity or a language barrier. Discharge team member inclusion criteria included the bedside nurse, attending physician, and coordinator (a unit-dedicated discharge nurse or social worker) caring for the patient participant at the time of hospital discharge. Each discharge team was unique: The same three individuals could not be included as a “team” for more than one discharge event, although individual members could be included as a part of other teams with a different set of individuals. Appendix A provides an enrollment flowchart.

Conceptual Framework

We applied the SMM conceptual framework to the context of hospital discharge. As shown in the Figure, SMMs are examined at the team level and contain the critical knowledge held by the team to be effective.^15,16 From a patient-centered perspective, patients are considered the expert on how ready they feel to be discharged home.^20,23,24 In this case, the SMM content is the discharge team members’ shared assessment of how ready the patient is for hospital discharge (Figure, B).¹⁰ Convergence is the degree of agreement among individual mental models.^10-13 In this study we examined two types of convergence: (1) team convergence, or the team members degree of agreement on the patient’s readiness for discharge (Figure, C), and (2) team-patient convergence, or the degree to which the team’s SMM aligns with the patient’s mental model (Figure, D).^10-13

Measures and Variables

Readiness for Hospital Discharge Scales/Short Form

We used parallel clinician and patient versions of the Readiness for Hospital Discharge Scale/Short Form (RHDS/SF)^25-28 to determine the teams’ assessment of discharge readiness, team SMM convergence, and team-patient SMM convergence.

The RHDS/SF scales are 8-item validated instruments that use a Likert scale (0 for not ready to 10 for totally ready) to assess the individual clinician’s or patient’s perceptions of how ready the patient is to be discharged.^20,25,27 The RHDS/SF instruments include four dimensions conceptualized as crucial to patient readiness for discharge and important to anticipatory planning: (1) Personal Status, physical-emotional state of the patient before discharge; (2) Knowledge, perceived adequacy of information needed to respond to common posthospitalization concerns/problems; (3) Coping Ability, perceived ability to self-manage health care needs; and (4) Expected Support, emotional-physical assistance available (Appendix B).^20,25,27 The RHDS/SF instruments’ results are calculated as a mean of item scores, with higher individual scores indicating the rater assessed the patient as being more ready for hospital discharge.²⁰ The RHDS/SF scales have undergone rigorous psychometric testing and are linked to patient outcomes (eg, readmissions, emergency room visits, patient coping difficulties after discharge, and patient-rated quality of preparation for posthospital care).^20,25-28 For example, predictive validity assessments for adult medical-surgical patients found lower Nurse-RHDS/SF scores are associated with a six- to ninefold increase in 30-day readmission risk.²⁰

 

Contextual Variables

We reviewed the literature to identify potential patient and system factors associated with adverse transitional care outcomes^1-8 and/or higher quality SMMs in other settings.^10-19 For example, patient characteristics included age, principal diagnosis, length of stay, number of comorbidities, and cognition impairment (using the Short Portable Mini Mental Status Questionnaire²⁹).^2,22,30 Examples of system factor include teamwork and communication quality^1-6 on day of discharge, as well as educational background and experience of clinicians on the team.^31-33 We adapted a validated survey using 7-point Likert scale questions to determine teamwork quality and communication quality during individual patent discharges.³³ Appendix C provides descriptions of all variables.

Data Collection

Patient recruitment occurred from February to October 2017 in a single community hospital in Iowa.⁹ We identified potentially eligible events in collaboration with the unit charge nurses. Patients were screened 24 to 48 hours prior to anticipated day of discharge; those interested/eligible underwent informed consent procedures.⁹ We collected data from the patient and their corresponding bedside nurse, coordinator, and attending physician on the day of discharge. After the discharge order was placed and care instructions were provided, the patient completed a demographic survey, Short Portable Mini Mental Status Questionnaire, and the Patient-RHDS/SF. Individual team members completed a survey with the demographic information, their respective versions of RHDS/SF, and day-of-discharge teamwork-related questions. On average, the survey took clinicians less than 5 minutes to complete. We performed a chart review to determine additional patient characteristics such as principal diagnosis, length of stay, and number of comorbidities.

Data Analysis

Team Assessment of Patient Discharge Readiness

The teams’ shared assessment (SMM content) was determined by averaging the members’ individual scores on the Clinician-RHDS/SF.³⁴ Discharge events with higher team assessments indicated the team perceived the patient as being readier for hospital discharge. Guided by prior research, we examined the RHDS/SF scores as a continuous variable and as a four-level categorical variable of readiness: low (<7), moderate (7-7.9), high (8-8.9), and very high (9-10).²⁰

Team SMM Convergence

To determine the teams’ convergence on patient discharge readiness, we calculated an adjusted interrater agreement index (r*_wg(j))^35,36 for each team using the individual clinicians’ scores on the RHDS/SF. These convergence values were categorized into four agreement levels: low agreement (<0.7), moderate agreement (0.7-0.79), high agreement (0.8-0.89), and very high agreement (0.9-1). See Appendix D for the r*_wg(j) equation.^35,36

Team-Patient SMM Convergence

To determine the team-patient SMM convergence, we subtracted the team’s assessment of patient discharge readiness from the Patient-RHDS/SF score. We used a one-unit change on the RHDS/SF (1 point on the 0-10 scale) as a meaningful difference between the patient’s self-assessment and teams’ assessment on readiness for hospital discharge. This definition for divergence aligns with prior RHDS psychometric testing studies^20,27 and research examining convergence between patient and nurse assessments.²⁸ For example, Weiss and colleagues²⁷ found a 1-point decrease in the RN-RHDS/SF item mean was associated with a 45% increase in likelihood of postdischarge utilization (hospital readmission and emergency room department visits). Therefore, we defined convergence of team-patient SMMs (or similar patient and team scores) as those with an absolute difference score less than 1 point, whereas teams with low team-patient SMM convergence (or divergent patient and team scores) were defined as having an absolute difference score greater than 1 point.

 

Prediction Models

For the exploratory aim, we first examined the bivariate relationship between the outcome variables (discharge teams’ assessment of patient readiness, team convergence, and team-patient convergence) and the identified contextual variables. We also checked for potential collinearity among the explanatory variables. Then we used a linear stepwise regression procedure to identify factors associated with each continuous outcome variable. Due to the small sample size, we performed separate backward stepwise regression selection analyses for the three outcomes of interest. The candidate explanatory variables were evaluated using P < .20 for model entry. Final models were evaluated using leave-one-out cross validation. STATA (v.15.1, StataCorp; 2017) was used for analysis.

RESULTS

Sample

A total of 64 discharge events were included in this study. All discharge teams had a unique composition including 64 patients and varying combinations of 56 individual nurses (n = 27), physicians (n = 23), and coordinators (n = 6). Each event had three team members (ie, a nurse, a coordinator, and a physician) with no missing data. The majority of the 64 patient participants were White, retired, had a high school education, and lived in their own home with only one other person (Table 1).

Interprofessional Teams’ SMM of Readiness for Hospital Discharge

While the majority of teams perceived patients had high readiness for hospital discharge (mean, 8.5 out of 10; SD, 0.91), patients scores were nearly a full point lower (mean, 7.7; SD, 1.6; Table 2). The largest difference across categories was in the low-readiness category with 27% of patient scores falling into this category vs only 9.4% of discharge team mean scores. The mean SMM convergence of team perception of patients’ readiness for discharge was 0.90 (SD, 0.10); however, scores ranged from 0.66 (low agreement) to 1 (complete agreement). The average SMM team-patient convergence, or the discrepancy between the discharge team mean scores and the patient total scores across domains, was 1.16 (SD, 0.82). Of the 64 discharge events, 42.2% had similar team-patient perceptions of readiness for discharge, 9.4% had the patient reporting higher readiness for discharge than the team, and 48.4% had a team assessment rating of higher readiness for discharge than the patient’s self-assessment.

Prediction Models

In the exploratory analysis, we created individual linear regression models to predict the teams’ assessment, team convergence, and team-patient convergence for readiness of hospital discharge (Table 3; Appendix E). Factors associated with the teams’ assessment of discharge readiness included whether the patient was married and had less cognitive impairment, both of which were positively related to a higher-­rated readiness among teams. An important system factor was higher quality of communication among team members, which was positively associated with teams’ assessment of patient discharge readiness. In contrast, only patient factors—married patients and those with a principal diagnosis of heart failure—were associated with more convergent team SMMs. Team-patient convergence was positively associated with two patient factors: marital status (married) and fewer comorbidities. However, team-patient convergence was also associated with two system factors: teams with a bachelor’s level–trained nurse (compared with a nurse with an associate degree ) and teams reporting a higher quality of teamwork on day of discharge.

 

DISCUSSION

Our study applied novel approaches to explore the interprofessional teams’ understanding of discharge readiness, a concept known to be an important predictor of patient outcomes after discharge, including readmission.^20,28 We found that discharge teams frequently had poor quality SMMs of hospital discharge readiness. Despite having a discharge order and receiving home care instructions, one in four patients reported low readiness for hospital discharge. Additionally, discharge teams frequently overestimated patient’s readiness for hospital discharge. Misalignment on patient readiness for discharge occurred both within the discharge team (ie, low team convergence) and between patients and their care teams (ie, low team-patient convergence). The potential importance of this disagreement is substantiated by prior work suggesting that divergence in readiness ratings between nurses and patients are associated with postdischarge coping difficulties.²⁸

Previous readiness for discharge has been measured from the perspective of the patient,^20,21,27,28 nurse,^20,25-28 and physician,³⁷ yet rarely has the teams’ perspective been examined. We add to this literature by measuring the team’s perspective, as well as agreement between team and patient, on the individual patient’s readiness for discharge. Notably, we found that higher-quality communication is positively related to teams’ assessment of discharge readiness, with teams that reported higher quality teamwork having more convergent team-patient SMMs. Our results support many qualitative studies identifying communication and teamwork as major factors in teams’ effectiveness in discharge planning.^1-7,9 However, given the small sample size in this study, additional research is needed to further understand these relationships, as well as link SMMs to patient outcomes such as hospital readmission.

In an attempt to improve discharge planning, hospitals are increasingly assessing readiness for discharge as a low-intensity, low-cost intervention.^26,27 Yet, recent evidence suggests that readiness assessments alone have minimal impact on reducing hospital readmissions.²⁶ To be successful, these assessments likely depend on quality interprofessional communication and ensuring the patient’s voice is incorporated into the discharge decision process.²⁶ However, there have been few ways to effectively evaluate these types of team interventions.⁹ Measuring SMM properties holds promise for identifying specific team mechanisms that may influence the effectiveness and fidelity of interventions for team-based discharge planning. As our findings indicate, SMMs provide a theoretical and methodological basis for evaluating if readiness for discharge was team based (convergence among team members) and patient centered (convergence among team assessment and patient self-assessment). Researchers and improvement scientists can use the approach outlined to evaluate team-based patient-centered interventions for hospital discharge planning.⁹

This study provides a unique contribution to the growing work in the team science of SMMs.^9,10 We rigorously evaluated SMMs of key stakeholders (patients and their interprofessional team) in “real-time” clinical practice using a patient-centered assessment linked with postdischarge outcomes.^20,27,28 However, it is still unknown how much convergence is needed (and with whom) to safely discharge patients.¹³ Prior studies suggest highly convergent SMMs increase team performance when they are also accurate.^10-13 Convergence alone should not be sought because this may reflect groupthink or clinical inertia.^10,15 To improve discharge team performance over time,^10‑13 it is important to assess not only patient’s readiness on the day of discharge but also how prepared the patient actually was for the recovery period following acute care. In the larger mixed-methods study, we found that teams’ with more convergent SMMs on teamwork quality were associated with patient’s reported quality of transition 30-days after discharge.⁹ Together, these findings further highlight the importance of aligning patient and interprofessional team members perspectives during the discharge planning, as well as providing clinicians with regular feedback about patient’s postdischarge experiences and outcomes.

To optimize team performance, the discharge planning process must be considered from an interprofessional team perspective as it functions in real-world practice settings. There are increasing pressures to discharge patients “quicker and sicker,” to simplify and standardize clinical process, and to provide patient-centered care.^3,5-8 Without thoughtful interventions to facilitate communication during discharge planning, these pressures likely reinforce inaccurate assumptions regarding the work of fellow team members and force teams to think “fast” instead of “slow.”^38-40 One approach to overcome such barriers is to focus on building a high-quality interprofessional SMM around discharge readiness. For example, the RHDS/SF questions could be integrated into the electronic medical records, displayed on dashboards, and discussed regularly during discharge rounds. In particular, to strengthen the team’s SMM and quality of teamwork, together the staff can ask three practical questions (Appendix F): (1) Do we think the patient is ready for discharge? (2) To what extent do we all agree the patient is ready for discharge? (3) Does our assessment of discharge readiness match the patient’s? During this high-risk transition point, asking these questions might allow the team to move from thinking fast to thinking slowly so they can more effectively identify heuristics they may be using inaccurately, prevent blind spots, and move toward high reliability.^{10,13,18,38-40}

This study has limitations. First, events were recruited from patients with any of only six conditions at a single hospital. Other settings, patient condition types, or team compositions of other clinicians may differ in results. Second, in this study the SMM content was focused on readiness for hospital discharge among four key stakeholders. It is possible other SMM content needs to be shared among the interprofessional discharge team (eg, caregivers’ perspectives,^2,6-8 resource availability,^3-6 clinicians’ roles^4,9) or additional members should be included (eg, physical therapists, nursing assistants, home health consultants, or primary care clinicians). Although this study focused on a patient-centered outcome (Patient-RHDS/SF), we did not examine other important outcomes such as hospital readmission. Additionally, due to the small sample size, these results have limited generalizability and should be interpreted with caution. Last, we limited data collection to the day of hospital discharge; future studies might consider assessing discharge readiness throughout hospitalization.

 

CONCLUSION

Readying patients for hospital discharge is a time-sensitivehigh-risk task requiring multiple healthcare professionals to concurrently assess patient needs, formulate an anticipatory care plan, provide education, and arrange for postdischarge needs.^20,21 Despite this, few studies have analyzed teamwork aspects to understand how these transitions could be improved.⁹ By piloting SMM measurement and describing factors that affect SMMs, we provide a step toward identifying and evaluating strategies to assist interprofessional care teams in preparing patients for a safe, high-quality, patient-centered hospital discharge.

Presentations

This work was presented at the Midwest Nursing Research Society’s 2018 Annual Research Conference in Cleveland, Ohio, as well as at AcademyHealth’s 2019 Annual Research Meeting in Washington, District of Columbia.

Preparing patients for hospital discharge requires multiple tasks that cross professional boundaries. Clinician’s roles may be ambiguous, and responsibility for a safe high-quality discharge is often diffused among the team rather than being defined as the core responsibility of a single member.^1-8 Without a shared understanding of patient resources and tasks involved in anticipatory planning, lapses in discharge preparation can occur, which places patients at increased risk for harm after hospitalization.^3-7 As a result, organizations like the Centers for Medicare & Medicaid Services (CMS) have called for team-based patient-centered discharge planning.⁸ Yet to develop more effective team-based discharge planning interventions, a more nuanced understanding of how healthcare teams work together is needed.^2,3,9

Shared mental models (SMMs) provide a useful theoretical framework and measurement approach for examining how interprofessional teams coordinate complex tasks like hospital discharge.^10-13 SMMs represent the team members’ collective understanding and organized knowledge of key elements needed for teams to perform effectively.^9-11 Validated questionnaires can be used to measure two key properties of SMMs: the degree to which team members have a similar understanding of the situation at hand (team SMM convergence) and to what extent this understanding is aligned with the patient (team-­patient SMM convergence).^10,11 Researchers have found that teams with higher-quality SMMs have a better understanding of what is happening and why, have clearer expectations of their roles and tasks, and can better predict what might happen next.^10,12 As a result, these teams more effectively coordinate actions and adapt to task demands even in cases of high complexity, uncertainty, and stress.^10-13 Prior studies examining healthcare teams in emergency departments,^14-16 critical care units,^17,18 and operating rooms¹⁹ suggest high-quality SMMs are needed to safely care for patients.¹³ Yet there has been limited evaluation of SMMs in general internal medicine, much less during hospital discharge.^9,13 The purpose of this study was to examine SMMs for a critical task of the inpatient team: developing a shared understanding of the patient’s readiness for hospital discharge.^20,21

 

METHODS

Design

We used a cross-sectional survey design at a single Midwestern community hospital to determine inpatient care teams’ SMMs of patient hospital discharge readiness. This study is part of a larger mixed-methods evaluation of interprofessional hospital discharge teamwork in older adult patients at risk for a poor transition to home.⁹ Data were collected using questionnaires from patients and their team (nurse, coordinator, and physician) within 4 hours of the patient leaving the hospital. First, we measured the teams’ assessment, team convergence, and team-­patient convergence on patient readiness for discharge from the hospital. Then, after identifying relevant potential predictors from the literature, we developed regression models to predict the teams’ assessment, team convergence, and team-patient convergence of discharge readiness based on these variables. Our local institutional review board approved this study.

Sample and Participants

We used a convenience sampling approach to identify eligible discharge events consisting of the patient and care team.⁹ We focused on patients at high-risk for poor hospital-to-home transitions.^3,22 Eligible events included older patients (≥65 years) who were discharged home without home health or hospice services and admitted with a primary diagnosis of heart failure, acute myocardial infarction, hip replacement, knee replacement, pneumonia, or chronic obstructive pulmonary disease. Patient exclusion criteria included inability to complete study forms because of mental incapacity or a language barrier. Discharge team member inclusion criteria included the bedside nurse, attending physician, and coordinator (a unit-dedicated discharge nurse or social worker) caring for the patient participant at the time of hospital discharge. Each discharge team was unique: The same three individuals could not be included as a “team” for more than one discharge event, although individual members could be included as a part of other teams with a different set of individuals. Appendix A provides an enrollment flowchart.

Conceptual Framework

We applied the SMM conceptual framework to the context of hospital discharge. As shown in the Figure, SMMs are examined at the team level and contain the critical knowledge held by the team to be effective.^15,16 From a patient-centered perspective, patients are considered the expert on how ready they feel to be discharged home.^20,23,24 In this case, the SMM content is the discharge team members’ shared assessment of how ready the patient is for hospital discharge (Figure, B).¹⁰ Convergence is the degree of agreement among individual mental models.^10-13 In this study we examined two types of convergence: (1) team convergence, or the team members degree of agreement on the patient’s readiness for discharge (Figure, C), and (2) team-patient convergence, or the degree to which the team’s SMM aligns with the patient’s mental model (Figure, D).^10-13

Measures and Variables

Readiness for Hospital Discharge Scales/Short Form

We used parallel clinician and patient versions of the Readiness for Hospital Discharge Scale/Short Form (RHDS/SF)^25-28 to determine the teams’ assessment of discharge readiness, team SMM convergence, and team-patient SMM convergence.

The RHDS/SF scales are 8-item validated instruments that use a Likert scale (0 for not ready to 10 for totally ready) to assess the individual clinician’s or patient’s perceptions of how ready the patient is to be discharged.^20,25,27 The RHDS/SF instruments include four dimensions conceptualized as crucial to patient readiness for discharge and important to anticipatory planning: (1) Personal Status, physical-emotional state of the patient before discharge; (2) Knowledge, perceived adequacy of information needed to respond to common posthospitalization concerns/problems; (3) Coping Ability, perceived ability to self-manage health care needs; and (4) Expected Support, emotional-physical assistance available (Appendix B).^20,25,27 The RHDS/SF instruments’ results are calculated as a mean of item scores, with higher individual scores indicating the rater assessed the patient as being more ready for hospital discharge.²⁰ The RHDS/SF scales have undergone rigorous psychometric testing and are linked to patient outcomes (eg, readmissions, emergency room visits, patient coping difficulties after discharge, and patient-rated quality of preparation for posthospital care).^20,25-28 For example, predictive validity assessments for adult medical-surgical patients found lower Nurse-RHDS/SF scores are associated with a six- to ninefold increase in 30-day readmission risk.²⁰

 

Contextual Variables

We reviewed the literature to identify potential patient and system factors associated with adverse transitional care outcomes^1-8 and/or higher quality SMMs in other settings.^10-19 For example, patient characteristics included age, principal diagnosis, length of stay, number of comorbidities, and cognition impairment (using the Short Portable Mini Mental Status Questionnaire²⁹).^2,22,30 Examples of system factor include teamwork and communication quality^1-6 on day of discharge, as well as educational background and experience of clinicians on the team.^31-33 We adapted a validated survey using 7-point Likert scale questions to determine teamwork quality and communication quality during individual patent discharges.³³ Appendix C provides descriptions of all variables.

Data Collection

Patient recruitment occurred from February to October 2017 in a single community hospital in Iowa.⁹ We identified potentially eligible events in collaboration with the unit charge nurses. Patients were screened 24 to 48 hours prior to anticipated day of discharge; those interested/eligible underwent informed consent procedures.⁹ We collected data from the patient and their corresponding bedside nurse, coordinator, and attending physician on the day of discharge. After the discharge order was placed and care instructions were provided, the patient completed a demographic survey, Short Portable Mini Mental Status Questionnaire, and the Patient-RHDS/SF. Individual team members completed a survey with the demographic information, their respective versions of RHDS/SF, and day-of-discharge teamwork-related questions. On average, the survey took clinicians less than 5 minutes to complete. We performed a chart review to determine additional patient characteristics such as principal diagnosis, length of stay, and number of comorbidities.

Data Analysis

Team Assessment of Patient Discharge Readiness

The teams’ shared assessment (SMM content) was determined by averaging the members’ individual scores on the Clinician-RHDS/SF.³⁴ Discharge events with higher team assessments indicated the team perceived the patient as being readier for hospital discharge. Guided by prior research, we examined the RHDS/SF scores as a continuous variable and as a four-level categorical variable of readiness: low (<7), moderate (7-7.9), high (8-8.9), and very high (9-10).²⁰

Team SMM Convergence

To determine the teams’ convergence on patient discharge readiness, we calculated an adjusted interrater agreement index (r*_wg(j))^35,36 for each team using the individual clinicians’ scores on the RHDS/SF. These convergence values were categorized into four agreement levels: low agreement (<0.7), moderate agreement (0.7-0.79), high agreement (0.8-0.89), and very high agreement (0.9-1). See Appendix D for the r*_wg(j) equation.^35,36

Team-Patient SMM Convergence

To determine the team-patient SMM convergence, we subtracted the team’s assessment of patient discharge readiness from the Patient-RHDS/SF score. We used a one-unit change on the RHDS/SF (1 point on the 0-10 scale) as a meaningful difference between the patient’s self-assessment and teams’ assessment on readiness for hospital discharge. This definition for divergence aligns with prior RHDS psychometric testing studies^20,27 and research examining convergence between patient and nurse assessments.²⁸ For example, Weiss and colleagues²⁷ found a 1-point decrease in the RN-RHDS/SF item mean was associated with a 45% increase in likelihood of postdischarge utilization (hospital readmission and emergency room department visits). Therefore, we defined convergence of team-patient SMMs (or similar patient and team scores) as those with an absolute difference score less than 1 point, whereas teams with low team-patient SMM convergence (or divergent patient and team scores) were defined as having an absolute difference score greater than 1 point.

 

Prediction Models

For the exploratory aim, we first examined the bivariate relationship between the outcome variables (discharge teams’ assessment of patient readiness, team convergence, and team-patient convergence) and the identified contextual variables. We also checked for potential collinearity among the explanatory variables. Then we used a linear stepwise regression procedure to identify factors associated with each continuous outcome variable. Due to the small sample size, we performed separate backward stepwise regression selection analyses for the three outcomes of interest. The candidate explanatory variables were evaluated using P < .20 for model entry. Final models were evaluated using leave-one-out cross validation. STATA (v.15.1, StataCorp; 2017) was used for analysis.

RESULTS

Sample

A total of 64 discharge events were included in this study. All discharge teams had a unique composition including 64 patients and varying combinations of 56 individual nurses (n = 27), physicians (n = 23), and coordinators (n = 6). Each event had three team members (ie, a nurse, a coordinator, and a physician) with no missing data. The majority of the 64 patient participants were White, retired, had a high school education, and lived in their own home with only one other person (Table 1).

Interprofessional Teams’ SMM of Readiness for Hospital Discharge

While the majority of teams perceived patients had high readiness for hospital discharge (mean, 8.5 out of 10; SD, 0.91), patients scores were nearly a full point lower (mean, 7.7; SD, 1.6; Table 2). The largest difference across categories was in the low-readiness category with 27% of patient scores falling into this category vs only 9.4% of discharge team mean scores. The mean SMM convergence of team perception of patients’ readiness for discharge was 0.90 (SD, 0.10); however, scores ranged from 0.66 (low agreement) to 1 (complete agreement). The average SMM team-patient convergence, or the discrepancy between the discharge team mean scores and the patient total scores across domains, was 1.16 (SD, 0.82). Of the 64 discharge events, 42.2% had similar team-patient perceptions of readiness for discharge, 9.4% had the patient reporting higher readiness for discharge than the team, and 48.4% had a team assessment rating of higher readiness for discharge than the patient’s self-assessment.

Prediction Models

In the exploratory analysis, we created individual linear regression models to predict the teams’ assessment, team convergence, and team-patient convergence for readiness of hospital discharge (Table 3; Appendix E). Factors associated with the teams’ assessment of discharge readiness included whether the patient was married and had less cognitive impairment, both of which were positively related to a higher-­rated readiness among teams. An important system factor was higher quality of communication among team members, which was positively associated with teams’ assessment of patient discharge readiness. In contrast, only patient factors—married patients and those with a principal diagnosis of heart failure—were associated with more convergent team SMMs. Team-patient convergence was positively associated with two patient factors: marital status (married) and fewer comorbidities. However, team-patient convergence was also associated with two system factors: teams with a bachelor’s level–trained nurse (compared with a nurse with an associate degree ) and teams reporting a higher quality of teamwork on day of discharge.

 

DISCUSSION

Our study applied novel approaches to explore the interprofessional teams’ understanding of discharge readiness, a concept known to be an important predictor of patient outcomes after discharge, including readmission.^20,28 We found that discharge teams frequently had poor quality SMMs of hospital discharge readiness. Despite having a discharge order and receiving home care instructions, one in four patients reported low readiness for hospital discharge. Additionally, discharge teams frequently overestimated patient’s readiness for hospital discharge. Misalignment on patient readiness for discharge occurred both within the discharge team (ie, low team convergence) and between patients and their care teams (ie, low team-patient convergence). The potential importance of this disagreement is substantiated by prior work suggesting that divergence in readiness ratings between nurses and patients are associated with postdischarge coping difficulties.²⁸

Previous readiness for discharge has been measured from the perspective of the patient,^20,21,27,28 nurse,^20,25-28 and physician,³⁷ yet rarely has the teams’ perspective been examined. We add to this literature by measuring the team’s perspective, as well as agreement between team and patient, on the individual patient’s readiness for discharge. Notably, we found that higher-quality communication is positively related to teams’ assessment of discharge readiness, with teams that reported higher quality teamwork having more convergent team-patient SMMs. Our results support many qualitative studies identifying communication and teamwork as major factors in teams’ effectiveness in discharge planning.^1-7,9 However, given the small sample size in this study, additional research is needed to further understand these relationships, as well as link SMMs to patient outcomes such as hospital readmission.

In an attempt to improve discharge planning, hospitals are increasingly assessing readiness for discharge as a low-intensity, low-cost intervention.^26,27 Yet, recent evidence suggests that readiness assessments alone have minimal impact on reducing hospital readmissions.²⁶ To be successful, these assessments likely depend on quality interprofessional communication and ensuring the patient’s voice is incorporated into the discharge decision process.²⁶ However, there have been few ways to effectively evaluate these types of team interventions.⁹ Measuring SMM properties holds promise for identifying specific team mechanisms that may influence the effectiveness and fidelity of interventions for team-based discharge planning. As our findings indicate, SMMs provide a theoretical and methodological basis for evaluating if readiness for discharge was team based (convergence among team members) and patient centered (convergence among team assessment and patient self-assessment). Researchers and improvement scientists can use the approach outlined to evaluate team-based patient-centered interventions for hospital discharge planning.⁹

This study provides a unique contribution to the growing work in the team science of SMMs.^9,10 We rigorously evaluated SMMs of key stakeholders (patients and their interprofessional team) in “real-time” clinical practice using a patient-centered assessment linked with postdischarge outcomes.^20,27,28 However, it is still unknown how much convergence is needed (and with whom) to safely discharge patients.¹³ Prior studies suggest highly convergent SMMs increase team performance when they are also accurate.^10-13 Convergence alone should not be sought because this may reflect groupthink or clinical inertia.^10,15 To improve discharge team performance over time,^10‑13 it is important to assess not only patient’s readiness on the day of discharge but also how prepared the patient actually was for the recovery period following acute care. In the larger mixed-methods study, we found that teams’ with more convergent SMMs on teamwork quality were associated with patient’s reported quality of transition 30-days after discharge.⁹ Together, these findings further highlight the importance of aligning patient and interprofessional team members perspectives during the discharge planning, as well as providing clinicians with regular feedback about patient’s postdischarge experiences and outcomes.

To optimize team performance, the discharge planning process must be considered from an interprofessional team perspective as it functions in real-world practice settings. There are increasing pressures to discharge patients “quicker and sicker,” to simplify and standardize clinical process, and to provide patient-centered care.^3,5-8 Without thoughtful interventions to facilitate communication during discharge planning, these pressures likely reinforce inaccurate assumptions regarding the work of fellow team members and force teams to think “fast” instead of “slow.”^38-40 One approach to overcome such barriers is to focus on building a high-quality interprofessional SMM around discharge readiness. For example, the RHDS/SF questions could be integrated into the electronic medical records, displayed on dashboards, and discussed regularly during discharge rounds. In particular, to strengthen the team’s SMM and quality of teamwork, together the staff can ask three practical questions (Appendix F): (1) Do we think the patient is ready for discharge? (2) To what extent do we all agree the patient is ready for discharge? (3) Does our assessment of discharge readiness match the patient’s? During this high-risk transition point, asking these questions might allow the team to move from thinking fast to thinking slowly so they can more effectively identify heuristics they may be using inaccurately, prevent blind spots, and move toward high reliability.^{10,13,18,38-40}

This study has limitations. First, events were recruited from patients with any of only six conditions at a single hospital. Other settings, patient condition types, or team compositions of other clinicians may differ in results. Second, in this study the SMM content was focused on readiness for hospital discharge among four key stakeholders. It is possible other SMM content needs to be shared among the interprofessional discharge team (eg, caregivers’ perspectives,^2,6-8 resource availability,^3-6 clinicians’ roles^4,9) or additional members should be included (eg, physical therapists, nursing assistants, home health consultants, or primary care clinicians). Although this study focused on a patient-centered outcome (Patient-RHDS/SF), we did not examine other important outcomes such as hospital readmission. Additionally, due to the small sample size, these results have limited generalizability and should be interpreted with caution. Last, we limited data collection to the day of hospital discharge; future studies might consider assessing discharge readiness throughout hospitalization.

 

CONCLUSION

Readying patients for hospital discharge is a time-sensitivehigh-risk task requiring multiple healthcare professionals to concurrently assess patient needs, formulate an anticipatory care plan, provide education, and arrange for postdischarge needs.^20,21 Despite this, few studies have analyzed teamwork aspects to understand how these transitions could be improved.⁹ By piloting SMM measurement and describing factors that affect SMMs, we provide a step toward identifying and evaluating strategies to assist interprofessional care teams in preparing patients for a safe, high-quality, patient-centered hospital discharge.

Presentations

This work was presented at the Midwest Nursing Research Society’s 2018 Annual Research Conference in Cleveland, Ohio, as well as at AcademyHealth’s 2019 Annual Research Meeting in Washington, District of Columbia.

Continuous pulse oximetry monitoring (cSpO₂) in children with bronchiolitis is associated with increased rates of hospital admission, longer lengths of stay, more frequent treatment with supplemental oxygen, alarm fatigue, and higher hospital cost. There is no evidence that it improves clinical outcomes.^1-7 The safety of reducing cSpO₂ for stable bronchiolitis patients (ie, those who are clinically well and not requiring supplemental oxygen) has been assessed in quality improvement initiatives^8-10 and a randomized controlled trial.² These studies showed no increase in intensive care unit transfers, codes, or readmissions associated with reduced cSpO₂. Current national guidelines from the American Academy of Pediatrics⁵ and the Society of Hospital Medicine Choosing Wisely in Pediatric Hospital Medicine workgroup⁴ support limiting monitoring of children with bronchiolitis. Despite this, the practice of cSpO₂ in stable bronchiolitis patients off supplemental oxygen remains widespread.^11,12

Deimplementation, defined as reducing or stopping low-value or ineffective healthcare practices,^13,14 is a discrete focus area within implementation science. Deimplementation research involves the reduction of unnecessary and overused services for which there is potential for harm or no benefit.^15,16 In pediatrics, there are a number of potential targets for deimplementation,^4,17-20 including cSpO₂ for stable infants with bronchiolitis, but efforts to reduce low-value practices have met limited success to date. ^21,22

Implementation science offers rigorous methods for advancing the development and evaluation of strategies for deimplementation.²³ In particular, implementation science frameworks can facilitate our understanding of relevant contextual factors that may hinder or help efforts to deimplement low-value practices. To develop broadly applicable strategies to reduce monitoring overuse, it is important to understand the barriers, facilitators, and contextual factors (eg, clinical, political, interpersonal) that contribute to guideline-discordant cSpO₂ in hospitalized bronchiolitis patients. Further, the process by which one can develop a rigorous understanding of these factors and how they may impact deimplementation efforts could generalize to other scenarios in pediatrics where overuse remains an issue.

The goal of this study was to use semistructured interviews, informed by an established implementation science framework, specifically the Consolidated Framework for Implementation Research (CFIR),²⁴ to (1) identify barriers and facilitators to deimplementing unnecessary cSpO₂, and (2) develop strategies to deimplement cSpO₂ in a multicenter cohort of hospital-based clinician and administrative stakeholders.

METHODS

Study Setting

This multicenter qualitative study using semistructured interviews took place within the Eliminating Monitor Overuse (EMO) SpO₂ study. The EMO SpO₂ study established rates of cSpO₂ in bronchiolitis patients not receiving supplemental oxygen or not receiving room air flow at 56 hospitals across the United States and in Canada from December 1, 2018, through March 31, 2019.¹² The study identified hospital-level risk-adjusted cSpO₂ rates ranging from 6% to 82%. A description of the EMO SpO₂ study methods²⁵ and its findings¹² have been published elsewhere.

Participants

We approached EMO study site principal investigators at 12 hospitals: the two highest- and two lowest-use hospitals within three hospital types (ie, freestanding children’s hospitals, children’s hospitals within large general hospitals, and community hospitals). We collaborated with the participating site principal investigators (n = 12), who were primarily hospitalist physicians in leadership roles, to recruit a purposive sample of additional stakeholders including bedside nurses (n = 12), hospitalist physicians (n = 15), respiratory therapists (n = 9), and hospital administrators (n = 8) to participate in semistructured interviews. Interviews were conducted until we achieved thematic saturation within each stakeholder group and within the high and low performing strata (total 56 interviews). Participants were asked to self-report basic demographic information (see Appendix, interview guide) as required by the study funder and to allow us to comment on the representativeness of the participant group.

 

Procedure

The interview guide was informed by the CFIR, a comprehensive framework detailing contextual factors that require consideration when planning for the implementation of a health service intervention. Table 1 details the CFIR domains with study-related examples. The interview guide (Appendix) provided limited clinical context apart from the age, diagnosis, and oxygen requirement for the population of interest to promote a broad array of responses and to avoid anchoring on specific clinical scenarios. Interviews were conducted by master’s degree or doctoral-level research coordinators with qualitative interviewing experience and supervised by a medical anthropologist and qualitative methods expert (F.K.B.). Prior to engaging in audio recorded phone interviews, the interviewer explained the risks and benefits of participating. Participants were compensated $50. Audio recordings were transcribed, deidentified, and uploaded to NVivo 12 Plus (QSR International) for data management.

The Institutional Review Boards of Children’s Hospital of Philadelphia, Pennsylvania, and the University of Pennsylvania in Philadelphia determined that the study met eligibility criteria for IRB exemption.

Data Analysis

Using an integrated approach to codebook development,²⁶ a priori codes were developed using constructs from the CFIR. Additional codes were added by the research team following a close reading of the first five transcripts.^27,28 Each code was defined, including decision rules for its application. Two research coordinators independently coded each transcript. Using the intercoder reliability function within NVivo, the coders established strong interrater reliability accordance scores (k > .8) by double coding 20% of the transcripts. Data were stratified by sites with low and high use of cSpO₂ to examine differences in barriers and facilitators to deimplementation. Each code was subcoded, summarized, and examined for patterns within and across participating disciplines, which yielded themes related to barriers and facilitators. We conducted member checking and reviewed our conclusions with a multidisciplinary group of clinical stakeholders (n = 13) to validate our analyses.

RESULTS

Barriers and facilitators to deimplementation were identified in multiple domains of the CFIR: outer setting, inner setting, characteristics of the individuals, and intervention characteristics (Table 1). Participants also suggested strategies to facilitate deimplementation in response to some identified barriers. See Table 2 for participant demographics and Table 3 for illustrative participant quotations.

Barriers

Outer Setting: Clinician Perceptions of Parental Discomfort With Discontinuing Monitoring

Participants mentioned parental preferences as a barrier to discontinuing cSpO₂, noting that parents seem to take comfort in watching the numbers on the monitor screen and are reluctant to have it withdrawn. Clinicians noted that parents sometimes put the monitor back on their child after a clinician removed it or have expressed concern that their unmonitored child was not receiving the same level of care as other patients who were being monitored. In these scenarios, clinicians reported they have found it helpful to educate caregivers about when cSpO₂ is and is not appropriate.

Inner Setting: Unclear or Nonexistent Guideline to Discontinue cSpO₂

Guidelines to discontinue cSpO₂ reportedly did not exist at all institutions. If a guideline did exist, lack of clarity or conflicting guidelines about when to use oxygen presented a barrier. Participants suggested that a clear guideline or additional oversight to ensure all clinicians are informed of the procedure for discontinuing cSpO₂ may help prevent miscommunication. Participants noted that their electronic health record (EHR) order sets commonly included cSpO₂ orders and that removing that option would facilitate deimplementation.

 

Inner Setting: Difficulty Educating All Staff

Participants noted difficulty with incorporating education about discontinuing cSpO₂ to all clinicians, particularly to those who are nightshift only or to rotating staff or trainees. This created barriers for frequent re-education because these staff are not familiar with the policies and procedures of the unit, which is crucial to developing a culture that supports the deimplementation of cSpO₂. Participants suggested that recurring education about procedures for discontinuing cSpO₂ should target trainees, new nurses, and overnight nurses. This would help to ensure that the guideline is uniformly followed.

Inner Setting: Culture of High cSpO₂ Use

Participants from high-use sites discussed a culture driven by readily available monitoring features or an expectation that monitoring indicates higher-quality care. Participants from low-use sites discussed increased cSpO₂ driven by clinicians who were accustomed to caring for higher-acuity patients, for whom continuous monitoring is likely appropriate, and were simultaneously caring for stable bronchiolitis patients.

Some suggested that visual cues would be useful to clinicians to sustain awareness about a cSpO₂ deimplementation guideline. It was also suggested that audit and feedback techniques like posting unit deimplementation statistics and creating a competition among units by posting unit performance could facilitate deimplementation. Additionally, some noted that visual aids in common spaces would be useful to remind clinicians and to engage caregivers about discontinuing cSpO₂.

Characteristics of Individuals: Clinician Discomfort Discontinuing cSpO₂

One frequently cited barrier across participants is that cSpO₂ provides “peace of mind” to alert clinicians to patients with low oxygen saturations that might otherwise be missed. Participants identified that clinician discomfort with reducing cSpO₂ may be driven by inexperienced clinicians less familiar with the bronchiolitis disease process, such as trainees, new nurses, or rotating clinicians unaccustomed to pediatric care. Trainees and new nurses were perceived as being more likely to work at night when there are fewer clinicians to provide patient care. Additionally, participants perceived that night shift clinicians favored cSpO₂ because they could measure vital signs without waking patients and families.

Clinicians discussed that discontinuing cSpO₂ would require alternative methods for assessing patient status, particularly for night shift nurses. Participants suggested strategies including changes to pulse oximetry assessment procedures to include more frequent “spot checks,” incorporation of assessments during sleep events (eg, naps) to ensure the patient does not experience desaturations during sleep, and training nurses to become more comfortable with suctioning patients. Suggestions also included education on the typical features of transient oxygen desaturations in otherwise stable patients with bronchiolitis² to bolster clinical confidence for clinicians unfamiliar with caring for bronchiolitis patients. Participants perceived that education about appropriate vs inappropriate use may help to empower clinicians to employ cSpO₂ appropriately.

Facilitators

Outer Setting: Standards and Evidence From Research, Professional Organizations, and Leaders in the Field

Many participants expressed the importance of consistent guidelines that are advocated by thought leaders in the field, supported by robust evidence, and consistent with approaches at peer hospitals. The more authoritative support a guideline has, the more comfortable people are adopting it and taking it seriously. Additionally, consistent education about guidelines was desired. Participants noted that all clinicians should be receiving education related to the American Academy of Pediatrics (AAP) Bronchiolitis and Choosing Wisely^® guidelines, ranging from a one-time update to annually. Continual updates and re-education sessions for clinicians who shared evidence about how cSpO₂ deimplementation could improve the quality of patient care by shortening hospital length of stay and lowering cost were suggested strategies.

 

Inner Setting: Leadership

Participants noted that successful deimplementation depends upon the presence of a champion or educator who will be able to lead the institutional charge in making practice change. This is typically an individual who is trusted at the institution, experienced in their field, or already doing implementation work. This could be either a single individual (champion) or a team. The most commonly noted clinician roles to engage in a leadership role or team were physicians and nurses.

Participants noted that a change in related clinical care pathways or EHR order sets would require cooperation from multiple clinical disciplines, administrators, and information technology leaders and explained that messaging and education about the value of the change would facilitate buy-in from those clinicians.

Inner Setting: EHR Support for Guidelines

Participants often endorsed the use of an order set within the EHR that supports guidelines and includes reminders to decrease cSpO₂. These reminders could come up when supplemental oxygen is discontinued or occur regularly throughout the patient’s stay to prompt the clinician to consider discontinuing cSpO₂.

Intervention Characteristics/Inner Setting: Clear Bronchiolitis Guidelines

The presence of a well-articulated hospital policy that delineates the appropriate and inappropriate use of cSpO₂ in bronchiolitis was mentioned as another facilitator of deimplementation.

DISCUSSION

Results of this qualitative study of stakeholders across hospitals with high and low cSpO₂ use illustrated the complexities involved with deimplementation of cSpO₂ in pediatric patients hospitalized with bronchiolitis. We identified numerous barriers spanning the CFIR constructs, including unclear or absent guidelines for stopping cSpO₂, clinician knowledge and comfort with bronchiolitis disease features, and unit culture. This suggests that multicomponent strategies that target various domains and a variety of stakeholders are needed to deimplement cSpO₂ use for stable bronchiolitis patients. Participants also identified facilitators, including clear cSpO₂ guidelines, supportive leaders and champions, and EHR modifications, that provide insight into strategies that may help sites reduce their use of cSpO₂. Additionally, participants also provided concrete, actionable suggestions for ways to reduce unnecessary monitoring that will be useful in informing promising deimplementation strategies for subsequent trials.

The importance of having specific and well-known guidelines from trusted sources, such as the AAP, about cSpO₂ and bronchiolitis treatment that are thoughtfully integrated in the EHR came through in multiple themes of our analysis. Prior studies on the effect of guidelines on clinical practice have suggested that rigorously designed guidelines can positively impact practice.²⁹ Participants also noted that cSpO₂ guidelines should be authoritative and that knowledge of guideline adoption by peer institutions was a facilitator of adoption. Usability issues negatively impact clinicians’ ability to follow guidelines.³⁰ Further, prior studies have demonstrated that EHR integration of guidelines can change practice.^31-33 Based on our findings, incorporating clear guidelines into commonly used formats, such as EHR order sets, could be an important deimplementation tool for cSpO₂ in stable bronchiolitis patients.

Education about and awareness of cSpO₂ guidelines was described as an important facilitator for appropriate cSpO₂ use and was suggested as a potential deimplementation strategy. Participants noted that educational need may vary by stakeholder group. For example, education may facilitate obtaining buy-in from hospital leaders, which is necessary to support changes to the EHR. Education incorporating information on the typical features of bronchiolitis and examples of appropriate and inappropriate cSpO₂ use was suggested for clinical team members. The limitations of education as a stand-alone deimplementation strategy were also noted, and participants highlighted challenges such as time needed for education and the need for ongoing education for rotating trainees. Inner and outer setting barriers, such as a perceived “culture of high pulse oximetryuse” and patient and family expectations, could also make education less effective as a stand-alone strategy. That—coupled with evidence that education and training alone are generally insufficient for producing reliable, sustained behavior change^34,35—suggests that a multifaceted approach will be important.

Our respondents consider parental perceptions and preference in their practice, which provides nuance to recent studies suggesting that parents prefer continuous monitors when their child is hospitalized with bronchiolitis. Chi et al described the impact of a brief educational intervention on parental preferences for monitoring children hospitalized for bronchiolitis.³⁶ This work suggests that educational interventions aimed at families should be considered in future (de)implementation studies because they may indirectly impact clinician behavior. Future studies should directly assess parental discomfort with discontinuing monitoring. Participants highlighted the link between knowledge and confidence in caring for typical bronchiolitis patients and monitoring practice, perceiving that less experienced clinicians are more likely to rely on cSpO₂. Participants at high-use sites emphasized the expectation that monitoring should occur during hospitalizations. This reflection is particularly pertinent for bronchiolitis, a disease characterized by frequent, self-resolving desaturations even after hospital discharge.³ This may reinforce a perceived need to capture and react to these desaturation events even though they are expected in bronchiolitis and can occur in healthy infants.³⁷ Some participants suggested that continuous monitoring be replaced with “nap tests” (ie, assessment for desaturations during a nap prior to discharge); however, like cSpO₂ in stable infants with bronchiolitis, this is another low-value practice. Otherwise healthy infants with mild to moderate disease are unlikely to subsequently worsen after showing signs of clinical improvement.³⁸ Nap tests are likely to lead to infants who are clinically improving being placed unnecessarily back on oxygen in reaction to the transient desaturations. Participants’ perception about the importance of cSpO₂ in bronchiolitis management, despite evidence suggesting it is a low-value practice, underscores the importance of not simply telling clinicians to stop cSpO₂. Employing strategies that replace continuous monitoring with another acceptable and feasible alternative (eg, regular clinician assessments including intermittent pulse oximetry checks) should be considered when planning for deimplementation.³⁹

Previous studies indicate that continuous monitoring can affect clinician decision-making, independent of other factors,^6,40 despite limited evidence that continuous monitors improve patient outcomes.^1-7 Studies have demonstrated noticeable increase in admissions based purely on pulse oximetry values,⁴⁰ with no evidence that this type of admission changes outcomes for bronchiolitis patients.⁶ One previous, single-center study identified inexperience as a potential driver for monitor use,⁴¹ and studies in adult populations have suggested that clinicians overestimate the value that continuous monitoring contributes to patient care,^42,43 which promotes guideline-discordant use. Our study provides novel insight into the issue of monitoring in bronchiolitis. Our results suggest that there is a need to shift organizational cultures around monitoring (which likely vary based on a range of factors) and that educational strategies addressing typical disease course, especially desaturations, in bronchiolitis will be an essential component in any deimplementation effort.

This study is strengthened by its sample of diverse stakeholder groups from multiple US health systems. Additionally, we interviewed individuals at sites with high cSpO₂ rates and at sites with low rates, as well as from community hospitals, children’s hospitals within general hospitals, and freestanding children’s hospitals, which allows us to understand barriers high-use sites encounter and facilitators of lower cSpO₂ rates at low-use sites. We also employed an interview approach informed by an established implementation science framework. Nonetheless, several limitations exist. First, participants at low-use sites did not necessarily have direct experience with a previous deimplementation effort to reduce cSpO₂. Additionally, participants were predominantly White and female; more diverse perspectives would strengthen confidence in the generalizability of our findings. While thematic saturation was achieved within each stakeholder group and within the high- and low-use strata, we interviewed fewer administrators and respiratory therapists relative to other stakeholder groups. Nevertheless, our conclusions were validated by our interdisciplinary stakeholder panel. As noted by participants, family preferences may influence clinician practice, and parents were not interviewed for this study. The information gleaned from the present study will inform the development of strategies to deimplement unnecessary cSpO₂ in pediatric hospitals, which we aim to rigorously evaluate in a future trial.

 

CONCLUSION

We identified barriers and facilitators to deimplementation of cSpO₂ for stable patients with bronchiolitis across children’s hospitals with high and low utilization of cSpO₂. These themes map to multiple CFIR domains and, along with participant-suggested strategies, can directly inform an approach to cSpO₂ deimplementation in a range of inpatient settings. Based on these data, future deimplementation efforts should focus on clear protocols for use and discontinuation of cSpO₂, EHR changes, and regular bronchiolitis education for hospital staff that emphasizes reducing unnecessary cSpO₂ utilization.

ACKNOWLEDGMENTS

We acknowledge the NHLBI scientists who contributed their expertise to this project as part of the U01 Cooperative Agreement funding mechanism as federal employees conducting their official job duties: Lora Reineck, MD, MS, Karen Bienstock, MS, and Cheryl Boyce, PhD. We thank the Executive Council of the Pediatric Research in Inpatient Settings (PRIS) Network for their contributions to the early scientific development of this project. The Network assessed a Collaborative Support Fee for access to the hospitals and support of this project. We thank the PRIS Network collaborators for their major contributions to data collection measuring utilization to identify the hospitals we subsequently chose for this project. We thank Claire Bocage and the Mixed Methods Research Lab for major help in data management and data analysis.

Continuous pulse oximetry monitoring (cSpO₂) in children with bronchiolitis is associated with increased rates of hospital admission, longer lengths of stay, more frequent treatment with supplemental oxygen, alarm fatigue, and higher hospital cost. There is no evidence that it improves clinical outcomes.^1-7 The safety of reducing cSpO₂ for stable bronchiolitis patients (ie, those who are clinically well and not requiring supplemental oxygen) has been assessed in quality improvement initiatives^8-10 and a randomized controlled trial.² These studies showed no increase in intensive care unit transfers, codes, or readmissions associated with reduced cSpO₂. Current national guidelines from the American Academy of Pediatrics⁵ and the Society of Hospital Medicine Choosing Wisely in Pediatric Hospital Medicine workgroup⁴ support limiting monitoring of children with bronchiolitis. Despite this, the practice of cSpO₂ in stable bronchiolitis patients off supplemental oxygen remains widespread.^11,12

Deimplementation, defined as reducing or stopping low-value or ineffective healthcare practices,^13,14 is a discrete focus area within implementation science. Deimplementation research involves the reduction of unnecessary and overused services for which there is potential for harm or no benefit.^15,16 In pediatrics, there are a number of potential targets for deimplementation,^4,17-20 including cSpO₂ for stable infants with bronchiolitis, but efforts to reduce low-value practices have met limited success to date. ^21,22

Implementation science offers rigorous methods for advancing the development and evaluation of strategies for deimplementation.²³ In particular, implementation science frameworks can facilitate our understanding of relevant contextual factors that may hinder or help efforts to deimplement low-value practices. To develop broadly applicable strategies to reduce monitoring overuse, it is important to understand the barriers, facilitators, and contextual factors (eg, clinical, political, interpersonal) that contribute to guideline-discordant cSpO₂ in hospitalized bronchiolitis patients. Further, the process by which one can develop a rigorous understanding of these factors and how they may impact deimplementation efforts could generalize to other scenarios in pediatrics where overuse remains an issue.

The goal of this study was to use semistructured interviews, informed by an established implementation science framework, specifically the Consolidated Framework for Implementation Research (CFIR),²⁴ to (1) identify barriers and facilitators to deimplementing unnecessary cSpO₂, and (2) develop strategies to deimplement cSpO₂ in a multicenter cohort of hospital-based clinician and administrative stakeholders.

METHODS

Study Setting

This multicenter qualitative study using semistructured interviews took place within the Eliminating Monitor Overuse (EMO) SpO₂ study. The EMO SpO₂ study established rates of cSpO₂ in bronchiolitis patients not receiving supplemental oxygen or not receiving room air flow at 56 hospitals across the United States and in Canada from December 1, 2018, through March 31, 2019.¹² The study identified hospital-level risk-adjusted cSpO₂ rates ranging from 6% to 82%. A description of the EMO SpO₂ study methods²⁵ and its findings¹² have been published elsewhere.

Participants

We approached EMO study site principal investigators at 12 hospitals: the two highest- and two lowest-use hospitals within three hospital types (ie, freestanding children’s hospitals, children’s hospitals within large general hospitals, and community hospitals). We collaborated with the participating site principal investigators (n = 12), who were primarily hospitalist physicians in leadership roles, to recruit a purposive sample of additional stakeholders including bedside nurses (n = 12), hospitalist physicians (n = 15), respiratory therapists (n = 9), and hospital administrators (n = 8) to participate in semistructured interviews. Interviews were conducted until we achieved thematic saturation within each stakeholder group and within the high and low performing strata (total 56 interviews). Participants were asked to self-report basic demographic information (see Appendix, interview guide) as required by the study funder and to allow us to comment on the representativeness of the participant group.

 

Procedure

The interview guide was informed by the CFIR, a comprehensive framework detailing contextual factors that require consideration when planning for the implementation of a health service intervention. Table 1 details the CFIR domains with study-related examples. The interview guide (Appendix) provided limited clinical context apart from the age, diagnosis, and oxygen requirement for the population of interest to promote a broad array of responses and to avoid anchoring on specific clinical scenarios. Interviews were conducted by master’s degree or doctoral-level research coordinators with qualitative interviewing experience and supervised by a medical anthropologist and qualitative methods expert (F.K.B.). Prior to engaging in audio recorded phone interviews, the interviewer explained the risks and benefits of participating. Participants were compensated $50. Audio recordings were transcribed, deidentified, and uploaded to NVivo 12 Plus (QSR International) for data management.

The Institutional Review Boards of Children’s Hospital of Philadelphia, Pennsylvania, and the University of Pennsylvania in Philadelphia determined that the study met eligibility criteria for IRB exemption.

Data Analysis

Using an integrated approach to codebook development,²⁶ a priori codes were developed using constructs from the CFIR. Additional codes were added by the research team following a close reading of the first five transcripts.^27,28 Each code was defined, including decision rules for its application. Two research coordinators independently coded each transcript. Using the intercoder reliability function within NVivo, the coders established strong interrater reliability accordance scores (k > .8) by double coding 20% of the transcripts. Data were stratified by sites with low and high use of cSpO₂ to examine differences in barriers and facilitators to deimplementation. Each code was subcoded, summarized, and examined for patterns within and across participating disciplines, which yielded themes related to barriers and facilitators. We conducted member checking and reviewed our conclusions with a multidisciplinary group of clinical stakeholders (n = 13) to validate our analyses.

RESULTS

Barriers and facilitators to deimplementation were identified in multiple domains of the CFIR: outer setting, inner setting, characteristics of the individuals, and intervention characteristics (Table 1). Participants also suggested strategies to facilitate deimplementation in response to some identified barriers. See Table 2 for participant demographics and Table 3 for illustrative participant quotations.

Barriers

Outer Setting: Clinician Perceptions of Parental Discomfort With Discontinuing Monitoring

Participants mentioned parental preferences as a barrier to discontinuing cSpO₂, noting that parents seem to take comfort in watching the numbers on the monitor screen and are reluctant to have it withdrawn. Clinicians noted that parents sometimes put the monitor back on their child after a clinician removed it or have expressed concern that their unmonitored child was not receiving the same level of care as other patients who were being monitored. In these scenarios, clinicians reported they have found it helpful to educate caregivers about when cSpO₂ is and is not appropriate.

Inner Setting: Unclear or Nonexistent Guideline to Discontinue cSpO₂

Guidelines to discontinue cSpO₂ reportedly did not exist at all institutions. If a guideline did exist, lack of clarity or conflicting guidelines about when to use oxygen presented a barrier. Participants suggested that a clear guideline or additional oversight to ensure all clinicians are informed of the procedure for discontinuing cSpO₂ may help prevent miscommunication. Participants noted that their electronic health record (EHR) order sets commonly included cSpO₂ orders and that removing that option would facilitate deimplementation.

 

Inner Setting: Difficulty Educating All Staff

Participants noted difficulty with incorporating education about discontinuing cSpO₂ to all clinicians, particularly to those who are nightshift only or to rotating staff or trainees. This created barriers for frequent re-education because these staff are not familiar with the policies and procedures of the unit, which is crucial to developing a culture that supports the deimplementation of cSpO₂. Participants suggested that recurring education about procedures for discontinuing cSpO₂ should target trainees, new nurses, and overnight nurses. This would help to ensure that the guideline is uniformly followed.

Inner Setting: Culture of High cSpO₂ Use

Participants from high-use sites discussed a culture driven by readily available monitoring features or an expectation that monitoring indicates higher-quality care. Participants from low-use sites discussed increased cSpO₂ driven by clinicians who were accustomed to caring for higher-acuity patients, for whom continuous monitoring is likely appropriate, and were simultaneously caring for stable bronchiolitis patients.

Some suggested that visual cues would be useful to clinicians to sustain awareness about a cSpO₂ deimplementation guideline. It was also suggested that audit and feedback techniques like posting unit deimplementation statistics and creating a competition among units by posting unit performance could facilitate deimplementation. Additionally, some noted that visual aids in common spaces would be useful to remind clinicians and to engage caregivers about discontinuing cSpO₂.

Characteristics of Individuals: Clinician Discomfort Discontinuing cSpO₂

One frequently cited barrier across participants is that cSpO₂ provides “peace of mind” to alert clinicians to patients with low oxygen saturations that might otherwise be missed. Participants identified that clinician discomfort with reducing cSpO₂ may be driven by inexperienced clinicians less familiar with the bronchiolitis disease process, such as trainees, new nurses, or rotating clinicians unaccustomed to pediatric care. Trainees and new nurses were perceived as being more likely to work at night when there are fewer clinicians to provide patient care. Additionally, participants perceived that night shift clinicians favored cSpO₂ because they could measure vital signs without waking patients and families.

Clinicians discussed that discontinuing cSpO₂ would require alternative methods for assessing patient status, particularly for night shift nurses. Participants suggested strategies including changes to pulse oximetry assessment procedures to include more frequent “spot checks,” incorporation of assessments during sleep events (eg, naps) to ensure the patient does not experience desaturations during sleep, and training nurses to become more comfortable with suctioning patients. Suggestions also included education on the typical features of transient oxygen desaturations in otherwise stable patients with bronchiolitis² to bolster clinical confidence for clinicians unfamiliar with caring for bronchiolitis patients. Participants perceived that education about appropriate vs inappropriate use may help to empower clinicians to employ cSpO₂ appropriately.

Facilitators

Outer Setting: Standards and Evidence From Research, Professional Organizations, and Leaders in the Field

Many participants expressed the importance of consistent guidelines that are advocated by thought leaders in the field, supported by robust evidence, and consistent with approaches at peer hospitals. The more authoritative support a guideline has, the more comfortable people are adopting it and taking it seriously. Additionally, consistent education about guidelines was desired. Participants noted that all clinicians should be receiving education related to the American Academy of Pediatrics (AAP) Bronchiolitis and Choosing Wisely^® guidelines, ranging from a one-time update to annually. Continual updates and re-education sessions for clinicians who shared evidence about how cSpO₂ deimplementation could improve the quality of patient care by shortening hospital length of stay and lowering cost were suggested strategies.

 

Inner Setting: Leadership

Participants noted that successful deimplementation depends upon the presence of a champion or educator who will be able to lead the institutional charge in making practice change. This is typically an individual who is trusted at the institution, experienced in their field, or already doing implementation work. This could be either a single individual (champion) or a team. The most commonly noted clinician roles to engage in a leadership role or team were physicians and nurses.

Participants noted that a change in related clinical care pathways or EHR order sets would require cooperation from multiple clinical disciplines, administrators, and information technology leaders and explained that messaging and education about the value of the change would facilitate buy-in from those clinicians.

Inner Setting: EHR Support for Guidelines

Participants often endorsed the use of an order set within the EHR that supports guidelines and includes reminders to decrease cSpO₂. These reminders could come up when supplemental oxygen is discontinued or occur regularly throughout the patient’s stay to prompt the clinician to consider discontinuing cSpO₂.

Intervention Characteristics/Inner Setting: Clear Bronchiolitis Guidelines

The presence of a well-articulated hospital policy that delineates the appropriate and inappropriate use of cSpO₂ in bronchiolitis was mentioned as another facilitator of deimplementation.

DISCUSSION

Results of this qualitative study of stakeholders across hospitals with high and low cSpO₂ use illustrated the complexities involved with deimplementation of cSpO₂ in pediatric patients hospitalized with bronchiolitis. We identified numerous barriers spanning the CFIR constructs, including unclear or absent guidelines for stopping cSpO₂, clinician knowledge and comfort with bronchiolitis disease features, and unit culture. This suggests that multicomponent strategies that target various domains and a variety of stakeholders are needed to deimplement cSpO₂ use for stable bronchiolitis patients. Participants also identified facilitators, including clear cSpO₂ guidelines, supportive leaders and champions, and EHR modifications, that provide insight into strategies that may help sites reduce their use of cSpO₂. Additionally, participants also provided concrete, actionable suggestions for ways to reduce unnecessary monitoring that will be useful in informing promising deimplementation strategies for subsequent trials.

The importance of having specific and well-known guidelines from trusted sources, such as the AAP, about cSpO₂ and bronchiolitis treatment that are thoughtfully integrated in the EHR came through in multiple themes of our analysis. Prior studies on the effect of guidelines on clinical practice have suggested that rigorously designed guidelines can positively impact practice.²⁹ Participants also noted that cSpO₂ guidelines should be authoritative and that knowledge of guideline adoption by peer institutions was a facilitator of adoption. Usability issues negatively impact clinicians’ ability to follow guidelines.³⁰ Further, prior studies have demonstrated that EHR integration of guidelines can change practice.^31-33 Based on our findings, incorporating clear guidelines into commonly used formats, such as EHR order sets, could be an important deimplementation tool for cSpO₂ in stable bronchiolitis patients.

Education about and awareness of cSpO₂ guidelines was described as an important facilitator for appropriate cSpO₂ use and was suggested as a potential deimplementation strategy. Participants noted that educational need may vary by stakeholder group. For example, education may facilitate obtaining buy-in from hospital leaders, which is necessary to support changes to the EHR. Education incorporating information on the typical features of bronchiolitis and examples of appropriate and inappropriate cSpO₂ use was suggested for clinical team members. The limitations of education as a stand-alone deimplementation strategy were also noted, and participants highlighted challenges such as time needed for education and the need for ongoing education for rotating trainees. Inner and outer setting barriers, such as a perceived “culture of high pulse oximetryuse” and patient and family expectations, could also make education less effective as a stand-alone strategy. That—coupled with evidence that education and training alone are generally insufficient for producing reliable, sustained behavior change^34,35—suggests that a multifaceted approach will be important.

Our respondents consider parental perceptions and preference in their practice, which provides nuance to recent studies suggesting that parents prefer continuous monitors when their child is hospitalized with bronchiolitis. Chi et al described the impact of a brief educational intervention on parental preferences for monitoring children hospitalized for bronchiolitis.³⁶ This work suggests that educational interventions aimed at families should be considered in future (de)implementation studies because they may indirectly impact clinician behavior. Future studies should directly assess parental discomfort with discontinuing monitoring. Participants highlighted the link between knowledge and confidence in caring for typical bronchiolitis patients and monitoring practice, perceiving that less experienced clinicians are more likely to rely on cSpO₂. Participants at high-use sites emphasized the expectation that monitoring should occur during hospitalizations. This reflection is particularly pertinent for bronchiolitis, a disease characterized by frequent, self-resolving desaturations even after hospital discharge.³ This may reinforce a perceived need to capture and react to these desaturation events even though they are expected in bronchiolitis and can occur in healthy infants.³⁷ Some participants suggested that continuous monitoring be replaced with “nap tests” (ie, assessment for desaturations during a nap prior to discharge); however, like cSpO₂ in stable infants with bronchiolitis, this is another low-value practice. Otherwise healthy infants with mild to moderate disease are unlikely to subsequently worsen after showing signs of clinical improvement.³⁸ Nap tests are likely to lead to infants who are clinically improving being placed unnecessarily back on oxygen in reaction to the transient desaturations. Participants’ perception about the importance of cSpO₂ in bronchiolitis management, despite evidence suggesting it is a low-value practice, underscores the importance of not simply telling clinicians to stop cSpO₂. Employing strategies that replace continuous monitoring with another acceptable and feasible alternative (eg, regular clinician assessments including intermittent pulse oximetry checks) should be considered when planning for deimplementation.³⁹

Previous studies indicate that continuous monitoring can affect clinician decision-making, independent of other factors,^6,40 despite limited evidence that continuous monitors improve patient outcomes.^1-7 Studies have demonstrated noticeable increase in admissions based purely on pulse oximetry values,⁴⁰ with no evidence that this type of admission changes outcomes for bronchiolitis patients.⁶ One previous, single-center study identified inexperience as a potential driver for monitor use,⁴¹ and studies in adult populations have suggested that clinicians overestimate the value that continuous monitoring contributes to patient care,^42,43 which promotes guideline-discordant use. Our study provides novel insight into the issue of monitoring in bronchiolitis. Our results suggest that there is a need to shift organizational cultures around monitoring (which likely vary based on a range of factors) and that educational strategies addressing typical disease course, especially desaturations, in bronchiolitis will be an essential component in any deimplementation effort.

This study is strengthened by its sample of diverse stakeholder groups from multiple US health systems. Additionally, we interviewed individuals at sites with high cSpO₂ rates and at sites with low rates, as well as from community hospitals, children’s hospitals within general hospitals, and freestanding children’s hospitals, which allows us to understand barriers high-use sites encounter and facilitators of lower cSpO₂ rates at low-use sites. We also employed an interview approach informed by an established implementation science framework. Nonetheless, several limitations exist. First, participants at low-use sites did not necessarily have direct experience with a previous deimplementation effort to reduce cSpO₂. Additionally, participants were predominantly White and female; more diverse perspectives would strengthen confidence in the generalizability of our findings. While thematic saturation was achieved within each stakeholder group and within the high- and low-use strata, we interviewed fewer administrators and respiratory therapists relative to other stakeholder groups. Nevertheless, our conclusions were validated by our interdisciplinary stakeholder panel. As noted by participants, family preferences may influence clinician practice, and parents were not interviewed for this study. The information gleaned from the present study will inform the development of strategies to deimplement unnecessary cSpO₂ in pediatric hospitals, which we aim to rigorously evaluate in a future trial.

 

CONCLUSION

We identified barriers and facilitators to deimplementation of cSpO₂ for stable patients with bronchiolitis across children’s hospitals with high and low utilization of cSpO₂. These themes map to multiple CFIR domains and, along with participant-suggested strategies, can directly inform an approach to cSpO₂ deimplementation in a range of inpatient settings. Based on these data, future deimplementation efforts should focus on clear protocols for use and discontinuation of cSpO₂, EHR changes, and regular bronchiolitis education for hospital staff that emphasizes reducing unnecessary cSpO₂ utilization.

ACKNOWLEDGMENTS

We acknowledge the NHLBI scientists who contributed their expertise to this project as part of the U01 Cooperative Agreement funding mechanism as federal employees conducting their official job duties: Lora Reineck, MD, MS, Karen Bienstock, MS, and Cheryl Boyce, PhD. We thank the Executive Council of the Pediatric Research in Inpatient Settings (PRIS) Network for their contributions to the early scientific development of this project. The Network assessed a Collaborative Support Fee for access to the hospitals and support of this project. We thank the PRIS Network collaborators for their major contributions to data collection measuring utilization to identify the hospitals we subsequently chose for this project. We thank Claire Bocage and the Mixed Methods Research Lab for major help in data management and data analysis.

Continuous pulse oximetry monitoring (cSpO₂) in children with bronchiolitis is associated with increased rates of hospital admission, longer lengths of stay, more frequent treatment with supplemental oxygen, alarm fatigue, and higher hospital cost. There is no evidence that it improves clinical outcomes.^1-7 The safety of reducing cSpO₂ for stable bronchiolitis patients (ie, those who are clinically well and not requiring supplemental oxygen) has been assessed in quality improvement initiatives^8-10 and a randomized controlled trial.² These studies showed no increase in intensive care unit transfers, codes, or readmissions associated with reduced cSpO₂. Current national guidelines from the American Academy of Pediatrics⁵ and the Society of Hospital Medicine Choosing Wisely in Pediatric Hospital Medicine workgroup⁴ support limiting monitoring of children with bronchiolitis. Despite this, the practice of cSpO₂ in stable bronchiolitis patients off supplemental oxygen remains widespread.^11,12

Deimplementation, defined as reducing or stopping low-value or ineffective healthcare practices,^13,14 is a discrete focus area within implementation science. Deimplementation research involves the reduction of unnecessary and overused services for which there is potential for harm or no benefit.^15,16 In pediatrics, there are a number of potential targets for deimplementation,^4,17-20 including cSpO₂ for stable infants with bronchiolitis, but efforts to reduce low-value practices have met limited success to date. ^21,22

Implementation science offers rigorous methods for advancing the development and evaluation of strategies for deimplementation.²³ In particular, implementation science frameworks can facilitate our understanding of relevant contextual factors that may hinder or help efforts to deimplement low-value practices. To develop broadly applicable strategies to reduce monitoring overuse, it is important to understand the barriers, facilitators, and contextual factors (eg, clinical, political, interpersonal) that contribute to guideline-discordant cSpO₂ in hospitalized bronchiolitis patients. Further, the process by which one can develop a rigorous understanding of these factors and how they may impact deimplementation efforts could generalize to other scenarios in pediatrics where overuse remains an issue.

The goal of this study was to use semistructured interviews, informed by an established implementation science framework, specifically the Consolidated Framework for Implementation Research (CFIR),²⁴ to (1) identify barriers and facilitators to deimplementing unnecessary cSpO₂, and (2) develop strategies to deimplement cSpO₂ in a multicenter cohort of hospital-based clinician and administrative stakeholders.

METHODS

Study Setting

This multicenter qualitative study using semistructured interviews took place within the Eliminating Monitor Overuse (EMO) SpO₂ study. The EMO SpO₂ study established rates of cSpO₂ in bronchiolitis patients not receiving supplemental oxygen or not receiving room air flow at 56 hospitals across the United States and in Canada from December 1, 2018, through March 31, 2019.¹² The study identified hospital-level risk-adjusted cSpO₂ rates ranging from 6% to 82%. A description of the EMO SpO₂ study methods²⁵ and its findings¹² have been published elsewhere.

Participants

We approached EMO study site principal investigators at 12 hospitals: the two highest- and two lowest-use hospitals within three hospital types (ie, freestanding children’s hospitals, children’s hospitals within large general hospitals, and community hospitals). We collaborated with the participating site principal investigators (n = 12), who were primarily hospitalist physicians in leadership roles, to recruit a purposive sample of additional stakeholders including bedside nurses (n = 12), hospitalist physicians (n = 15), respiratory therapists (n = 9), and hospital administrators (n = 8) to participate in semistructured interviews. Interviews were conducted until we achieved thematic saturation within each stakeholder group and within the high and low performing strata (total 56 interviews). Participants were asked to self-report basic demographic information (see Appendix, interview guide) as required by the study funder and to allow us to comment on the representativeness of the participant group.

 

Procedure

The interview guide was informed by the CFIR, a comprehensive framework detailing contextual factors that require consideration when planning for the implementation of a health service intervention. Table 1 details the CFIR domains with study-related examples. The interview guide (Appendix) provided limited clinical context apart from the age, diagnosis, and oxygen requirement for the population of interest to promote a broad array of responses and to avoid anchoring on specific clinical scenarios. Interviews were conducted by master’s degree or doctoral-level research coordinators with qualitative interviewing experience and supervised by a medical anthropologist and qualitative methods expert (F.K.B.). Prior to engaging in audio recorded phone interviews, the interviewer explained the risks and benefits of participating. Participants were compensated $50. Audio recordings were transcribed, deidentified, and uploaded to NVivo 12 Plus (QSR International) for data management.

The Institutional Review Boards of Children’s Hospital of Philadelphia, Pennsylvania, and the University of Pennsylvania in Philadelphia determined that the study met eligibility criteria for IRB exemption.

Data Analysis

Using an integrated approach to codebook development,²⁶ a priori codes were developed using constructs from the CFIR. Additional codes were added by the research team following a close reading of the first five transcripts.^27,28 Each code was defined, including decision rules for its application. Two research coordinators independently coded each transcript. Using the intercoder reliability function within NVivo, the coders established strong interrater reliability accordance scores (k > .8) by double coding 20% of the transcripts. Data were stratified by sites with low and high use of cSpO₂ to examine differences in barriers and facilitators to deimplementation. Each code was subcoded, summarized, and examined for patterns within and across participating disciplines, which yielded themes related to barriers and facilitators. We conducted member checking and reviewed our conclusions with a multidisciplinary group of clinical stakeholders (n = 13) to validate our analyses.

RESULTS

Barriers and facilitators to deimplementation were identified in multiple domains of the CFIR: outer setting, inner setting, characteristics of the individuals, and intervention characteristics (Table 1). Participants also suggested strategies to facilitate deimplementation in response to some identified barriers. See Table 2 for participant demographics and Table 3 for illustrative participant quotations.

Barriers

Outer Setting: Clinician Perceptions of Parental Discomfort With Discontinuing Monitoring

Participants mentioned parental preferences as a barrier to discontinuing cSpO₂, noting that parents seem to take comfort in watching the numbers on the monitor screen and are reluctant to have it withdrawn. Clinicians noted that parents sometimes put the monitor back on their child after a clinician removed it or have expressed concern that their unmonitored child was not receiving the same level of care as other patients who were being monitored. In these scenarios, clinicians reported they have found it helpful to educate caregivers about when cSpO₂ is and is not appropriate.

Inner Setting: Unclear or Nonexistent Guideline to Discontinue cSpO₂

Guidelines to discontinue cSpO₂ reportedly did not exist at all institutions. If a guideline did exist, lack of clarity or conflicting guidelines about when to use oxygen presented a barrier. Participants suggested that a clear guideline or additional oversight to ensure all clinicians are informed of the procedure for discontinuing cSpO₂ may help prevent miscommunication. Participants noted that their electronic health record (EHR) order sets commonly included cSpO₂ orders and that removing that option would facilitate deimplementation.

 

Inner Setting: Difficulty Educating All Staff

Participants noted difficulty with incorporating education about discontinuing cSpO₂ to all clinicians, particularly to those who are nightshift only or to rotating staff or trainees. This created barriers for frequent re-education because these staff are not familiar with the policies and procedures of the unit, which is crucial to developing a culture that supports the deimplementation of cSpO₂. Participants suggested that recurring education about procedures for discontinuing cSpO₂ should target trainees, new nurses, and overnight nurses. This would help to ensure that the guideline is uniformly followed.

Inner Setting: Culture of High cSpO₂ Use

Participants from high-use sites discussed a culture driven by readily available monitoring features or an expectation that monitoring indicates higher-quality care. Participants from low-use sites discussed increased cSpO₂ driven by clinicians who were accustomed to caring for higher-acuity patients, for whom continuous monitoring is likely appropriate, and were simultaneously caring for stable bronchiolitis patients.

Some suggested that visual cues would be useful to clinicians to sustain awareness about a cSpO₂ deimplementation guideline. It was also suggested that audit and feedback techniques like posting unit deimplementation statistics and creating a competition among units by posting unit performance could facilitate deimplementation. Additionally, some noted that visual aids in common spaces would be useful to remind clinicians and to engage caregivers about discontinuing cSpO₂.

Characteristics of Individuals: Clinician Discomfort Discontinuing cSpO₂

One frequently cited barrier across participants is that cSpO₂ provides “peace of mind” to alert clinicians to patients with low oxygen saturations that might otherwise be missed. Participants identified that clinician discomfort with reducing cSpO₂ may be driven by inexperienced clinicians less familiar with the bronchiolitis disease process, such as trainees, new nurses, or rotating clinicians unaccustomed to pediatric care. Trainees and new nurses were perceived as being more likely to work at night when there are fewer clinicians to provide patient care. Additionally, participants perceived that night shift clinicians favored cSpO₂ because they could measure vital signs without waking patients and families.

Clinicians discussed that discontinuing cSpO₂ would require alternative methods for assessing patient status, particularly for night shift nurses. Participants suggested strategies including changes to pulse oximetry assessment procedures to include more frequent “spot checks,” incorporation of assessments during sleep events (eg, naps) to ensure the patient does not experience desaturations during sleep, and training nurses to become more comfortable with suctioning patients. Suggestions also included education on the typical features of transient oxygen desaturations in otherwise stable patients with bronchiolitis² to bolster clinical confidence for clinicians unfamiliar with caring for bronchiolitis patients. Participants perceived that education about appropriate vs inappropriate use may help to empower clinicians to employ cSpO₂ appropriately.

Facilitators

Outer Setting: Standards and Evidence From Research, Professional Organizations, and Leaders in the Field

Many participants expressed the importance of consistent guidelines that are advocated by thought leaders in the field, supported by robust evidence, and consistent with approaches at peer hospitals. The more authoritative support a guideline has, the more comfortable people are adopting it and taking it seriously. Additionally, consistent education about guidelines was desired. Participants noted that all clinicians should be receiving education related to the American Academy of Pediatrics (AAP) Bronchiolitis and Choosing Wisely^® guidelines, ranging from a one-time update to annually. Continual updates and re-education sessions for clinicians who shared evidence about how cSpO₂ deimplementation could improve the quality of patient care by shortening hospital length of stay and lowering cost were suggested strategies.

 

Inner Setting: Leadership

Participants noted that successful deimplementation depends upon the presence of a champion or educator who will be able to lead the institutional charge in making practice change. This is typically an individual who is trusted at the institution, experienced in their field, or already doing implementation work. This could be either a single individual (champion) or a team. The most commonly noted clinician roles to engage in a leadership role or team were physicians and nurses.

Participants noted that a change in related clinical care pathways or EHR order sets would require cooperation from multiple clinical disciplines, administrators, and information technology leaders and explained that messaging and education about the value of the change would facilitate buy-in from those clinicians.

Inner Setting: EHR Support for Guidelines

Participants often endorsed the use of an order set within the EHR that supports guidelines and includes reminders to decrease cSpO₂. These reminders could come up when supplemental oxygen is discontinued or occur regularly throughout the patient’s stay to prompt the clinician to consider discontinuing cSpO₂.

Intervention Characteristics/Inner Setting: Clear Bronchiolitis Guidelines

The presence of a well-articulated hospital policy that delineates the appropriate and inappropriate use of cSpO₂ in bronchiolitis was mentioned as another facilitator of deimplementation.

DISCUSSION

Results of this qualitative study of stakeholders across hospitals with high and low cSpO₂ use illustrated the complexities involved with deimplementation of cSpO₂ in pediatric patients hospitalized with bronchiolitis. We identified numerous barriers spanning the CFIR constructs, including unclear or absent guidelines for stopping cSpO₂, clinician knowledge and comfort with bronchiolitis disease features, and unit culture. This suggests that multicomponent strategies that target various domains and a variety of stakeholders are needed to deimplement cSpO₂ use for stable bronchiolitis patients. Participants also identified facilitators, including clear cSpO₂ guidelines, supportive leaders and champions, and EHR modifications, that provide insight into strategies that may help sites reduce their use of cSpO₂. Additionally, participants also provided concrete, actionable suggestions for ways to reduce unnecessary monitoring that will be useful in informing promising deimplementation strategies for subsequent trials.

The importance of having specific and well-known guidelines from trusted sources, such as the AAP, about cSpO₂ and bronchiolitis treatment that are thoughtfully integrated in the EHR came through in multiple themes of our analysis. Prior studies on the effect of guidelines on clinical practice have suggested that rigorously designed guidelines can positively impact practice.²⁹ Participants also noted that cSpO₂ guidelines should be authoritative and that knowledge of guideline adoption by peer institutions was a facilitator of adoption. Usability issues negatively impact clinicians’ ability to follow guidelines.³⁰ Further, prior studies have demonstrated that EHR integration of guidelines can change practice.^31-33 Based on our findings, incorporating clear guidelines into commonly used formats, such as EHR order sets, could be an important deimplementation tool for cSpO₂ in stable bronchiolitis patients.

Education about and awareness of cSpO₂ guidelines was described as an important facilitator for appropriate cSpO₂ use and was suggested as a potential deimplementation strategy. Participants noted that educational need may vary by stakeholder group. For example, education may facilitate obtaining buy-in from hospital leaders, which is necessary to support changes to the EHR. Education incorporating information on the typical features of bronchiolitis and examples of appropriate and inappropriate cSpO₂ use was suggested for clinical team members. The limitations of education as a stand-alone deimplementation strategy were also noted, and participants highlighted challenges such as time needed for education and the need for ongoing education for rotating trainees. Inner and outer setting barriers, such as a perceived “culture of high pulse oximetryuse” and patient and family expectations, could also make education less effective as a stand-alone strategy. That—coupled with evidence that education and training alone are generally insufficient for producing reliable, sustained behavior change^34,35—suggests that a multifaceted approach will be important.

Our respondents consider parental perceptions and preference in their practice, which provides nuance to recent studies suggesting that parents prefer continuous monitors when their child is hospitalized with bronchiolitis. Chi et al described the impact of a brief educational intervention on parental preferences for monitoring children hospitalized for bronchiolitis.³⁶ This work suggests that educational interventions aimed at families should be considered in future (de)implementation studies because they may indirectly impact clinician behavior. Future studies should directly assess parental discomfort with discontinuing monitoring. Participants highlighted the link between knowledge and confidence in caring for typical bronchiolitis patients and monitoring practice, perceiving that less experienced clinicians are more likely to rely on cSpO₂. Participants at high-use sites emphasized the expectation that monitoring should occur during hospitalizations. This reflection is particularly pertinent for bronchiolitis, a disease characterized by frequent, self-resolving desaturations even after hospital discharge.³ This may reinforce a perceived need to capture and react to these desaturation events even though they are expected in bronchiolitis and can occur in healthy infants.³⁷ Some participants suggested that continuous monitoring be replaced with “nap tests” (ie, assessment for desaturations during a nap prior to discharge); however, like cSpO₂ in stable infants with bronchiolitis, this is another low-value practice. Otherwise healthy infants with mild to moderate disease are unlikely to subsequently worsen after showing signs of clinical improvement.³⁸ Nap tests are likely to lead to infants who are clinically improving being placed unnecessarily back on oxygen in reaction to the transient desaturations. Participants’ perception about the importance of cSpO₂ in bronchiolitis management, despite evidence suggesting it is a low-value practice, underscores the importance of not simply telling clinicians to stop cSpO₂. Employing strategies that replace continuous monitoring with another acceptable and feasible alternative (eg, regular clinician assessments including intermittent pulse oximetry checks) should be considered when planning for deimplementation.³⁹

Previous studies indicate that continuous monitoring can affect clinician decision-making, independent of other factors,^6,40 despite limited evidence that continuous monitors improve patient outcomes.^1-7 Studies have demonstrated noticeable increase in admissions based purely on pulse oximetry values,⁴⁰ with no evidence that this type of admission changes outcomes for bronchiolitis patients.⁶ One previous, single-center study identified inexperience as a potential driver for monitor use,⁴¹ and studies in adult populations have suggested that clinicians overestimate the value that continuous monitoring contributes to patient care,^42,43 which promotes guideline-discordant use. Our study provides novel insight into the issue of monitoring in bronchiolitis. Our results suggest that there is a need to shift organizational cultures around monitoring (which likely vary based on a range of factors) and that educational strategies addressing typical disease course, especially desaturations, in bronchiolitis will be an essential component in any deimplementation effort.

This study is strengthened by its sample of diverse stakeholder groups from multiple US health systems. Additionally, we interviewed individuals at sites with high cSpO₂ rates and at sites with low rates, as well as from community hospitals, children’s hospitals within general hospitals, and freestanding children’s hospitals, which allows us to understand barriers high-use sites encounter and facilitators of lower cSpO₂ rates at low-use sites. We also employed an interview approach informed by an established implementation science framework. Nonetheless, several limitations exist. First, participants at low-use sites did not necessarily have direct experience with a previous deimplementation effort to reduce cSpO₂. Additionally, participants were predominantly White and female; more diverse perspectives would strengthen confidence in the generalizability of our findings. While thematic saturation was achieved within each stakeholder group and within the high- and low-use strata, we interviewed fewer administrators and respiratory therapists relative to other stakeholder groups. Nevertheless, our conclusions were validated by our interdisciplinary stakeholder panel. As noted by participants, family preferences may influence clinician practice, and parents were not interviewed for this study. The information gleaned from the present study will inform the development of strategies to deimplement unnecessary cSpO₂ in pediatric hospitals, which we aim to rigorously evaluate in a future trial.

 

CONCLUSION

We identified barriers and facilitators to deimplementation of cSpO₂ for stable patients with bronchiolitis across children’s hospitals with high and low utilization of cSpO₂. These themes map to multiple CFIR domains and, along with participant-suggested strategies, can directly inform an approach to cSpO₂ deimplementation in a range of inpatient settings. Based on these data, future deimplementation efforts should focus on clear protocols for use and discontinuation of cSpO₂, EHR changes, and regular bronchiolitis education for hospital staff that emphasizes reducing unnecessary cSpO₂ utilization.

ACKNOWLEDGMENTS

We acknowledge the NHLBI scientists who contributed their expertise to this project as part of the U01 Cooperative Agreement funding mechanism as federal employees conducting their official job duties: Lora Reineck, MD, MS, Karen Bienstock, MS, and Cheryl Boyce, PhD. We thank the Executive Council of the Pediatric Research in Inpatient Settings (PRIS) Network for their contributions to the early scientific development of this project. The Network assessed a Collaborative Support Fee for access to the hospitals and support of this project. We thank the PRIS Network collaborators for their major contributions to data collection measuring utilization to identify the hospitals we subsequently chose for this project. We thank Claire Bocage and the Mixed Methods Research Lab for major help in data management and data analysis.

Transitions from the hospital to the ambulatory setting are high-risk periods for patients in terms of adverse events, poor clinical outcomes, and readmission. Processes of care during care transitions are suboptimal, including poor communication among inpatient providers, patients, and ambulatory providers^1,2; suboptimal communication of postdischarge plans of care to patients and their ability to carry out these plans³; medication discrepancies and nonadherence after discharge⁴; and lack of timely follow-up with ambulatory providers.⁵ Healthcare organizations continue to struggle with the question of which interventions to implement and how best to implement them.

Interventions to improve care transitions typically focus on readmission rates, but some studies have focused on postdischarge adverse events, defined as injuries in the 30 days after discharge caused by medical management rather than underlying disease processes.² These adverse events cause psychological distress, out-of-pocket expenses, decreases in functional status, and caregiver burden. An estimated 20% of hospitalized patients suffer a postdischarge adverse event.^1,2 Approximately two-thirds of these may be preventable or ameliorable.

The advent of Accountable Care Organizations (ACOs), defined as “groups of doctors, hospitals, and other health care providers who come together voluntarily to give coordinated high quality care to their patients,” creates an opportunity for improvements in patient safety during care transitions.⁶ Another opportunity has been the advent of Patient-Centered Medical Homes (PCMH), consisting of patient-oriented, comprehensive, team-based primary care enhanced by health information technology and population-based disease management tools.^7,8 In theory, a hospital-PCMH collaboration within an ACO can improve transitional interventions since optimal communication and collaboration are more likely when both inpatient and primary care providers (PCPs) share infrastructure and are similarly incentivized. The objectives of this study were to design and implement a collaborative hospital-PCMH care transitions intervention within an ACO and evaluate its effects.

 

METHODS

This study was a two-arm, single-blind (blinded outcomes assessor), stepped-wedge, multisite cluster-randomized clinical trial (NCT02130570) approved by the institutional review board of Partners HealthCare.

Study Design and Randomization

The study employed a “stepped-wedge” design, which is a cluster-randomized study design in which an intervention is sequentially rolled out to different groups at different, prespecified, randomly determined times.⁹ Each cluster (in this case, each primary care practice) served as its own control, while still allowing for adjustment for temporal trends. Originally, 18 practices participated, but one withdrew due to the low number of patients enrolled in the study, leaving 17 clusters and 16 sequences; see Figure 1 of Appendix 1 for a full description of the sample size and timeline for each cluster. Practices were not aware of this timeline until after recruitment.

Study Setting and Participants

Conducted within a large Pioneer ACO in Boston and funded by the Patient-Centered Outcomes Research Institute (PCORI), the Partners-PCORI Transitions Study was designed as a “real-world” quality improvement project. Potential participants were adult patients who were admitted to medical and surgical services of two large academic hospitals (Hospital A and Hospital B) affiliated with an ACO, who were likely to be discharged back to the community, and whose PCP belonged to a primary care practice that was affiliated with the ACO, agreed to participate, and were designated PCMHs or on their way to being designated by meeting certain criteria: electronic health record, patient portal, team-based care, practice redesign, care management, and identification of high-risk patients. See Study Protocol (Appendix 2) for detailed patient and primary care practice inclusion criteria.

Patient Enrollment

Study staff screened participants from a daily automated list of patients admitted the day before, using medical records to determine eligibility, which was then confirmed by the patient’s nurse. Exclusion criteria included likely discharge to a location other than home, being in police custody, lack of a telephone, being homeless, previous enrollment in the study, and being unable to communicate in English or Spanish. Allocation to study arm was concealed until the patient or proxy provided informed written consent. The research assistant administered questionnaires to all study subjects to assess potential confounders and functional status 1 month prior to admission (Medical Outcomes Study 12-Item Short Form Health Survey [SF-12]).¹⁰ Patients were recruited between March 2013 and October 2015.

Intervention

The intervention was based on a conceptual model of an ideal discharge¹¹ that we developed based on work by Naylor et al,¹² work by Coleman and Berenson,³ best practices in medication reconciliation and information transfer according to our own research,^13-15 the best examples of interventions to improve the discharge process,^12,16,17 and a systematic review of discharge interventions.¹⁸ Some of the factors necessary for an ideal care transition include complete, organized, and timely documentation of the patient’s hospital course and postdischarge plan; effective discharge planning; coordination of care among the patient’s providers; methods to ensure medication safety; advanced care planning in appropriate patients; and education and “coaching” of patients and their caregivers so they learn how to manage their conditions. The final multifaceted intervention addressed each component of the ideal discharge and included inpatient and outpatient components (Table 1 and Table 1 of Appendix 1).

 

Patient and Public Involvement in Research

As with all PCORI-funded studies, this study involved a patient-family advisory council (PFAC). Our PFAC included six recently hospitalized patients or caregivers of recently hospitalized patients. The PFAC participated in monthly meetings throughout the study period. They helped inform the research questions, including confirmation that the endpoints were patient centered, and provided valuable input for the design of the intervention and the patient-facing components of the data collection instruments. They also interviewed several patient participants in the study regarding their experiences with the intervention. Lastly, they helped develop plans for dissemination of study results to the public.¹⁹

We also formed a steering committee consisting of physician, nursing, pharmacy, information technology, and administrative leadership representing primary care, inpatient care, and transitional care at both hospitals and Partners Healthcare. PFAC members took turns participating in quarterly steering committee meetings.

Evolution of the Intervention and Implementation

The intervention was iteratively refined during the course of the study in response to input from the PFAC, steering committee, and members of the intervention team; cases of adverse events and readmissions from patients despite being in the intervention arm; exit interviews of patients who had recently completed the intervention; and informal feedback from inpatient and outpatient clinicians. For example, we learned that the more complicated a patient’s conditions are, the sooner the clinical team wanted them to be seen after discharge. However, these patients were also less likely to feel well enough to keep that appointment. Therefore, the timing of follow-up of appointments needed to be a negotiation among the inpatient team, the patient, any caregivers, and the outpatient provider. PFAC members also emphasized that patients wanted one person to trust and to be the “point person” during a complicated transition such as hospital discharge.

At the same time, the intervention components evolved because of factors outside our control (eg, resource limitations). In keeping with the real-world nature of the research, the aim was for the intervention to be internally supported because incentives were theoretically more aligned with improvement of care transitions under the ACO model. By design, the PCORI contract only paid for limited parts of the intervention, such as a nurse practitioner to act as the discharge advocate at one hospital, overtime costs of inpatient pharmacists, and project manager time to facilitate inpatient-outpatient provider communication. (See Table 1 of Appendix 1 for details about the modifications to the intervention.)

Lastly, in keeping with PCORI’s methodology standards for studies of complex interventions,²⁰ we strove to standardize the intervention by function across hospitals, units, and practices, while still allowing for local adaptation in the form. In other words, rather than specifying exactly how a task (eg, medication counseling) needed to be performed, the study design offered sites flexibility in how they implemented the task given their available personnel and institutional culture.

Intervention Fidelity

To determine the extent to which each patient in the intervention arm received each intervention component, a project manager unblinded to treatment arm reviewed the electronic medical record for documentation of each component implemented by providers (eg, inpatient pharmacists, outpatient nurses). Because each intervention component produced documentation, this provided an accurate assessment of intervention fidelity, ie, the extent to which the intervention was implemented as intended.

 

Outcome Assessment

Postdischarge Follow-up

Based on previous studies,^2,21 a trained research assistant attempted to contact all study subjects 30 days (±5 days) after discharge and administered a questionnaire to identify any new or worsening symptoms since discharge, any healthcare use since discharge, and functional status in the previous week. Follow-up questions used branching logic to determine the relationship of any new or worsening symptoms to medications or other aspects of medical management. Research assistants followed up any positive responses with directed medical record review for objective findings, diagnoses, treatments, and responses. If patients could not be reached after five attempts, the research assistant instead conducted a thorough review of the outpatient medical record alone for provider reports of any new or worsening symptoms noted during follow-up within the 30-day postdischarge period. Research assistants also reviewed laboratory test results in all patients for evidence of postdischarge renal failure, elevated liver function tests, or new/worsening anemia.

Hospital Readmissions

We measured nonelective hospital readmissions within 30 days of discharge using a combination of administrative data for hospitalizations within the ACO network plus patient report during the 30-day phone call for all other readmissions.²²

Adjudication of Outcomes

Adverse events and preventable adverse events: All cases of new or worsening symptoms or signs, along with all supporting documentation, were then presented to teams of two trained blinded physician adjudicators through application of methods established in previous studies.^4,21 Each of the two adjudicators independently reviewed the information, along with the medical record, and completed a standardized form to confirm or deny the presence of any adverse events (ie, patient injury due to medical management) and to classify the type of event (eg, adverse drug event, hospital-acquired infection, procedural complication, diagnostic or management error), the severity and duration of the event, and whether the event was preventable or ameliorable. The two adjudicators then met to resolve any differences in their findings and come to consensus.

Preventable readmissions: If patients were readmitted to either study hospital, we conducted an evaluation, based on previous studies,²³ to determine if and how the readmission could have been prevented including (a) a standardized patient and caregiver interview to identify possible problems with the transitions process and (b) an email questionnaire to the patient’s PCP and the inpatient teams who cared for the patient during the index admission and readmission regarding possible deficiencies with the transitions process. As with adverse event adjudications, two physician adjudicators worked independently to classify the preventability of the readmission and then met to come to consensus. Conflicts were resolved by a third adjudicator.

Analysis Plan

To evaluate the effects of the intervention on the primary outcome, the number of postdischarge adverse events per patient, we used multivariable Poisson regression, with study arm as the main predictor. A similar approach was used to evaluate the number of new or worsening postdischarge signs or symptoms and the number of preventable adverse events per patient. We used an intention-to-treat analysis: If a practice did not start the intervention when they were scheduled to, based on our randomization, we counted all patients in that practice admitted after that point as intervention patients. We adjusted for patient demographics, clinical characteristics, month, inpatient unit, and primary care practice as fixed effects. We clustered by PCP using general linear models. Intervention effects were expressed as both unadjusted and adjusted incidence rate ratios (IRRs). We also conducted a limited number of subgroup analyses, determined a priori, to determine whether the intervention was more effective in certain patient populations; we used interaction terms (intervention × subgroup) to determine the statistical significance of any effect modification.

 

To evaluate the effects of the intervention on nonelective readmissions and preventable readmissions, we used a similar approach, using multivariable logistic regression. Postdischarge functional status, adjusted for status prior to admission, was analyzed using multivariable linear regression and random effects by primary care practice. The general linear mixed model (GLIMMIX) procedure in the SAS 9.3 statistical package (SAS Institute) was used to carry out all analyses.

Power and Sample Size

We assumed a baseline rate of postdischarge adverse events of 0.30 per patient.²¹ We conservatively assumed an effect size of a change from 0.30 in the control group to 0.23 in the intervention group (a relative reduction of 22%, which was based on studies of preventability rates²³ and close to the minimum clinically important difference). Based on prior studies,^4,22 we assumed an intraclass correlation coefficient of 0.01 with an average cluster size of seven patients per PCP. Assuming a 10% loss to follow-up rate and an alpha of 0.05, we targeted a sample size of 1,800 patients to achieve 80% power, with one-third of the patients in the usual care arm and two-thirds in the intervention arm.

RESULTS

We enrolled 18 PCMH primary care practices to participate in the study, including 8 from Hospital A (out of 13 approached), 8 from Hospital B (out of 11), and 2 from other ACO practices (out of 9) (plus two pilot practices). Reasons for not participating included not having dedicated personnel to play the role of the responsible outpatient clinician, undergoing recent turn-over in practice leadership, and not having enough patients admitted to the two hospitals. One practice only enrolled 5 patients in the study and withdrew from participation, which left 17 practices.

Study Patients

We enrolled 1,679 patients (Figure 1). Reasons for nonenrollment included being unable to complete the screen prior to discharge, not meeting inclusion criteria or meeting exclusion criteria, being assigned to a pilot practice, and declining informed written consent. The baseline characteristics of enrolled patients are presented in Table 2. Differences between the two study arms were small. About 47% of the cohort was not reachable by phone after five attempts for the 30-day phone call, but only 69 (4.1%) were truly lost to follow-up because they were unreachable by phone and had no documentation in the electronic medical record in the 30-days after discharge.

Intervention Fidelity

The majority of patients did not receive most intervention components, even those components that were supposed to be delivered to all intervention patients (Table 3). A minority of patients were referred to visiting nurse services and to the home pharmacy program. However, 855 patients (87%) in the intervention arm received at least one intervention component.

Outcome Measures

The intervention was associated with a statistically significant reduction in several of the outcomes of interest, including the primary outcome, number of postdischarge adverse events (45% reduction), and new or worsening postdischarge signs or symptoms (22% reduction), as well as preventable postdischarge adverse events (58% reduction) (Table 4). There was a nonsignificant difference in functional status. There was no significant effect on total nonelective or on preventable readmission rates. When analyzed by type of adverse event, the intervention was associated with a reduction in adverse drug events and in procedural complications (Table 2 of Appendix 1). Of note, there was no significant difference in the proportion of patients with at least one adverse event whether the outcome was determined by phone call and medical record review (49%) or medical record review alone (51%) (P = .48).

In subgroup analyses, there was no evidence of effect modification by service, hospital, patient age, readmission risk, health literacy, or comorbidity score (Table 3 of Appendix 1). Table 4 of Appendix 1 provides examples of postdischarge adverse events seen in the usual care arm that might have been prevented in the intervention.

 

 

 

DISCUSSION

This intervention was associated with a reduction in postdischarge adverse events. The relative improvement in each outcome aligned with the hypothesized sensitivity to change: the smallest improvement was seen in new or worsening signs or symptoms, followed by postdischarge adverse events and then by preventable postdischarge adverse events. The intervention was not associated with a difference in readmissions. The lack of effect on hospital readmissions may have been caused by the low proportion of readmissions that are preventable, as well as low intervention fidelity and lack of resources to implement facets such as postdischarge coaching, an evidence-based intervention that was never adopted.^16,23 One lesson of this study is that it may be easier to reduce postdischarge injury (still an important outcome) than readmissions.

Putting this study in context, we should note that the literature on interventions to improve care transitions is mixed.¹⁸ While there are several reports of successful interventions, there are many reports of unsuccessful ones, often using similar components. Success is often the result of adequate resources and attention to myriad details regarding implementation.²⁴ The intervention in our study likely contributed to improvements in patient and caregiver engagement in the hospital, enhancements of communication between inpatient and outpatient clinicians, and implementation of pharmacist-led interventions to improve medication safety. Regarding the latter, several prior studies have shown the benefits of pharmacist interventions in decreasing postdischarge adverse drug events.^4,25,26 Therefore, even an intervention with incomplete intervention fidelity can reduce postdischarge adverse events, especially because adverse drug events make up the majority of adverse events.^1,2,21

Perhaps the biggest lesson we learned was regarding the limitations of the hospital-led ACO model to incentivize sufficient up-front investments in transitional care interventions. By design, we chose a real-world approach in which interventions were integrated with existing ACO efforts, which were paid for internally by the institution. As a result, many of the interventions had to be scaled back because of resource constraints. The ACO model theoretically incentivizes more integrated care, but this may not always be true in practice. Emerging evidence suggests that physician group–led ACOs are associated with lower costs and use compared with hospital-led ACOs, likely because of more aligned incentives in physician group–led ACOs to reduce use of inpatient care.^27,28

An unresolved question is whether the ideal implementation approach is to protect the time of existing clinical personnel to carry out transitional care tasks or to hire external personnel to do these tasks. We purposely spread the intervention over several clinician types to minimize the additional burden on any one of them, minimize additional costs, and play to each clinician’s expertise, but in retrospect, this may not have been the right approach. By providing additional personnel with dedicated time, interest, and training in care transitions, the intervention may be delivered with higher quantitative and qualitative fidelity, and it could create a single point of contact for patients, which was considered highly desirable by our PFAC.

This study has several limitations. A large proportion of patients (44%) were unavailable for postdischarge phone calls. However, we were able to perform medical record review for worsening signs (eg, lab abnormalities) and symptoms (as reported by patients’ providers) in the postdischarge period and adjudicate them for adverse events for all but 69 of these patients. Because all these patients had ACO-affiliated PCPs, we would expect most of their utilization to have been within the system and, therefore, to be present in the medical record. The proportion of patients with at least one adverse event did not vary by the method of follow-up, which suggests that this issue is an unlikely source of bias. Assessment of readmission was imperfect because we do not have statewide or national data. However, our combination of administrative data for Partners readmissions plus self-report for non-Partners readmission has been shown to be fairly complete in previous studies.²⁹ Adjudicators could not be fully blinded to intervention status due to the lack of blinding of admission date. We did not calculate a kappa value for interrater reliability of individual assessments of adverse events; rather, coming to consensus among the two adjudicators was part of the process. In only a handful of cases was a third adjudicator required. Lastly, this study was conducted at two academic medical centers and their affiliated primary care clinics, which potentially limits generalizability; however, the results are likely generalizable to other ACOs that include major academic medical centers.

 

CONCLUSION

In conclusion, in this real-world clinical trial, we designed, implemented, and iteratively refined a multifaceted intervention to improve care transitions within a hospital-based academic ACO. Evolution of the intervention components was the result of stakeholder input, experience with the intervention, and ACO resource constraints. The intervention reduced postdischarge adverse events. However, across the ACO network, intervention fidelity was low, and this may have contributed to the lack of effect on readmission rates. ACOs that implement interventions without hiring new personnel or protecting the time of existing personnel to conduct transitional tasks are likely to face the same challenges of low fidelity.

Acknowledgments

The authors would like to acknowledge the many people who worked on designing, implementing, and evaluating this intervention, including but not limited to: Natasha Isaac, Hilary Heyison, Jacqueline Minahan, Molly O’Reilly, Michelle Potter, Nailah Khoory, Maureen Fagan, David Bates, Laura Carr, Joseph Frolkis, Eric Weil, Jacqueline Somerville, Stephanie Ahmed, Marcy Bergeron, Jessica Smith, and Jane Millett. We would also like to thank the members of our Patient-Family Advisory Council: Maureen Fagan, Karen Spikes, Margie Hodges, Win Hodges, Aureldon Henderson, Dena Salzberg, and Kay Bander.

Transitions from the hospital to the ambulatory setting are high-risk periods for patients in terms of adverse events, poor clinical outcomes, and readmission. Processes of care during care transitions are suboptimal, including poor communication among inpatient providers, patients, and ambulatory providers^1,2; suboptimal communication of postdischarge plans of care to patients and their ability to carry out these plans³; medication discrepancies and nonadherence after discharge⁴; and lack of timely follow-up with ambulatory providers.⁵ Healthcare organizations continue to struggle with the question of which interventions to implement and how best to implement them.

Interventions to improve care transitions typically focus on readmission rates, but some studies have focused on postdischarge adverse events, defined as injuries in the 30 days after discharge caused by medical management rather than underlying disease processes.² These adverse events cause psychological distress, out-of-pocket expenses, decreases in functional status, and caregiver burden. An estimated 20% of hospitalized patients suffer a postdischarge adverse event.^1,2 Approximately two-thirds of these may be preventable or ameliorable.

The advent of Accountable Care Organizations (ACOs), defined as “groups of doctors, hospitals, and other health care providers who come together voluntarily to give coordinated high quality care to their patients,” creates an opportunity for improvements in patient safety during care transitions.⁶ Another opportunity has been the advent of Patient-Centered Medical Homes (PCMH), consisting of patient-oriented, comprehensive, team-based primary care enhanced by health information technology and population-based disease management tools.^7,8 In theory, a hospital-PCMH collaboration within an ACO can improve transitional interventions since optimal communication and collaboration are more likely when both inpatient and primary care providers (PCPs) share infrastructure and are similarly incentivized. The objectives of this study were to design and implement a collaborative hospital-PCMH care transitions intervention within an ACO and evaluate its effects.

 

METHODS

This study was a two-arm, single-blind (blinded outcomes assessor), stepped-wedge, multisite cluster-randomized clinical trial (NCT02130570) approved by the institutional review board of Partners HealthCare.

Study Design and Randomization

The study employed a “stepped-wedge” design, which is a cluster-randomized study design in which an intervention is sequentially rolled out to different groups at different, prespecified, randomly determined times.⁹ Each cluster (in this case, each primary care practice) served as its own control, while still allowing for adjustment for temporal trends. Originally, 18 practices participated, but one withdrew due to the low number of patients enrolled in the study, leaving 17 clusters and 16 sequences; see Figure 1 of Appendix 1 for a full description of the sample size and timeline for each cluster. Practices were not aware of this timeline until after recruitment.

Study Setting and Participants

Conducted within a large Pioneer ACO in Boston and funded by the Patient-Centered Outcomes Research Institute (PCORI), the Partners-PCORI Transitions Study was designed as a “real-world” quality improvement project. Potential participants were adult patients who were admitted to medical and surgical services of two large academic hospitals (Hospital A and Hospital B) affiliated with an ACO, who were likely to be discharged back to the community, and whose PCP belonged to a primary care practice that was affiliated with the ACO, agreed to participate, and were designated PCMHs or on their way to being designated by meeting certain criteria: electronic health record, patient portal, team-based care, practice redesign, care management, and identification of high-risk patients. See Study Protocol (Appendix 2) for detailed patient and primary care practice inclusion criteria.

Patient Enrollment

Study staff screened participants from a daily automated list of patients admitted the day before, using medical records to determine eligibility, which was then confirmed by the patient’s nurse. Exclusion criteria included likely discharge to a location other than home, being in police custody, lack of a telephone, being homeless, previous enrollment in the study, and being unable to communicate in English or Spanish. Allocation to study arm was concealed until the patient or proxy provided informed written consent. The research assistant administered questionnaires to all study subjects to assess potential confounders and functional status 1 month prior to admission (Medical Outcomes Study 12-Item Short Form Health Survey [SF-12]).¹⁰ Patients were recruited between March 2013 and October 2015.

Intervention

The intervention was based on a conceptual model of an ideal discharge¹¹ that we developed based on work by Naylor et al,¹² work by Coleman and Berenson,³ best practices in medication reconciliation and information transfer according to our own research,^13-15 the best examples of interventions to improve the discharge process,^12,16,17 and a systematic review of discharge interventions.¹⁸ Some of the factors necessary for an ideal care transition include complete, organized, and timely documentation of the patient’s hospital course and postdischarge plan; effective discharge planning; coordination of care among the patient’s providers; methods to ensure medication safety; advanced care planning in appropriate patients; and education and “coaching” of patients and their caregivers so they learn how to manage their conditions. The final multifaceted intervention addressed each component of the ideal discharge and included inpatient and outpatient components (Table 1 and Table 1 of Appendix 1).

 

Patient and Public Involvement in Research

As with all PCORI-funded studies, this study involved a patient-family advisory council (PFAC). Our PFAC included six recently hospitalized patients or caregivers of recently hospitalized patients. The PFAC participated in monthly meetings throughout the study period. They helped inform the research questions, including confirmation that the endpoints were patient centered, and provided valuable input for the design of the intervention and the patient-facing components of the data collection instruments. They also interviewed several patient participants in the study regarding their experiences with the intervention. Lastly, they helped develop plans for dissemination of study results to the public.¹⁹

We also formed a steering committee consisting of physician, nursing, pharmacy, information technology, and administrative leadership representing primary care, inpatient care, and transitional care at both hospitals and Partners Healthcare. PFAC members took turns participating in quarterly steering committee meetings.

Evolution of the Intervention and Implementation

The intervention was iteratively refined during the course of the study in response to input from the PFAC, steering committee, and members of the intervention team; cases of adverse events and readmissions from patients despite being in the intervention arm; exit interviews of patients who had recently completed the intervention; and informal feedback from inpatient and outpatient clinicians. For example, we learned that the more complicated a patient’s conditions are, the sooner the clinical team wanted them to be seen after discharge. However, these patients were also less likely to feel well enough to keep that appointment. Therefore, the timing of follow-up of appointments needed to be a negotiation among the inpatient team, the patient, any caregivers, and the outpatient provider. PFAC members also emphasized that patients wanted one person to trust and to be the “point person” during a complicated transition such as hospital discharge.

At the same time, the intervention components evolved because of factors outside our control (eg, resource limitations). In keeping with the real-world nature of the research, the aim was for the intervention to be internally supported because incentives were theoretically more aligned with improvement of care transitions under the ACO model. By design, the PCORI contract only paid for limited parts of the intervention, such as a nurse practitioner to act as the discharge advocate at one hospital, overtime costs of inpatient pharmacists, and project manager time to facilitate inpatient-outpatient provider communication. (See Table 1 of Appendix 1 for details about the modifications to the intervention.)

Lastly, in keeping with PCORI’s methodology standards for studies of complex interventions,²⁰ we strove to standardize the intervention by function across hospitals, units, and practices, while still allowing for local adaptation in the form. In other words, rather than specifying exactly how a task (eg, medication counseling) needed to be performed, the study design offered sites flexibility in how they implemented the task given their available personnel and institutional culture.

Intervention Fidelity

To determine the extent to which each patient in the intervention arm received each intervention component, a project manager unblinded to treatment arm reviewed the electronic medical record for documentation of each component implemented by providers (eg, inpatient pharmacists, outpatient nurses). Because each intervention component produced documentation, this provided an accurate assessment of intervention fidelity, ie, the extent to which the intervention was implemented as intended.

 

Outcome Assessment

Postdischarge Follow-up

Based on previous studies,^2,21 a trained research assistant attempted to contact all study subjects 30 days (±5 days) after discharge and administered a questionnaire to identify any new or worsening symptoms since discharge, any healthcare use since discharge, and functional status in the previous week. Follow-up questions used branching logic to determine the relationship of any new or worsening symptoms to medications or other aspects of medical management. Research assistants followed up any positive responses with directed medical record review for objective findings, diagnoses, treatments, and responses. If patients could not be reached after five attempts, the research assistant instead conducted a thorough review of the outpatient medical record alone for provider reports of any new or worsening symptoms noted during follow-up within the 30-day postdischarge period. Research assistants also reviewed laboratory test results in all patients for evidence of postdischarge renal failure, elevated liver function tests, or new/worsening anemia.

Hospital Readmissions

We measured nonelective hospital readmissions within 30 days of discharge using a combination of administrative data for hospitalizations within the ACO network plus patient report during the 30-day phone call for all other readmissions.²²

Adjudication of Outcomes

Adverse events and preventable adverse events: All cases of new or worsening symptoms or signs, along with all supporting documentation, were then presented to teams of two trained blinded physician adjudicators through application of methods established in previous studies.^4,21 Each of the two adjudicators independently reviewed the information, along with the medical record, and completed a standardized form to confirm or deny the presence of any adverse events (ie, patient injury due to medical management) and to classify the type of event (eg, adverse drug event, hospital-acquired infection, procedural complication, diagnostic or management error), the severity and duration of the event, and whether the event was preventable or ameliorable. The two adjudicators then met to resolve any differences in their findings and come to consensus.

Preventable readmissions: If patients were readmitted to either study hospital, we conducted an evaluation, based on previous studies,²³ to determine if and how the readmission could have been prevented including (a) a standardized patient and caregiver interview to identify possible problems with the transitions process and (b) an email questionnaire to the patient’s PCP and the inpatient teams who cared for the patient during the index admission and readmission regarding possible deficiencies with the transitions process. As with adverse event adjudications, two physician adjudicators worked independently to classify the preventability of the readmission and then met to come to consensus. Conflicts were resolved by a third adjudicator.

Analysis Plan

To evaluate the effects of the intervention on the primary outcome, the number of postdischarge adverse events per patient, we used multivariable Poisson regression, with study arm as the main predictor. A similar approach was used to evaluate the number of new or worsening postdischarge signs or symptoms and the number of preventable adverse events per patient. We used an intention-to-treat analysis: If a practice did not start the intervention when they were scheduled to, based on our randomization, we counted all patients in that practice admitted after that point as intervention patients. We adjusted for patient demographics, clinical characteristics, month, inpatient unit, and primary care practice as fixed effects. We clustered by PCP using general linear models. Intervention effects were expressed as both unadjusted and adjusted incidence rate ratios (IRRs). We also conducted a limited number of subgroup analyses, determined a priori, to determine whether the intervention was more effective in certain patient populations; we used interaction terms (intervention × subgroup) to determine the statistical significance of any effect modification.

 

To evaluate the effects of the intervention on nonelective readmissions and preventable readmissions, we used a similar approach, using multivariable logistic regression. Postdischarge functional status, adjusted for status prior to admission, was analyzed using multivariable linear regression and random effects by primary care practice. The general linear mixed model (GLIMMIX) procedure in the SAS 9.3 statistical package (SAS Institute) was used to carry out all analyses.

Power and Sample Size

We assumed a baseline rate of postdischarge adverse events of 0.30 per patient.²¹ We conservatively assumed an effect size of a change from 0.30 in the control group to 0.23 in the intervention group (a relative reduction of 22%, which was based on studies of preventability rates²³ and close to the minimum clinically important difference). Based on prior studies,^4,22 we assumed an intraclass correlation coefficient of 0.01 with an average cluster size of seven patients per PCP. Assuming a 10% loss to follow-up rate and an alpha of 0.05, we targeted a sample size of 1,800 patients to achieve 80% power, with one-third of the patients in the usual care arm and two-thirds in the intervention arm.

RESULTS

We enrolled 18 PCMH primary care practices to participate in the study, including 8 from Hospital A (out of 13 approached), 8 from Hospital B (out of 11), and 2 from other ACO practices (out of 9) (plus two pilot practices). Reasons for not participating included not having dedicated personnel to play the role of the responsible outpatient clinician, undergoing recent turn-over in practice leadership, and not having enough patients admitted to the two hospitals. One practice only enrolled 5 patients in the study and withdrew from participation, which left 17 practices.

Study Patients

We enrolled 1,679 patients (Figure 1). Reasons for nonenrollment included being unable to complete the screen prior to discharge, not meeting inclusion criteria or meeting exclusion criteria, being assigned to a pilot practice, and declining informed written consent. The baseline characteristics of enrolled patients are presented in Table 2. Differences between the two study arms were small. About 47% of the cohort was not reachable by phone after five attempts for the 30-day phone call, but only 69 (4.1%) were truly lost to follow-up because they were unreachable by phone and had no documentation in the electronic medical record in the 30-days after discharge.

Intervention Fidelity

The majority of patients did not receive most intervention components, even those components that were supposed to be delivered to all intervention patients (Table 3). A minority of patients were referred to visiting nurse services and to the home pharmacy program. However, 855 patients (87%) in the intervention arm received at least one intervention component.

Outcome Measures

The intervention was associated with a statistically significant reduction in several of the outcomes of interest, including the primary outcome, number of postdischarge adverse events (45% reduction), and new or worsening postdischarge signs or symptoms (22% reduction), as well as preventable postdischarge adverse events (58% reduction) (Table 4). There was a nonsignificant difference in functional status. There was no significant effect on total nonelective or on preventable readmission rates. When analyzed by type of adverse event, the intervention was associated with a reduction in adverse drug events and in procedural complications (Table 2 of Appendix 1). Of note, there was no significant difference in the proportion of patients with at least one adverse event whether the outcome was determined by phone call and medical record review (49%) or medical record review alone (51%) (P = .48).

In subgroup analyses, there was no evidence of effect modification by service, hospital, patient age, readmission risk, health literacy, or comorbidity score (Table 3 of Appendix 1). Table 4 of Appendix 1 provides examples of postdischarge adverse events seen in the usual care arm that might have been prevented in the intervention.

 

 

 

DISCUSSION

This intervention was associated with a reduction in postdischarge adverse events. The relative improvement in each outcome aligned with the hypothesized sensitivity to change: the smallest improvement was seen in new or worsening signs or symptoms, followed by postdischarge adverse events and then by preventable postdischarge adverse events. The intervention was not associated with a difference in readmissions. The lack of effect on hospital readmissions may have been caused by the low proportion of readmissions that are preventable, as well as low intervention fidelity and lack of resources to implement facets such as postdischarge coaching, an evidence-based intervention that was never adopted.^16,23 One lesson of this study is that it may be easier to reduce postdischarge injury (still an important outcome) than readmissions.

Putting this study in context, we should note that the literature on interventions to improve care transitions is mixed.¹⁸ While there are several reports of successful interventions, there are many reports of unsuccessful ones, often using similar components. Success is often the result of adequate resources and attention to myriad details regarding implementation.²⁴ The intervention in our study likely contributed to improvements in patient and caregiver engagement in the hospital, enhancements of communication between inpatient and outpatient clinicians, and implementation of pharmacist-led interventions to improve medication safety. Regarding the latter, several prior studies have shown the benefits of pharmacist interventions in decreasing postdischarge adverse drug events.^4,25,26 Therefore, even an intervention with incomplete intervention fidelity can reduce postdischarge adverse events, especially because adverse drug events make up the majority of adverse events.^1,2,21

Perhaps the biggest lesson we learned was regarding the limitations of the hospital-led ACO model to incentivize sufficient up-front investments in transitional care interventions. By design, we chose a real-world approach in which interventions were integrated with existing ACO efforts, which were paid for internally by the institution. As a result, many of the interventions had to be scaled back because of resource constraints. The ACO model theoretically incentivizes more integrated care, but this may not always be true in practice. Emerging evidence suggests that physician group–led ACOs are associated with lower costs and use compared with hospital-led ACOs, likely because of more aligned incentives in physician group–led ACOs to reduce use of inpatient care.^27,28

An unresolved question is whether the ideal implementation approach is to protect the time of existing clinical personnel to carry out transitional care tasks or to hire external personnel to do these tasks. We purposely spread the intervention over several clinician types to minimize the additional burden on any one of them, minimize additional costs, and play to each clinician’s expertise, but in retrospect, this may not have been the right approach. By providing additional personnel with dedicated time, interest, and training in care transitions, the intervention may be delivered with higher quantitative and qualitative fidelity, and it could create a single point of contact for patients, which was considered highly desirable by our PFAC.

This study has several limitations. A large proportion of patients (44%) were unavailable for postdischarge phone calls. However, we were able to perform medical record review for worsening signs (eg, lab abnormalities) and symptoms (as reported by patients’ providers) in the postdischarge period and adjudicate them for adverse events for all but 69 of these patients. Because all these patients had ACO-affiliated PCPs, we would expect most of their utilization to have been within the system and, therefore, to be present in the medical record. The proportion of patients with at least one adverse event did not vary by the method of follow-up, which suggests that this issue is an unlikely source of bias. Assessment of readmission was imperfect because we do not have statewide or national data. However, our combination of administrative data for Partners readmissions plus self-report for non-Partners readmission has been shown to be fairly complete in previous studies.²⁹ Adjudicators could not be fully blinded to intervention status due to the lack of blinding of admission date. We did not calculate a kappa value for interrater reliability of individual assessments of adverse events; rather, coming to consensus among the two adjudicators was part of the process. In only a handful of cases was a third adjudicator required. Lastly, this study was conducted at two academic medical centers and their affiliated primary care clinics, which potentially limits generalizability; however, the results are likely generalizable to other ACOs that include major academic medical centers.

 

CONCLUSION

In conclusion, in this real-world clinical trial, we designed, implemented, and iteratively refined a multifaceted intervention to improve care transitions within a hospital-based academic ACO. Evolution of the intervention components was the result of stakeholder input, experience with the intervention, and ACO resource constraints. The intervention reduced postdischarge adverse events. However, across the ACO network, intervention fidelity was low, and this may have contributed to the lack of effect on readmission rates. ACOs that implement interventions without hiring new personnel or protecting the time of existing personnel to conduct transitional tasks are likely to face the same challenges of low fidelity.

Acknowledgments

The authors would like to acknowledge the many people who worked on designing, implementing, and evaluating this intervention, including but not limited to: Natasha Isaac, Hilary Heyison, Jacqueline Minahan, Molly O’Reilly, Michelle Potter, Nailah Khoory, Maureen Fagan, David Bates, Laura Carr, Joseph Frolkis, Eric Weil, Jacqueline Somerville, Stephanie Ahmed, Marcy Bergeron, Jessica Smith, and Jane Millett. We would also like to thank the members of our Patient-Family Advisory Council: Maureen Fagan, Karen Spikes, Margie Hodges, Win Hodges, Aureldon Henderson, Dena Salzberg, and Kay Bander.

Transitions from the hospital to the ambulatory setting are high-risk periods for patients in terms of adverse events, poor clinical outcomes, and readmission. Processes of care during care transitions are suboptimal, including poor communication among inpatient providers, patients, and ambulatory providers^1,2; suboptimal communication of postdischarge plans of care to patients and their ability to carry out these plans³; medication discrepancies and nonadherence after discharge⁴; and lack of timely follow-up with ambulatory providers.⁵ Healthcare organizations continue to struggle with the question of which interventions to implement and how best to implement them.

Interventions to improve care transitions typically focus on readmission rates, but some studies have focused on postdischarge adverse events, defined as injuries in the 30 days after discharge caused by medical management rather than underlying disease processes.² These adverse events cause psychological distress, out-of-pocket expenses, decreases in functional status, and caregiver burden. An estimated 20% of hospitalized patients suffer a postdischarge adverse event.^1,2 Approximately two-thirds of these may be preventable or ameliorable.

The advent of Accountable Care Organizations (ACOs), defined as “groups of doctors, hospitals, and other health care providers who come together voluntarily to give coordinated high quality care to their patients,” creates an opportunity for improvements in patient safety during care transitions.⁶ Another opportunity has been the advent of Patient-Centered Medical Homes (PCMH), consisting of patient-oriented, comprehensive, team-based primary care enhanced by health information technology and population-based disease management tools.^7,8 In theory, a hospital-PCMH collaboration within an ACO can improve transitional interventions since optimal communication and collaboration are more likely when both inpatient and primary care providers (PCPs) share infrastructure and are similarly incentivized. The objectives of this study were to design and implement a collaborative hospital-PCMH care transitions intervention within an ACO and evaluate its effects.

 

METHODS

This study was a two-arm, single-blind (blinded outcomes assessor), stepped-wedge, multisite cluster-randomized clinical trial (NCT02130570) approved by the institutional review board of Partners HealthCare.

Study Design and Randomization

The study employed a “stepped-wedge” design, which is a cluster-randomized study design in which an intervention is sequentially rolled out to different groups at different, prespecified, randomly determined times.⁹ Each cluster (in this case, each primary care practice) served as its own control, while still allowing for adjustment for temporal trends. Originally, 18 practices participated, but one withdrew due to the low number of patients enrolled in the study, leaving 17 clusters and 16 sequences; see Figure 1 of Appendix 1 for a full description of the sample size and timeline for each cluster. Practices were not aware of this timeline until after recruitment.

Study Setting and Participants

Conducted within a large Pioneer ACO in Boston and funded by the Patient-Centered Outcomes Research Institute (PCORI), the Partners-PCORI Transitions Study was designed as a “real-world” quality improvement project. Potential participants were adult patients who were admitted to medical and surgical services of two large academic hospitals (Hospital A and Hospital B) affiliated with an ACO, who were likely to be discharged back to the community, and whose PCP belonged to a primary care practice that was affiliated with the ACO, agreed to participate, and were designated PCMHs or on their way to being designated by meeting certain criteria: electronic health record, patient portal, team-based care, practice redesign, care management, and identification of high-risk patients. See Study Protocol (Appendix 2) for detailed patient and primary care practice inclusion criteria.

Patient Enrollment

Study staff screened participants from a daily automated list of patients admitted the day before, using medical records to determine eligibility, which was then confirmed by the patient’s nurse. Exclusion criteria included likely discharge to a location other than home, being in police custody, lack of a telephone, being homeless, previous enrollment in the study, and being unable to communicate in English or Spanish. Allocation to study arm was concealed until the patient or proxy provided informed written consent. The research assistant administered questionnaires to all study subjects to assess potential confounders and functional status 1 month prior to admission (Medical Outcomes Study 12-Item Short Form Health Survey [SF-12]).¹⁰ Patients were recruited between March 2013 and October 2015.

Intervention

The intervention was based on a conceptual model of an ideal discharge¹¹ that we developed based on work by Naylor et al,¹² work by Coleman and Berenson,³ best practices in medication reconciliation and information transfer according to our own research,^13-15 the best examples of interventions to improve the discharge process,^12,16,17 and a systematic review of discharge interventions.¹⁸ Some of the factors necessary for an ideal care transition include complete, organized, and timely documentation of the patient’s hospital course and postdischarge plan; effective discharge planning; coordination of care among the patient’s providers; methods to ensure medication safety; advanced care planning in appropriate patients; and education and “coaching” of patients and their caregivers so they learn how to manage their conditions. The final multifaceted intervention addressed each component of the ideal discharge and included inpatient and outpatient components (Table 1 and Table 1 of Appendix 1).

 

Patient and Public Involvement in Research

As with all PCORI-funded studies, this study involved a patient-family advisory council (PFAC). Our PFAC included six recently hospitalized patients or caregivers of recently hospitalized patients. The PFAC participated in monthly meetings throughout the study period. They helped inform the research questions, including confirmation that the endpoints were patient centered, and provided valuable input for the design of the intervention and the patient-facing components of the data collection instruments. They also interviewed several patient participants in the study regarding their experiences with the intervention. Lastly, they helped develop plans for dissemination of study results to the public.¹⁹

We also formed a steering committee consisting of physician, nursing, pharmacy, information technology, and administrative leadership representing primary care, inpatient care, and transitional care at both hospitals and Partners Healthcare. PFAC members took turns participating in quarterly steering committee meetings.

Evolution of the Intervention and Implementation

The intervention was iteratively refined during the course of the study in response to input from the PFAC, steering committee, and members of the intervention team; cases of adverse events and readmissions from patients despite being in the intervention arm; exit interviews of patients who had recently completed the intervention; and informal feedback from inpatient and outpatient clinicians. For example, we learned that the more complicated a patient’s conditions are, the sooner the clinical team wanted them to be seen after discharge. However, these patients were also less likely to feel well enough to keep that appointment. Therefore, the timing of follow-up of appointments needed to be a negotiation among the inpatient team, the patient, any caregivers, and the outpatient provider. PFAC members also emphasized that patients wanted one person to trust and to be the “point person” during a complicated transition such as hospital discharge.

At the same time, the intervention components evolved because of factors outside our control (eg, resource limitations). In keeping with the real-world nature of the research, the aim was for the intervention to be internally supported because incentives were theoretically more aligned with improvement of care transitions under the ACO model. By design, the PCORI contract only paid for limited parts of the intervention, such as a nurse practitioner to act as the discharge advocate at one hospital, overtime costs of inpatient pharmacists, and project manager time to facilitate inpatient-outpatient provider communication. (See Table 1 of Appendix 1 for details about the modifications to the intervention.)

Lastly, in keeping with PCORI’s methodology standards for studies of complex interventions,²⁰ we strove to standardize the intervention by function across hospitals, units, and practices, while still allowing for local adaptation in the form. In other words, rather than specifying exactly how a task (eg, medication counseling) needed to be performed, the study design offered sites flexibility in how they implemented the task given their available personnel and institutional culture.

Intervention Fidelity

To determine the extent to which each patient in the intervention arm received each intervention component, a project manager unblinded to treatment arm reviewed the electronic medical record for documentation of each component implemented by providers (eg, inpatient pharmacists, outpatient nurses). Because each intervention component produced documentation, this provided an accurate assessment of intervention fidelity, ie, the extent to which the intervention was implemented as intended.

 

Outcome Assessment

Postdischarge Follow-up

Based on previous studies,^2,21 a trained research assistant attempted to contact all study subjects 30 days (±5 days) after discharge and administered a questionnaire to identify any new or worsening symptoms since discharge, any healthcare use since discharge, and functional status in the previous week. Follow-up questions used branching logic to determine the relationship of any new or worsening symptoms to medications or other aspects of medical management. Research assistants followed up any positive responses with directed medical record review for objective findings, diagnoses, treatments, and responses. If patients could not be reached after five attempts, the research assistant instead conducted a thorough review of the outpatient medical record alone for provider reports of any new or worsening symptoms noted during follow-up within the 30-day postdischarge period. Research assistants also reviewed laboratory test results in all patients for evidence of postdischarge renal failure, elevated liver function tests, or new/worsening anemia.

Hospital Readmissions

We measured nonelective hospital readmissions within 30 days of discharge using a combination of administrative data for hospitalizations within the ACO network plus patient report during the 30-day phone call for all other readmissions.²²

Adjudication of Outcomes

Adverse events and preventable adverse events: All cases of new or worsening symptoms or signs, along with all supporting documentation, were then presented to teams of two trained blinded physician adjudicators through application of methods established in previous studies.^4,21 Each of the two adjudicators independently reviewed the information, along with the medical record, and completed a standardized form to confirm or deny the presence of any adverse events (ie, patient injury due to medical management) and to classify the type of event (eg, adverse drug event, hospital-acquired infection, procedural complication, diagnostic or management error), the severity and duration of the event, and whether the event was preventable or ameliorable. The two adjudicators then met to resolve any differences in their findings and come to consensus.

Preventable readmissions: If patients were readmitted to either study hospital, we conducted an evaluation, based on previous studies,²³ to determine if and how the readmission could have been prevented including (a) a standardized patient and caregiver interview to identify possible problems with the transitions process and (b) an email questionnaire to the patient’s PCP and the inpatient teams who cared for the patient during the index admission and readmission regarding possible deficiencies with the transitions process. As with adverse event adjudications, two physician adjudicators worked independently to classify the preventability of the readmission and then met to come to consensus. Conflicts were resolved by a third adjudicator.

Analysis Plan

To evaluate the effects of the intervention on the primary outcome, the number of postdischarge adverse events per patient, we used multivariable Poisson regression, with study arm as the main predictor. A similar approach was used to evaluate the number of new or worsening postdischarge signs or symptoms and the number of preventable adverse events per patient. We used an intention-to-treat analysis: If a practice did not start the intervention when they were scheduled to, based on our randomization, we counted all patients in that practice admitted after that point as intervention patients. We adjusted for patient demographics, clinical characteristics, month, inpatient unit, and primary care practice as fixed effects. We clustered by PCP using general linear models. Intervention effects were expressed as both unadjusted and adjusted incidence rate ratios (IRRs). We also conducted a limited number of subgroup analyses, determined a priori, to determine whether the intervention was more effective in certain patient populations; we used interaction terms (intervention × subgroup) to determine the statistical significance of any effect modification.

 

To evaluate the effects of the intervention on nonelective readmissions and preventable readmissions, we used a similar approach, using multivariable logistic regression. Postdischarge functional status, adjusted for status prior to admission, was analyzed using multivariable linear regression and random effects by primary care practice. The general linear mixed model (GLIMMIX) procedure in the SAS 9.3 statistical package (SAS Institute) was used to carry out all analyses.

Power and Sample Size

We assumed a baseline rate of postdischarge adverse events of 0.30 per patient.²¹ We conservatively assumed an effect size of a change from 0.30 in the control group to 0.23 in the intervention group (a relative reduction of 22%, which was based on studies of preventability rates²³ and close to the minimum clinically important difference). Based on prior studies,^4,22 we assumed an intraclass correlation coefficient of 0.01 with an average cluster size of seven patients per PCP. Assuming a 10% loss to follow-up rate and an alpha of 0.05, we targeted a sample size of 1,800 patients to achieve 80% power, with one-third of the patients in the usual care arm and two-thirds in the intervention arm.

RESULTS

We enrolled 18 PCMH primary care practices to participate in the study, including 8 from Hospital A (out of 13 approached), 8 from Hospital B (out of 11), and 2 from other ACO practices (out of 9) (plus two pilot practices). Reasons for not participating included not having dedicated personnel to play the role of the responsible outpatient clinician, undergoing recent turn-over in practice leadership, and not having enough patients admitted to the two hospitals. One practice only enrolled 5 patients in the study and withdrew from participation, which left 17 practices.

Study Patients

We enrolled 1,679 patients (Figure 1). Reasons for nonenrollment included being unable to complete the screen prior to discharge, not meeting inclusion criteria or meeting exclusion criteria, being assigned to a pilot practice, and declining informed written consent. The baseline characteristics of enrolled patients are presented in Table 2. Differences between the two study arms were small. About 47% of the cohort was not reachable by phone after five attempts for the 30-day phone call, but only 69 (4.1%) were truly lost to follow-up because they were unreachable by phone and had no documentation in the electronic medical record in the 30-days after discharge.

Intervention Fidelity

The majority of patients did not receive most intervention components, even those components that were supposed to be delivered to all intervention patients (Table 3). A minority of patients were referred to visiting nurse services and to the home pharmacy program. However, 855 patients (87%) in the intervention arm received at least one intervention component.

Outcome Measures

The intervention was associated with a statistically significant reduction in several of the outcomes of interest, including the primary outcome, number of postdischarge adverse events (45% reduction), and new or worsening postdischarge signs or symptoms (22% reduction), as well as preventable postdischarge adverse events (58% reduction) (Table 4). There was a nonsignificant difference in functional status. There was no significant effect on total nonelective or on preventable readmission rates. When analyzed by type of adverse event, the intervention was associated with a reduction in adverse drug events and in procedural complications (Table 2 of Appendix 1). Of note, there was no significant difference in the proportion of patients with at least one adverse event whether the outcome was determined by phone call and medical record review (49%) or medical record review alone (51%) (P = .48).

In subgroup analyses, there was no evidence of effect modification by service, hospital, patient age, readmission risk, health literacy, or comorbidity score (Table 3 of Appendix 1). Table 4 of Appendix 1 provides examples of postdischarge adverse events seen in the usual care arm that might have been prevented in the intervention.

 

 

 

DISCUSSION

This intervention was associated with a reduction in postdischarge adverse events. The relative improvement in each outcome aligned with the hypothesized sensitivity to change: the smallest improvement was seen in new or worsening signs or symptoms, followed by postdischarge adverse events and then by preventable postdischarge adverse events. The intervention was not associated with a difference in readmissions. The lack of effect on hospital readmissions may have been caused by the low proportion of readmissions that are preventable, as well as low intervention fidelity and lack of resources to implement facets such as postdischarge coaching, an evidence-based intervention that was never adopted.^16,23 One lesson of this study is that it may be easier to reduce postdischarge injury (still an important outcome) than readmissions.

Putting this study in context, we should note that the literature on interventions to improve care transitions is mixed.¹⁸ While there are several reports of successful interventions, there are many reports of unsuccessful ones, often using similar components. Success is often the result of adequate resources and attention to myriad details regarding implementation.²⁴ The intervention in our study likely contributed to improvements in patient and caregiver engagement in the hospital, enhancements of communication between inpatient and outpatient clinicians, and implementation of pharmacist-led interventions to improve medication safety. Regarding the latter, several prior studies have shown the benefits of pharmacist interventions in decreasing postdischarge adverse drug events.^4,25,26 Therefore, even an intervention with incomplete intervention fidelity can reduce postdischarge adverse events, especially because adverse drug events make up the majority of adverse events.^1,2,21

Perhaps the biggest lesson we learned was regarding the limitations of the hospital-led ACO model to incentivize sufficient up-front investments in transitional care interventions. By design, we chose a real-world approach in which interventions were integrated with existing ACO efforts, which were paid for internally by the institution. As a result, many of the interventions had to be scaled back because of resource constraints. The ACO model theoretically incentivizes more integrated care, but this may not always be true in practice. Emerging evidence suggests that physician group–led ACOs are associated with lower costs and use compared with hospital-led ACOs, likely because of more aligned incentives in physician group–led ACOs to reduce use of inpatient care.^27,28

An unresolved question is whether the ideal implementation approach is to protect the time of existing clinical personnel to carry out transitional care tasks or to hire external personnel to do these tasks. We purposely spread the intervention over several clinician types to minimize the additional burden on any one of them, minimize additional costs, and play to each clinician’s expertise, but in retrospect, this may not have been the right approach. By providing additional personnel with dedicated time, interest, and training in care transitions, the intervention may be delivered with higher quantitative and qualitative fidelity, and it could create a single point of contact for patients, which was considered highly desirable by our PFAC.

This study has several limitations. A large proportion of patients (44%) were unavailable for postdischarge phone calls. However, we were able to perform medical record review for worsening signs (eg, lab abnormalities) and symptoms (as reported by patients’ providers) in the postdischarge period and adjudicate them for adverse events for all but 69 of these patients. Because all these patients had ACO-affiliated PCPs, we would expect most of their utilization to have been within the system and, therefore, to be present in the medical record. The proportion of patients with at least one adverse event did not vary by the method of follow-up, which suggests that this issue is an unlikely source of bias. Assessment of readmission was imperfect because we do not have statewide or national data. However, our combination of administrative data for Partners readmissions plus self-report for non-Partners readmission has been shown to be fairly complete in previous studies.²⁹ Adjudicators could not be fully blinded to intervention status due to the lack of blinding of admission date. We did not calculate a kappa value for interrater reliability of individual assessments of adverse events; rather, coming to consensus among the two adjudicators was part of the process. In only a handful of cases was a third adjudicator required. Lastly, this study was conducted at two academic medical centers and their affiliated primary care clinics, which potentially limits generalizability; however, the results are likely generalizable to other ACOs that include major academic medical centers.

 

CONCLUSION

In conclusion, in this real-world clinical trial, we designed, implemented, and iteratively refined a multifaceted intervention to improve care transitions within a hospital-based academic ACO. Evolution of the intervention components was the result of stakeholder input, experience with the intervention, and ACO resource constraints. The intervention reduced postdischarge adverse events. However, across the ACO network, intervention fidelity was low, and this may have contributed to the lack of effect on readmission rates. ACOs that implement interventions without hiring new personnel or protecting the time of existing personnel to conduct transitional tasks are likely to face the same challenges of low fidelity.

Acknowledgments

The authors would like to acknowledge the many people who worked on designing, implementing, and evaluating this intervention, including but not limited to: Natasha Isaac, Hilary Heyison, Jacqueline Minahan, Molly O’Reilly, Michelle Potter, Nailah Khoory, Maureen Fagan, David Bates, Laura Carr, Joseph Frolkis, Eric Weil, Jacqueline Somerville, Stephanie Ahmed, Marcy Bergeron, Jessica Smith, and Jane Millett. We would also like to thank the members of our Patient-Family Advisory Council: Maureen Fagan, Karen Spikes, Margie Hodges, Win Hodges, Aureldon Henderson, Dena Salzberg, and Kay Bander.

To the Editor:

Onychomycosis is the most common nail disorder, affecting approximately 5.5% of the world’s population.¹ There are a limited number of topical and systemic therapies approved by the US Food and Drug Administration (FDA), but no consensus guidelines exist for the management of onychomycosis. Therefore, we hypothesized that prescribing patterns would vary among different groups.

We examined data from the Centers for Medicare & Medicaid Services’ Part D Prescriber Public Use Files for 2013 to 2016.² Prescribing patterns were assessed for dermatologists, nurse practitioners, physician assistants, and podiatrists prescribing systemic (ie, terbinafine, itraconazole) or topical (ie, efinaconazole, tavaborole, ciclopirox) therapies. A cut-off of systemic therapy lasting 84 days or more (reflecting FDA-approved treatment regimens for toenail onychomycosis) was used to exclude prescriptions for other fungal conditions that require shorter treatment courses. Statistical analysis with χ² tests identified differences among specialties’ prescribing patterns.

Overall, onychomycosis medications accounted for $85.4 million in expenditures from 2013 to 2016, with spending increasing at a rate of 21.2% annually (Table 1). The greatest single-year increase was observed from 2014 to 2015, with a 40.6% surge in overall expenditures for onychomycosis medications—increasing from $17.8 million to $25.0 million in spending. Dermatologists’ prescriptions accounted for 14.8% of all claims for onychomycosis medications and 18.3% of total expenditures during the study period, totaling $15.7 million in costs. Dermatologists’ claims increased at a rate of 7.4% annually, while expenditures increased at 15.4% annually. A greater proportion of dermatologists (96.4%) prescribed topicals for onychomycosis relative to nurse practitioners (90.2%) and podiatrists (91.3%)(P<.01)(Table 2). No significant difference was observed in the prescribing patterns of dermatologists and physician assistants (P=.99).

Per-claim spending for treating onychomycosis increased 7.4% annually for dermatologists, second only to podiatrists at 17.2% annually. Each analyzed group reported at least a 7% annual increase in the amount of topicals prescribed for onychomycosis. Following their FDA approvals in 2014, tavaborole and efinaconazole accounted for 0.9% and 2.3% of onychomycosis claims in 2016, respectively, and 15.0% and 25.1% of total Medicare expenditures on onychomycosis treatments that same year, respectively. Itraconazole also disproportionately contributed to expenditures, accounting for 1.3% of onychomycosis claims in 2016 while accounting for 9.5% of total expenditures.

The introduction of efinaconazole and tavaborole in 2014 resulted in large increases in Medicare spending for onychomycosis. Limited manufacturer competition due to patents may contribute to increased spending on these topicals in the future.³ A prior analysis demonstrated that podiatrists prescribe topicals more often than other clinicians,⁴ but after adjusting for the number of dermatologists managing onychomycosis, we found that a greater proportion of dermatologists (96.4%) are prescribing topicals for onychomycosis than other clinicians. This includes these newly approved, high-cost topicals, thus disproportionately contributing to the cost burden of onychomycosis treatment.

Ciclopirox is the most commonly prescribed therapy for onychomycosis across all groups, prescribed by more than 88% of prescribers in all studied specialties. Although ciclopirox is one of the least expensive treatment options available for onychomycosis, it has the lowest relative cure rate.⁵ Onychomycosis management requires understanding of drug efficacy and disease severity.⁶ Inappropriate treatment selection may result in prolonged treatment courses and increased costs. Consensus guidelines for onychomycosis therapies across specialties may yield more cost-effective treatment for this common nail condition.

 

Acknowledgment
The authors thank Paul J. Christos, DrPH, MS (New York, New York), for his advisement regarding statistical analysis for this manuscript.

To the Editor:

Onychomycosis is the most common nail disorder, affecting approximately 5.5% of the world’s population.¹ There are a limited number of topical and systemic therapies approved by the US Food and Drug Administration (FDA), but no consensus guidelines exist for the management of onychomycosis. Therefore, we hypothesized that prescribing patterns would vary among different groups.

We examined data from the Centers for Medicare & Medicaid Services’ Part D Prescriber Public Use Files for 2013 to 2016.² Prescribing patterns were assessed for dermatologists, nurse practitioners, physician assistants, and podiatrists prescribing systemic (ie, terbinafine, itraconazole) or topical (ie, efinaconazole, tavaborole, ciclopirox) therapies. A cut-off of systemic therapy lasting 84 days or more (reflecting FDA-approved treatment regimens for toenail onychomycosis) was used to exclude prescriptions for other fungal conditions that require shorter treatment courses. Statistical analysis with χ² tests identified differences among specialties’ prescribing patterns.

Overall, onychomycosis medications accounted for $85.4 million in expenditures from 2013 to 2016, with spending increasing at a rate of 21.2% annually (Table 1). The greatest single-year increase was observed from 2014 to 2015, with a 40.6% surge in overall expenditures for onychomycosis medications—increasing from $17.8 million to $25.0 million in spending. Dermatologists’ prescriptions accounted for 14.8% of all claims for onychomycosis medications and 18.3% of total expenditures during the study period, totaling $15.7 million in costs. Dermatologists’ claims increased at a rate of 7.4% annually, while expenditures increased at 15.4% annually. A greater proportion of dermatologists (96.4%) prescribed topicals for onychomycosis relative to nurse practitioners (90.2%) and podiatrists (91.3%)(P<.01)(Table 2). No significant difference was observed in the prescribing patterns of dermatologists and physician assistants (P=.99).

Per-claim spending for treating onychomycosis increased 7.4% annually for dermatologists, second only to podiatrists at 17.2% annually. Each analyzed group reported at least a 7% annual increase in the amount of topicals prescribed for onychomycosis. Following their FDA approvals in 2014, tavaborole and efinaconazole accounted for 0.9% and 2.3% of onychomycosis claims in 2016, respectively, and 15.0% and 25.1% of total Medicare expenditures on onychomycosis treatments that same year, respectively. Itraconazole also disproportionately contributed to expenditures, accounting for 1.3% of onychomycosis claims in 2016 while accounting for 9.5% of total expenditures.

The introduction of efinaconazole and tavaborole in 2014 resulted in large increases in Medicare spending for onychomycosis. Limited manufacturer competition due to patents may contribute to increased spending on these topicals in the future.³ A prior analysis demonstrated that podiatrists prescribe topicals more often than other clinicians,⁴ but after adjusting for the number of dermatologists managing onychomycosis, we found that a greater proportion of dermatologists (96.4%) are prescribing topicals for onychomycosis than other clinicians. This includes these newly approved, high-cost topicals, thus disproportionately contributing to the cost burden of onychomycosis treatment.

Ciclopirox is the most commonly prescribed therapy for onychomycosis across all groups, prescribed by more than 88% of prescribers in all studied specialties. Although ciclopirox is one of the least expensive treatment options available for onychomycosis, it has the lowest relative cure rate.⁵ Onychomycosis management requires understanding of drug efficacy and disease severity.⁶ Inappropriate treatment selection may result in prolonged treatment courses and increased costs. Consensus guidelines for onychomycosis therapies across specialties may yield more cost-effective treatment for this common nail condition.

 

Acknowledgment
The authors thank Paul J. Christos, DrPH, MS (New York, New York), for his advisement regarding statistical analysis for this manuscript.

To the Editor:

Onychomycosis is the most common nail disorder, affecting approximately 5.5% of the world’s population.¹ There are a limited number of topical and systemic therapies approved by the US Food and Drug Administration (FDA), but no consensus guidelines exist for the management of onychomycosis. Therefore, we hypothesized that prescribing patterns would vary among different groups.

We examined data from the Centers for Medicare & Medicaid Services’ Part D Prescriber Public Use Files for 2013 to 2016.² Prescribing patterns were assessed for dermatologists, nurse practitioners, physician assistants, and podiatrists prescribing systemic (ie, terbinafine, itraconazole) or topical (ie, efinaconazole, tavaborole, ciclopirox) therapies. A cut-off of systemic therapy lasting 84 days or more (reflecting FDA-approved treatment regimens for toenail onychomycosis) was used to exclude prescriptions for other fungal conditions that require shorter treatment courses. Statistical analysis with χ² tests identified differences among specialties’ prescribing patterns.

Overall, onychomycosis medications accounted for $85.4 million in expenditures from 2013 to 2016, with spending increasing at a rate of 21.2% annually (Table 1). The greatest single-year increase was observed from 2014 to 2015, with a 40.6% surge in overall expenditures for onychomycosis medications—increasing from $17.8 million to $25.0 million in spending. Dermatologists’ prescriptions accounted for 14.8% of all claims for onychomycosis medications and 18.3% of total expenditures during the study period, totaling $15.7 million in costs. Dermatologists’ claims increased at a rate of 7.4% annually, while expenditures increased at 15.4% annually. A greater proportion of dermatologists (96.4%) prescribed topicals for onychomycosis relative to nurse practitioners (90.2%) and podiatrists (91.3%)(P<.01)(Table 2). No significant difference was observed in the prescribing patterns of dermatologists and physician assistants (P=.99).

Per-claim spending for treating onychomycosis increased 7.4% annually for dermatologists, second only to podiatrists at 17.2% annually. Each analyzed group reported at least a 7% annual increase in the amount of topicals prescribed for onychomycosis. Following their FDA approvals in 2014, tavaborole and efinaconazole accounted for 0.9% and 2.3% of onychomycosis claims in 2016, respectively, and 15.0% and 25.1% of total Medicare expenditures on onychomycosis treatments that same year, respectively. Itraconazole also disproportionately contributed to expenditures, accounting for 1.3% of onychomycosis claims in 2016 while accounting for 9.5% of total expenditures.

The introduction of efinaconazole and tavaborole in 2014 resulted in large increases in Medicare spending for onychomycosis. Limited manufacturer competition due to patents may contribute to increased spending on these topicals in the future.³ A prior analysis demonstrated that podiatrists prescribe topicals more often than other clinicians,⁴ but after adjusting for the number of dermatologists managing onychomycosis, we found that a greater proportion of dermatologists (96.4%) are prescribing topicals for onychomycosis than other clinicians. This includes these newly approved, high-cost topicals, thus disproportionately contributing to the cost burden of onychomycosis treatment.

Ciclopirox is the most commonly prescribed therapy for onychomycosis across all groups, prescribed by more than 88% of prescribers in all studied specialties. Although ciclopirox is one of the least expensive treatment options available for onychomycosis, it has the lowest relative cure rate.⁵ Onychomycosis management requires understanding of drug efficacy and disease severity.⁶ Inappropriate treatment selection may result in prolonged treatment courses and increased costs. Consensus guidelines for onychomycosis therapies across specialties may yield more cost-effective treatment for this common nail condition.

 

Acknowledgment
The authors thank Paul J. Christos, DrPH, MS (New York, New York), for his advisement regarding statistical analysis for this manuscript.