Original Research

From Banner Health Corporation, Phoenix, AZ.

Background: Health care providers are routinely incentivized with pay-for-performance (P4P) metrics to increase the quality of care. In an inpatient setting, P4P models typically measure quality by attributing each patient’s outcome to a single provider even though many providers routinely care for the patient. This study investigates a new attribution approach aiming to distribute each outcome across all providers who provided care.

Methods: The methodology relies on a multi-membership model and is demonstrated in the Banner Health system using 3 clinical outcome measures (length of stay, 30-day readmissions, and mortality) and responses to 3 survey questions that measure a patient’s perception of their care. The new approach is compared to the “standard” method, which attributes each patient to only 1 provider.

Results: When ranking by clinical outcomes, both methods were concordant 72.1% to 82.1% of the time for top-half/bottom-half rankings, with a median percentile difference between 7 and 15. When ranking by survey scores, there was more agreement, with concordance between 84.1% and 86.6% and a median percentile difference between 11 and 13. Last, Pearson correlation coefficients of the paired percentiles ranged from 0.56 to 0.78.

Conclusion: The new approach provides a fairer solution when measuring provider performance.

Keywords: patient attribution; PAMM; PAPR; random effect model; pay for performance.

Providers practicing in hospitals are routinely evaluated based on their performance and, in many cases, are financially incentivized for a better-than-average performance within a pay-for-performance (P4P) model. The use of P4P models is based on the belief that they will “improve, motivate, and enhance providers to pursue aggressively and ultimately achieve the quality performance targets thus decreasing the number of medical errors with less malpractice events.”¹ Although P4P models continue to be a movement in health care, they have been challenging to implement.

One concern involves the general quality of implementation, such as defining metrics and targets, setting payout amounts, managing technology and market conditions, and gauging the level of transparency to the provider.² Another challenge, and the focus of this project, are concerns around measuring performance to avoid perceptions of unfairness. This concern can be minimized if the attribution is handled in a fairer way, by spreading it across all providers who affected the outcome, both in a positive or negative direction.³

To implement these models, the performance of providers needs to be measured and tracked periodically. This requires linking, or attributing, a patient’s outcome to a provider, which is almost always the attending or discharging provider (ie, a single provider).³ In this single-provider attribution approach, one provider will receive all the credit (good or bad) for their respective patients’ outcomes, even though the provider may have seen the patient only a fraction of the time during the hospitalization. Attributing outcomes—for example, length of stay (LOS), readmission rate, mortality rate, net promoter score (NPS)—using this approach reduces the validity of metrics designed to measure provider performance, especially in a rotating provider environment where many providers interact with and care for a patient. For example, the quality of providers’ interpersonal skills and competence were among the strongest determinants of patient satisfaction,⁴ but it is not credible that this is solely based on the last provider during a hospitalization.

Proportionally distributing the attribution of an outcome has been used successfully in other contexts. Typically, a statistical modeling approach using a multi-membership framework is used because it can handle the sometimes-complicated relationships within the hierarchy. It also allows for auxiliary variables to be introduced, which can help explain and control for exogenous effects.^5-7 For example, in the education setting, standardized testing is administered to students at defined years of schooling: at grades 4, 8, and 10, for instance. The progress of students, measured as the academic gains between test years, are proportionally attributed to all the teachers who the student has had between the test years. These partial attributions are combined to evaluate an overall teacher performance.^8,9

Although the multi-membership framework has been used in other industries, it has yet to be applied in measuring provider performance. The purpose of this project is to investigate the impact of using a multi-provider approach compared to the standard single-provider approach. The findings may lead to modifications in the way a provider’s performance is measured and, thus, how providers are compensated. A similar study investigated the impact of proportionally distributing patients’ outcomes across all rotating providers using a weighting method based on billing practices to measure the partial impact of each provider.³

This study is different in 2 fundamental ways. First, attribution is weighted based on the number of clinically documented interactions (via clinical notes) between a patient and all rotating providers during the hospitalization. Second, performance is measured via multi-membership models, which can estimate the effect (both positive and negative) that a provider has on an outcome, even when caring for a patient a fraction of the time during the hospitalization.

 

Methods

Setting

Banner Health is a non-profit, multi-hospital health care system across 6 states in the western United States that is uniquely positioned to study provider quality attribution models. It not only has a large number of providers and serves a broad patient population, but Banner Health also uses an instance of Cerner (Kansas City, MO), an enterprise-level electronic health record (EHR) system that connects all its facilities and allows for advanced analytics across its system.

For this study, we included only general medicine and surgery patients admitted and discharged from the inpatient setting between January 1, 2018, and December 31, 2018, who were between 18 and 89 years old at admission, and who had a LOS between 1 and 14 days. Visit- and patient-level data were collected from Cerner, while outcome data, and corresponding expected outcome data, were obtained from Premier, Inc. (Charlotte, NC) using their CareScience methodologies.¹⁰ To measure patient experience, response data were extracted from post-discharge surveys administered by InMoment (Salt Lake City, UT).

Provider Attribution Models

Provider Attribution by Physician of Record (PAPR). In the standard approach, denoted here as the PAPR model, 1 provider—typically the attending or discharging provider, which may be the same person—is attributed to the entire hospitalization. This provider is responsible for the patient’s care, and all patient outcomes are aggregated and attributed to the provider to gauge his or her performance. The PAPR model is the most popular form of attribution across many health care systems and is routinely used for P4P incentives.

In this study, the discharging provider was used when attributing hospitalizations using the PAPR model. Providers responsible for fewer than 12 discharges in the calendar year were excluded. Because of the directness of this type of attribution, the performance of 1 provider does not account for the performance of the other rotating providers during hospitalizations.

Provider Attribution by Multiple Membership (PAMM). In contrast, we introduce another attribution approach here that is designed to assign partial attribution to each provider who cares for the patient during the hospitalization. To aggregate the partial attributions, and possibly control for any exogenous or risk-based factors, a multiple-membership, or multi-member (MM), model is used. The MM model can measure the effect of a provider on an outcome even when the patient-to-provider relationship is complex, such as in a rotating provider environment.⁸

The purpose of this study is to compare attribution models and to determine whether there are meaningful differences between them. Therefore, for comparison purposes, the same discharging providers using the PAPR approach are eligible for the PAMM approach, so that both attribution models are using the same set of providers. All other providers are excluded because their performance would not be comparable to the PAPR approach.

While there are many ways to document provider-to-patient interactions, 2 methods are available in almost all health care systems. The first method is to link a provider’s billing charges to each patient-day combination. This approach limits the attribution to 1 provider per patient per day because multiple rotating providers cannot charge for the same patient-day combination.³ However, many providers interact with a patient on the same day, so using this approach excludes non-billed provider-to-patient interactions.

The second method, which was used in this study, relies on documented clinical notes within the EHR to determine how attribution is shared. In this approach, attribution is weighted based on the authorship of 3 types of eligible clinical notes: admitting history/physical notes (during admission), progress notes (during subsequent days), and discharge summary notes (during final discharge). This will (likely) result in many providers being linked to a patient on each day, which better reflects the clinical setting (Figure). Recently, clinical notes were used to attribute care of patients in an inpatient setting, and it was found that this approach provides a reliable way of tracking interactions and assigning ownership.¹¹

The provider-level attribution weights are based on the share of authorships of eligible note types. Specifically, for each provider j, let a_ij be the total count of eligible note types for hospitalization i authored by provider j, and let a_i be the overall total count of eligible note types for hospitalization i. Then the attribution weight is

(Eq. 1)

for hospitalization i and provider j. Note that ∑_jw_ij = 1: in other words, the total attribution, summed across all providers, is constrained to be 1 for each hospitalization.

Patient Outcomes

Outcomes were chosen based on their routine use in health care systems as standards when evaluating provider performance. This study included 6 outcomes: inpatient LOS, inpatient mortality, 30-day inpatient readmission, and patient responses from 3 survey questions. These outcomes can be collected without any manual chart reviews, and therefore are viewed as objective outcomes of provider performance.

Each outcome was aggregated for each provider using both attribution methods independently. For the PAPR method, observed-to-expected (OE) indices for LOS, mortality, and readmissions were calculated along with average patient survey scores. For the PAMM method, provider-level random effects from the fitted models were used. In both cases, the calculated measures were used for ranking purposes when determining top (or bottom) providers for each outcome.

 

Individual Provider Metrics for the PAPR Method

Inpatient LOS Index. Hospital inpatient LOS was measured as the number of days between admission date and discharge date. For each hospital visit, an expected LOS was determined using Premier’s CareScience Analytics (CSA) risk-adjustment methodology.¹⁰ The CSA methodology for LOS incorporates a patient’s clinical history, demographics, and visit-related administrative information.

Let n_j be the number of hospitalizations attributed to provider j. Let o_ij and e_ij be the observed and expected LOS, respectively, for hospitalization i = 1,…,n_j attributed to provider j. Then the inpatient LOS index for provider j is L_j = ∑_io_ij⁄∑_ie_ij.

Inpatient Mortality Index. Inpatient mortality was defined as the death of the patient during hospitalization. For each hospitalization, an expected mortality probability was determined using Premier’s CSA risk-adjustment methodology.¹⁰ The CSA methodology for mortality incorporates a patient’s demographics and comorbidities.

Just as before, let n_j be the number of hospitalizations attributed to provider j. Let m_ij = 1 if the patient died during hospitalization i = 1, … , n_j attributed to provider j; m_ij = 0 otherwise. Let p_ij^(m) be the corresponding expected mortality probability. Then the inpatient mortality index for provider j is M_j = ∑_im_ij⁄∑_ip_ij^(m).

30-Day Inpatient Readmission Index. A 30-day inpatient readmission was defined as the event when a patient is discharged and readmits back into the inpatient setting within 30 days. The inclusion criteria defined by the Centers for Medicare and Medicaid Services (CMS) all-cause hospital-wide readmission measure was used and, consequently, planned readmissions were excluded.¹² Readmissions could occur at any Banner hospital, including the same hospital. For each hospital visit, an expected readmission probability was derived using Premier’s CSA risk-adjustment methodology.¹⁰ The CSA methodology for readmissions incorporates a patient’s clinical history, demographics, and visit-related administrative information.

Let n_j be the number of hospitalizations attributed to provider j. Let r_ij = 1 if the patient had a readmission following hospitalization i = 1, … , n_j attributed to provider j; r_ij = 0 otherwise. Let p_ij^(r) be the corresponding expected readmission probability. Then the 30-day inpatient readmission index for provider j is R_j = ∑_ir_ij ⁄∑_ip_ij^(r).

Patient Survey Scores. The satisfaction of the patient’s experience during hospitalization was measured via post-discharge surveys administered by InMoment. Two survey questions were selected because they related directly to a provider’s interaction with the patient: “My interactions with doctors were excellent” (Doctor) and “I received the best possible care” (Care). A third question, “I would recommend this hospital to my family and friends,” was selected as a proxy measure of the overall experience and, in the aggregate, is referred to as the net promoter score (NPS).^13,14 The responses were measured on an 11-point Likert scale, ranging from “Strongly Disagree” (0) to “Strongly Agree” (10); “N/A” or missing responses were excluded.

The Likert responses were coded to 3 discrete values as follows: if the value was between 0 and 6, then -1 (ie, detractor); between 7 and 8 (ie, neutral), then 0; otherwise 1 (ie, promoter). Averaging these coded responses results in a patient survey score for each question. Specifically, let n_j be the number of hospitalizations attributed to provider j in which the patient responded to the survey question. Let s_ij ∈{−1, 0, 1} be the coded response linked to hospitalization i = 1, … , n_j attributed to provider j. Then the patient experience score for provider j is S_j = ∑_is_ij⁄n_j.

Handling Ties in Provider Performance Measures. Because ties can occur in the PAPR approach for all measures, a tie-breaking strategy is needed. For LOS indices, ties are less likely because their numerator is strictly greater than 0, and expected LOS values are typically distinct enough. Indeed, no ties were found in this study for LOS indices. However, mortality and readmission indices can routinely result in ties when the best possible index is achieved, such as 0 deaths or readmissions among attributed hospitalizations. To help differentiate between those indices in the PAPR approach, the total estimated risk (denominator) was utilized as a secondary scoring criterion.

Mortality and readmission metrics were addressed by sorting first by the outcome (mortality index), and second by the denominator (total estimated risk). For example, if provider A has the same mortality rate as provider B, then provider A would be ranked higher if the denominator was larger, indicating a higher risk for mortality.

Similarly, it was very common for providers to have the same overall average rating for a survey question. Therefore, the denominator (number of respondents) was used to break ties. However, the denominator sorting was bidirectional. For example, if the tied score was positive (more promoters than detractors) for providers A and B, then provider A would be ranked higher if the denominator was larger. Conversely, if the tied score between providers A and B was neutral or negative (more detractors than promoters), then provider A would be ranked lower if the denominator was larger.

 

Individual Provider Metrics for the PAMM Method

For the PAMM method, model-based metrics were derived using a MM model.⁸ Specifically, let J be the number of rotating providers in a health care system. Let Y_i be an outcome of interest from hospitalization i, X_1i, …, X_pi be fixed effects or covariates, and ß₁, …, ß_p be the coefficients for the respective covariates. Then the generalized MM statistical model is

(Eq. 2)

where g(μ_i) is a link function between the mean of the outcome, μ_i, and its linear predictor, ß₀, is the marginal intercept, w_ij represents the attribution weight of provider j on hospitalization i (described in Equation 1), and γ_j represents the random effect of provider j on the outcome with γ_j~N(0,σ_γ²).

For the mortality and readmission binary outcomes, logistic regression was performed using a logit link function, with the corresponding expected probability as the only fixed covariate. The expected probabilities were first converted into odds and then log-transformed before entering the model. For LOS, Poisson regression was performed using a log link function with the log-transformed expected LOS as the only fixed covariate. For coded patient experience responses, an ordered logistic regression was performed using a cumulative logit link function (no fixed effects were added).

MM Model-based Metrics. Each fitted MM model produces a predicted random effect for each provider. The provider-specific random effects can be interpreted as the unobserved influence of each provider on the outcome after controlling for any fixed effect included in the model. Therefore, the provider-specific random effects were used to evaluate the relative provider performance, which is analogous to the individual provider-level metrics used in the PAPR method.

Measuring provider performance using a MM model is more flexible and robust to outliers compared to the standard approach using OE indices or simple averages. First, although not investigated here, the effect of patient-, visit-, provider-, and/or temporal-level covariates can be controlled when evaluating provider performance. For example, a patient’s socioeconomic status, a provider’s workload, and seasonal factors can be added to the MM model. These external factors are not accounted for in OE indices.

Another advantage of using predicted random effects is the concept of “shrinkage.” The process of estimating random effects inherently accounts for small sample sizes (when providers do not treat a large enough sample of patients) and/or when there is a large ratio of patient variance to provider variance (for instance, when patient outcome variability is much higher compared to provider performance variability). In both cases, the estimation of the random effect is pulled ever closer to 0, signaling that the provider performance is closer to the population average. See Henderson¹⁵ and Mood¹⁶ for further details.

In contrast, OE indices can result in unreliable estimates when a provider has not cared for many patients. This is especially prevalent when the outcome is binary with a low probability of occurring, such as mortality. Indeed, provider-level mortality OE indices are routinely 0 when the patient counts are low, which skews performance rankings unfairly. Finally, OE indices also ignore the magnitude of the variance of an outcome between providers and patients, which can be large.

Comparison Methodology

In this study, we seek to compare the 2 methods of attribution, PAPR and PAMM, to determine whether there are meaningful differences between them when measuring provider performance. Using retrospective data described in the next section, each attribution method was used independently to derive provider-level metrics. To assess relative performance, percentiles were assigned to each provider based on their metric values so that, in the end, there were 2 percentile ranks for each provider for each metric.

Using these paired percentiles, we derived the following measures of concordance, similar to Herzke, Michtalik³: (1) the percent concordance measure—defined as the number of providers who landed in the top half (greater than the median) or bottom half under both attribution models—divided by the total number of providers; (2) the median of the absolute difference in percentiles under both attribution models; and (3) the Pearson correlation coefficient of the paired provider ranks. The first measure is a global measure of concordance between the 2 approaches and would be expected to be 50% by chance. The second measure gauges how an individual provider’s rank is affected by the change in attribution methodologies. The third measure is a statistical measure of linear correlation of the paired percentiles and was not included in the Herzke, Michtalik³ study.

All statistical analyses were performed on SAS (version 9.4; Cary, NC) and the MM models were fitted using PROC GLIMMIX with the EFFECT statement. The Banner Health Institutional Review Board approved this study.

 

Results

Descriptive Statistics

A total of 58,730 hospitalizations were included, of which care was provided by 963 unique providers across 25 acute care and critical access hospitals. Table 1 contains patient characteristics, and Table 2 depicts overall unadjusted outcomes. Providers responsible for less than 12 discharges in the calendar year were excluded from both approaches. Also, some hospitalizations were excluded when expected values were not available.

Multi-Membership Model Results

Table 3 displays the results after independently fitting MM models to each of the 3 clinical outcomes. Along with a marginal intercept, the only covariate in each model was the corresponding expected value after a transformation. This was added to use the same information that is typically used in OE indices, therefore allowing for a proper comparison between the 2 attribution methods. The provider-level variance represents the between-provider variation and measures the amount of influence providers have on the corresponding outcome after controlling for any covariates in the model. A provider-level variance of 0 would indicate that providers do not have any influence on the outcome. While the mortality and readmission model results can be compared to each other, the LOS model cannot given its different scale and transformation altogether.

The results in Table 3 suggest that each expected value covariate is highly correlated with its corresponding outcome, which is the anticipated conclusion given that they are constructed in this fashion. The estimated provider-level variances indicate that, after including an expected value in the model, providers have less of an influence on a patient’s LOS and likelihood of being readmitted. On the other hand, the results suggest that providers have much more influence on the likelihood of a patient dying in the hospital, even after controlling for an expected mortality covariate.

Table 4 shows the results after independently fitting MM-ordered logistic models to each of the 3 survey questions. The similar provider-level variances suggest that providers have the same influence on the patient’s perception of the quality of their interactions with the doctor (Doctor), the quality of the care they received (Care), and their likelihood to recommend a friend or family member to the hospital (NPS).

 

Comparison Results Between Both Attribution Methods

Table 5 compares the 2 attribution methods when ranking providers based on their performance on each outcome measure. The comparison metrics gauge how well the 2 methods agree overall (percent concordance), agree at the provider level (absolute percentile difference and interquartile range [IQR]), and how the paired percentiles linearly correlate to each other (Pearson correlation coefficient).

LOS, by a small margin, had the lowest concordance of clinical outcomes (72.1%), followed by mortality (75.9%) and readmissions (82.1%). Generally, the survey scores had higher percent concordance than the clinical outcome measures, with Doctor at 84.1%, Care at 85.9%, and NPS having the highest percent concordance at 86.6%. Given that by chance the percent concordance is expected to be 50%, there was notable discordance, especially with the clinical outcome measures. Using LOS performance as an example, one attribution methodology would rank a provider in the top half or bottom half, while the other attribution methodology would rank the same provider exactly the opposite way about 28% of the time.

The median absolute percentile difference between the 2 methods was more modest (between 7 and 15). Still, there were some providers whose performance ranking was heavily impacted by the attribution methodology that was used. This was especially true when evaluating performance for certain clinical measures, where the attribution method that was used could change the provider performance percentile by up to 90 levels.

The paired percentiles were positively correlated when ranking performance using any of the 6 measures. This suggests that both methodologies assess performance generally in the same direction, irrespective of the methodology and measure. We did not investigate more complex correlation measures and left this for future research.

It should be noted that ties occurred much more frequently with the PAPR method than when using PAMM and therefore required tie-breaking rules to be designed. Given the nature of OE indices, PAPR methodology is especially sensitive to ties whenever the measure includes counting the number of events (for example, mortality and readmissions) and whenever there are many providers with very few attributed patients. On the other hand, using the PAMM method is much more robust against ties given that the summation of all the weighted attributed outcomes will rarely result in ties, even with a nominal set of providers.

 

 

Discussion

In this study, the PAMM methodology was introduced and was used to assess relative provider performance on 3 clinical outcome measures and 3 patient survey scores. The new approach aims to distribute each outcome among all providers who provided care for a patient in an inpatient setting. Clinical notes were used to account for patient-to-provider interactions, and fitted MM statistical models were used to compute the effects that each provider had on each outcome. The provider effect was introduced as a random effect, and the set of predicted random effects was used to rank the performance of each provider.

The PAMM approach was compared to the more traditional methodology, PAPR, where each patient is attributed to only 1 provider: the discharging physician in this study. Using this approach, OE indices of clinical outcomes and averages of survey scores were used to rank the performance of each provider. This approach resulted in many ties, which were broken based on the number of hospitalizations, although other tie-breaking methods may be used in practice.

Both methodologies showed modest concordance with each other for the clinical outcomes, but higher concordance for the patient survey scores. This was also true when using the Pearson correlation coefficient to assess agreement. The 1 outcome measure that showed the least concordance and least linear correlation between methods was LOS, which would suggest that LOS performance is more sensitive to the attribution methodology that is used. However, it was the least concordant by a small margin.

Furthermore, although the medians of the absolute percentile differences were small, there were some providers who had large deviations, suggesting that some providers would move from being shown as high-performers to low-performers and vice versa based on the chosen attribution method. We investigated examples of this and determined that the root cause was the difference in effective sample sizes for a provider. For the PAPR method, the effective sample size is simply the number of hospitalizations attributed to the provider. For the PAMM method, the effective sample size is the sum of all non-zero weights across all hospitalizations where the provider cared for a patient. By and large, the PAMM methodology provides more information of the provider effect on an outcome than the PAPR approach because every provider-patient interaction is considered. For example, providers who do not routinely discharge patients, but often care for patients, will have rankings that differ dramatically between the 2 methods.

The PAMM methodology has many statistical advantages that were not fully utilized in this comparative study. For example, we did not include any covariates in the MM models except for the expected value of the outcome, when it was available. Still, it is known that other covariates can impact an outcome as well, such as the patient’s age, socioeconomic indicators, existing chronic conditions, and severity of hospitalization, which can be added to the MM models as fixed effects. In this way, the PAMM approach can control for these other covariates, which are typically outside of the control of providers but typically ignored using OE indices. Therefore, using the PAMM approach would provide a fairer comparison of provider performance.

Using the PAMM method, most providers had a large sample size to assess their performance once all the weighted interactions were included. Still, there were a few who did not care for many patients for a variety of reasons. In these scenarios, MM models “borrow” strength from other providers to produce a more robust predicted provider effect by using a weighted average between the overall population trend and the specific provider outcomes (see Rao and Molina¹⁷). As a result, PAMM is a more suitable approach when the sample sizes of patients attributed to providers can be small.

One of the most interesting findings of this study was the relative size of the provider-level variance to the size of the fixed effect in each model (Table 3). Except for mortality, these variances suggest that there is a small difference in performance from one provider to another. However, these should be interpreted as the variance when only 1 provider is involved in the care of a patient. When multiple providers are involved, using basic statistical theory, the overall provider-level variance will be σ_γ² ∑w_ij² (see Equation 2). For example, the estimated variance among providers for LOS was 0.03 (on a log scale), but, using the scenario in the Figure, the overall provider-level variance for this hospitalization will be 0.03 (0.375²+ 0.125²+ 0.5²) = 0.012. Hence, the combined effect of providers on LOS is less than would be expected. Indeed, as more providers are involved with a patient’s care, the more their combined influence on an outcome is diluted.

In this study, the PAMM approach placed an equal weight on all provider-patient interactions via clinical note authorship, but that may not be optimal in some settings. For example, it may make more sense to set a higher weight on the provider who admitted or discharged the patient while placing less (or 0) weight on all other interactions. In the extreme, if the full weight were placed on 1 provider interaction (eg, during discharge, then the MM model would be reduced to a one-way random effects model. The flexibility of weighting interactions is a feature of the PAMM approach, but any weighting framework must be transparent to the providers before implementation.

Conclusion

This study demonstrates that the PAMM approach is a feasible option within a large health care organization. For P4P programs to be successful, providers must be able to trust that their performance will be fairly assessed and that all provider-patient interactions are captured to provide a full comparison amongst their peers. The PAMM methodology is one solution to spread the positive (and negative) outcomes across all providers who cared for a patient and therefore, if implemented, would add trust and fairness when measuring and assessing provider performance.

Acknowledgments: The authors thank Barrie Bradley for his support in the initial stages of this research and Dr. Syed Ismail Jafri for his help and support on the standard approaches of assessing and measuring provider performances.

Corresponding author: Rachel Ginn, MS, Banner Health Corporation, 2901 N. Central Ave., Phoenix, AZ 85012; rachel.ginn@gmail.com.

Financial disclosures: None.

From Banner Health Corporation, Phoenix, AZ.

Background: Health care providers are routinely incentivized with pay-for-performance (P4P) metrics to increase the quality of care. In an inpatient setting, P4P models typically measure quality by attributing each patient’s outcome to a single provider even though many providers routinely care for the patient. This study investigates a new attribution approach aiming to distribute each outcome across all providers who provided care.

Methods: The methodology relies on a multi-membership model and is demonstrated in the Banner Health system using 3 clinical outcome measures (length of stay, 30-day readmissions, and mortality) and responses to 3 survey questions that measure a patient’s perception of their care. The new approach is compared to the “standard” method, which attributes each patient to only 1 provider.

Results: When ranking by clinical outcomes, both methods were concordant 72.1% to 82.1% of the time for top-half/bottom-half rankings, with a median percentile difference between 7 and 15. When ranking by survey scores, there was more agreement, with concordance between 84.1% and 86.6% and a median percentile difference between 11 and 13. Last, Pearson correlation coefficients of the paired percentiles ranged from 0.56 to 0.78.

Conclusion: The new approach provides a fairer solution when measuring provider performance.

Keywords: patient attribution; PAMM; PAPR; random effect model; pay for performance.

Providers practicing in hospitals are routinely evaluated based on their performance and, in many cases, are financially incentivized for a better-than-average performance within a pay-for-performance (P4P) model. The use of P4P models is based on the belief that they will “improve, motivate, and enhance providers to pursue aggressively and ultimately achieve the quality performance targets thus decreasing the number of medical errors with less malpractice events.”¹ Although P4P models continue to be a movement in health care, they have been challenging to implement.

One concern involves the general quality of implementation, such as defining metrics and targets, setting payout amounts, managing technology and market conditions, and gauging the level of transparency to the provider.² Another challenge, and the focus of this project, are concerns around measuring performance to avoid perceptions of unfairness. This concern can be minimized if the attribution is handled in a fairer way, by spreading it across all providers who affected the outcome, both in a positive or negative direction.³

To implement these models, the performance of providers needs to be measured and tracked periodically. This requires linking, or attributing, a patient’s outcome to a provider, which is almost always the attending or discharging provider (ie, a single provider).³ In this single-provider attribution approach, one provider will receive all the credit (good or bad) for their respective patients’ outcomes, even though the provider may have seen the patient only a fraction of the time during the hospitalization. Attributing outcomes—for example, length of stay (LOS), readmission rate, mortality rate, net promoter score (NPS)—using this approach reduces the validity of metrics designed to measure provider performance, especially in a rotating provider environment where many providers interact with and care for a patient. For example, the quality of providers’ interpersonal skills and competence were among the strongest determinants of patient satisfaction,⁴ but it is not credible that this is solely based on the last provider during a hospitalization.

Proportionally distributing the attribution of an outcome has been used successfully in other contexts. Typically, a statistical modeling approach using a multi-membership framework is used because it can handle the sometimes-complicated relationships within the hierarchy. It also allows for auxiliary variables to be introduced, which can help explain and control for exogenous effects.^5-7 For example, in the education setting, standardized testing is administered to students at defined years of schooling: at grades 4, 8, and 10, for instance. The progress of students, measured as the academic gains between test years, are proportionally attributed to all the teachers who the student has had between the test years. These partial attributions are combined to evaluate an overall teacher performance.^8,9

Although the multi-membership framework has been used in other industries, it has yet to be applied in measuring provider performance. The purpose of this project is to investigate the impact of using a multi-provider approach compared to the standard single-provider approach. The findings may lead to modifications in the way a provider’s performance is measured and, thus, how providers are compensated. A similar study investigated the impact of proportionally distributing patients’ outcomes across all rotating providers using a weighting method based on billing practices to measure the partial impact of each provider.³

This study is different in 2 fundamental ways. First, attribution is weighted based on the number of clinically documented interactions (via clinical notes) between a patient and all rotating providers during the hospitalization. Second, performance is measured via multi-membership models, which can estimate the effect (both positive and negative) that a provider has on an outcome, even when caring for a patient a fraction of the time during the hospitalization.

 

Methods

Setting

Banner Health is a non-profit, multi-hospital health care system across 6 states in the western United States that is uniquely positioned to study provider quality attribution models. It not only has a large number of providers and serves a broad patient population, but Banner Health also uses an instance of Cerner (Kansas City, MO), an enterprise-level electronic health record (EHR) system that connects all its facilities and allows for advanced analytics across its system.

For this study, we included only general medicine and surgery patients admitted and discharged from the inpatient setting between January 1, 2018, and December 31, 2018, who were between 18 and 89 years old at admission, and who had a LOS between 1 and 14 days. Visit- and patient-level data were collected from Cerner, while outcome data, and corresponding expected outcome data, were obtained from Premier, Inc. (Charlotte, NC) using their CareScience methodologies.¹⁰ To measure patient experience, response data were extracted from post-discharge surveys administered by InMoment (Salt Lake City, UT).

Provider Attribution Models

Provider Attribution by Physician of Record (PAPR). In the standard approach, denoted here as the PAPR model, 1 provider—typically the attending or discharging provider, which may be the same person—is attributed to the entire hospitalization. This provider is responsible for the patient’s care, and all patient outcomes are aggregated and attributed to the provider to gauge his or her performance. The PAPR model is the most popular form of attribution across many health care systems and is routinely used for P4P incentives.

In this study, the discharging provider was used when attributing hospitalizations using the PAPR model. Providers responsible for fewer than 12 discharges in the calendar year were excluded. Because of the directness of this type of attribution, the performance of 1 provider does not account for the performance of the other rotating providers during hospitalizations.

Provider Attribution by Multiple Membership (PAMM). In contrast, we introduce another attribution approach here that is designed to assign partial attribution to each provider who cares for the patient during the hospitalization. To aggregate the partial attributions, and possibly control for any exogenous or risk-based factors, a multiple-membership, or multi-member (MM), model is used. The MM model can measure the effect of a provider on an outcome even when the patient-to-provider relationship is complex, such as in a rotating provider environment.⁸

The purpose of this study is to compare attribution models and to determine whether there are meaningful differences between them. Therefore, for comparison purposes, the same discharging providers using the PAPR approach are eligible for the PAMM approach, so that both attribution models are using the same set of providers. All other providers are excluded because their performance would not be comparable to the PAPR approach.

While there are many ways to document provider-to-patient interactions, 2 methods are available in almost all health care systems. The first method is to link a provider’s billing charges to each patient-day combination. This approach limits the attribution to 1 provider per patient per day because multiple rotating providers cannot charge for the same patient-day combination.³ However, many providers interact with a patient on the same day, so using this approach excludes non-billed provider-to-patient interactions.

The second method, which was used in this study, relies on documented clinical notes within the EHR to determine how attribution is shared. In this approach, attribution is weighted based on the authorship of 3 types of eligible clinical notes: admitting history/physical notes (during admission), progress notes (during subsequent days), and discharge summary notes (during final discharge). This will (likely) result in many providers being linked to a patient on each day, which better reflects the clinical setting (Figure). Recently, clinical notes were used to attribute care of patients in an inpatient setting, and it was found that this approach provides a reliable way of tracking interactions and assigning ownership.¹¹

The provider-level attribution weights are based on the share of authorships of eligible note types. Specifically, for each provider j, let a_ij be the total count of eligible note types for hospitalization i authored by provider j, and let a_i be the overall total count of eligible note types for hospitalization i. Then the attribution weight is

(Eq. 1)

for hospitalization i and provider j. Note that ∑_jw_ij = 1: in other words, the total attribution, summed across all providers, is constrained to be 1 for each hospitalization.

Patient Outcomes

Outcomes were chosen based on their routine use in health care systems as standards when evaluating provider performance. This study included 6 outcomes: inpatient LOS, inpatient mortality, 30-day inpatient readmission, and patient responses from 3 survey questions. These outcomes can be collected without any manual chart reviews, and therefore are viewed as objective outcomes of provider performance.

Each outcome was aggregated for each provider using both attribution methods independently. For the PAPR method, observed-to-expected (OE) indices for LOS, mortality, and readmissions were calculated along with average patient survey scores. For the PAMM method, provider-level random effects from the fitted models were used. In both cases, the calculated measures were used for ranking purposes when determining top (or bottom) providers for each outcome.

 

Individual Provider Metrics for the PAPR Method

Inpatient LOS Index. Hospital inpatient LOS was measured as the number of days between admission date and discharge date. For each hospital visit, an expected LOS was determined using Premier’s CareScience Analytics (CSA) risk-adjustment methodology.¹⁰ The CSA methodology for LOS incorporates a patient’s clinical history, demographics, and visit-related administrative information.

Let n_j be the number of hospitalizations attributed to provider j. Let o_ij and e_ij be the observed and expected LOS, respectively, for hospitalization i = 1,…,n_j attributed to provider j. Then the inpatient LOS index for provider j is L_j = ∑_io_ij⁄∑_ie_ij.

Inpatient Mortality Index. Inpatient mortality was defined as the death of the patient during hospitalization. For each hospitalization, an expected mortality probability was determined using Premier’s CSA risk-adjustment methodology.¹⁰ The CSA methodology for mortality incorporates a patient’s demographics and comorbidities.

Just as before, let n_j be the number of hospitalizations attributed to provider j. Let m_ij = 1 if the patient died during hospitalization i = 1, … , n_j attributed to provider j; m_ij = 0 otherwise. Let p_ij^(m) be the corresponding expected mortality probability. Then the inpatient mortality index for provider j is M_j = ∑_im_ij⁄∑_ip_ij^(m).

30-Day Inpatient Readmission Index. A 30-day inpatient readmission was defined as the event when a patient is discharged and readmits back into the inpatient setting within 30 days. The inclusion criteria defined by the Centers for Medicare and Medicaid Services (CMS) all-cause hospital-wide readmission measure was used and, consequently, planned readmissions were excluded.¹² Readmissions could occur at any Banner hospital, including the same hospital. For each hospital visit, an expected readmission probability was derived using Premier’s CSA risk-adjustment methodology.¹⁰ The CSA methodology for readmissions incorporates a patient’s clinical history, demographics, and visit-related administrative information.

Let n_j be the number of hospitalizations attributed to provider j. Let r_ij = 1 if the patient had a readmission following hospitalization i = 1, … , n_j attributed to provider j; r_ij = 0 otherwise. Let p_ij^(r) be the corresponding expected readmission probability. Then the 30-day inpatient readmission index for provider j is R_j = ∑_ir_ij ⁄∑_ip_ij^(r).

Patient Survey Scores. The satisfaction of the patient’s experience during hospitalization was measured via post-discharge surveys administered by InMoment. Two survey questions were selected because they related directly to a provider’s interaction with the patient: “My interactions with doctors were excellent” (Doctor) and “I received the best possible care” (Care). A third question, “I would recommend this hospital to my family and friends,” was selected as a proxy measure of the overall experience and, in the aggregate, is referred to as the net promoter score (NPS).^13,14 The responses were measured on an 11-point Likert scale, ranging from “Strongly Disagree” (0) to “Strongly Agree” (10); “N/A” or missing responses were excluded.

The Likert responses were coded to 3 discrete values as follows: if the value was between 0 and 6, then -1 (ie, detractor); between 7 and 8 (ie, neutral), then 0; otherwise 1 (ie, promoter). Averaging these coded responses results in a patient survey score for each question. Specifically, let n_j be the number of hospitalizations attributed to provider j in which the patient responded to the survey question. Let s_ij ∈{−1, 0, 1} be the coded response linked to hospitalization i = 1, … , n_j attributed to provider j. Then the patient experience score for provider j is S_j = ∑_is_ij⁄n_j.

Handling Ties in Provider Performance Measures. Because ties can occur in the PAPR approach for all measures, a tie-breaking strategy is needed. For LOS indices, ties are less likely because their numerator is strictly greater than 0, and expected LOS values are typically distinct enough. Indeed, no ties were found in this study for LOS indices. However, mortality and readmission indices can routinely result in ties when the best possible index is achieved, such as 0 deaths or readmissions among attributed hospitalizations. To help differentiate between those indices in the PAPR approach, the total estimated risk (denominator) was utilized as a secondary scoring criterion.

Mortality and readmission metrics were addressed by sorting first by the outcome (mortality index), and second by the denominator (total estimated risk). For example, if provider A has the same mortality rate as provider B, then provider A would be ranked higher if the denominator was larger, indicating a higher risk for mortality.

Similarly, it was very common for providers to have the same overall average rating for a survey question. Therefore, the denominator (number of respondents) was used to break ties. However, the denominator sorting was bidirectional. For example, if the tied score was positive (more promoters than detractors) for providers A and B, then provider A would be ranked higher if the denominator was larger. Conversely, if the tied score between providers A and B was neutral or negative (more detractors than promoters), then provider A would be ranked lower if the denominator was larger.

 

Individual Provider Metrics for the PAMM Method

For the PAMM method, model-based metrics were derived using a MM model.⁸ Specifically, let J be the number of rotating providers in a health care system. Let Y_i be an outcome of interest from hospitalization i, X_1i, …, X_pi be fixed effects or covariates, and ß₁, …, ß_p be the coefficients for the respective covariates. Then the generalized MM statistical model is

(Eq. 2)

where g(μ_i) is a link function between the mean of the outcome, μ_i, and its linear predictor, ß₀, is the marginal intercept, w_ij represents the attribution weight of provider j on hospitalization i (described in Equation 1), and γ_j represents the random effect of provider j on the outcome with γ_j~N(0,σ_γ²).

For the mortality and readmission binary outcomes, logistic regression was performed using a logit link function, with the corresponding expected probability as the only fixed covariate. The expected probabilities were first converted into odds and then log-transformed before entering the model. For LOS, Poisson regression was performed using a log link function with the log-transformed expected LOS as the only fixed covariate. For coded patient experience responses, an ordered logistic regression was performed using a cumulative logit link function (no fixed effects were added).

MM Model-based Metrics. Each fitted MM model produces a predicted random effect for each provider. The provider-specific random effects can be interpreted as the unobserved influence of each provider on the outcome after controlling for any fixed effect included in the model. Therefore, the provider-specific random effects were used to evaluate the relative provider performance, which is analogous to the individual provider-level metrics used in the PAPR method.

Measuring provider performance using a MM model is more flexible and robust to outliers compared to the standard approach using OE indices or simple averages. First, although not investigated here, the effect of patient-, visit-, provider-, and/or temporal-level covariates can be controlled when evaluating provider performance. For example, a patient’s socioeconomic status, a provider’s workload, and seasonal factors can be added to the MM model. These external factors are not accounted for in OE indices.

Another advantage of using predicted random effects is the concept of “shrinkage.” The process of estimating random effects inherently accounts for small sample sizes (when providers do not treat a large enough sample of patients) and/or when there is a large ratio of patient variance to provider variance (for instance, when patient outcome variability is much higher compared to provider performance variability). In both cases, the estimation of the random effect is pulled ever closer to 0, signaling that the provider performance is closer to the population average. See Henderson¹⁵ and Mood¹⁶ for further details.

In contrast, OE indices can result in unreliable estimates when a provider has not cared for many patients. This is especially prevalent when the outcome is binary with a low probability of occurring, such as mortality. Indeed, provider-level mortality OE indices are routinely 0 when the patient counts are low, which skews performance rankings unfairly. Finally, OE indices also ignore the magnitude of the variance of an outcome between providers and patients, which can be large.

Comparison Methodology

In this study, we seek to compare the 2 methods of attribution, PAPR and PAMM, to determine whether there are meaningful differences between them when measuring provider performance. Using retrospective data described in the next section, each attribution method was used independently to derive provider-level metrics. To assess relative performance, percentiles were assigned to each provider based on their metric values so that, in the end, there were 2 percentile ranks for each provider for each metric.

Using these paired percentiles, we derived the following measures of concordance, similar to Herzke, Michtalik³: (1) the percent concordance measure—defined as the number of providers who landed in the top half (greater than the median) or bottom half under both attribution models—divided by the total number of providers; (2) the median of the absolute difference in percentiles under both attribution models; and (3) the Pearson correlation coefficient of the paired provider ranks. The first measure is a global measure of concordance between the 2 approaches and would be expected to be 50% by chance. The second measure gauges how an individual provider’s rank is affected by the change in attribution methodologies. The third measure is a statistical measure of linear correlation of the paired percentiles and was not included in the Herzke, Michtalik³ study.

All statistical analyses were performed on SAS (version 9.4; Cary, NC) and the MM models were fitted using PROC GLIMMIX with the EFFECT statement. The Banner Health Institutional Review Board approved this study.

 

Results

Descriptive Statistics

A total of 58,730 hospitalizations were included, of which care was provided by 963 unique providers across 25 acute care and critical access hospitals. Table 1 contains patient characteristics, and Table 2 depicts overall unadjusted outcomes. Providers responsible for less than 12 discharges in the calendar year were excluded from both approaches. Also, some hospitalizations were excluded when expected values were not available.

Multi-Membership Model Results

Table 3 displays the results after independently fitting MM models to each of the 3 clinical outcomes. Along with a marginal intercept, the only covariate in each model was the corresponding expected value after a transformation. This was added to use the same information that is typically used in OE indices, therefore allowing for a proper comparison between the 2 attribution methods. The provider-level variance represents the between-provider variation and measures the amount of influence providers have on the corresponding outcome after controlling for any covariates in the model. A provider-level variance of 0 would indicate that providers do not have any influence on the outcome. While the mortality and readmission model results can be compared to each other, the LOS model cannot given its different scale and transformation altogether.

The results in Table 3 suggest that each expected value covariate is highly correlated with its corresponding outcome, which is the anticipated conclusion given that they are constructed in this fashion. The estimated provider-level variances indicate that, after including an expected value in the model, providers have less of an influence on a patient’s LOS and likelihood of being readmitted. On the other hand, the results suggest that providers have much more influence on the likelihood of a patient dying in the hospital, even after controlling for an expected mortality covariate.

Table 4 shows the results after independently fitting MM-ordered logistic models to each of the 3 survey questions. The similar provider-level variances suggest that providers have the same influence on the patient’s perception of the quality of their interactions with the doctor (Doctor), the quality of the care they received (Care), and their likelihood to recommend a friend or family member to the hospital (NPS).

 

Comparison Results Between Both Attribution Methods

Table 5 compares the 2 attribution methods when ranking providers based on their performance on each outcome measure. The comparison metrics gauge how well the 2 methods agree overall (percent concordance), agree at the provider level (absolute percentile difference and interquartile range [IQR]), and how the paired percentiles linearly correlate to each other (Pearson correlation coefficient).

LOS, by a small margin, had the lowest concordance of clinical outcomes (72.1%), followed by mortality (75.9%) and readmissions (82.1%). Generally, the survey scores had higher percent concordance than the clinical outcome measures, with Doctor at 84.1%, Care at 85.9%, and NPS having the highest percent concordance at 86.6%. Given that by chance the percent concordance is expected to be 50%, there was notable discordance, especially with the clinical outcome measures. Using LOS performance as an example, one attribution methodology would rank a provider in the top half or bottom half, while the other attribution methodology would rank the same provider exactly the opposite way about 28% of the time.

The median absolute percentile difference between the 2 methods was more modest (between 7 and 15). Still, there were some providers whose performance ranking was heavily impacted by the attribution methodology that was used. This was especially true when evaluating performance for certain clinical measures, where the attribution method that was used could change the provider performance percentile by up to 90 levels.

The paired percentiles were positively correlated when ranking performance using any of the 6 measures. This suggests that both methodologies assess performance generally in the same direction, irrespective of the methodology and measure. We did not investigate more complex correlation measures and left this for future research.

It should be noted that ties occurred much more frequently with the PAPR method than when using PAMM and therefore required tie-breaking rules to be designed. Given the nature of OE indices, PAPR methodology is especially sensitive to ties whenever the measure includes counting the number of events (for example, mortality and readmissions) and whenever there are many providers with very few attributed patients. On the other hand, using the PAMM method is much more robust against ties given that the summation of all the weighted attributed outcomes will rarely result in ties, even with a nominal set of providers.

 

 

Discussion

In this study, the PAMM methodology was introduced and was used to assess relative provider performance on 3 clinical outcome measures and 3 patient survey scores. The new approach aims to distribute each outcome among all providers who provided care for a patient in an inpatient setting. Clinical notes were used to account for patient-to-provider interactions, and fitted MM statistical models were used to compute the effects that each provider had on each outcome. The provider effect was introduced as a random effect, and the set of predicted random effects was used to rank the performance of each provider.

The PAMM approach was compared to the more traditional methodology, PAPR, where each patient is attributed to only 1 provider: the discharging physician in this study. Using this approach, OE indices of clinical outcomes and averages of survey scores were used to rank the performance of each provider. This approach resulted in many ties, which were broken based on the number of hospitalizations, although other tie-breaking methods may be used in practice.

Both methodologies showed modest concordance with each other for the clinical outcomes, but higher concordance for the patient survey scores. This was also true when using the Pearson correlation coefficient to assess agreement. The 1 outcome measure that showed the least concordance and least linear correlation between methods was LOS, which would suggest that LOS performance is more sensitive to the attribution methodology that is used. However, it was the least concordant by a small margin.

Furthermore, although the medians of the absolute percentile differences were small, there were some providers who had large deviations, suggesting that some providers would move from being shown as high-performers to low-performers and vice versa based on the chosen attribution method. We investigated examples of this and determined that the root cause was the difference in effective sample sizes for a provider. For the PAPR method, the effective sample size is simply the number of hospitalizations attributed to the provider. For the PAMM method, the effective sample size is the sum of all non-zero weights across all hospitalizations where the provider cared for a patient. By and large, the PAMM methodology provides more information of the provider effect on an outcome than the PAPR approach because every provider-patient interaction is considered. For example, providers who do not routinely discharge patients, but often care for patients, will have rankings that differ dramatically between the 2 methods.

The PAMM methodology has many statistical advantages that were not fully utilized in this comparative study. For example, we did not include any covariates in the MM models except for the expected value of the outcome, when it was available. Still, it is known that other covariates can impact an outcome as well, such as the patient’s age, socioeconomic indicators, existing chronic conditions, and severity of hospitalization, which can be added to the MM models as fixed effects. In this way, the PAMM approach can control for these other covariates, which are typically outside of the control of providers but typically ignored using OE indices. Therefore, using the PAMM approach would provide a fairer comparison of provider performance.

Using the PAMM method, most providers had a large sample size to assess their performance once all the weighted interactions were included. Still, there were a few who did not care for many patients for a variety of reasons. In these scenarios, MM models “borrow” strength from other providers to produce a more robust predicted provider effect by using a weighted average between the overall population trend and the specific provider outcomes (see Rao and Molina¹⁷). As a result, PAMM is a more suitable approach when the sample sizes of patients attributed to providers can be small.

One of the most interesting findings of this study was the relative size of the provider-level variance to the size of the fixed effect in each model (Table 3). Except for mortality, these variances suggest that there is a small difference in performance from one provider to another. However, these should be interpreted as the variance when only 1 provider is involved in the care of a patient. When multiple providers are involved, using basic statistical theory, the overall provider-level variance will be σ_γ² ∑w_ij² (see Equation 2). For example, the estimated variance among providers for LOS was 0.03 (on a log scale), but, using the scenario in the Figure, the overall provider-level variance for this hospitalization will be 0.03 (0.375²+ 0.125²+ 0.5²) = 0.012. Hence, the combined effect of providers on LOS is less than would be expected. Indeed, as more providers are involved with a patient’s care, the more their combined influence on an outcome is diluted.

In this study, the PAMM approach placed an equal weight on all provider-patient interactions via clinical note authorship, but that may not be optimal in some settings. For example, it may make more sense to set a higher weight on the provider who admitted or discharged the patient while placing less (or 0) weight on all other interactions. In the extreme, if the full weight were placed on 1 provider interaction (eg, during discharge, then the MM model would be reduced to a one-way random effects model. The flexibility of weighting interactions is a feature of the PAMM approach, but any weighting framework must be transparent to the providers before implementation.

Conclusion

This study demonstrates that the PAMM approach is a feasible option within a large health care organization. For P4P programs to be successful, providers must be able to trust that their performance will be fairly assessed and that all provider-patient interactions are captured to provide a full comparison amongst their peers. The PAMM methodology is one solution to spread the positive (and negative) outcomes across all providers who cared for a patient and therefore, if implemented, would add trust and fairness when measuring and assessing provider performance.

Acknowledgments: The authors thank Barrie Bradley for his support in the initial stages of this research and Dr. Syed Ismail Jafri for his help and support on the standard approaches of assessing and measuring provider performances.

Corresponding author: Rachel Ginn, MS, Banner Health Corporation, 2901 N. Central Ave., Phoenix, AZ 85012; rachel.ginn@gmail.com.

Financial disclosures: None.

From Banner Health Corporation, Phoenix, AZ.

Background: Health care providers are routinely incentivized with pay-for-performance (P4P) metrics to increase the quality of care. In an inpatient setting, P4P models typically measure quality by attributing each patient’s outcome to a single provider even though many providers routinely care for the patient. This study investigates a new attribution approach aiming to distribute each outcome across all providers who provided care.

Methods: The methodology relies on a multi-membership model and is demonstrated in the Banner Health system using 3 clinical outcome measures (length of stay, 30-day readmissions, and mortality) and responses to 3 survey questions that measure a patient’s perception of their care. The new approach is compared to the “standard” method, which attributes each patient to only 1 provider.

Results: When ranking by clinical outcomes, both methods were concordant 72.1% to 82.1% of the time for top-half/bottom-half rankings, with a median percentile difference between 7 and 15. When ranking by survey scores, there was more agreement, with concordance between 84.1% and 86.6% and a median percentile difference between 11 and 13. Last, Pearson correlation coefficients of the paired percentiles ranged from 0.56 to 0.78.

Conclusion: The new approach provides a fairer solution when measuring provider performance.

Keywords: patient attribution; PAMM; PAPR; random effect model; pay for performance.

Providers practicing in hospitals are routinely evaluated based on their performance and, in many cases, are financially incentivized for a better-than-average performance within a pay-for-performance (P4P) model. The use of P4P models is based on the belief that they will “improve, motivate, and enhance providers to pursue aggressively and ultimately achieve the quality performance targets thus decreasing the number of medical errors with less malpractice events.”¹ Although P4P models continue to be a movement in health care, they have been challenging to implement.

One concern involves the general quality of implementation, such as defining metrics and targets, setting payout amounts, managing technology and market conditions, and gauging the level of transparency to the provider.² Another challenge, and the focus of this project, are concerns around measuring performance to avoid perceptions of unfairness. This concern can be minimized if the attribution is handled in a fairer way, by spreading it across all providers who affected the outcome, both in a positive or negative direction.³

To implement these models, the performance of providers needs to be measured and tracked periodically. This requires linking, or attributing, a patient’s outcome to a provider, which is almost always the attending or discharging provider (ie, a single provider).³ In this single-provider attribution approach, one provider will receive all the credit (good or bad) for their respective patients’ outcomes, even though the provider may have seen the patient only a fraction of the time during the hospitalization. Attributing outcomes—for example, length of stay (LOS), readmission rate, mortality rate, net promoter score (NPS)—using this approach reduces the validity of metrics designed to measure provider performance, especially in a rotating provider environment where many providers interact with and care for a patient. For example, the quality of providers’ interpersonal skills and competence were among the strongest determinants of patient satisfaction,⁴ but it is not credible that this is solely based on the last provider during a hospitalization.

Proportionally distributing the attribution of an outcome has been used successfully in other contexts. Typically, a statistical modeling approach using a multi-membership framework is used because it can handle the sometimes-complicated relationships within the hierarchy. It also allows for auxiliary variables to be introduced, which can help explain and control for exogenous effects.^5-7 For example, in the education setting, standardized testing is administered to students at defined years of schooling: at grades 4, 8, and 10, for instance. The progress of students, measured as the academic gains between test years, are proportionally attributed to all the teachers who the student has had between the test years. These partial attributions are combined to evaluate an overall teacher performance.^8,9

Although the multi-membership framework has been used in other industries, it has yet to be applied in measuring provider performance. The purpose of this project is to investigate the impact of using a multi-provider approach compared to the standard single-provider approach. The findings may lead to modifications in the way a provider’s performance is measured and, thus, how providers are compensated. A similar study investigated the impact of proportionally distributing patients’ outcomes across all rotating providers using a weighting method based on billing practices to measure the partial impact of each provider.³

This study is different in 2 fundamental ways. First, attribution is weighted based on the number of clinically documented interactions (via clinical notes) between a patient and all rotating providers during the hospitalization. Second, performance is measured via multi-membership models, which can estimate the effect (both positive and negative) that a provider has on an outcome, even when caring for a patient a fraction of the time during the hospitalization.

 

Methods

Setting

Banner Health is a non-profit, multi-hospital health care system across 6 states in the western United States that is uniquely positioned to study provider quality attribution models. It not only has a large number of providers and serves a broad patient population, but Banner Health also uses an instance of Cerner (Kansas City, MO), an enterprise-level electronic health record (EHR) system that connects all its facilities and allows for advanced analytics across its system.

For this study, we included only general medicine and surgery patients admitted and discharged from the inpatient setting between January 1, 2018, and December 31, 2018, who were between 18 and 89 years old at admission, and who had a LOS between 1 and 14 days. Visit- and patient-level data were collected from Cerner, while outcome data, and corresponding expected outcome data, were obtained from Premier, Inc. (Charlotte, NC) using their CareScience methodologies.¹⁰ To measure patient experience, response data were extracted from post-discharge surveys administered by InMoment (Salt Lake City, UT).

Provider Attribution Models

Provider Attribution by Physician of Record (PAPR). In the standard approach, denoted here as the PAPR model, 1 provider—typically the attending or discharging provider, which may be the same person—is attributed to the entire hospitalization. This provider is responsible for the patient’s care, and all patient outcomes are aggregated and attributed to the provider to gauge his or her performance. The PAPR model is the most popular form of attribution across many health care systems and is routinely used for P4P incentives.

In this study, the discharging provider was used when attributing hospitalizations using the PAPR model. Providers responsible for fewer than 12 discharges in the calendar year were excluded. Because of the directness of this type of attribution, the performance of 1 provider does not account for the performance of the other rotating providers during hospitalizations.

Provider Attribution by Multiple Membership (PAMM). In contrast, we introduce another attribution approach here that is designed to assign partial attribution to each provider who cares for the patient during the hospitalization. To aggregate the partial attributions, and possibly control for any exogenous or risk-based factors, a multiple-membership, or multi-member (MM), model is used. The MM model can measure the effect of a provider on an outcome even when the patient-to-provider relationship is complex, such as in a rotating provider environment.⁸

The purpose of this study is to compare attribution models and to determine whether there are meaningful differences between them. Therefore, for comparison purposes, the same discharging providers using the PAPR approach are eligible for the PAMM approach, so that both attribution models are using the same set of providers. All other providers are excluded because their performance would not be comparable to the PAPR approach.

While there are many ways to document provider-to-patient interactions, 2 methods are available in almost all health care systems. The first method is to link a provider’s billing charges to each patient-day combination. This approach limits the attribution to 1 provider per patient per day because multiple rotating providers cannot charge for the same patient-day combination.³ However, many providers interact with a patient on the same day, so using this approach excludes non-billed provider-to-patient interactions.

The second method, which was used in this study, relies on documented clinical notes within the EHR to determine how attribution is shared. In this approach, attribution is weighted based on the authorship of 3 types of eligible clinical notes: admitting history/physical notes (during admission), progress notes (during subsequent days), and discharge summary notes (during final discharge). This will (likely) result in many providers being linked to a patient on each day, which better reflects the clinical setting (Figure). Recently, clinical notes were used to attribute care of patients in an inpatient setting, and it was found that this approach provides a reliable way of tracking interactions and assigning ownership.¹¹

The provider-level attribution weights are based on the share of authorships of eligible note types. Specifically, for each provider j, let a_ij be the total count of eligible note types for hospitalization i authored by provider j, and let a_i be the overall total count of eligible note types for hospitalization i. Then the attribution weight is

(Eq. 1)

for hospitalization i and provider j. Note that ∑_jw_ij = 1: in other words, the total attribution, summed across all providers, is constrained to be 1 for each hospitalization.

Patient Outcomes

Outcomes were chosen based on their routine use in health care systems as standards when evaluating provider performance. This study included 6 outcomes: inpatient LOS, inpatient mortality, 30-day inpatient readmission, and patient responses from 3 survey questions. These outcomes can be collected without any manual chart reviews, and therefore are viewed as objective outcomes of provider performance.

Each outcome was aggregated for each provider using both attribution methods independently. For the PAPR method, observed-to-expected (OE) indices for LOS, mortality, and readmissions were calculated along with average patient survey scores. For the PAMM method, provider-level random effects from the fitted models were used. In both cases, the calculated measures were used for ranking purposes when determining top (or bottom) providers for each outcome.

 

Individual Provider Metrics for the PAPR Method

Inpatient LOS Index. Hospital inpatient LOS was measured as the number of days between admission date and discharge date. For each hospital visit, an expected LOS was determined using Premier’s CareScience Analytics (CSA) risk-adjustment methodology.¹⁰ The CSA methodology for LOS incorporates a patient’s clinical history, demographics, and visit-related administrative information.

Let n_j be the number of hospitalizations attributed to provider j. Let o_ij and e_ij be the observed and expected LOS, respectively, for hospitalization i = 1,…,n_j attributed to provider j. Then the inpatient LOS index for provider j is L_j = ∑_io_ij⁄∑_ie_ij.

Inpatient Mortality Index. Inpatient mortality was defined as the death of the patient during hospitalization. For each hospitalization, an expected mortality probability was determined using Premier’s CSA risk-adjustment methodology.¹⁰ The CSA methodology for mortality incorporates a patient’s demographics and comorbidities.

Just as before, let n_j be the number of hospitalizations attributed to provider j. Let m_ij = 1 if the patient died during hospitalization i = 1, … , n_j attributed to provider j; m_ij = 0 otherwise. Let p_ij^(m) be the corresponding expected mortality probability. Then the inpatient mortality index for provider j is M_j = ∑_im_ij⁄∑_ip_ij^(m).

30-Day Inpatient Readmission Index. A 30-day inpatient readmission was defined as the event when a patient is discharged and readmits back into the inpatient setting within 30 days. The inclusion criteria defined by the Centers for Medicare and Medicaid Services (CMS) all-cause hospital-wide readmission measure was used and, consequently, planned readmissions were excluded.¹² Readmissions could occur at any Banner hospital, including the same hospital. For each hospital visit, an expected readmission probability was derived using Premier’s CSA risk-adjustment methodology.¹⁰ The CSA methodology for readmissions incorporates a patient’s clinical history, demographics, and visit-related administrative information.

Let n_j be the number of hospitalizations attributed to provider j. Let r_ij = 1 if the patient had a readmission following hospitalization i = 1, … , n_j attributed to provider j; r_ij = 0 otherwise. Let p_ij^(r) be the corresponding expected readmission probability. Then the 30-day inpatient readmission index for provider j is R_j = ∑_ir_ij ⁄∑_ip_ij^(r).

Patient Survey Scores. The satisfaction of the patient’s experience during hospitalization was measured via post-discharge surveys administered by InMoment. Two survey questions were selected because they related directly to a provider’s interaction with the patient: “My interactions with doctors were excellent” (Doctor) and “I received the best possible care” (Care). A third question, “I would recommend this hospital to my family and friends,” was selected as a proxy measure of the overall experience and, in the aggregate, is referred to as the net promoter score (NPS).^13,14 The responses were measured on an 11-point Likert scale, ranging from “Strongly Disagree” (0) to “Strongly Agree” (10); “N/A” or missing responses were excluded.

The Likert responses were coded to 3 discrete values as follows: if the value was between 0 and 6, then -1 (ie, detractor); between 7 and 8 (ie, neutral), then 0; otherwise 1 (ie, promoter). Averaging these coded responses results in a patient survey score for each question. Specifically, let n_j be the number of hospitalizations attributed to provider j in which the patient responded to the survey question. Let s_ij ∈{−1, 0, 1} be the coded response linked to hospitalization i = 1, … , n_j attributed to provider j. Then the patient experience score for provider j is S_j = ∑_is_ij⁄n_j.

Handling Ties in Provider Performance Measures. Because ties can occur in the PAPR approach for all measures, a tie-breaking strategy is needed. For LOS indices, ties are less likely because their numerator is strictly greater than 0, and expected LOS values are typically distinct enough. Indeed, no ties were found in this study for LOS indices. However, mortality and readmission indices can routinely result in ties when the best possible index is achieved, such as 0 deaths or readmissions among attributed hospitalizations. To help differentiate between those indices in the PAPR approach, the total estimated risk (denominator) was utilized as a secondary scoring criterion.

Mortality and readmission metrics were addressed by sorting first by the outcome (mortality index), and second by the denominator (total estimated risk). For example, if provider A has the same mortality rate as provider B, then provider A would be ranked higher if the denominator was larger, indicating a higher risk for mortality.

Similarly, it was very common for providers to have the same overall average rating for a survey question. Therefore, the denominator (number of respondents) was used to break ties. However, the denominator sorting was bidirectional. For example, if the tied score was positive (more promoters than detractors) for providers A and B, then provider A would be ranked higher if the denominator was larger. Conversely, if the tied score between providers A and B was neutral or negative (more detractors than promoters), then provider A would be ranked lower if the denominator was larger.

 

Individual Provider Metrics for the PAMM Method

For the PAMM method, model-based metrics were derived using a MM model.⁸ Specifically, let J be the number of rotating providers in a health care system. Let Y_i be an outcome of interest from hospitalization i, X_1i, …, X_pi be fixed effects or covariates, and ß₁, …, ß_p be the coefficients for the respective covariates. Then the generalized MM statistical model is

(Eq. 2)

where g(μ_i) is a link function between the mean of the outcome, μ_i, and its linear predictor, ß₀, is the marginal intercept, w_ij represents the attribution weight of provider j on hospitalization i (described in Equation 1), and γ_j represents the random effect of provider j on the outcome with γ_j~N(0,σ_γ²).

For the mortality and readmission binary outcomes, logistic regression was performed using a logit link function, with the corresponding expected probability as the only fixed covariate. The expected probabilities were first converted into odds and then log-transformed before entering the model. For LOS, Poisson regression was performed using a log link function with the log-transformed expected LOS as the only fixed covariate. For coded patient experience responses, an ordered logistic regression was performed using a cumulative logit link function (no fixed effects were added).

MM Model-based Metrics. Each fitted MM model produces a predicted random effect for each provider. The provider-specific random effects can be interpreted as the unobserved influence of each provider on the outcome after controlling for any fixed effect included in the model. Therefore, the provider-specific random effects were used to evaluate the relative provider performance, which is analogous to the individual provider-level metrics used in the PAPR method.

Measuring provider performance using a MM model is more flexible and robust to outliers compared to the standard approach using OE indices or simple averages. First, although not investigated here, the effect of patient-, visit-, provider-, and/or temporal-level covariates can be controlled when evaluating provider performance. For example, a patient’s socioeconomic status, a provider’s workload, and seasonal factors can be added to the MM model. These external factors are not accounted for in OE indices.

Another advantage of using predicted random effects is the concept of “shrinkage.” The process of estimating random effects inherently accounts for small sample sizes (when providers do not treat a large enough sample of patients) and/or when there is a large ratio of patient variance to provider variance (for instance, when patient outcome variability is much higher compared to provider performance variability). In both cases, the estimation of the random effect is pulled ever closer to 0, signaling that the provider performance is closer to the population average. See Henderson¹⁵ and Mood¹⁶ for further details.

In contrast, OE indices can result in unreliable estimates when a provider has not cared for many patients. This is especially prevalent when the outcome is binary with a low probability of occurring, such as mortality. Indeed, provider-level mortality OE indices are routinely 0 when the patient counts are low, which skews performance rankings unfairly. Finally, OE indices also ignore the magnitude of the variance of an outcome between providers and patients, which can be large.

Comparison Methodology

In this study, we seek to compare the 2 methods of attribution, PAPR and PAMM, to determine whether there are meaningful differences between them when measuring provider performance. Using retrospective data described in the next section, each attribution method was used independently to derive provider-level metrics. To assess relative performance, percentiles were assigned to each provider based on their metric values so that, in the end, there were 2 percentile ranks for each provider for each metric.

Using these paired percentiles, we derived the following measures of concordance, similar to Herzke, Michtalik³: (1) the percent concordance measure—defined as the number of providers who landed in the top half (greater than the median) or bottom half under both attribution models—divided by the total number of providers; (2) the median of the absolute difference in percentiles under both attribution models; and (3) the Pearson correlation coefficient of the paired provider ranks. The first measure is a global measure of concordance between the 2 approaches and would be expected to be 50% by chance. The second measure gauges how an individual provider’s rank is affected by the change in attribution methodologies. The third measure is a statistical measure of linear correlation of the paired percentiles and was not included in the Herzke, Michtalik³ study.

All statistical analyses were performed on SAS (version 9.4; Cary, NC) and the MM models were fitted using PROC GLIMMIX with the EFFECT statement. The Banner Health Institutional Review Board approved this study.

 

Results

Descriptive Statistics

A total of 58,730 hospitalizations were included, of which care was provided by 963 unique providers across 25 acute care and critical access hospitals. Table 1 contains patient characteristics, and Table 2 depicts overall unadjusted outcomes. Providers responsible for less than 12 discharges in the calendar year were excluded from both approaches. Also, some hospitalizations were excluded when expected values were not available.

Multi-Membership Model Results

Table 3 displays the results after independently fitting MM models to each of the 3 clinical outcomes. Along with a marginal intercept, the only covariate in each model was the corresponding expected value after a transformation. This was added to use the same information that is typically used in OE indices, therefore allowing for a proper comparison between the 2 attribution methods. The provider-level variance represents the between-provider variation and measures the amount of influence providers have on the corresponding outcome after controlling for any covariates in the model. A provider-level variance of 0 would indicate that providers do not have any influence on the outcome. While the mortality and readmission model results can be compared to each other, the LOS model cannot given its different scale and transformation altogether.

The results in Table 3 suggest that each expected value covariate is highly correlated with its corresponding outcome, which is the anticipated conclusion given that they are constructed in this fashion. The estimated provider-level variances indicate that, after including an expected value in the model, providers have less of an influence on a patient’s LOS and likelihood of being readmitted. On the other hand, the results suggest that providers have much more influence on the likelihood of a patient dying in the hospital, even after controlling for an expected mortality covariate.

Table 4 shows the results after independently fitting MM-ordered logistic models to each of the 3 survey questions. The similar provider-level variances suggest that providers have the same influence on the patient’s perception of the quality of their interactions with the doctor (Doctor), the quality of the care they received (Care), and their likelihood to recommend a friend or family member to the hospital (NPS).

 

Comparison Results Between Both Attribution Methods

Table 5 compares the 2 attribution methods when ranking providers based on their performance on each outcome measure. The comparison metrics gauge how well the 2 methods agree overall (percent concordance), agree at the provider level (absolute percentile difference and interquartile range [IQR]), and how the paired percentiles linearly correlate to each other (Pearson correlation coefficient).

LOS, by a small margin, had the lowest concordance of clinical outcomes (72.1%), followed by mortality (75.9%) and readmissions (82.1%). Generally, the survey scores had higher percent concordance than the clinical outcome measures, with Doctor at 84.1%, Care at 85.9%, and NPS having the highest percent concordance at 86.6%. Given that by chance the percent concordance is expected to be 50%, there was notable discordance, especially with the clinical outcome measures. Using LOS performance as an example, one attribution methodology would rank a provider in the top half or bottom half, while the other attribution methodology would rank the same provider exactly the opposite way about 28% of the time.

The median absolute percentile difference between the 2 methods was more modest (between 7 and 15). Still, there were some providers whose performance ranking was heavily impacted by the attribution methodology that was used. This was especially true when evaluating performance for certain clinical measures, where the attribution method that was used could change the provider performance percentile by up to 90 levels.

The paired percentiles were positively correlated when ranking performance using any of the 6 measures. This suggests that both methodologies assess performance generally in the same direction, irrespective of the methodology and measure. We did not investigate more complex correlation measures and left this for future research.

It should be noted that ties occurred much more frequently with the PAPR method than when using PAMM and therefore required tie-breaking rules to be designed. Given the nature of OE indices, PAPR methodology is especially sensitive to ties whenever the measure includes counting the number of events (for example, mortality and readmissions) and whenever there are many providers with very few attributed patients. On the other hand, using the PAMM method is much more robust against ties given that the summation of all the weighted attributed outcomes will rarely result in ties, even with a nominal set of providers.

 

 

Discussion

In this study, the PAMM methodology was introduced and was used to assess relative provider performance on 3 clinical outcome measures and 3 patient survey scores. The new approach aims to distribute each outcome among all providers who provided care for a patient in an inpatient setting. Clinical notes were used to account for patient-to-provider interactions, and fitted MM statistical models were used to compute the effects that each provider had on each outcome. The provider effect was introduced as a random effect, and the set of predicted random effects was used to rank the performance of each provider.

The PAMM approach was compared to the more traditional methodology, PAPR, where each patient is attributed to only 1 provider: the discharging physician in this study. Using this approach, OE indices of clinical outcomes and averages of survey scores were used to rank the performance of each provider. This approach resulted in many ties, which were broken based on the number of hospitalizations, although other tie-breaking methods may be used in practice.

Both methodologies showed modest concordance with each other for the clinical outcomes, but higher concordance for the patient survey scores. This was also true when using the Pearson correlation coefficient to assess agreement. The 1 outcome measure that showed the least concordance and least linear correlation between methods was LOS, which would suggest that LOS performance is more sensitive to the attribution methodology that is used. However, it was the least concordant by a small margin.

Furthermore, although the medians of the absolute percentile differences were small, there were some providers who had large deviations, suggesting that some providers would move from being shown as high-performers to low-performers and vice versa based on the chosen attribution method. We investigated examples of this and determined that the root cause was the difference in effective sample sizes for a provider. For the PAPR method, the effective sample size is simply the number of hospitalizations attributed to the provider. For the PAMM method, the effective sample size is the sum of all non-zero weights across all hospitalizations where the provider cared for a patient. By and large, the PAMM methodology provides more information of the provider effect on an outcome than the PAPR approach because every provider-patient interaction is considered. For example, providers who do not routinely discharge patients, but often care for patients, will have rankings that differ dramatically between the 2 methods.

The PAMM methodology has many statistical advantages that were not fully utilized in this comparative study. For example, we did not include any covariates in the MM models except for the expected value of the outcome, when it was available. Still, it is known that other covariates can impact an outcome as well, such as the patient’s age, socioeconomic indicators, existing chronic conditions, and severity of hospitalization, which can be added to the MM models as fixed effects. In this way, the PAMM approach can control for these other covariates, which are typically outside of the control of providers but typically ignored using OE indices. Therefore, using the PAMM approach would provide a fairer comparison of provider performance.

Using the PAMM method, most providers had a large sample size to assess their performance once all the weighted interactions were included. Still, there were a few who did not care for many patients for a variety of reasons. In these scenarios, MM models “borrow” strength from other providers to produce a more robust predicted provider effect by using a weighted average between the overall population trend and the specific provider outcomes (see Rao and Molina¹⁷). As a result, PAMM is a more suitable approach when the sample sizes of patients attributed to providers can be small.

One of the most interesting findings of this study was the relative size of the provider-level variance to the size of the fixed effect in each model (Table 3). Except for mortality, these variances suggest that there is a small difference in performance from one provider to another. However, these should be interpreted as the variance when only 1 provider is involved in the care of a patient. When multiple providers are involved, using basic statistical theory, the overall provider-level variance will be σ_γ² ∑w_ij² (see Equation 2). For example, the estimated variance among providers for LOS was 0.03 (on a log scale), but, using the scenario in the Figure, the overall provider-level variance for this hospitalization will be 0.03 (0.375²+ 0.125²+ 0.5²) = 0.012. Hence, the combined effect of providers on LOS is less than would be expected. Indeed, as more providers are involved with a patient’s care, the more their combined influence on an outcome is diluted.

In this study, the PAMM approach placed an equal weight on all provider-patient interactions via clinical note authorship, but that may not be optimal in some settings. For example, it may make more sense to set a higher weight on the provider who admitted or discharged the patient while placing less (or 0) weight on all other interactions. In the extreme, if the full weight were placed on 1 provider interaction (eg, during discharge, then the MM model would be reduced to a one-way random effects model. The flexibility of weighting interactions is a feature of the PAMM approach, but any weighting framework must be transparent to the providers before implementation.

Conclusion

This study demonstrates that the PAMM approach is a feasible option within a large health care organization. For P4P programs to be successful, providers must be able to trust that their performance will be fairly assessed and that all provider-patient interactions are captured to provide a full comparison amongst their peers. The PAMM methodology is one solution to spread the positive (and negative) outcomes across all providers who cared for a patient and therefore, if implemented, would add trust and fairness when measuring and assessing provider performance.

Acknowledgments: The authors thank Barrie Bradley for his support in the initial stages of this research and Dr. Syed Ismail Jafri for his help and support on the standard approaches of assessing and measuring provider performances.

Corresponding author: Rachel Ginn, MS, Banner Health Corporation, 2901 N. Central Ave., Phoenix, AZ 85012; rachel.ginn@gmail.com.

Financial disclosures: None.

From Nova Southeastern University College of Pharmacy, Fort Lauderdale, FL.

Abstract

• Objective: To determine pharmacists’ preferences in bleed risk tool (BRT) usage and gastroprotection when bleed risk was lower than or equal to stroke risk in patients with nonvalvular atrial fibrillation and who were candidates for oral anticoagulation therapy (warfarin or direct oral anticoagulants [DOACs]).
• Methods: A survey consisting of 4 domains (demographics, clinical experience, BRT usage, and treatment preferences based on cases where bleed risk was lower than or equal to stroke risk) was developed. The anonymous survey was disseminated via REDCap software to members of the American College of Clinical Pharmacy ambulatory care and cardiology Practice-based Research Networks. Descriptive statistics were calculated for all study variables and inferential statistics were employed as necessary.
• Results: Of 165 BRT users, 97% preferred HAS-BLED. When bleed risk was lower than stroke risk, 151 respondents chose either DOACs (65%) or warfarin (35%); 15% added gastroprotection. When bleed risk was equal to stroke risk, 141 respondents chose DOACs (50%), warfarin (45%), or aspirin (5%); 40% added gastroprotection.
• Conclusion: In addition to BRT usage, pharmacists were judicious in their recommendation to add gastroprotection and would consider doing so if there was a specific indication. As more than 80% of extracranial bleeds are gastrointestinal bleeds and most BRTs are nonspecific for predicting these bleeds, randomized, prospective studies stratified by HAS-BLED and stroke risk scores are needed to provide further guidance on the efficacy and safety of oral anticoagulation therapy with or without gastroprotection.

Keywords: NVAF; gastroprotection; proton pump inhibitors; warfarin; oral anticoagulants.

Management of patients with nonvalvular atrial fibrillation (NVAF) with oral anticoagulation therapy (OACT) requires constant attention to maintain a balance between preventing strokes and minimizing bleeds. Several validated bleed risk tools (BRTs) available for use in NVAF patients include HAS-BLED, HEMORR₂HAGES, ATRIA, and mOBRI.^1,2 A high bleed risk score is not a contraindication to OACT, but, prior to and throughout therapy, bleed risk should be assessed and modifiable risk factors addressed.³ While intraluminal gastrointestinal (GI) bleeds are not considered a critical bleed site, they are a common complication of chronic OACT and can result in hemodynamic compromise and permanent discontinuation of therapy.^4,5 In 3233 patients with nonvariceal upper GI bleeds (2005-2016), the adjusted odds ratio of hospital admission, transfusion, and re-bleeding while on OACT (warfarin, heparin, or apixaban) was 3.48, 2.53, and 2.26, respectively.⁶ Addition of acid-suppressive therapy with a proton pump inhibitor (PPI) or histamine-2 receptor antagonist (H₂RA) in NVAF patients at increased risk for upper GI bleeds and receiving OACT may result in fewer bleeds.^7,8

Pharmacists play an integral part in managing patients on warfarin,^9-11 and data on their role in managing patients receiving direct oral anticoagulants (DOACs) are increasing.^12-16 Inpatient pharmacists actively participate in multidisciplinary collaborative teams and use clinical decision-support systems or enhanced monitoring to ensure safe prescribing of high-risk medications.^12,15,16 Pharmacist-managed, outpatient-based anticoagulation services in patients on warfarin were associated with lower rates of bleeding and thromboembolic events and lower health care utilization versus routine care.¹⁷ However, it is unclear how pharmacists manage patients who are candidates for OACT but who may be at increased risk for upper GI bleeds. Using a US-based survey, the investigators sought to determine pharmacists’ preferences in BRT usage and gastroprotection when bleed risk was lower than or equal to stroke risk.

 

Methods

This cross-sectional study was conducted after receiving approval by Nova Southeastern University’s Institutional Review Board. The survey consisted of 16 items divided into 4 domains: demographics, clinical experience, use of BRTs, and treatment preferences based on cases where bleed risk was lower than or equal to stroke risk (Figure 1). Queries were multiple choice and allowed for free-text input when “Other” was selected. Licensed pharmacists ≥ 18 years of age who routinely provided care to patients with NVAF were eligible to participate in the study. Participants who reported using a BRT (users) completed all study domains, while participants who reported not using a BRT (nonusers) completed domains 1 through 3 only.

An invitation containing the survey link was sent to the American College of Clinical Pharmacy ambulatory care (n = 2237) and cardiology (n = 1318) pharmacists listed in the organization’s Practice-based Research Networks. The survey was administered in the United States between April and June 2016 via Research Electronic Data Capture (REDCap) software, a secure Web application for building and managing online surveys designed to support data collection for research studies.¹⁸

Survey responses were downloaded, and data were analyzed using NCSS 2019 Statistical Software, LLC (Kaysville, UT). Descriptive statistics were calculated for all study variables. Demographic and clinical experience data for the group that used a BRT versus the group that did not were compared using Pearson’s chi-square, ANOVA, or the Cochran-Armitage test for trends. Logistic regression with hierarchical forward selection with switching was used to identify predictors of drug selection and use of gastroprotection.

Results

Of 230 respondents who completed the survey (response rate 6.5%), 165 (72%) used a BRT and 65 (28%) did not. No significant differences were found for age, gender, duration in clinical practice, the percentage of time spent in patient care, or practice specialty between users and nonusers (Table). The median age of users was 32 years; 68% were females; the median duration in clinical practice was 6 years; 75% of their time was spent in clinical practice; and clinical settings included ambulatory care, cardiology, and internal medicine. A significant difference was found for practice region between users versus nonusers (P = 0.014). Respondents who managed more than 200 NVAF patients per year used a BRT more often than those who managed fewer than 100 NVAF patients per year (P = 0.001).

Of those who used a BRT, 97% utilized the HAS-BLED tool (n = 160). The remainder used HEMORR₂HAGES (n = 3), ATRIA (n = 1), and mOBRI (n = 1). Reasons for choosing HAS-BLED included “familiarity/ease-of-use,” “preference by institution/clinical team,” and the fact that it was a “validated tool for NVAF.”

When bleed risk was lower than stroke risk, 151 of 165 users (92%) chose a treatment option (Figure 2). Of those, 65% chose a DOAC and 35% chose warfarin. Fourteen respondents chose “other” and explained that they “would initiate OACT after weighing patient factors and preferences.” When a DOAC was selected, 9% (n = 9) chose PPI co-therapy and 4% (n = 4) chose a H₂RA. When warfarin was selected, 13% (n = 7) chose PPI co-therapy and 4% (n = 2) chose a H₂RA. Respondents who chose gastroprotection did not provide reasons for doing so, but those who did not add it explained that they “would add gastroprotection only if patient is also on an NSAID or has a history of GI bleed” or cited “patient preference.” Specific to warfarin, some respondents would not add gastroprotection, as anticoagulation with warfarin is “easily reversed.”

When bleed risk was equal to stroke risk, 141 of 165 users (85%) chose a treatment option (Figure 3). Fifty percent chose DOACs, 45% chose warfarin, and 5% chose aspirin. Logistic regression analysis (outcome DOAC versus warfarin area-under-ROC curve, 0.67) showed that as the number of NVAF patients seen in 12 months increased, respondents were more likely to select a DOAC over warfarin (odds ratio, 1.7; 95% CI, 1.1-2.5). Therefore, for every 50-patient increase per year, the probability of recommending a DOAC increased 1.7-fold.

Of respondents who selected either a DOAC or warfarin, 38% (n = 50) also added gastroprotection (Figure 3). When a DOAC was selected, 34% (n = 24) favored PPI co-therapy and 7% (n = 5) chose a H₂RA. When warfarin was selected, 19% (n = 12) favored PPI co-therapy, while 13% (n = 8) chose a H₂RA. Rationale for choosing gastroprotection, regardless of OACT selection, included “stroke is more devastating, so if patient wants to continue treatment, but knew risks of bleeding were similar, would recommend gastroprotection to help minimize bleeding risk” and “patient-specific consideration.” Rationales for not choosing gastroprotection included “would add gastroprotection only if patient is on dual antiplatelet therapy or has another indication”; “in most patients, stroke risk outweighs bleed risk so no need for gastroprotection unless there is a stated reason”; “would use apixaban as has lowest bleeding rate of all DOACs in clinical trials”; and “gastroprotection has not been shown to be beneficial in large scale trials.” 

Eight respondents chose aspirin because it was “easy and relatively low cost.” Twenty-four respondents chose “other” and explained that the choice of OACT depended on patient preference after they had discussed stroke and bleed risk with the patient and/or determined the etiology driving bleed risk.

 

Discussion

This is the first national survey exploring US pharmacists’ preferences in BRT usage and treatment based on bleed risk. Pharmacists preferred the HAS-BLED tool and considered patient-specific factors and evidence-based data when weighing the risk-benefit of OACT with or without gastroprotective therapy.

Similar to our findings, where three-quarters of pharmacists used a BRT, a recent Medscape/American College of Cardiology (ACC) survey reported that 74% of cardiologists used a BRT (eg, HAS-BLED) always/most of the time or sometimes to assess a patient’s overall risk of bleeding prior to initiating DOAC therapy; 27% never or rarely used a bleed risk score before prescribing DOACs.¹⁹ Although reasons for BRT preference were not provided, they may be similar to those reported by our respondents (ie, familiarity/ease-of-use). In both surveys, rationales for not using a BRT were not obtained, but possible reasons include lack of confidence with bleed risk calculators,²⁰ inconsistent implementation of comprehensive assessments (stroke risk, bleed risk, and medication-related issues prior to decision-making),²¹ and nonspecific guideline recommendations.²²

More recently, a network meta-analysis found that HAS-BLED and HEMORR₂HAGES had modest but balanced sensitivity (defined as the ratio between the number of major bleeding events in high-risk stratification and the total number of bleeding events) and specificity (defined as the ratio between the number of nonmajor bleeding events in the low-risk population and total nonbleeding events) for predicting major bleeding events.^2,3 Several respondents did comment that, although HAS-BLED was imprecise and only studied with warfarin, it was necessary to identify bleed risk in a patient starting a high-risk medication, and that the ACC anticoagulation application uses HAS-BLED with CHA₂DS₂VASc along with clinical trial data to estimate stroke risk and bleed risk, with projected risk reduction (strokes) and risk increases (bleeds) expected with each treatment (www.acc.org/tools-and-practice-support/mobile-resources/features/anticoag-evaluator). The 2019 AHA/ACC/HRS atrial fibrillation guideline recommends that HAS-BLED scores be used to assess bleed risk in patients for whom anticoagulation is being considered, and that the need for and choice of OACT should be periodically reevaluated to reassess stroke and bleed risks.²³

Although more than 80% of extracranial bleeds are GI bleeds,²⁴ most BRTs are nonspecific for predicting GI bleeds. Indeed, one respondent used a spreadsheet with several BRTs to maximize treatment guidance for patients with multiple risk factors for strokes and bleeds. A comprehensive approach to determining factors that increase bleed risk should be adopted. These factors include age (HAS-BLED, HEMORR₂HAGES, mOBRI, ATRIA); anemia (mOBRI, HEMORR₂HAGES, ATRIA); hepatic/renal disease (HAS-BLED, HEMORR₂HAGES, ATRIA, mOBRI); concomitant medications/alcohol use, including NSAIDs, corticosteroids, and antiplatelet therapy (HAS-BLED, HEMORR₂HAGES); bleed history/rebleeding risk (HEMORR₂HAGES, HAS-BLED, ATRIA); and GI bleeds (mOBRI).^1,2 Additional risk factors for GI bleeds include being a tobacco smoker and/or being infected with Helicobacter pylori. A prospective cohort study that analyzed data from questionnaires completed by 99,359 individuals from the Copenhagen General Population Study reported that the multivariable adjusted hazard ratio for current smokers versus never smokers was 2.20 (95% CI, 1.84-2.62) for GI bleeds.²⁵ Presence of H pylori should be investigated, with a subsequent eradication regimen implemented, as patients with warfarin-associated upper GI bleeds who were H pylori-positive had lower HAS-BLED scores versus those who were negative.²⁶

When bleed risk was lower than stroke risk (eg, HAS-BLED < 3, CHA₂DS₂VASc ≥ 1), respondents appropriately initiated therapy with an OAC (predominantly apixaban); a small proportion also added gastroprotection. If the patient did not have any other GI bleed risk factors (eg, a previous GI bleed or on chronic antiplatelet or NSAID therapy), the choice of OACT depended on the attributes of each OAC and patient preference.²⁷ Selection of warfarin was appropriate if cost, formulary restrictions, and availability of an inexpensive reversal agent were important concerns to patients and/or their health care providers. Rivaroxaban was selected because of its once-daily dosing and low risk for GI bleeding.

 

The recently published ARISTOPHANES study provides evidence that apixaban is an appropriate choice in patients with a HAS-BLED score < 3. In this retrospective observational study, more than 70% of patients received standard doses of DOACs (apixaban 5 mg, dabigatran 150 mg, or rivaroxaban 20 mg) and about 20% had a bleeding history, about 30% were on PPIs, less than 25% were on NSAIDs, and about 40% had a HAS-BLED score < 3. The study found that apixaban was more effective (reduced rates of ischemic or hemorrhagic strokes/systemic embolism) and safer (reduced rates of major GI bleed or intracranial bleed) than warfarin.²⁸ Dabigatran and rivaroxaban were also more effective than warfarin for stroke prevention and had a lower risk for major intracranial bleed risk; while the risk of major GI bleed was similar between dabigatran and warfarin, major GI bleed risk was higher for rivaroxaban. When compared with each other, the 3 DOACs were effective at stroke prevention, with apixaban more effective than dabigatran and rivaroxaban; similar efficacy was noted for dabigatran versus rivaroxaban. Apixaban was associated with fewer GI bleeds versus dabigatran and rivaroxaban, but with similar intracranial bleed risks; dabigatran was associated with fewer GI bleeds but similar intracranial bleed risks versus rivaroxaban.²⁸ Efficacy and safety findings from a subgroup analysis based on HAS-BLED scores < 3 and ≥ 3 were generally consistent with the main results.

When bleed risk was equal to stroke risk, the difficulty was determining how OACT in a patient at high stroke risk (CHA₂DS₂VASc score ≥ 2) and high bleed risk (HAS-BLED score ≥ 3) should be managed. Eight respondents chose aspirin and added gastroprotection with either a PPI or H₂RA; however, currently, aspirin is not recommended as the sole antithrombotic for patients with NVAF.²³ With the OACT, an interesting finding was that as the number of patients seen in 12 months increased, pharmacists were almost twice as likely to select a DOAC over warfarin. Moreover, pharmacists were judicious in their recommendation to add gastroprotection, and would consider doing so if there was a specific indication. At the time of our survey, several studies described DOAC-associated GI bleeds,^29-31 but data on the effectiveness of acid-suppressive therapy, specifically with PPIs, in the prevention of upper GI bleeds were sparse.^4,7,32 Respondents most likely were familiar with GI bleed risk factors and prevention strategies from various guidelines published between 2009 and 2010, which did not include DOACs.^33-35

Another important finding was pharmacists’ uncertainty as to the effectiveness of PPIs in preventing GI bleeds in combination with DOACs. The data are conflicting. A meta-analysis of older studies (2007-2015) showed that PPIs (but not H₂RAs) reduced the risk of upper GI bleeds in patients on warfarin but not for dabigatran.³⁶ A retrospective cohort study of Medicare beneficiaries on OACTs (2011-2015) showed the adjusted incidence of hospitalization for upper GI bleeds in patients on PPI co-therapy was significantly lower compared with patients not on PPI co-therapy (76 versus 115 per 10,000 person-years, respectively).⁸ Apixaban without PPI co-therapy was associated with the lowest risk of upper GI bleed hospitalizations (73/10,000 person-years), and PPI co-therapy further reduced this risk (49/10,000 person-years). Warfarin without PPI co-therapy was associated with the next lowest risk (113/10,000 person-years), followed by dabigatran (120/10,000 person-years) and rivaroxaban (144/10,000 person-years). PPI co-therapy significantly reduced the risk of upper GI bleed hospitalizations with all OACTs, but the incidence of upper GI bleed hospitalizations with rivaroxaban was significantly greater than with the other OACTs.⁸ Therefore, if there are concerns about the safety of PPIs,^37-39 or the patient is unable to tolerate a PPI, then apixaban may be the most appropriate DOAC for a patient with high bleed risk. Notably, a 2020 review of data from the PINNACLE registry (average age, 75-77 years; 31% on PPIs) found that the relative GI bleed safety advantage of apixaban and dabigatran versus warfarin was attenuated in patients ≥ 75 years.⁴⁰ Last, since the risk for lower GI bleeds is not reduced by PPIs,⁴¹ consideration of their use should be accompanied by an assessment to detect bleeds (eg, low hemoglobin/hematocrit, presence of bright red blood, hematochezia/melena, fecal occult testing), with prompt management as necessary.⁵

Limitations

Limitations of our survey included an overall low response rate, which can generate a biased sample if respondents are systematically different from nonrespondents. In addition, to maintain simplicity and reduce respondents’ time commitment, our survey did not include actual CHA₂DS₂VASc stroke risk scores, HAS-BLED bleed risk scores, or specific GI bleed risk factors when querying pharmacists about treatment options based on bleed risk. The addition of these variables would have improved the robustness of the data.

Conclusion

In addition to applying BRTs in the management of NVAF patients, pharmacists considered patient-specific variables, prescriber preferences, and evidence-based guidance when recommending OACT with or without gastroprotection. To avoid suboptimal patient management, busy pharmacists should be granted time to attend continuing education programs describing optimal OACT selection and formulation of individualized, evidence-based plans to address modifiable risk factors for bleeding, including the appropriate use of gastroprotection. Randomized, prospective, long-term studies stratified by HAS-BLED and CHA₂DS₂VASc scores are needed to further clarify efficacy, safety, and cost-effectiveness of OACT, with and without PPIs, in patients who may be at risk for upper GI bleeds.

Acknowledgments: The authors thank Robin J. Jacobs, PhD, MSW, MS, MPH, Patrick C. Hardigan, PhD, Steven Brettler, PharmD, MPH, Maria-Isabel A. Cabral, PharmD, and Reginald Gyapong, PharmD, for their participation in this project. The authors also sincerely thank Fabio Franco, BS Computer Science, who organized the database to enable efficient data management.

Corresponding author: Devada Singh-Franco, PharmD, CDE, Nova Southeastern University College of Pharmacy, 3200 S University Drive, Fort Lauderdale, FL 33328; singh@nova.edu

Disclosures: None.

Funding: The study was supported by Nova Southeastern University’s Health Professions Division Internal Research Grant.

From Nova Southeastern University College of Pharmacy, Fort Lauderdale, FL.

Abstract

• Objective: To determine pharmacists’ preferences in bleed risk tool (BRT) usage and gastroprotection when bleed risk was lower than or equal to stroke risk in patients with nonvalvular atrial fibrillation and who were candidates for oral anticoagulation therapy (warfarin or direct oral anticoagulants [DOACs]).
• Methods: A survey consisting of 4 domains (demographics, clinical experience, BRT usage, and treatment preferences based on cases where bleed risk was lower than or equal to stroke risk) was developed. The anonymous survey was disseminated via REDCap software to members of the American College of Clinical Pharmacy ambulatory care and cardiology Practice-based Research Networks. Descriptive statistics were calculated for all study variables and inferential statistics were employed as necessary.
• Results: Of 165 BRT users, 97% preferred HAS-BLED. When bleed risk was lower than stroke risk, 151 respondents chose either DOACs (65%) or warfarin (35%); 15% added gastroprotection. When bleed risk was equal to stroke risk, 141 respondents chose DOACs (50%), warfarin (45%), or aspirin (5%); 40% added gastroprotection.
• Conclusion: In addition to BRT usage, pharmacists were judicious in their recommendation to add gastroprotection and would consider doing so if there was a specific indication. As more than 80% of extracranial bleeds are gastrointestinal bleeds and most BRTs are nonspecific for predicting these bleeds, randomized, prospective studies stratified by HAS-BLED and stroke risk scores are needed to provide further guidance on the efficacy and safety of oral anticoagulation therapy with or without gastroprotection.

Keywords: NVAF; gastroprotection; proton pump inhibitors; warfarin; oral anticoagulants.

Management of patients with nonvalvular atrial fibrillation (NVAF) with oral anticoagulation therapy (OACT) requires constant attention to maintain a balance between preventing strokes and minimizing bleeds. Several validated bleed risk tools (BRTs) available for use in NVAF patients include HAS-BLED, HEMORR₂HAGES, ATRIA, and mOBRI.^1,2 A high bleed risk score is not a contraindication to OACT, but, prior to and throughout therapy, bleed risk should be assessed and modifiable risk factors addressed.³ While intraluminal gastrointestinal (GI) bleeds are not considered a critical bleed site, they are a common complication of chronic OACT and can result in hemodynamic compromise and permanent discontinuation of therapy.^4,5 In 3233 patients with nonvariceal upper GI bleeds (2005-2016), the adjusted odds ratio of hospital admission, transfusion, and re-bleeding while on OACT (warfarin, heparin, or apixaban) was 3.48, 2.53, and 2.26, respectively.⁶ Addition of acid-suppressive therapy with a proton pump inhibitor (PPI) or histamine-2 receptor antagonist (H₂RA) in NVAF patients at increased risk for upper GI bleeds and receiving OACT may result in fewer bleeds.^7,8

Pharmacists play an integral part in managing patients on warfarin,^9-11 and data on their role in managing patients receiving direct oral anticoagulants (DOACs) are increasing.^12-16 Inpatient pharmacists actively participate in multidisciplinary collaborative teams and use clinical decision-support systems or enhanced monitoring to ensure safe prescribing of high-risk medications.^12,15,16 Pharmacist-managed, outpatient-based anticoagulation services in patients on warfarin were associated with lower rates of bleeding and thromboembolic events and lower health care utilization versus routine care.¹⁷ However, it is unclear how pharmacists manage patients who are candidates for OACT but who may be at increased risk for upper GI bleeds. Using a US-based survey, the investigators sought to determine pharmacists’ preferences in BRT usage and gastroprotection when bleed risk was lower than or equal to stroke risk.

 

Methods

This cross-sectional study was conducted after receiving approval by Nova Southeastern University’s Institutional Review Board. The survey consisted of 16 items divided into 4 domains: demographics, clinical experience, use of BRTs, and treatment preferences based on cases where bleed risk was lower than or equal to stroke risk (Figure 1). Queries were multiple choice and allowed for free-text input when “Other” was selected. Licensed pharmacists ≥ 18 years of age who routinely provided care to patients with NVAF were eligible to participate in the study. Participants who reported using a BRT (users) completed all study domains, while participants who reported not using a BRT (nonusers) completed domains 1 through 3 only.

An invitation containing the survey link was sent to the American College of Clinical Pharmacy ambulatory care (n = 2237) and cardiology (n = 1318) pharmacists listed in the organization’s Practice-based Research Networks. The survey was administered in the United States between April and June 2016 via Research Electronic Data Capture (REDCap) software, a secure Web application for building and managing online surveys designed to support data collection for research studies.¹⁸

Survey responses were downloaded, and data were analyzed using NCSS 2019 Statistical Software, LLC (Kaysville, UT). Descriptive statistics were calculated for all study variables. Demographic and clinical experience data for the group that used a BRT versus the group that did not were compared using Pearson’s chi-square, ANOVA, or the Cochran-Armitage test for trends. Logistic regression with hierarchical forward selection with switching was used to identify predictors of drug selection and use of gastroprotection.

Results

Of 230 respondents who completed the survey (response rate 6.5%), 165 (72%) used a BRT and 65 (28%) did not. No significant differences were found for age, gender, duration in clinical practice, the percentage of time spent in patient care, or practice specialty between users and nonusers (Table). The median age of users was 32 years; 68% were females; the median duration in clinical practice was 6 years; 75% of their time was spent in clinical practice; and clinical settings included ambulatory care, cardiology, and internal medicine. A significant difference was found for practice region between users versus nonusers (P = 0.014). Respondents who managed more than 200 NVAF patients per year used a BRT more often than those who managed fewer than 100 NVAF patients per year (P = 0.001).

Of those who used a BRT, 97% utilized the HAS-BLED tool (n = 160). The remainder used HEMORR₂HAGES (n = 3), ATRIA (n = 1), and mOBRI (n = 1). Reasons for choosing HAS-BLED included “familiarity/ease-of-use,” “preference by institution/clinical team,” and the fact that it was a “validated tool for NVAF.”

When bleed risk was lower than stroke risk, 151 of 165 users (92%) chose a treatment option (Figure 2). Of those, 65% chose a DOAC and 35% chose warfarin. Fourteen respondents chose “other” and explained that they “would initiate OACT after weighing patient factors and preferences.” When a DOAC was selected, 9% (n = 9) chose PPI co-therapy and 4% (n = 4) chose a H₂RA. When warfarin was selected, 13% (n = 7) chose PPI co-therapy and 4% (n = 2) chose a H₂RA. Respondents who chose gastroprotection did not provide reasons for doing so, but those who did not add it explained that they “would add gastroprotection only if patient is also on an NSAID or has a history of GI bleed” or cited “patient preference.” Specific to warfarin, some respondents would not add gastroprotection, as anticoagulation with warfarin is “easily reversed.”

When bleed risk was equal to stroke risk, 141 of 165 users (85%) chose a treatment option (Figure 3). Fifty percent chose DOACs, 45% chose warfarin, and 5% chose aspirin. Logistic regression analysis (outcome DOAC versus warfarin area-under-ROC curve, 0.67) showed that as the number of NVAF patients seen in 12 months increased, respondents were more likely to select a DOAC over warfarin (odds ratio, 1.7; 95% CI, 1.1-2.5). Therefore, for every 50-patient increase per year, the probability of recommending a DOAC increased 1.7-fold.

Of respondents who selected either a DOAC or warfarin, 38% (n = 50) also added gastroprotection (Figure 3). When a DOAC was selected, 34% (n = 24) favored PPI co-therapy and 7% (n = 5) chose a H₂RA. When warfarin was selected, 19% (n = 12) favored PPI co-therapy, while 13% (n = 8) chose a H₂RA. Rationale for choosing gastroprotection, regardless of OACT selection, included “stroke is more devastating, so if patient wants to continue treatment, but knew risks of bleeding were similar, would recommend gastroprotection to help minimize bleeding risk” and “patient-specific consideration.” Rationales for not choosing gastroprotection included “would add gastroprotection only if patient is on dual antiplatelet therapy or has another indication”; “in most patients, stroke risk outweighs bleed risk so no need for gastroprotection unless there is a stated reason”; “would use apixaban as has lowest bleeding rate of all DOACs in clinical trials”; and “gastroprotection has not been shown to be beneficial in large scale trials.” 

Eight respondents chose aspirin because it was “easy and relatively low cost.” Twenty-four respondents chose “other” and explained that the choice of OACT depended on patient preference after they had discussed stroke and bleed risk with the patient and/or determined the etiology driving bleed risk.

 

Discussion

This is the first national survey exploring US pharmacists’ preferences in BRT usage and treatment based on bleed risk. Pharmacists preferred the HAS-BLED tool and considered patient-specific factors and evidence-based data when weighing the risk-benefit of OACT with or without gastroprotective therapy.

Similar to our findings, where three-quarters of pharmacists used a BRT, a recent Medscape/American College of Cardiology (ACC) survey reported that 74% of cardiologists used a BRT (eg, HAS-BLED) always/most of the time or sometimes to assess a patient’s overall risk of bleeding prior to initiating DOAC therapy; 27% never or rarely used a bleed risk score before prescribing DOACs.¹⁹ Although reasons for BRT preference were not provided, they may be similar to those reported by our respondents (ie, familiarity/ease-of-use). In both surveys, rationales for not using a BRT were not obtained, but possible reasons include lack of confidence with bleed risk calculators,²⁰ inconsistent implementation of comprehensive assessments (stroke risk, bleed risk, and medication-related issues prior to decision-making),²¹ and nonspecific guideline recommendations.²²

More recently, a network meta-analysis found that HAS-BLED and HEMORR₂HAGES had modest but balanced sensitivity (defined as the ratio between the number of major bleeding events in high-risk stratification and the total number of bleeding events) and specificity (defined as the ratio between the number of nonmajor bleeding events in the low-risk population and total nonbleeding events) for predicting major bleeding events.^2,3 Several respondents did comment that, although HAS-BLED was imprecise and only studied with warfarin, it was necessary to identify bleed risk in a patient starting a high-risk medication, and that the ACC anticoagulation application uses HAS-BLED with CHA₂DS₂VASc along with clinical trial data to estimate stroke risk and bleed risk, with projected risk reduction (strokes) and risk increases (bleeds) expected with each treatment (www.acc.org/tools-and-practice-support/mobile-resources/features/anticoag-evaluator). The 2019 AHA/ACC/HRS atrial fibrillation guideline recommends that HAS-BLED scores be used to assess bleed risk in patients for whom anticoagulation is being considered, and that the need for and choice of OACT should be periodically reevaluated to reassess stroke and bleed risks.²³

Although more than 80% of extracranial bleeds are GI bleeds,²⁴ most BRTs are nonspecific for predicting GI bleeds. Indeed, one respondent used a spreadsheet with several BRTs to maximize treatment guidance for patients with multiple risk factors for strokes and bleeds. A comprehensive approach to determining factors that increase bleed risk should be adopted. These factors include age (HAS-BLED, HEMORR₂HAGES, mOBRI, ATRIA); anemia (mOBRI, HEMORR₂HAGES, ATRIA); hepatic/renal disease (HAS-BLED, HEMORR₂HAGES, ATRIA, mOBRI); concomitant medications/alcohol use, including NSAIDs, corticosteroids, and antiplatelet therapy (HAS-BLED, HEMORR₂HAGES); bleed history/rebleeding risk (HEMORR₂HAGES, HAS-BLED, ATRIA); and GI bleeds (mOBRI).^1,2 Additional risk factors for GI bleeds include being a tobacco smoker and/or being infected with Helicobacter pylori. A prospective cohort study that analyzed data from questionnaires completed by 99,359 individuals from the Copenhagen General Population Study reported that the multivariable adjusted hazard ratio for current smokers versus never smokers was 2.20 (95% CI, 1.84-2.62) for GI bleeds.²⁵ Presence of H pylori should be investigated, with a subsequent eradication regimen implemented, as patients with warfarin-associated upper GI bleeds who were H pylori-positive had lower HAS-BLED scores versus those who were negative.²⁶

When bleed risk was lower than stroke risk (eg, HAS-BLED < 3, CHA₂DS₂VASc ≥ 1), respondents appropriately initiated therapy with an OAC (predominantly apixaban); a small proportion also added gastroprotection. If the patient did not have any other GI bleed risk factors (eg, a previous GI bleed or on chronic antiplatelet or NSAID therapy), the choice of OACT depended on the attributes of each OAC and patient preference.²⁷ Selection of warfarin was appropriate if cost, formulary restrictions, and availability of an inexpensive reversal agent were important concerns to patients and/or their health care providers. Rivaroxaban was selected because of its once-daily dosing and low risk for GI bleeding.

 

The recently published ARISTOPHANES study provides evidence that apixaban is an appropriate choice in patients with a HAS-BLED score < 3. In this retrospective observational study, more than 70% of patients received standard doses of DOACs (apixaban 5 mg, dabigatran 150 mg, or rivaroxaban 20 mg) and about 20% had a bleeding history, about 30% were on PPIs, less than 25% were on NSAIDs, and about 40% had a HAS-BLED score < 3. The study found that apixaban was more effective (reduced rates of ischemic or hemorrhagic strokes/systemic embolism) and safer (reduced rates of major GI bleed or intracranial bleed) than warfarin.²⁸ Dabigatran and rivaroxaban were also more effective than warfarin for stroke prevention and had a lower risk for major intracranial bleed risk; while the risk of major GI bleed was similar between dabigatran and warfarin, major GI bleed risk was higher for rivaroxaban. When compared with each other, the 3 DOACs were effective at stroke prevention, with apixaban more effective than dabigatran and rivaroxaban; similar efficacy was noted for dabigatran versus rivaroxaban. Apixaban was associated with fewer GI bleeds versus dabigatran and rivaroxaban, but with similar intracranial bleed risks; dabigatran was associated with fewer GI bleeds but similar intracranial bleed risks versus rivaroxaban.²⁸ Efficacy and safety findings from a subgroup analysis based on HAS-BLED scores < 3 and ≥ 3 were generally consistent with the main results.

When bleed risk was equal to stroke risk, the difficulty was determining how OACT in a patient at high stroke risk (CHA₂DS₂VASc score ≥ 2) and high bleed risk (HAS-BLED score ≥ 3) should be managed. Eight respondents chose aspirin and added gastroprotection with either a PPI or H₂RA; however, currently, aspirin is not recommended as the sole antithrombotic for patients with NVAF.²³ With the OACT, an interesting finding was that as the number of patients seen in 12 months increased, pharmacists were almost twice as likely to select a DOAC over warfarin. Moreover, pharmacists were judicious in their recommendation to add gastroprotection, and would consider doing so if there was a specific indication. At the time of our survey, several studies described DOAC-associated GI bleeds,^29-31 but data on the effectiveness of acid-suppressive therapy, specifically with PPIs, in the prevention of upper GI bleeds were sparse.^4,7,32 Respondents most likely were familiar with GI bleed risk factors and prevention strategies from various guidelines published between 2009 and 2010, which did not include DOACs.^33-35

Another important finding was pharmacists’ uncertainty as to the effectiveness of PPIs in preventing GI bleeds in combination with DOACs. The data are conflicting. A meta-analysis of older studies (2007-2015) showed that PPIs (but not H₂RAs) reduced the risk of upper GI bleeds in patients on warfarin but not for dabigatran.³⁶ A retrospective cohort study of Medicare beneficiaries on OACTs (2011-2015) showed the adjusted incidence of hospitalization for upper GI bleeds in patients on PPI co-therapy was significantly lower compared with patients not on PPI co-therapy (76 versus 115 per 10,000 person-years, respectively).⁸ Apixaban without PPI co-therapy was associated with the lowest risk of upper GI bleed hospitalizations (73/10,000 person-years), and PPI co-therapy further reduced this risk (49/10,000 person-years). Warfarin without PPI co-therapy was associated with the next lowest risk (113/10,000 person-years), followed by dabigatran (120/10,000 person-years) and rivaroxaban (144/10,000 person-years). PPI co-therapy significantly reduced the risk of upper GI bleed hospitalizations with all OACTs, but the incidence of upper GI bleed hospitalizations with rivaroxaban was significantly greater than with the other OACTs.⁸ Therefore, if there are concerns about the safety of PPIs,^37-39 or the patient is unable to tolerate a PPI, then apixaban may be the most appropriate DOAC for a patient with high bleed risk. Notably, a 2020 review of data from the PINNACLE registry (average age, 75-77 years; 31% on PPIs) found that the relative GI bleed safety advantage of apixaban and dabigatran versus warfarin was attenuated in patients ≥ 75 years.⁴⁰ Last, since the risk for lower GI bleeds is not reduced by PPIs,⁴¹ consideration of their use should be accompanied by an assessment to detect bleeds (eg, low hemoglobin/hematocrit, presence of bright red blood, hematochezia/melena, fecal occult testing), with prompt management as necessary.⁵

Limitations

Limitations of our survey included an overall low response rate, which can generate a biased sample if respondents are systematically different from nonrespondents. In addition, to maintain simplicity and reduce respondents’ time commitment, our survey did not include actual CHA₂DS₂VASc stroke risk scores, HAS-BLED bleed risk scores, or specific GI bleed risk factors when querying pharmacists about treatment options based on bleed risk. The addition of these variables would have improved the robustness of the data.

Conclusion

In addition to applying BRTs in the management of NVAF patients, pharmacists considered patient-specific variables, prescriber preferences, and evidence-based guidance when recommending OACT with or without gastroprotection. To avoid suboptimal patient management, busy pharmacists should be granted time to attend continuing education programs describing optimal OACT selection and formulation of individualized, evidence-based plans to address modifiable risk factors for bleeding, including the appropriate use of gastroprotection. Randomized, prospective, long-term studies stratified by HAS-BLED and CHA₂DS₂VASc scores are needed to further clarify efficacy, safety, and cost-effectiveness of OACT, with and without PPIs, in patients who may be at risk for upper GI bleeds.

Acknowledgments: The authors thank Robin J. Jacobs, PhD, MSW, MS, MPH, Patrick C. Hardigan, PhD, Steven Brettler, PharmD, MPH, Maria-Isabel A. Cabral, PharmD, and Reginald Gyapong, PharmD, for their participation in this project. The authors also sincerely thank Fabio Franco, BS Computer Science, who organized the database to enable efficient data management.

Corresponding author: Devada Singh-Franco, PharmD, CDE, Nova Southeastern University College of Pharmacy, 3200 S University Drive, Fort Lauderdale, FL 33328; singh@nova.edu

Disclosures: None.

Funding: The study was supported by Nova Southeastern University’s Health Professions Division Internal Research Grant.

From Nova Southeastern University College of Pharmacy, Fort Lauderdale, FL.

Abstract

• Objective: To determine pharmacists’ preferences in bleed risk tool (BRT) usage and gastroprotection when bleed risk was lower than or equal to stroke risk in patients with nonvalvular atrial fibrillation and who were candidates for oral anticoagulation therapy (warfarin or direct oral anticoagulants [DOACs]).
• Methods: A survey consisting of 4 domains (demographics, clinical experience, BRT usage, and treatment preferences based on cases where bleed risk was lower than or equal to stroke risk) was developed. The anonymous survey was disseminated via REDCap software to members of the American College of Clinical Pharmacy ambulatory care and cardiology Practice-based Research Networks. Descriptive statistics were calculated for all study variables and inferential statistics were employed as necessary.
• Results: Of 165 BRT users, 97% preferred HAS-BLED. When bleed risk was lower than stroke risk, 151 respondents chose either DOACs (65%) or warfarin (35%); 15% added gastroprotection. When bleed risk was equal to stroke risk, 141 respondents chose DOACs (50%), warfarin (45%), or aspirin (5%); 40% added gastroprotection.
• Conclusion: In addition to BRT usage, pharmacists were judicious in their recommendation to add gastroprotection and would consider doing so if there was a specific indication. As more than 80% of extracranial bleeds are gastrointestinal bleeds and most BRTs are nonspecific for predicting these bleeds, randomized, prospective studies stratified by HAS-BLED and stroke risk scores are needed to provide further guidance on the efficacy and safety of oral anticoagulation therapy with or without gastroprotection.

Keywords: NVAF; gastroprotection; proton pump inhibitors; warfarin; oral anticoagulants.

Management of patients with nonvalvular atrial fibrillation (NVAF) with oral anticoagulation therapy (OACT) requires constant attention to maintain a balance between preventing strokes and minimizing bleeds. Several validated bleed risk tools (BRTs) available for use in NVAF patients include HAS-BLED, HEMORR₂HAGES, ATRIA, and mOBRI.^1,2 A high bleed risk score is not a contraindication to OACT, but, prior to and throughout therapy, bleed risk should be assessed and modifiable risk factors addressed.³ While intraluminal gastrointestinal (GI) bleeds are not considered a critical bleed site, they are a common complication of chronic OACT and can result in hemodynamic compromise and permanent discontinuation of therapy.^4,5 In 3233 patients with nonvariceal upper GI bleeds (2005-2016), the adjusted odds ratio of hospital admission, transfusion, and re-bleeding while on OACT (warfarin, heparin, or apixaban) was 3.48, 2.53, and 2.26, respectively.⁶ Addition of acid-suppressive therapy with a proton pump inhibitor (PPI) or histamine-2 receptor antagonist (H₂RA) in NVAF patients at increased risk for upper GI bleeds and receiving OACT may result in fewer bleeds.^7,8

Pharmacists play an integral part in managing patients on warfarin,^9-11 and data on their role in managing patients receiving direct oral anticoagulants (DOACs) are increasing.^12-16 Inpatient pharmacists actively participate in multidisciplinary collaborative teams and use clinical decision-support systems or enhanced monitoring to ensure safe prescribing of high-risk medications.^12,15,16 Pharmacist-managed, outpatient-based anticoagulation services in patients on warfarin were associated with lower rates of bleeding and thromboembolic events and lower health care utilization versus routine care.¹⁷ However, it is unclear how pharmacists manage patients who are candidates for OACT but who may be at increased risk for upper GI bleeds. Using a US-based survey, the investigators sought to determine pharmacists’ preferences in BRT usage and gastroprotection when bleed risk was lower than or equal to stroke risk.

 

Methods

This cross-sectional study was conducted after receiving approval by Nova Southeastern University’s Institutional Review Board. The survey consisted of 16 items divided into 4 domains: demographics, clinical experience, use of BRTs, and treatment preferences based on cases where bleed risk was lower than or equal to stroke risk (Figure 1). Queries were multiple choice and allowed for free-text input when “Other” was selected. Licensed pharmacists ≥ 18 years of age who routinely provided care to patients with NVAF were eligible to participate in the study. Participants who reported using a BRT (users) completed all study domains, while participants who reported not using a BRT (nonusers) completed domains 1 through 3 only.

An invitation containing the survey link was sent to the American College of Clinical Pharmacy ambulatory care (n = 2237) and cardiology (n = 1318) pharmacists listed in the organization’s Practice-based Research Networks. The survey was administered in the United States between April and June 2016 via Research Electronic Data Capture (REDCap) software, a secure Web application for building and managing online surveys designed to support data collection for research studies.¹⁸

Survey responses were downloaded, and data were analyzed using NCSS 2019 Statistical Software, LLC (Kaysville, UT). Descriptive statistics were calculated for all study variables. Demographic and clinical experience data for the group that used a BRT versus the group that did not were compared using Pearson’s chi-square, ANOVA, or the Cochran-Armitage test for trends. Logistic regression with hierarchical forward selection with switching was used to identify predictors of drug selection and use of gastroprotection.

Results

Of 230 respondents who completed the survey (response rate 6.5%), 165 (72%) used a BRT and 65 (28%) did not. No significant differences were found for age, gender, duration in clinical practice, the percentage of time spent in patient care, or practice specialty between users and nonusers (Table). The median age of users was 32 years; 68% were females; the median duration in clinical practice was 6 years; 75% of their time was spent in clinical practice; and clinical settings included ambulatory care, cardiology, and internal medicine. A significant difference was found for practice region between users versus nonusers (P = 0.014). Respondents who managed more than 200 NVAF patients per year used a BRT more often than those who managed fewer than 100 NVAF patients per year (P = 0.001).

Of those who used a BRT, 97% utilized the HAS-BLED tool (n = 160). The remainder used HEMORR₂HAGES (n = 3), ATRIA (n = 1), and mOBRI (n = 1). Reasons for choosing HAS-BLED included “familiarity/ease-of-use,” “preference by institution/clinical team,” and the fact that it was a “validated tool for NVAF.”

When bleed risk was lower than stroke risk, 151 of 165 users (92%) chose a treatment option (Figure 2). Of those, 65% chose a DOAC and 35% chose warfarin. Fourteen respondents chose “other” and explained that they “would initiate OACT after weighing patient factors and preferences.” When a DOAC was selected, 9% (n = 9) chose PPI co-therapy and 4% (n = 4) chose a H₂RA. When warfarin was selected, 13% (n = 7) chose PPI co-therapy and 4% (n = 2) chose a H₂RA. Respondents who chose gastroprotection did not provide reasons for doing so, but those who did not add it explained that they “would add gastroprotection only if patient is also on an NSAID or has a history of GI bleed” or cited “patient preference.” Specific to warfarin, some respondents would not add gastroprotection, as anticoagulation with warfarin is “easily reversed.”

When bleed risk was equal to stroke risk, 141 of 165 users (85%) chose a treatment option (Figure 3). Fifty percent chose DOACs, 45% chose warfarin, and 5% chose aspirin. Logistic regression analysis (outcome DOAC versus warfarin area-under-ROC curve, 0.67) showed that as the number of NVAF patients seen in 12 months increased, respondents were more likely to select a DOAC over warfarin (odds ratio, 1.7; 95% CI, 1.1-2.5). Therefore, for every 50-patient increase per year, the probability of recommending a DOAC increased 1.7-fold.

Of respondents who selected either a DOAC or warfarin, 38% (n = 50) also added gastroprotection (Figure 3). When a DOAC was selected, 34% (n = 24) favored PPI co-therapy and 7% (n = 5) chose a H₂RA. When warfarin was selected, 19% (n = 12) favored PPI co-therapy, while 13% (n = 8) chose a H₂RA. Rationale for choosing gastroprotection, regardless of OACT selection, included “stroke is more devastating, so if patient wants to continue treatment, but knew risks of bleeding were similar, would recommend gastroprotection to help minimize bleeding risk” and “patient-specific consideration.” Rationales for not choosing gastroprotection included “would add gastroprotection only if patient is on dual antiplatelet therapy or has another indication”; “in most patients, stroke risk outweighs bleed risk so no need for gastroprotection unless there is a stated reason”; “would use apixaban as has lowest bleeding rate of all DOACs in clinical trials”; and “gastroprotection has not been shown to be beneficial in large scale trials.” 

Eight respondents chose aspirin because it was “easy and relatively low cost.” Twenty-four respondents chose “other” and explained that the choice of OACT depended on patient preference after they had discussed stroke and bleed risk with the patient and/or determined the etiology driving bleed risk.

 

Discussion

This is the first national survey exploring US pharmacists’ preferences in BRT usage and treatment based on bleed risk. Pharmacists preferred the HAS-BLED tool and considered patient-specific factors and evidence-based data when weighing the risk-benefit of OACT with or without gastroprotective therapy.

Similar to our findings, where three-quarters of pharmacists used a BRT, a recent Medscape/American College of Cardiology (ACC) survey reported that 74% of cardiologists used a BRT (eg, HAS-BLED) always/most of the time or sometimes to assess a patient’s overall risk of bleeding prior to initiating DOAC therapy; 27% never or rarely used a bleed risk score before prescribing DOACs.¹⁹ Although reasons for BRT preference were not provided, they may be similar to those reported by our respondents (ie, familiarity/ease-of-use). In both surveys, rationales for not using a BRT were not obtained, but possible reasons include lack of confidence with bleed risk calculators,²⁰ inconsistent implementation of comprehensive assessments (stroke risk, bleed risk, and medication-related issues prior to decision-making),²¹ and nonspecific guideline recommendations.²²

More recently, a network meta-analysis found that HAS-BLED and HEMORR₂HAGES had modest but balanced sensitivity (defined as the ratio between the number of major bleeding events in high-risk stratification and the total number of bleeding events) and specificity (defined as the ratio between the number of nonmajor bleeding events in the low-risk population and total nonbleeding events) for predicting major bleeding events.^2,3 Several respondents did comment that, although HAS-BLED was imprecise and only studied with warfarin, it was necessary to identify bleed risk in a patient starting a high-risk medication, and that the ACC anticoagulation application uses HAS-BLED with CHA₂DS₂VASc along with clinical trial data to estimate stroke risk and bleed risk, with projected risk reduction (strokes) and risk increases (bleeds) expected with each treatment (www.acc.org/tools-and-practice-support/mobile-resources/features/anticoag-evaluator). The 2019 AHA/ACC/HRS atrial fibrillation guideline recommends that HAS-BLED scores be used to assess bleed risk in patients for whom anticoagulation is being considered, and that the need for and choice of OACT should be periodically reevaluated to reassess stroke and bleed risks.²³

Although more than 80% of extracranial bleeds are GI bleeds,²⁴ most BRTs are nonspecific for predicting GI bleeds. Indeed, one respondent used a spreadsheet with several BRTs to maximize treatment guidance for patients with multiple risk factors for strokes and bleeds. A comprehensive approach to determining factors that increase bleed risk should be adopted. These factors include age (HAS-BLED, HEMORR₂HAGES, mOBRI, ATRIA); anemia (mOBRI, HEMORR₂HAGES, ATRIA); hepatic/renal disease (HAS-BLED, HEMORR₂HAGES, ATRIA, mOBRI); concomitant medications/alcohol use, including NSAIDs, corticosteroids, and antiplatelet therapy (HAS-BLED, HEMORR₂HAGES); bleed history/rebleeding risk (HEMORR₂HAGES, HAS-BLED, ATRIA); and GI bleeds (mOBRI).^1,2 Additional risk factors for GI bleeds include being a tobacco smoker and/or being infected with Helicobacter pylori. A prospective cohort study that analyzed data from questionnaires completed by 99,359 individuals from the Copenhagen General Population Study reported that the multivariable adjusted hazard ratio for current smokers versus never smokers was 2.20 (95% CI, 1.84-2.62) for GI bleeds.²⁵ Presence of H pylori should be investigated, with a subsequent eradication regimen implemented, as patients with warfarin-associated upper GI bleeds who were H pylori-positive had lower HAS-BLED scores versus those who were negative.²⁶

When bleed risk was lower than stroke risk (eg, HAS-BLED < 3, CHA₂DS₂VASc ≥ 1), respondents appropriately initiated therapy with an OAC (predominantly apixaban); a small proportion also added gastroprotection. If the patient did not have any other GI bleed risk factors (eg, a previous GI bleed or on chronic antiplatelet or NSAID therapy), the choice of OACT depended on the attributes of each OAC and patient preference.²⁷ Selection of warfarin was appropriate if cost, formulary restrictions, and availability of an inexpensive reversal agent were important concerns to patients and/or their health care providers. Rivaroxaban was selected because of its once-daily dosing and low risk for GI bleeding.

 

The recently published ARISTOPHANES study provides evidence that apixaban is an appropriate choice in patients with a HAS-BLED score < 3. In this retrospective observational study, more than 70% of patients received standard doses of DOACs (apixaban 5 mg, dabigatran 150 mg, or rivaroxaban 20 mg) and about 20% had a bleeding history, about 30% were on PPIs, less than 25% were on NSAIDs, and about 40% had a HAS-BLED score < 3. The study found that apixaban was more effective (reduced rates of ischemic or hemorrhagic strokes/systemic embolism) and safer (reduced rates of major GI bleed or intracranial bleed) than warfarin.²⁸ Dabigatran and rivaroxaban were also more effective than warfarin for stroke prevention and had a lower risk for major intracranial bleed risk; while the risk of major GI bleed was similar between dabigatran and warfarin, major GI bleed risk was higher for rivaroxaban. When compared with each other, the 3 DOACs were effective at stroke prevention, with apixaban more effective than dabigatran and rivaroxaban; similar efficacy was noted for dabigatran versus rivaroxaban. Apixaban was associated with fewer GI bleeds versus dabigatran and rivaroxaban, but with similar intracranial bleed risks; dabigatran was associated with fewer GI bleeds but similar intracranial bleed risks versus rivaroxaban.²⁸ Efficacy and safety findings from a subgroup analysis based on HAS-BLED scores < 3 and ≥ 3 were generally consistent with the main results.

When bleed risk was equal to stroke risk, the difficulty was determining how OACT in a patient at high stroke risk (CHA₂DS₂VASc score ≥ 2) and high bleed risk (HAS-BLED score ≥ 3) should be managed. Eight respondents chose aspirin and added gastroprotection with either a PPI or H₂RA; however, currently, aspirin is not recommended as the sole antithrombotic for patients with NVAF.²³ With the OACT, an interesting finding was that as the number of patients seen in 12 months increased, pharmacists were almost twice as likely to select a DOAC over warfarin. Moreover, pharmacists were judicious in their recommendation to add gastroprotection, and would consider doing so if there was a specific indication. At the time of our survey, several studies described DOAC-associated GI bleeds,^29-31 but data on the effectiveness of acid-suppressive therapy, specifically with PPIs, in the prevention of upper GI bleeds were sparse.^4,7,32 Respondents most likely were familiar with GI bleed risk factors and prevention strategies from various guidelines published between 2009 and 2010, which did not include DOACs.^33-35

Another important finding was pharmacists’ uncertainty as to the effectiveness of PPIs in preventing GI bleeds in combination with DOACs. The data are conflicting. A meta-analysis of older studies (2007-2015) showed that PPIs (but not H₂RAs) reduced the risk of upper GI bleeds in patients on warfarin but not for dabigatran.³⁶ A retrospective cohort study of Medicare beneficiaries on OACTs (2011-2015) showed the adjusted incidence of hospitalization for upper GI bleeds in patients on PPI co-therapy was significantly lower compared with patients not on PPI co-therapy (76 versus 115 per 10,000 person-years, respectively).⁸ Apixaban without PPI co-therapy was associated with the lowest risk of upper GI bleed hospitalizations (73/10,000 person-years), and PPI co-therapy further reduced this risk (49/10,000 person-years). Warfarin without PPI co-therapy was associated with the next lowest risk (113/10,000 person-years), followed by dabigatran (120/10,000 person-years) and rivaroxaban (144/10,000 person-years). PPI co-therapy significantly reduced the risk of upper GI bleed hospitalizations with all OACTs, but the incidence of upper GI bleed hospitalizations with rivaroxaban was significantly greater than with the other OACTs.⁸ Therefore, if there are concerns about the safety of PPIs,^37-39 or the patient is unable to tolerate a PPI, then apixaban may be the most appropriate DOAC for a patient with high bleed risk. Notably, a 2020 review of data from the PINNACLE registry (average age, 75-77 years; 31% on PPIs) found that the relative GI bleed safety advantage of apixaban and dabigatran versus warfarin was attenuated in patients ≥ 75 years.⁴⁰ Last, since the risk for lower GI bleeds is not reduced by PPIs,⁴¹ consideration of their use should be accompanied by an assessment to detect bleeds (eg, low hemoglobin/hematocrit, presence of bright red blood, hematochezia/melena, fecal occult testing), with prompt management as necessary.⁵

Limitations

Limitations of our survey included an overall low response rate, which can generate a biased sample if respondents are systematically different from nonrespondents. In addition, to maintain simplicity and reduce respondents’ time commitment, our survey did not include actual CHA₂DS₂VASc stroke risk scores, HAS-BLED bleed risk scores, or specific GI bleed risk factors when querying pharmacists about treatment options based on bleed risk. The addition of these variables would have improved the robustness of the data.

Conclusion

In addition to applying BRTs in the management of NVAF patients, pharmacists considered patient-specific variables, prescriber preferences, and evidence-based guidance when recommending OACT with or without gastroprotection. To avoid suboptimal patient management, busy pharmacists should be granted time to attend continuing education programs describing optimal OACT selection and formulation of individualized, evidence-based plans to address modifiable risk factors for bleeding, including the appropriate use of gastroprotection. Randomized, prospective, long-term studies stratified by HAS-BLED and CHA₂DS₂VASc scores are needed to further clarify efficacy, safety, and cost-effectiveness of OACT, with and without PPIs, in patients who may be at risk for upper GI bleeds.

Acknowledgments: The authors thank Robin J. Jacobs, PhD, MSW, MS, MPH, Patrick C. Hardigan, PhD, Steven Brettler, PharmD, MPH, Maria-Isabel A. Cabral, PharmD, and Reginald Gyapong, PharmD, for their participation in this project. The authors also sincerely thank Fabio Franco, BS Computer Science, who organized the database to enable efficient data management.

Corresponding author: Devada Singh-Franco, PharmD, CDE, Nova Southeastern University College of Pharmacy, 3200 S University Drive, Fort Lauderdale, FL 33328; singh@nova.edu

Disclosures: None.

Funding: The study was supported by Nova Southeastern University’s Health Professions Division Internal Research Grant.

Bacterial infections are a common reason for pediatric hospital admissions in the United States.¹ Antibiotics are the mainstay of treatment, and whether to administer them intravenously (IV) or enterally is an important and, at times, challenging decision. Not all hospitalized patients with infections require IV antibiotics, and safe, effective early transitions to enteral therapy have been described for numerous infections.^2-7 However, guidelines describing the ideal initial route of antibiotic administration and when to transition to oral therapy are lacking.^5,7,8 This lack of high-quality evidence-based guidance may contribute to overuse of IV antibiotics for many hospitalized pediatric patients, even when safe and effective enteral options exist.⁹

Significant costs and harms are associated with the use of IV antibiotics. In particular, studies have demonstrated longer length of stay (LOS), increased costs, and worsened pain or anxiety related to complications (eg, phlebitis, extravasation injury, thrombosis, catheter-associated bloodstream infections) associated with IV antibiotics.^3,4,10-13 Earlier transition to enteral therapy, however, can mitigate these increased risks and costs.

The Centers for Disease Control and Prevention lists the transition from IV to oral antibiotics as a key stewardship intervention for improving antibiotic use.¹⁴ The Infectious Diseases Society of America (IDSA) antibiotic stewardship program guidelines strongly recommend the timely conversion from IV to oral antibiotics, stating that efforts focusing on this transition should be integrated into routine practice.¹⁵ There are a few metrics in the literature to measure this intervention, but none is universally used, and a modified delphi process could not reach consensus on IV-to-oral transition metrics.¹⁶

Few studies describe the opportunity to transition to enteral antibiotics in hospitalized patients with common bacterial infections or explore variation across hospitals. It is critical to understand current practice of antibiotic administration in order to identify opportunities to optimize patient outcomes and promote high-value care. Furthermore, few studies have evaluated the feasibility of IV-to-oral transition metrics using an administrative database. Thus, the aims of this study were to (1) determine opportunities to transition from IV to enteral antibiotics for pediatric patients hospitalized with common bacterial infections based on their ability to tolerate other enteral medications, (2) describe variation in transition practices among children’s hospitals, and (3) evaluate the feasibility of novel IV-to-oral transition metrics using an administrative database to inform stewardship efforts.

METHODS

Study Design and Setting

This multicenter, retrospective cohort study used data from the Pediatric Health Information System (PHIS), an administrative and billing database containing encounter-level data from 52 tertiary care pediatric hospitals across the United States affiliated with the Children’s Hospital Association (Lenexa, Kansas). Hospitals submit encounter-level data, including demographics, medications, and diagnoses based on International Classification of Disease, Tenth Revision, Clinical Modification (ICD-10-CM) codes. Data were de-identified at the time of submission, and data quality and reliability were assured by joint efforts between the Children’s Hospital Association and participating hospitals.

Study Population

This study included pediatric patients aged 60 days to 18 years who were hospitalized (inpatient or observation status) at one of the participating hospitals between January 1, 2017, and December 31, 2018, for one of the following seven common bacterial infections: community-acquired pneumonia (CAP), neck infection (superficial and deep), periorbital/orbital infection, urinary tract infection (UTI), osteomyelitis, septic arthritis, or skin and soft tissue infection (SSTI). The diagnosis cohorts were defined based on ICD-10-CM discharge diagnoses adapted from previous studies (Appendix Table 1).^3,17-23 To define a cohort of generally healthy pediatric patients with an acute infection, we excluded patients hospitalized in the intensive care unit, patients with nonhome discharges, and patients with complex chronic conditions.²⁴ We also excluded hospitals with incomplete data during the study period (n=1). The Institutional Review Board at Cincinnati Children’s Hospital Medical Center determined this study to be non–human-subjects research.

Outcomes

The primary outcomes were the number of opportunity days and the percent of days with opportunity to transition from IV to enteral therapy. Opportunity days, or days in which there was a potential opportunity to transition from IV to enteral antibiotics, were defined as days patients received only IV antibiotic doses and at least one enteral nonantibiotic medication, suggesting an ability to take enteral medications.¹³ We excluded days patients received IV antibiotics for which there was no enteral alternative (eg, vancomycin, Appendix Table 2). When measuring opportunity, to be conservative (ie, to underestimate rather than overestimate opportunity), we did not count as an opportunity day any day in which patients received both IV and enteral antibiotics. Percent opportunity, or the percent of days patients received antibiotics in which there was potential opportunity to transition from IV to enteral antibiotics, was defined as the number of opportunity days divided by number of inpatient days patients received enteral antibiotics or IV antibiotics with at least one enteral nonantibiotic medication (antibiotic days). Similar to opportunity days, antibiotic days excluded days patients were on IV antibiotics for which there was no enteral alternative. Based on our definition, a lower percent opportunity indicates that a hospital is using enteral antibiotics earlier during the hospitalization (earlier transition), while a higher percent opportunity represents later enteral antibiotic use (later transition).

Statistical Analysis

Demographic and clinical characteristics were summarized by diagnosis with descriptive statistics, including frequency with percentage, mean with standard deviation, and median with interquartile range (IQR). For each diagnosis, we evaluated aggregate opportunity days (sum of opportunity days among all hospitals), opportunity days per encounter, and aggregate percent opportunity using frequencies, mean with standard deviation, and percentages, respectively. We also calculated aggregate opportunity days for diagnosis-antibiotic combinations. To visually show variation in the percent opportunity across hospitals, we displayed the percent opportunity on a heat map, and evaluated percent opportunity across hospitals using chi-square tests. To compare the variability in the percent opportunity across and within hospitals, we used a generalized linear model with two fixed effects (hospital and diagnosis), and parsed the variability using the sum of squares. We performed a sensitivity analysis and excluded days that patients received antiemetic medications (eg, ondansetron, granisetron, prochlorperazine, promethazine), as these suggest potential intolerance of enteral medications. All statistical analyses were performed using SAS v.9.4 (SAS Institute Inc, Cary, North Carolina) and GraphPad Prism 8.0 (GraphPad Software Inc., San Diego, California), and P values < .05 were considered statistically significant.

RESULTS

During the 2-year study period, 100,103 hospitalizations met our inclusion criteria across 51 hospitals and seven diagnosis categories (Table 1). Diagnosis cohorts ranged in size from 1,462 encounters for septic arthritis to 35,665 encounters for neck infections. Overall, we identified 88,522 aggregate opportunity days on which there was an opportunity to switch from IV to enteral treatment in the majority of participants (percent opportunity, 57%).

Opportunity by Diagnosis

The number of opportunity days (aggregate and mean per encounter) and percent opportunity varied by diagnosis (Table 2). The aggregate number of opportunity days ranged from 3,693 in patients with septic arthritis to 25,359 in patients with SSTI, and mean opportunity days per encounter ranged from 0.9 in CAP to 2.8 in septic arthritis. Percent opportunity was highest for septic arthritis at 72.7% and lowest for CAP at 39.7%.

Variation in Opportunity Among Hospitals

The variation in the percent opportunity across hospitals was statistically significant for all diagnoses (Figure). Within hospitals, we observed similar practice patterns across diagnoses. For example, hospitals with a higher percent opportunity for one diagnosis tended to have higher percent opportunity for the other diagnoses (as noted in the top portion of the Figure), and those with lower percent opportunity for one diagnosis tended to also have lower percent opportunity for the other diagnoses studied (as noted in the bottom portion of the Figure). When evaluating variability in the percent opportunity, 45% of the variability was attributable to the hospital-effect and 35% to the diagnosis; the remainder was unexplained variability. Sensitivity analysis excluding days when patients received an antiemetic medication yielded no differences in our results.

Opportunity by Antibiotic

The aggregate number of opportunity days varied by antibiotic (Table 3). Intravenous antibiotics with the largest number of opportunity days included clindamycin (44,293), ceftriaxone (23,896), and ampicillin-sulbactam (15,484). Antibiotic-diagnosis combinations with the largest number of opportunity days for each diagnosis included ceftriaxone and ampicillin in CAP; clindamycin in cellulitis, SSTI, and neck infections; ceftriaxone in UTI; and cefazolin in osteomyelitis and septic arthritis.

DISCUSSION

In this multicenter study of pediatric patients hospitalized with common bacterial infections, there was the potential to transition from IV to enteral treatment in over half of the antibiotic days. The degree of opportunity varied by infection, antibiotic, and hospital. Antibiotics with a large aggregate number of opportunity days for enteral transition included clindamycin, which has excellent bioavailability; and ampicillin and ampicillin-sulbactam, which can achieve pharmacodynamic targets with oral equivalents.^25-29 The across-hospital variation for a given diagnosis suggests that certain hospitals have strategies in place which permit an earlier transition to enteral antibiotics compared to other institutions in which there were likely missed opportunities to do so. This variability is likely due to limited evidence, emphasizing the need for robust studies to better understand the optimal initial antibiotic route and transition time. Our findings highlight the need for, and large potential impact of, stewardship efforts to promote earlier transition for specific drug targets. This study also demonstrates the feasibility of obtaining two metrics—percent opportunity and opportunity days—from administrative databases to inform stewardship efforts within and across hospitals.

Opportunity days and percent opportunity varied among diagnoses. The variation in aggregate opportunity days was largely a reflection of the number of encounters: Diagnoses such as SSTI, neck infections, and CAP had a large number of both aggregate opportunity days and encounters. The range of opportunity days per encounter (0.9-2.5) suggests potential missed opportunities to transition to enteral antibiotics across all diagnoses (Table 2). The higher opportunity days per encounter in osteomyelitis and septic arthritis may be related to longer LOS and higher percent opportunity. Percent opportunity likely varied among diagnoses due to differences in admission and discharge readiness criteria, diagnostic evaluation, frequency of antibiotic administration, and evidence on the optimal route of initial antibiotics and when to transition to oral formulations. For example, we hypothesize that certain diagnoses, such as osteomyelitis and septic arthritis, have admission and discharge readiness criteria directly tied to the perceived need for IV antibiotics, which may limit in-hospital days on enteral antibiotics and explain the high percent opportunity that we observed. The high percent opportunity seen in musculoskeletal infections also may be due to delays in initiating targeted treatment until culture results were available. Encounters for CAP had the lowest percent opportunity; we hypothesize that this is because admission and discharge readiness may be determined by factors other than the need for IV antibiotics (eg, need for supplemental oxygen), which may increase days on enteral antibiotics and lead to a lower percent opportunity.³⁰

Urinary tract infection encounters had a high percent opportunity. As with musculoskeletal infection, this may be related to delays in initiating targeted treatment until culture results became available. Another reason for the high percent opportunity in UTI could be the common use of ceftriaxone, which, dosed every 24 hours, likely reduced the opportunity to transition to enteral antibiotics. There is strong evidence demonstrating no difference in outcomes based on antibiotic routes for UTI, and we would expect this to result in a low percent opportunity.^2,31 While the observed high opportunity in UTI may relate to an initial unknown diagnosis or concern for systemic infection, this highlights potential opportunities for quality improvement initiatives to promote empiric oral antibiotics in clinically stable patients hospitalized with suspected UTI.

There was substantial variation in percent opportunity across hospitals for a given diagnosis, with less variation across diagnoses for a given hospital. Variation across hospitals but consistency within individual hospitals suggests that some hospitals may promote earlier transition from IV to enteral antibiotics as standard practice for all diagnoses, while other hospitals continue IV antibiotics for the entire hospitalization, highlighting potential missed opportunities at some institutions. While emerging data suggest that traditional long durations of IV antibiotics are not necessary for many infections, the limited evidence on the optimal time to switch to oral antibiotics may have influenced this variation.^2-7 Many guidelines recommend initial IV antibiotics for hospitalized pediatric patients, but there are few studies comparing IV and enteral therapy.^2,5,9 Limited evidence leaves significant room for hospital culture, antibiotic stewardship efforts, reimbursement considerations, and/or hospital workflow to influence transition timing and overall opportunity at individual hospitals.^7,8,32-34 These findings emphasize the importance of research to identify optimal transition time and comparative effectiveness studies to evaluate whether initial IV antibiotics are truly needed for mild—and even severe—disease presentations. Since many patients are admitted for the perceived need for IV antibiotics, earlier use of enteral antibiotics could reduce rates of hospitalizations, LOS, healthcare costs, and resource utilization.

Antibiotics with a high number of opportunity days included clindamycin, ceftriaxone, ampicillin-sublactam, and ampicillin. Our findings are consistent with another study which found that most bioavailable drugs, including clindamycin, were administered via the IV route and accounted for a large number of antibiotic days.³⁵ The Infectious Diseases Society of America recommends that hospitals promote earlier transition to oral formulations for highly bioavailable drugs.⁷ Given the high bioavailability of clindamycin, its common use in high-frequency encounters such as SSTI and neck infections, and the fact that it accounted for a large number of opportunity days, quality improvement initiatives promoting earlier transition to oral clindamycin could have a large impact across health systems.^25,26 Additionally, although beta-lactam antibiotics such as amoxicillin and amoxicillin-sulbactam are not highly bioavailable, oral dosing can achieve sufficient serum concentrations to reach pharmacodynamic targets for common clinical indications; this could be an important quality improvement initiative.^27-29 Several single-site studies have successfully implemented quality improvement initiatives to promote earlier IV-to-enteral transition, with resulting reductions in costs and no adverse events noted, highlighting the feasibility and impact of such efforts.^13,36-38

This study also demonstrates the feasibility of collecting two metrics (percent opportunity and opportunity days) from administrative databases to inform IV-to-oral transition benchmarking and stewardship efforts. While there are several metrics in the literature for evaluating antibiotic transition (eg, days of IV or oral therapy, percentage of antibiotics given via the oral route, time to switch from IV to oral, and acceptance rate of suggested changes to antibiotic route), none are universally used or agreed upon.^15,16,39 The opportunity metrics used in this study have several strengths, including the feasibility of obtaining them from existing databases and the ability to account for intake of other enteral medications; the latter is not evaluated in other metrics. These opportunity metrics can be used together to identify the percent of time in which there is opportunity to transition and total number of days to understand the full extent of potential opportunity for future interventions. As demonstrated in this study, these metrics can be measured by diagnosis, antibiotic, or diagnosis-antibiotic combination, and they can be used to evaluate stewardship efforts at a single institution over time or compare efforts across hospitals.

These findings should be interpreted in the context of important limitations. First, we attempted to characterize potential opportunity to transition to enteral medications based on a patient’s ability to tolerate nonenteral medications. However, there are other factors that could limit the opportunity to transition that we could not account for with an administrative dataset, including the use of antibiotics prior to admission, disease progression, severity of illness, and malabsorptive concerns. Thus, though we may have overestimated the true opportunity to transition to enteral antibiotics, it is unlikely that this would account for all of the variation in transition times that we observed across hospitals. Second, while our study required patients to have one of seven types of infection, we did not exclude any additional infectious diagnoses (eg, concurrent bacteremia, Clostridioides difficile, otitis media) that could have driven the choice of antibiotic type and modality. Although emerging evidence is supporting earlier transitions to oral therapy, bacteremia is typically treated with IV antibiotics; this may have led to an overestimation of true opportunity.⁴⁰ “Clostridioides” difficile and otitis media are typically treated with enteral therapy; concurrent infections such as these may have led to an underestimation of opportunity given the fact that, based on our definition, the days on which patients received both IV and enteral antibiotics were not counted as opportunity days. Third, because PHIS uses billing days to capture medication use, we were unable to distinguish transitions that occurred early in the day vs those that took place later in the day. This could have led to an underestimation of percent opportunity, particularly for diagnoses with a short LOS; it also likely led to an underestimation of the variability observed across hospitals. Fourth, because we used an administrative dataset, we are unable to understand reasoning behind transitioning time from IV to oral antibiotics, as well as provider, patient, and institutional level factors that influenced these decisions.

CONCLUSION

Children hospitalized with bacterial infections often receive IV antibiotics, and the timing of transition from IV to enteral antibiotics varies significantly across hospitals. Further research is needed to compare the effectiveness of IV and enteral antibiotics and better define criteria for transition to enteral therapy. We identified ample opportunities for quality improvement initiatives to promote earlier transition, which have the potential to reduce healthcare utilization and promote optimal patient-directed high-value care.

Bacterial infections are a common reason for pediatric hospital admissions in the United States.¹ Antibiotics are the mainstay of treatment, and whether to administer them intravenously (IV) or enterally is an important and, at times, challenging decision. Not all hospitalized patients with infections require IV antibiotics, and safe, effective early transitions to enteral therapy have been described for numerous infections.^2-7 However, guidelines describing the ideal initial route of antibiotic administration and when to transition to oral therapy are lacking.^5,7,8 This lack of high-quality evidence-based guidance may contribute to overuse of IV antibiotics for many hospitalized pediatric patients, even when safe and effective enteral options exist.⁹

Significant costs and harms are associated with the use of IV antibiotics. In particular, studies have demonstrated longer length of stay (LOS), increased costs, and worsened pain or anxiety related to complications (eg, phlebitis, extravasation injury, thrombosis, catheter-associated bloodstream infections) associated with IV antibiotics.^3,4,10-13 Earlier transition to enteral therapy, however, can mitigate these increased risks and costs.

The Centers for Disease Control and Prevention lists the transition from IV to oral antibiotics as a key stewardship intervention for improving antibiotic use.¹⁴ The Infectious Diseases Society of America (IDSA) antibiotic stewardship program guidelines strongly recommend the timely conversion from IV to oral antibiotics, stating that efforts focusing on this transition should be integrated into routine practice.¹⁵ There are a few metrics in the literature to measure this intervention, but none is universally used, and a modified delphi process could not reach consensus on IV-to-oral transition metrics.¹⁶

Few studies describe the opportunity to transition to enteral antibiotics in hospitalized patients with common bacterial infections or explore variation across hospitals. It is critical to understand current practice of antibiotic administration in order to identify opportunities to optimize patient outcomes and promote high-value care. Furthermore, few studies have evaluated the feasibility of IV-to-oral transition metrics using an administrative database. Thus, the aims of this study were to (1) determine opportunities to transition from IV to enteral antibiotics for pediatric patients hospitalized with common bacterial infections based on their ability to tolerate other enteral medications, (2) describe variation in transition practices among children’s hospitals, and (3) evaluate the feasibility of novel IV-to-oral transition metrics using an administrative database to inform stewardship efforts.

METHODS

Study Design and Setting

This multicenter, retrospective cohort study used data from the Pediatric Health Information System (PHIS), an administrative and billing database containing encounter-level data from 52 tertiary care pediatric hospitals across the United States affiliated with the Children’s Hospital Association (Lenexa, Kansas). Hospitals submit encounter-level data, including demographics, medications, and diagnoses based on International Classification of Disease, Tenth Revision, Clinical Modification (ICD-10-CM) codes. Data were de-identified at the time of submission, and data quality and reliability were assured by joint efforts between the Children’s Hospital Association and participating hospitals.

Study Population

This study included pediatric patients aged 60 days to 18 years who were hospitalized (inpatient or observation status) at one of the participating hospitals between January 1, 2017, and December 31, 2018, for one of the following seven common bacterial infections: community-acquired pneumonia (CAP), neck infection (superficial and deep), periorbital/orbital infection, urinary tract infection (UTI), osteomyelitis, septic arthritis, or skin and soft tissue infection (SSTI). The diagnosis cohorts were defined based on ICD-10-CM discharge diagnoses adapted from previous studies (Appendix Table 1).^3,17-23 To define a cohort of generally healthy pediatric patients with an acute infection, we excluded patients hospitalized in the intensive care unit, patients with nonhome discharges, and patients with complex chronic conditions.²⁴ We also excluded hospitals with incomplete data during the study period (n=1). The Institutional Review Board at Cincinnati Children’s Hospital Medical Center determined this study to be non–human-subjects research.

Outcomes

The primary outcomes were the number of opportunity days and the percent of days with opportunity to transition from IV to enteral therapy. Opportunity days, or days in which there was a potential opportunity to transition from IV to enteral antibiotics, were defined as days patients received only IV antibiotic doses and at least one enteral nonantibiotic medication, suggesting an ability to take enteral medications.¹³ We excluded days patients received IV antibiotics for which there was no enteral alternative (eg, vancomycin, Appendix Table 2). When measuring opportunity, to be conservative (ie, to underestimate rather than overestimate opportunity), we did not count as an opportunity day any day in which patients received both IV and enteral antibiotics. Percent opportunity, or the percent of days patients received antibiotics in which there was potential opportunity to transition from IV to enteral antibiotics, was defined as the number of opportunity days divided by number of inpatient days patients received enteral antibiotics or IV antibiotics with at least one enteral nonantibiotic medication (antibiotic days). Similar to opportunity days, antibiotic days excluded days patients were on IV antibiotics for which there was no enteral alternative. Based on our definition, a lower percent opportunity indicates that a hospital is using enteral antibiotics earlier during the hospitalization (earlier transition), while a higher percent opportunity represents later enteral antibiotic use (later transition).

Statistical Analysis

Demographic and clinical characteristics were summarized by diagnosis with descriptive statistics, including frequency with percentage, mean with standard deviation, and median with interquartile range (IQR). For each diagnosis, we evaluated aggregate opportunity days (sum of opportunity days among all hospitals), opportunity days per encounter, and aggregate percent opportunity using frequencies, mean with standard deviation, and percentages, respectively. We also calculated aggregate opportunity days for diagnosis-antibiotic combinations. To visually show variation in the percent opportunity across hospitals, we displayed the percent opportunity on a heat map, and evaluated percent opportunity across hospitals using chi-square tests. To compare the variability in the percent opportunity across and within hospitals, we used a generalized linear model with two fixed effects (hospital and diagnosis), and parsed the variability using the sum of squares. We performed a sensitivity analysis and excluded days that patients received antiemetic medications (eg, ondansetron, granisetron, prochlorperazine, promethazine), as these suggest potential intolerance of enteral medications. All statistical analyses were performed using SAS v.9.4 (SAS Institute Inc, Cary, North Carolina) and GraphPad Prism 8.0 (GraphPad Software Inc., San Diego, California), and P values < .05 were considered statistically significant.

RESULTS

During the 2-year study period, 100,103 hospitalizations met our inclusion criteria across 51 hospitals and seven diagnosis categories (Table 1). Diagnosis cohorts ranged in size from 1,462 encounters for septic arthritis to 35,665 encounters for neck infections. Overall, we identified 88,522 aggregate opportunity days on which there was an opportunity to switch from IV to enteral treatment in the majority of participants (percent opportunity, 57%).

Opportunity by Diagnosis

The number of opportunity days (aggregate and mean per encounter) and percent opportunity varied by diagnosis (Table 2). The aggregate number of opportunity days ranged from 3,693 in patients with septic arthritis to 25,359 in patients with SSTI, and mean opportunity days per encounter ranged from 0.9 in CAP to 2.8 in septic arthritis. Percent opportunity was highest for septic arthritis at 72.7% and lowest for CAP at 39.7%.

Variation in Opportunity Among Hospitals

The variation in the percent opportunity across hospitals was statistically significant for all diagnoses (Figure). Within hospitals, we observed similar practice patterns across diagnoses. For example, hospitals with a higher percent opportunity for one diagnosis tended to have higher percent opportunity for the other diagnoses (as noted in the top portion of the Figure), and those with lower percent opportunity for one diagnosis tended to also have lower percent opportunity for the other diagnoses studied (as noted in the bottom portion of the Figure). When evaluating variability in the percent opportunity, 45% of the variability was attributable to the hospital-effect and 35% to the diagnosis; the remainder was unexplained variability. Sensitivity analysis excluding days when patients received an antiemetic medication yielded no differences in our results.

Opportunity by Antibiotic

The aggregate number of opportunity days varied by antibiotic (Table 3). Intravenous antibiotics with the largest number of opportunity days included clindamycin (44,293), ceftriaxone (23,896), and ampicillin-sulbactam (15,484). Antibiotic-diagnosis combinations with the largest number of opportunity days for each diagnosis included ceftriaxone and ampicillin in CAP; clindamycin in cellulitis, SSTI, and neck infections; ceftriaxone in UTI; and cefazolin in osteomyelitis and septic arthritis.

DISCUSSION

In this multicenter study of pediatric patients hospitalized with common bacterial infections, there was the potential to transition from IV to enteral treatment in over half of the antibiotic days. The degree of opportunity varied by infection, antibiotic, and hospital. Antibiotics with a large aggregate number of opportunity days for enteral transition included clindamycin, which has excellent bioavailability; and ampicillin and ampicillin-sulbactam, which can achieve pharmacodynamic targets with oral equivalents.^25-29 The across-hospital variation for a given diagnosis suggests that certain hospitals have strategies in place which permit an earlier transition to enteral antibiotics compared to other institutions in which there were likely missed opportunities to do so. This variability is likely due to limited evidence, emphasizing the need for robust studies to better understand the optimal initial antibiotic route and transition time. Our findings highlight the need for, and large potential impact of, stewardship efforts to promote earlier transition for specific drug targets. This study also demonstrates the feasibility of obtaining two metrics—percent opportunity and opportunity days—from administrative databases to inform stewardship efforts within and across hospitals.

Opportunity days and percent opportunity varied among diagnoses. The variation in aggregate opportunity days was largely a reflection of the number of encounters: Diagnoses such as SSTI, neck infections, and CAP had a large number of both aggregate opportunity days and encounters. The range of opportunity days per encounter (0.9-2.5) suggests potential missed opportunities to transition to enteral antibiotics across all diagnoses (Table 2). The higher opportunity days per encounter in osteomyelitis and septic arthritis may be related to longer LOS and higher percent opportunity. Percent opportunity likely varied among diagnoses due to differences in admission and discharge readiness criteria, diagnostic evaluation, frequency of antibiotic administration, and evidence on the optimal route of initial antibiotics and when to transition to oral formulations. For example, we hypothesize that certain diagnoses, such as osteomyelitis and septic arthritis, have admission and discharge readiness criteria directly tied to the perceived need for IV antibiotics, which may limit in-hospital days on enteral antibiotics and explain the high percent opportunity that we observed. The high percent opportunity seen in musculoskeletal infections also may be due to delays in initiating targeted treatment until culture results were available. Encounters for CAP had the lowest percent opportunity; we hypothesize that this is because admission and discharge readiness may be determined by factors other than the need for IV antibiotics (eg, need for supplemental oxygen), which may increase days on enteral antibiotics and lead to a lower percent opportunity.³⁰

Urinary tract infection encounters had a high percent opportunity. As with musculoskeletal infection, this may be related to delays in initiating targeted treatment until culture results became available. Another reason for the high percent opportunity in UTI could be the common use of ceftriaxone, which, dosed every 24 hours, likely reduced the opportunity to transition to enteral antibiotics. There is strong evidence demonstrating no difference in outcomes based on antibiotic routes for UTI, and we would expect this to result in a low percent opportunity.^2,31 While the observed high opportunity in UTI may relate to an initial unknown diagnosis or concern for systemic infection, this highlights potential opportunities for quality improvement initiatives to promote empiric oral antibiotics in clinically stable patients hospitalized with suspected UTI.

There was substantial variation in percent opportunity across hospitals for a given diagnosis, with less variation across diagnoses for a given hospital. Variation across hospitals but consistency within individual hospitals suggests that some hospitals may promote earlier transition from IV to enteral antibiotics as standard practice for all diagnoses, while other hospitals continue IV antibiotics for the entire hospitalization, highlighting potential missed opportunities at some institutions. While emerging data suggest that traditional long durations of IV antibiotics are not necessary for many infections, the limited evidence on the optimal time to switch to oral antibiotics may have influenced this variation.^2-7 Many guidelines recommend initial IV antibiotics for hospitalized pediatric patients, but there are few studies comparing IV and enteral therapy.^2,5,9 Limited evidence leaves significant room for hospital culture, antibiotic stewardship efforts, reimbursement considerations, and/or hospital workflow to influence transition timing and overall opportunity at individual hospitals.^7,8,32-34 These findings emphasize the importance of research to identify optimal transition time and comparative effectiveness studies to evaluate whether initial IV antibiotics are truly needed for mild—and even severe—disease presentations. Since many patients are admitted for the perceived need for IV antibiotics, earlier use of enteral antibiotics could reduce rates of hospitalizations, LOS, healthcare costs, and resource utilization.

Antibiotics with a high number of opportunity days included clindamycin, ceftriaxone, ampicillin-sublactam, and ampicillin. Our findings are consistent with another study which found that most bioavailable drugs, including clindamycin, were administered via the IV route and accounted for a large number of antibiotic days.³⁵ The Infectious Diseases Society of America recommends that hospitals promote earlier transition to oral formulations for highly bioavailable drugs.⁷ Given the high bioavailability of clindamycin, its common use in high-frequency encounters such as SSTI and neck infections, and the fact that it accounted for a large number of opportunity days, quality improvement initiatives promoting earlier transition to oral clindamycin could have a large impact across health systems.^25,26 Additionally, although beta-lactam antibiotics such as amoxicillin and amoxicillin-sulbactam are not highly bioavailable, oral dosing can achieve sufficient serum concentrations to reach pharmacodynamic targets for common clinical indications; this could be an important quality improvement initiative.^27-29 Several single-site studies have successfully implemented quality improvement initiatives to promote earlier IV-to-enteral transition, with resulting reductions in costs and no adverse events noted, highlighting the feasibility and impact of such efforts.^13,36-38

This study also demonstrates the feasibility of collecting two metrics (percent opportunity and opportunity days) from administrative databases to inform IV-to-oral transition benchmarking and stewardship efforts. While there are several metrics in the literature for evaluating antibiotic transition (eg, days of IV or oral therapy, percentage of antibiotics given via the oral route, time to switch from IV to oral, and acceptance rate of suggested changes to antibiotic route), none are universally used or agreed upon.^15,16,39 The opportunity metrics used in this study have several strengths, including the feasibility of obtaining them from existing databases and the ability to account for intake of other enteral medications; the latter is not evaluated in other metrics. These opportunity metrics can be used together to identify the percent of time in which there is opportunity to transition and total number of days to understand the full extent of potential opportunity for future interventions. As demonstrated in this study, these metrics can be measured by diagnosis, antibiotic, or diagnosis-antibiotic combination, and they can be used to evaluate stewardship efforts at a single institution over time or compare efforts across hospitals.

These findings should be interpreted in the context of important limitations. First, we attempted to characterize potential opportunity to transition to enteral medications based on a patient’s ability to tolerate nonenteral medications. However, there are other factors that could limit the opportunity to transition that we could not account for with an administrative dataset, including the use of antibiotics prior to admission, disease progression, severity of illness, and malabsorptive concerns. Thus, though we may have overestimated the true opportunity to transition to enteral antibiotics, it is unlikely that this would account for all of the variation in transition times that we observed across hospitals. Second, while our study required patients to have one of seven types of infection, we did not exclude any additional infectious diagnoses (eg, concurrent bacteremia, Clostridioides difficile, otitis media) that could have driven the choice of antibiotic type and modality. Although emerging evidence is supporting earlier transitions to oral therapy, bacteremia is typically treated with IV antibiotics; this may have led to an overestimation of true opportunity.⁴⁰ “Clostridioides” difficile and otitis media are typically treated with enteral therapy; concurrent infections such as these may have led to an underestimation of opportunity given the fact that, based on our definition, the days on which patients received both IV and enteral antibiotics were not counted as opportunity days. Third, because PHIS uses billing days to capture medication use, we were unable to distinguish transitions that occurred early in the day vs those that took place later in the day. This could have led to an underestimation of percent opportunity, particularly for diagnoses with a short LOS; it also likely led to an underestimation of the variability observed across hospitals. Fourth, because we used an administrative dataset, we are unable to understand reasoning behind transitioning time from IV to oral antibiotics, as well as provider, patient, and institutional level factors that influenced these decisions.

CONCLUSION

Children hospitalized with bacterial infections often receive IV antibiotics, and the timing of transition from IV to enteral antibiotics varies significantly across hospitals. Further research is needed to compare the effectiveness of IV and enteral antibiotics and better define criteria for transition to enteral therapy. We identified ample opportunities for quality improvement initiatives to promote earlier transition, which have the potential to reduce healthcare utilization and promote optimal patient-directed high-value care.

Bacterial infections are a common reason for pediatric hospital admissions in the United States.¹ Antibiotics are the mainstay of treatment, and whether to administer them intravenously (IV) or enterally is an important and, at times, challenging decision. Not all hospitalized patients with infections require IV antibiotics, and safe, effective early transitions to enteral therapy have been described for numerous infections.^2-7 However, guidelines describing the ideal initial route of antibiotic administration and when to transition to oral therapy are lacking.^5,7,8 This lack of high-quality evidence-based guidance may contribute to overuse of IV antibiotics for many hospitalized pediatric patients, even when safe and effective enteral options exist.⁹

Significant costs and harms are associated with the use of IV antibiotics. In particular, studies have demonstrated longer length of stay (LOS), increased costs, and worsened pain or anxiety related to complications (eg, phlebitis, extravasation injury, thrombosis, catheter-associated bloodstream infections) associated with IV antibiotics.^3,4,10-13 Earlier transition to enteral therapy, however, can mitigate these increased risks and costs.

The Centers for Disease Control and Prevention lists the transition from IV to oral antibiotics as a key stewardship intervention for improving antibiotic use.¹⁴ The Infectious Diseases Society of America (IDSA) antibiotic stewardship program guidelines strongly recommend the timely conversion from IV to oral antibiotics, stating that efforts focusing on this transition should be integrated into routine practice.¹⁵ There are a few metrics in the literature to measure this intervention, but none is universally used, and a modified delphi process could not reach consensus on IV-to-oral transition metrics.¹⁶

Few studies describe the opportunity to transition to enteral antibiotics in hospitalized patients with common bacterial infections or explore variation across hospitals. It is critical to understand current practice of antibiotic administration in order to identify opportunities to optimize patient outcomes and promote high-value care. Furthermore, few studies have evaluated the feasibility of IV-to-oral transition metrics using an administrative database. Thus, the aims of this study were to (1) determine opportunities to transition from IV to enteral antibiotics for pediatric patients hospitalized with common bacterial infections based on their ability to tolerate other enteral medications, (2) describe variation in transition practices among children’s hospitals, and (3) evaluate the feasibility of novel IV-to-oral transition metrics using an administrative database to inform stewardship efforts.

METHODS

Study Design and Setting

This multicenter, retrospective cohort study used data from the Pediatric Health Information System (PHIS), an administrative and billing database containing encounter-level data from 52 tertiary care pediatric hospitals across the United States affiliated with the Children’s Hospital Association (Lenexa, Kansas). Hospitals submit encounter-level data, including demographics, medications, and diagnoses based on International Classification of Disease, Tenth Revision, Clinical Modification (ICD-10-CM) codes. Data were de-identified at the time of submission, and data quality and reliability were assured by joint efforts between the Children’s Hospital Association and participating hospitals.

Study Population

This study included pediatric patients aged 60 days to 18 years who were hospitalized (inpatient or observation status) at one of the participating hospitals between January 1, 2017, and December 31, 2018, for one of the following seven common bacterial infections: community-acquired pneumonia (CAP), neck infection (superficial and deep), periorbital/orbital infection, urinary tract infection (UTI), osteomyelitis, septic arthritis, or skin and soft tissue infection (SSTI). The diagnosis cohorts were defined based on ICD-10-CM discharge diagnoses adapted from previous studies (Appendix Table 1).^3,17-23 To define a cohort of generally healthy pediatric patients with an acute infection, we excluded patients hospitalized in the intensive care unit, patients with nonhome discharges, and patients with complex chronic conditions.²⁴ We also excluded hospitals with incomplete data during the study period (n=1). The Institutional Review Board at Cincinnati Children’s Hospital Medical Center determined this study to be non–human-subjects research.

Outcomes

The primary outcomes were the number of opportunity days and the percent of days with opportunity to transition from IV to enteral therapy. Opportunity days, or days in which there was a potential opportunity to transition from IV to enteral antibiotics, were defined as days patients received only IV antibiotic doses and at least one enteral nonantibiotic medication, suggesting an ability to take enteral medications.¹³ We excluded days patients received IV antibiotics for which there was no enteral alternative (eg, vancomycin, Appendix Table 2). When measuring opportunity, to be conservative (ie, to underestimate rather than overestimate opportunity), we did not count as an opportunity day any day in which patients received both IV and enteral antibiotics. Percent opportunity, or the percent of days patients received antibiotics in which there was potential opportunity to transition from IV to enteral antibiotics, was defined as the number of opportunity days divided by number of inpatient days patients received enteral antibiotics or IV antibiotics with at least one enteral nonantibiotic medication (antibiotic days). Similar to opportunity days, antibiotic days excluded days patients were on IV antibiotics for which there was no enteral alternative. Based on our definition, a lower percent opportunity indicates that a hospital is using enteral antibiotics earlier during the hospitalization (earlier transition), while a higher percent opportunity represents later enteral antibiotic use (later transition).

Statistical Analysis

Demographic and clinical characteristics were summarized by diagnosis with descriptive statistics, including frequency with percentage, mean with standard deviation, and median with interquartile range (IQR). For each diagnosis, we evaluated aggregate opportunity days (sum of opportunity days among all hospitals), opportunity days per encounter, and aggregate percent opportunity using frequencies, mean with standard deviation, and percentages, respectively. We also calculated aggregate opportunity days for diagnosis-antibiotic combinations. To visually show variation in the percent opportunity across hospitals, we displayed the percent opportunity on a heat map, and evaluated percent opportunity across hospitals using chi-square tests. To compare the variability in the percent opportunity across and within hospitals, we used a generalized linear model with two fixed effects (hospital and diagnosis), and parsed the variability using the sum of squares. We performed a sensitivity analysis and excluded days that patients received antiemetic medications (eg, ondansetron, granisetron, prochlorperazine, promethazine), as these suggest potential intolerance of enteral medications. All statistical analyses were performed using SAS v.9.4 (SAS Institute Inc, Cary, North Carolina) and GraphPad Prism 8.0 (GraphPad Software Inc., San Diego, California), and P values < .05 were considered statistically significant.

RESULTS

During the 2-year study period, 100,103 hospitalizations met our inclusion criteria across 51 hospitals and seven diagnosis categories (Table 1). Diagnosis cohorts ranged in size from 1,462 encounters for septic arthritis to 35,665 encounters for neck infections. Overall, we identified 88,522 aggregate opportunity days on which there was an opportunity to switch from IV to enteral treatment in the majority of participants (percent opportunity, 57%).

Opportunity by Diagnosis

The number of opportunity days (aggregate and mean per encounter) and percent opportunity varied by diagnosis (Table 2). The aggregate number of opportunity days ranged from 3,693 in patients with septic arthritis to 25,359 in patients with SSTI, and mean opportunity days per encounter ranged from 0.9 in CAP to 2.8 in septic arthritis. Percent opportunity was highest for septic arthritis at 72.7% and lowest for CAP at 39.7%.

Variation in Opportunity Among Hospitals

The variation in the percent opportunity across hospitals was statistically significant for all diagnoses (Figure). Within hospitals, we observed similar practice patterns across diagnoses. For example, hospitals with a higher percent opportunity for one diagnosis tended to have higher percent opportunity for the other diagnoses (as noted in the top portion of the Figure), and those with lower percent opportunity for one diagnosis tended to also have lower percent opportunity for the other diagnoses studied (as noted in the bottom portion of the Figure). When evaluating variability in the percent opportunity, 45% of the variability was attributable to the hospital-effect and 35% to the diagnosis; the remainder was unexplained variability. Sensitivity analysis excluding days when patients received an antiemetic medication yielded no differences in our results.

Opportunity by Antibiotic

The aggregate number of opportunity days varied by antibiotic (Table 3). Intravenous antibiotics with the largest number of opportunity days included clindamycin (44,293), ceftriaxone (23,896), and ampicillin-sulbactam (15,484). Antibiotic-diagnosis combinations with the largest number of opportunity days for each diagnosis included ceftriaxone and ampicillin in CAP; clindamycin in cellulitis, SSTI, and neck infections; ceftriaxone in UTI; and cefazolin in osteomyelitis and septic arthritis.

DISCUSSION

In this multicenter study of pediatric patients hospitalized with common bacterial infections, there was the potential to transition from IV to enteral treatment in over half of the antibiotic days. The degree of opportunity varied by infection, antibiotic, and hospital. Antibiotics with a large aggregate number of opportunity days for enteral transition included clindamycin, which has excellent bioavailability; and ampicillin and ampicillin-sulbactam, which can achieve pharmacodynamic targets with oral equivalents.^25-29 The across-hospital variation for a given diagnosis suggests that certain hospitals have strategies in place which permit an earlier transition to enteral antibiotics compared to other institutions in which there were likely missed opportunities to do so. This variability is likely due to limited evidence, emphasizing the need for robust studies to better understand the optimal initial antibiotic route and transition time. Our findings highlight the need for, and large potential impact of, stewardship efforts to promote earlier transition for specific drug targets. This study also demonstrates the feasibility of obtaining two metrics—percent opportunity and opportunity days—from administrative databases to inform stewardship efforts within and across hospitals.

Opportunity days and percent opportunity varied among diagnoses. The variation in aggregate opportunity days was largely a reflection of the number of encounters: Diagnoses such as SSTI, neck infections, and CAP had a large number of both aggregate opportunity days and encounters. The range of opportunity days per encounter (0.9-2.5) suggests potential missed opportunities to transition to enteral antibiotics across all diagnoses (Table 2). The higher opportunity days per encounter in osteomyelitis and septic arthritis may be related to longer LOS and higher percent opportunity. Percent opportunity likely varied among diagnoses due to differences in admission and discharge readiness criteria, diagnostic evaluation, frequency of antibiotic administration, and evidence on the optimal route of initial antibiotics and when to transition to oral formulations. For example, we hypothesize that certain diagnoses, such as osteomyelitis and septic arthritis, have admission and discharge readiness criteria directly tied to the perceived need for IV antibiotics, which may limit in-hospital days on enteral antibiotics and explain the high percent opportunity that we observed. The high percent opportunity seen in musculoskeletal infections also may be due to delays in initiating targeted treatment until culture results were available. Encounters for CAP had the lowest percent opportunity; we hypothesize that this is because admission and discharge readiness may be determined by factors other than the need for IV antibiotics (eg, need for supplemental oxygen), which may increase days on enteral antibiotics and lead to a lower percent opportunity.³⁰

Urinary tract infection encounters had a high percent opportunity. As with musculoskeletal infection, this may be related to delays in initiating targeted treatment until culture results became available. Another reason for the high percent opportunity in UTI could be the common use of ceftriaxone, which, dosed every 24 hours, likely reduced the opportunity to transition to enteral antibiotics. There is strong evidence demonstrating no difference in outcomes based on antibiotic routes for UTI, and we would expect this to result in a low percent opportunity.^2,31 While the observed high opportunity in UTI may relate to an initial unknown diagnosis or concern for systemic infection, this highlights potential opportunities for quality improvement initiatives to promote empiric oral antibiotics in clinically stable patients hospitalized with suspected UTI.

There was substantial variation in percent opportunity across hospitals for a given diagnosis, with less variation across diagnoses for a given hospital. Variation across hospitals but consistency within individual hospitals suggests that some hospitals may promote earlier transition from IV to enteral antibiotics as standard practice for all diagnoses, while other hospitals continue IV antibiotics for the entire hospitalization, highlighting potential missed opportunities at some institutions. While emerging data suggest that traditional long durations of IV antibiotics are not necessary for many infections, the limited evidence on the optimal time to switch to oral antibiotics may have influenced this variation.^2-7 Many guidelines recommend initial IV antibiotics for hospitalized pediatric patients, but there are few studies comparing IV and enteral therapy.^2,5,9 Limited evidence leaves significant room for hospital culture, antibiotic stewardship efforts, reimbursement considerations, and/or hospital workflow to influence transition timing and overall opportunity at individual hospitals.^7,8,32-34 These findings emphasize the importance of research to identify optimal transition time and comparative effectiveness studies to evaluate whether initial IV antibiotics are truly needed for mild—and even severe—disease presentations. Since many patients are admitted for the perceived need for IV antibiotics, earlier use of enteral antibiotics could reduce rates of hospitalizations, LOS, healthcare costs, and resource utilization.

Antibiotics with a high number of opportunity days included clindamycin, ceftriaxone, ampicillin-sublactam, and ampicillin. Our findings are consistent with another study which found that most bioavailable drugs, including clindamycin, were administered via the IV route and accounted for a large number of antibiotic days.³⁵ The Infectious Diseases Society of America recommends that hospitals promote earlier transition to oral formulations for highly bioavailable drugs.⁷ Given the high bioavailability of clindamycin, its common use in high-frequency encounters such as SSTI and neck infections, and the fact that it accounted for a large number of opportunity days, quality improvement initiatives promoting earlier transition to oral clindamycin could have a large impact across health systems.^25,26 Additionally, although beta-lactam antibiotics such as amoxicillin and amoxicillin-sulbactam are not highly bioavailable, oral dosing can achieve sufficient serum concentrations to reach pharmacodynamic targets for common clinical indications; this could be an important quality improvement initiative.^27-29 Several single-site studies have successfully implemented quality improvement initiatives to promote earlier IV-to-enteral transition, with resulting reductions in costs and no adverse events noted, highlighting the feasibility and impact of such efforts.^13,36-38

This study also demonstrates the feasibility of collecting two metrics (percent opportunity and opportunity days) from administrative databases to inform IV-to-oral transition benchmarking and stewardship efforts. While there are several metrics in the literature for evaluating antibiotic transition (eg, days of IV or oral therapy, percentage of antibiotics given via the oral route, time to switch from IV to oral, and acceptance rate of suggested changes to antibiotic route), none are universally used or agreed upon.^15,16,39 The opportunity metrics used in this study have several strengths, including the feasibility of obtaining them from existing databases and the ability to account for intake of other enteral medications; the latter is not evaluated in other metrics. These opportunity metrics can be used together to identify the percent of time in which there is opportunity to transition and total number of days to understand the full extent of potential opportunity for future interventions. As demonstrated in this study, these metrics can be measured by diagnosis, antibiotic, or diagnosis-antibiotic combination, and they can be used to evaluate stewardship efforts at a single institution over time or compare efforts across hospitals.

These findings should be interpreted in the context of important limitations. First, we attempted to characterize potential opportunity to transition to enteral medications based on a patient’s ability to tolerate nonenteral medications. However, there are other factors that could limit the opportunity to transition that we could not account for with an administrative dataset, including the use of antibiotics prior to admission, disease progression, severity of illness, and malabsorptive concerns. Thus, though we may have overestimated the true opportunity to transition to enteral antibiotics, it is unlikely that this would account for all of the variation in transition times that we observed across hospitals. Second, while our study required patients to have one of seven types of infection, we did not exclude any additional infectious diagnoses (eg, concurrent bacteremia, Clostridioides difficile, otitis media) that could have driven the choice of antibiotic type and modality. Although emerging evidence is supporting earlier transitions to oral therapy, bacteremia is typically treated with IV antibiotics; this may have led to an overestimation of true opportunity.⁴⁰ “Clostridioides” difficile and otitis media are typically treated with enteral therapy; concurrent infections such as these may have led to an underestimation of opportunity given the fact that, based on our definition, the days on which patients received both IV and enteral antibiotics were not counted as opportunity days. Third, because PHIS uses billing days to capture medication use, we were unable to distinguish transitions that occurred early in the day vs those that took place later in the day. This could have led to an underestimation of percent opportunity, particularly for diagnoses with a short LOS; it also likely led to an underestimation of the variability observed across hospitals. Fourth, because we used an administrative dataset, we are unable to understand reasoning behind transitioning time from IV to oral antibiotics, as well as provider, patient, and institutional level factors that influenced these decisions.

CONCLUSION

Children hospitalized with bacterial infections often receive IV antibiotics, and the timing of transition from IV to enteral antibiotics varies significantly across hospitals. Further research is needed to compare the effectiveness of IV and enteral antibiotics and better define criteria for transition to enteral therapy. We identified ample opportunities for quality improvement initiatives to promote earlier transition, which have the potential to reduce healthcare utilization and promote optimal patient-directed high-value care.

There is substantial geographic variation in intensity of healthcare use in the United States,¹ yet areas with higher healthcare utilization do not demonstrate superior clinical outcomes.² Low-value care exposes patients to unnecessary anxiety, radiation, and risk for adverse events.

Previous research has focused on measuring low-value care at the level of hospital referral regions,^3-6 metropolitan statistical areas,⁷ provider organizations,⁸ and individual physicians.^9,10 Hospital referral regions designate regional healthcare markets for tertiary care and generally include at least one major referral center.¹¹ Well-calibrated and validated hospital-level measures of diagnostic overuse are lacking.

We sought to construct a novel index to measure hospital level overuse of diagnostic testing. We focused on diagnostic intensity rather than other forms of overuse such as screening or treatment intensity. Moreover, we aimed to create a parsimonious index—one that is simple, relies on a small number of inputs, is derived from readily available administrative data without the need for chart review or complex logic, and does not require exclusion criteria.

METHODS

Conceptual Framework for Choosing Index Components

To create our overuse index, we took advantage of the requirements for International Classification of Diseases, 9th Revision-Clinical Modification (ICD-9-CM) billing codes 780-796; these codes are based on “symptoms, signs, and ill-defined conditions” and can only be listed as the primary discharge diagnosis if no more specific diagnosis is made.¹² As such, when coupled with expensive tests, a high prevalence of these symptom-based diagnosis codes at discharge may serve as a proxy for low-value care. One of the candidate metrics we selected was based on Choosing Wisely^® recommendations.¹³ The other candidate metrics were based on clinical experience and consensus of the study team.

Data Sources

We used hospital-level data on primary discharge diagnosis codes and utilization of testing data from the State Inpatient Databases (SID), which are part of the Agency for Healthcare Research and Quality Healthcare Cost and Utilization Project (HCUP). Our derivation cohort used data from acute care hospitals in Maryland, New Jersey, and Washington state. Our validation cohort used data from acute care hospitals in Kentucky, North Carolina, New York, and West Virginia. States were selected based on availability of data (certain states lacked complete testing utilization data) and cost of data acquisition. The SID contains hospital-level utilization of computed tomography (CT) scans (CT of the body and head) and diagnostic testing, including stress testing and esophagogastroduodenoscopy (EGD).

Data on three prespecified Dartmouth Atlas of Health Care metrics at the hospital service area (HSA) level were obtained from the Dartmouth Atlas website.¹⁴ These metrics were (1) rate of inpatient coronary angiograms per 1,000 Medicare enrollees, (2) price-adjusted physician reimbursement per fee-for-service Medicare enrollee per year (adjusted for patient sex, race, and age), and (3) mean inpatient spending per decedent in the last 6 months of life.¹⁵ Data on three prespecified Medicare metrics at the county level were obtained from the Centers for Medicare & Medicaid Services (CMS) website.¹⁶ These metrics were standardized per capita cost per (1) procedure, (2) imaging, and (3) test of Medicare fee-for-service patients. The CMS uses the Berenson-Eggers Type of Service Codes to classify fee-generating interventions into a number of categories, including procedure, imaging, and test.¹⁷

Components of the Overuse Index

We tested five candidate metrics for index inclusion (Table 1). We utilized Clinical Classifications Software (CCS) codes provided by HCUP, which combine several ICD-9-CM codes into a single primary CCS discharge code for ease of use. The components were (1) primary CCS diagnosis of “nausea and vomiting” coupled with body CT scan or EGD, (2) primary CCS diagnosis of abdominal pain and body CT scan or EGD, (3) primary CCS diagnosis of “nonspecific chest pain” and body CT scan or stress test, (4) primary CCS diagnosis of syncope and stress test, and (5) primary CCS diagnosis for syncope and CT of the brain. For a given metric, the denominator was all patients with the particular primary CCS discharge diagnosis code. The numerator was patients with the diagnostic code who also had the specific test or procedure. We characterized the denominators of each metric in terms of mean, SD, and range.

Index Inclusion Criteria and Construction

Specialty, pediatric, rehabilitation, and long-term care hospitals were excluded. Moreover, any hospital with an overall denominator (for the entire index, not an individual metric) of five or fewer observations was excluded. Admissions to acute care hospitals between January 2011 and September 2015 (time of transition from ICD-9-CM to ICD-10-CM) that had one of the specified diagnosis codes were included. For a given hospital, the value of each of the five candidate metrics was defined as the ratio of all admissions that had the given testing and all admissions during the observation period with inclusion CCS diagnosis codes.

Derivation and Validation of the Index

In our derivation cohort (hospitals in Maryland, New Jersey, and Washington state), we tested the temporal stability of each candidate metric by year using the intraclass correlation coefficient (ICC). Using exploratory factor analysis (EFA) and Cronbach’s alpha, we then tested internal consistency of the index candidate components to ensure that all measured a common underlying factor (ie, diagnostic overuse). To standardize data, test rates for both of these analyses were converted to z-scores. For the EFA, we expected that if the index was reflecting only a single underlying factor, the Eigenvalue for one factor should be much higher (typically above 1.0) than that for multiple factors. We calculated item-test correlation for each candidate metric and Cronbach’s alpha for the entire index. A high and stable value for item-test correlation for each index component, as well as a high Cronbach’s alpha, suggests that index components measure a single common factor. Given the small number of test items, we considered a Cronbach’s alpha above 0.6 to be satisfactory.

This analysis showed satisfactory temporal stability of each candidate metric and good internal consistency of the candidate metrics in the derivation cohort. Therefore, we decided to keep all metrics rather than discard any of them. This same process was repeated with the validation cohort (Kentucky, New York, North Carolina, and West Virginia) and then with the combined group of seven states. Tests on the validation and entire cohort further supported our decision to keep all five metrics.

To determine the overall index value for a hospital, all of its metric numerators and denominators were added to calculate one fraction. In this way for a given hospital, a metric for which there were no observations was effectively excluded from the index. This essentially weights each index component by frequency. We chose to count syncope admissions only once in the denominator to avoid the index being unduly influenced by this diagnosis. The hospital index values were combined into their HSAs by adding numerators and denominators from each hospital to calculate HSA index values, effectively giving higher weight to hospitals with more observations. Spearman’s correlation coefficients were measured for these Dartmouth Atlas metrics, also at the HSA level. For the county level analysis, we used a hospital-county crosswalk (available from the American Hospital Association [AHA] Annual Survey; https://www.ahadata.com/aha-annual-survey-database) to link a hospital overuse index value to a county level cost value rather than aggregating data at the county level. We felt this was appropriate, as HSAs were constructed to represent a local healthcare market, whereas counties are less likely to be homogenous from a healthcare perspective.

Analysis of Entire Hospital Sample

The mean index value and SD were calculated for the entire sample of hospitals and for each state. The mean index value for each year of data was calculated to measure the temporal change of the index (representing a change in diagnostic intensity over the study period) using linear regression. We divided the cohort of hospitals into tertiles based on their index value. This is consistent with the CMS categorization of hospital payments and value of care as being “at,” “significantly above,” or “significantly below” a mean value.¹⁸ The characteristics of hospitals by tertile were described by mean total hospital beds, mean annual admissions, teaching status (nonteaching hospital, minor teaching hospital, major teaching hospital), and critical access hospital (yes/no). We utilized the AHA Annual Survey for data on hospital characteristics. We calculated P values using analysis of variance for hospital bed size and a chi-square test for teaching status and critical access hospital.

The entire group of hospitals from seven states was then used to apply the index to the HSA level. Numerators and denominators for each hospital in an HSA were added to calculate an HSA-level proportion. Thus, the HSA level index value, though unweighted, is dominated by hospitals with larger numbers of observations. For each of the Dartmouth metrics, the adjusted odds ratio of being in a higher diagnostic overuse index tertile given being in a certain Dartmouth Atlas metric tertile was calculated using ordinal logistic regression. This model controlled for the mean number of beds of hospitals in the HSA (continuous variable), mean Elixhauser Comorbidity Index (ECI) score (continuous variable; unweighted average among hospitals in an HSA), whether the HSA had a major or minor teaching hospital (yes/no) or was a critical access hospital (yes/no), and state fixed effects. The ECI score is a validated score that uses the presence or absence of 29 comorbidities to predict in-hospital mortality.¹⁹ For discriminant validity, we also tested two variables not expected to be associated with overuse—hospital ownership and affiliation with the Catholic Church.

For the county-level analysis, ordinal logistic regression was used to predict the adjusted odds ratio of being in a higher diagnostic overuse index tertile given being in a certain tertile of a given county-level spending metric. This model controlled for hospital bed size (continuous variable), hospital ECI score (continuous variable), teaching status (major, minor, nonteaching), critical access hospital status (yes/no), and state fixed effects.

RESULTS

Descriptive Statistics for Metrics

A total of 620 acute care hospitals were included in the index. Thirteen hospitals were excluded because their denominator was five or fewer. The vast majority of HSAs (85.9%) had only one hospital, 8.2% had two hospitals, and 2.4% had three hospitals. Similarly, the majority of counties (68.7%) had only one hospital, 15.1% had two hospitals, and 6.6% had three hospitals (Appendix Tables 1.1 and 1.2). Nonspecific chest pain was the metric with largest denominator mean (650), SD (1,012), and range (0-10,725) (Appendix Table 2). Overall, the metric denominators were a small fraction of total hospital discharges, with means at the hospital level ranging from 0.69% for nausea and vomiting to 5.81% for nonspecific chest pain, suggesting that our index relies on a relatively small fraction of discharges.

Tests for Temporal Stability and Internal Consistency by Derivation and Validation Strategy

Overall, the ICCs for the derivation, validation, and entire cohort suggested strong temporal stability (Appendix Table 3). The EFA of the derivation, validation, and entire cohort showed high Eigenvalues for one principal component, with no other factors close to 1, indicating strong internal consistency (Appendix Table 4). The Cronbach’s alpha analysis also suggested strong internal consistency, with alpha values ranging from 0.73 for the validation cohort to 0.80 for the derivation cohort (Table 2).

Correlation With External Validation Measures

For the entire cohort, the Spearman’s rho for correlation between our overuse index and inpatient rate of coronary angiography at the HSA level was 0.186 (95% CI, 0.089-0.283), Medicare reimbursement at the HSA level was 0.355 (95% CI, 0.272-0.437), and Medicare spending during the last 6 months of life at the HSA level was 0.149 (95% CI, 0.061-0.236) (Appendix Figures 5.1-5.3). The Spearman’s rho for correlation between our overuse index and county level standardized procedure cost was 0.284 (95% CI, 0.210-0.358), imaging cost was 0.268 (95% CI, 0.195-0.342), and testing cost was 0.226 (95% CI, 0.152-0.300) (Appendix Figures 6.1-6.3).

Overall Index Values and Change Over Time

The mean hospital index value was 0.541 (SD, 0.178) (Appendix Table 7). There was a slight but statistically significant annual increase in the overall mean index value over the study period, suggesting a small rise in overuse of diagnostic testing (coefficient 0.011; P <.001) (Appendix Figure 8).

Diagnostic Overuse Index Tertiles

Hospitals in the lowest tertile of the index tended to be smaller (based on number of beds) (P < .0001) and were more likely to be critical access hospitals (P <.0001). There was a significant difference in the proportion of nonteaching, minor teaching, and major teaching hospitals, with more nonteaching hospitals in tertile 1 (P = .001) (Table 3). The median ECI score was not significantly different among tertiles. Neither of the variables tested for discriminant validity (hospital ownership and Catholic Church affiliation) was associated with our index.

Adjusted Multilevel Mixed-Effects Ordinal Logistic Regression

Our overuse index correlated most closely with physician reimbursement, with an odds ratio of 2.02 (95% CI, 1.11-3.66) of being in a higher tertile of the overuse index when comparing tertiles 3 and 1 of this Dartmouth metric. Of the Medicare county-level metrics, our index correlated most closely with cost of procedures, with an odds ratio of 2.03 (95% CI, 1.21-3.39) of being in a higher overuse index tertile when comparing tertiles 3 and 1 of the cost per procedure metric (Figure 1).

DISCUSSION

Previous research shows variation among hospitals for overall physician spending,²⁰ noninvasive cardiac imaging,²¹ and the rate of finding obstructive lesions during elective coronary angiography.²² However, there is a lack of standardized methods to study a broad range of diagnostic overuse at the hospital level. To our knowledge, no studies have attempted to develop a diagnostic overuse index at the hospital level. We used a derivation-validation approach to achieve our goal. Although the five metrics represent a range of conditions, the EFA and Cronbach’s alpha tests suggest that they measure a common phenomenon. To avoid systematically excluding smaller hospitals, we limited the extent to which we eliminated hospitals with few observations. Our findings suggest that it may be reasonable to make generalizations on the diagnostic intensity of a hospital based on a relatively small number of discharges. Moreover, our index is a proof of concept that rates of negative diagnostic testing can serve as a proxy for estimating diagnostic overuse.

Our hospital-level index values extrapolated to the HSA level weakly correlated with prespecified Dartmouth Atlas metrics. In a multivariate ordinal regression, there was a significant though weak association between hospitals in higher tertiles of the Dartmouth Atlas metrics and categorization in higher tertiles of our diagnostic overuse index. Similarly, our hospital-level index correlated with two of the three county-level metrics in a multivariate ordinal regression.

We do not assume that all of the metrics in our index track together. However, our results, including the wide dispersion of index values among the tertiles (Table 3), suggest that at least some hospitals are outliers in multiple metrics. We did not assume ex ante that our index should correlate with Dartmouth overuse metrics or Medicare county-level spending; however, we did believe that an association with these measures would assist in validating our index. Given that our index utilizes four common diagnoses, while the Dartmouth and Medicare cost metrics are based on a much broader range of conditions, we would not expect more than a weak correlation even if our index is a valid way to measure overuse.

All of the metrics were based on the concept that hospitals with high rates of negative testing are likely providing large amounts of low-value care. Prior studies on diagnostic yield of CT scans in the emergency department for pulmonary embolus (PE) found an increase in testing and decrease in yield over time; these studies also showed that physicians with more experience ordered fewer CT scans and had a higher yield.²³ A review of electronic health records and billing data also showed that hospitals with higher rates of D-dimer testing had higher yields on CT scans ordered to test for PE.²⁴

We took advantage of the coding convention that certain diagnoses only be listed as the primary discharge diagnosis if no more specific diagnosis is made. This allowed us to identify hospitals that likely had high rates of negative tests without granular data. Of course, the metrics are not measuring rates of negative testing per se, but a proxy for this, based instead on the proportion of patients with a symptom-based primary discharge diagnosis who underwent diagnostic testing.

Measuring diagnostic overuse at the hospital level may help to understand factors that drive overuse, given that institutional incentives and culture likely play important roles in ordering tests. There is evidence that financial incentives drive physicians’ decisions,^25-27 and there is also evidence that institutional culture impacts outcomes.²⁸ Further, quality improvement projects are typically designed at the hospital level and may be an effective way to curb overuse.^29,30

Previous studies have focused on measuring variation among providers and identifying outlier physicians.^9,10,20 Providing feedback to underperforming physicians has been shown to change practice habits.^31,32 Efforts to improve the practice habits of outlier hospitals may have a number of advantages, including economies of scale and scope and the added benefit of improving the habits of all providers—not just those who are underperforming.

Ordering expensive diagnostic tests on patients with a low pretest probability of having an organic etiology for their symptoms contributes to high healthcare costs. Of course, we do not believe that the ideal rate of negative testing is zero. However, hospitals with high rates of negative diagnostic testing are more likely to be those with clinicians who use expensive tests as a substitute for clinical judgment or less-expensive tests (eg, D-dimer testing to rule out PE).

One challenge we faced is that there is no gold standard of hospital-level overuse with which to validate our index. Our index is weakly correlated with a number of regional metrics that may be proxies for overuse. We are reassured that there is a statistically significant correlation with measures at both HSA and county levels. These correlations are weak, but these regional metrics are themselves imperfect surrogates for overuse. Furthermore, our index is preliminary and will need refinement in future studies.

Limitations

Our analysis has multiple limitations. First, since it relies heavily on primary ICD discharge diagnosis codes, biases could exist due to variations in coding practices. Second, the SID does not include observation stays or tests conducted in the ED, so differential use of observation stays among hospitals might impact results. Finally, based on utilization data, we were not able to distinguish between CT scans of the chest, abdomen, and pelvis because the SID labels each of these as body CT.

CONCLUSION

We developed a novel index to measure diagnostic intensity at the hospital level. This index relies on the concept that high rates of negative diagnostic testing likely indicate some degree of overuse. Our index is parsimonious, does not require granular claims data, and measures a range of potentially overused tests for common clinical scenarios. Our next steps include further refining the index, testing it with granular data, and validating it with other datasets. Thereafter, this index may be useful at identifying positive and negative outliers to understand what processes of care contribute to outlier high and low levels of diagnostic testing. We suspect our index is more useful at identifying extremes than comparing hospitals in the middle of the utilization curve. Additionally, exploring the relationship among individual metrics and the relationship between our index and quality measures like mortality and readmissions may be informative.

There is substantial geographic variation in intensity of healthcare use in the United States,¹ yet areas with higher healthcare utilization do not demonstrate superior clinical outcomes.² Low-value care exposes patients to unnecessary anxiety, radiation, and risk for adverse events.

Previous research has focused on measuring low-value care at the level of hospital referral regions,^3-6 metropolitan statistical areas,⁷ provider organizations,⁸ and individual physicians.^9,10 Hospital referral regions designate regional healthcare markets for tertiary care and generally include at least one major referral center.¹¹ Well-calibrated and validated hospital-level measures of diagnostic overuse are lacking.

We sought to construct a novel index to measure hospital level overuse of diagnostic testing. We focused on diagnostic intensity rather than other forms of overuse such as screening or treatment intensity. Moreover, we aimed to create a parsimonious index—one that is simple, relies on a small number of inputs, is derived from readily available administrative data without the need for chart review or complex logic, and does not require exclusion criteria.

METHODS

Conceptual Framework for Choosing Index Components

To create our overuse index, we took advantage of the requirements for International Classification of Diseases, 9th Revision-Clinical Modification (ICD-9-CM) billing codes 780-796; these codes are based on “symptoms, signs, and ill-defined conditions” and can only be listed as the primary discharge diagnosis if no more specific diagnosis is made.¹² As such, when coupled with expensive tests, a high prevalence of these symptom-based diagnosis codes at discharge may serve as a proxy for low-value care. One of the candidate metrics we selected was based on Choosing Wisely^® recommendations.¹³ The other candidate metrics were based on clinical experience and consensus of the study team.

Data Sources

We used hospital-level data on primary discharge diagnosis codes and utilization of testing data from the State Inpatient Databases (SID), which are part of the Agency for Healthcare Research and Quality Healthcare Cost and Utilization Project (HCUP). Our derivation cohort used data from acute care hospitals in Maryland, New Jersey, and Washington state. Our validation cohort used data from acute care hospitals in Kentucky, North Carolina, New York, and West Virginia. States were selected based on availability of data (certain states lacked complete testing utilization data) and cost of data acquisition. The SID contains hospital-level utilization of computed tomography (CT) scans (CT of the body and head) and diagnostic testing, including stress testing and esophagogastroduodenoscopy (EGD).

Data on three prespecified Dartmouth Atlas of Health Care metrics at the hospital service area (HSA) level were obtained from the Dartmouth Atlas website.¹⁴ These metrics were (1) rate of inpatient coronary angiograms per 1,000 Medicare enrollees, (2) price-adjusted physician reimbursement per fee-for-service Medicare enrollee per year (adjusted for patient sex, race, and age), and (3) mean inpatient spending per decedent in the last 6 months of life.¹⁵ Data on three prespecified Medicare metrics at the county level were obtained from the Centers for Medicare & Medicaid Services (CMS) website.¹⁶ These metrics were standardized per capita cost per (1) procedure, (2) imaging, and (3) test of Medicare fee-for-service patients. The CMS uses the Berenson-Eggers Type of Service Codes to classify fee-generating interventions into a number of categories, including procedure, imaging, and test.¹⁷

Components of the Overuse Index

We tested five candidate metrics for index inclusion (Table 1). We utilized Clinical Classifications Software (CCS) codes provided by HCUP, which combine several ICD-9-CM codes into a single primary CCS discharge code for ease of use. The components were (1) primary CCS diagnosis of “nausea and vomiting” coupled with body CT scan or EGD, (2) primary CCS diagnosis of abdominal pain and body CT scan or EGD, (3) primary CCS diagnosis of “nonspecific chest pain” and body CT scan or stress test, (4) primary CCS diagnosis of syncope and stress test, and (5) primary CCS diagnosis for syncope and CT of the brain. For a given metric, the denominator was all patients with the particular primary CCS discharge diagnosis code. The numerator was patients with the diagnostic code who also had the specific test or procedure. We characterized the denominators of each metric in terms of mean, SD, and range.

Index Inclusion Criteria and Construction

Specialty, pediatric, rehabilitation, and long-term care hospitals were excluded. Moreover, any hospital with an overall denominator (for the entire index, not an individual metric) of five or fewer observations was excluded. Admissions to acute care hospitals between January 2011 and September 2015 (time of transition from ICD-9-CM to ICD-10-CM) that had one of the specified diagnosis codes were included. For a given hospital, the value of each of the five candidate metrics was defined as the ratio of all admissions that had the given testing and all admissions during the observation period with inclusion CCS diagnosis codes.

Derivation and Validation of the Index

In our derivation cohort (hospitals in Maryland, New Jersey, and Washington state), we tested the temporal stability of each candidate metric by year using the intraclass correlation coefficient (ICC). Using exploratory factor analysis (EFA) and Cronbach’s alpha, we then tested internal consistency of the index candidate components to ensure that all measured a common underlying factor (ie, diagnostic overuse). To standardize data, test rates for both of these analyses were converted to z-scores. For the EFA, we expected that if the index was reflecting only a single underlying factor, the Eigenvalue for one factor should be much higher (typically above 1.0) than that for multiple factors. We calculated item-test correlation for each candidate metric and Cronbach’s alpha for the entire index. A high and stable value for item-test correlation for each index component, as well as a high Cronbach’s alpha, suggests that index components measure a single common factor. Given the small number of test items, we considered a Cronbach’s alpha above 0.6 to be satisfactory.

This analysis showed satisfactory temporal stability of each candidate metric and good internal consistency of the candidate metrics in the derivation cohort. Therefore, we decided to keep all metrics rather than discard any of them. This same process was repeated with the validation cohort (Kentucky, New York, North Carolina, and West Virginia) and then with the combined group of seven states. Tests on the validation and entire cohort further supported our decision to keep all five metrics.

To determine the overall index value for a hospital, all of its metric numerators and denominators were added to calculate one fraction. In this way for a given hospital, a metric for which there were no observations was effectively excluded from the index. This essentially weights each index component by frequency. We chose to count syncope admissions only once in the denominator to avoid the index being unduly influenced by this diagnosis. The hospital index values were combined into their HSAs by adding numerators and denominators from each hospital to calculate HSA index values, effectively giving higher weight to hospitals with more observations. Spearman’s correlation coefficients were measured for these Dartmouth Atlas metrics, also at the HSA level. For the county level analysis, we used a hospital-county crosswalk (available from the American Hospital Association [AHA] Annual Survey; https://www.ahadata.com/aha-annual-survey-database) to link a hospital overuse index value to a county level cost value rather than aggregating data at the county level. We felt this was appropriate, as HSAs were constructed to represent a local healthcare market, whereas counties are less likely to be homogenous from a healthcare perspective.

Analysis of Entire Hospital Sample

The mean index value and SD were calculated for the entire sample of hospitals and for each state. The mean index value for each year of data was calculated to measure the temporal change of the index (representing a change in diagnostic intensity over the study period) using linear regression. We divided the cohort of hospitals into tertiles based on their index value. This is consistent with the CMS categorization of hospital payments and value of care as being “at,” “significantly above,” or “significantly below” a mean value.¹⁸ The characteristics of hospitals by tertile were described by mean total hospital beds, mean annual admissions, teaching status (nonteaching hospital, minor teaching hospital, major teaching hospital), and critical access hospital (yes/no). We utilized the AHA Annual Survey for data on hospital characteristics. We calculated P values using analysis of variance for hospital bed size and a chi-square test for teaching status and critical access hospital.

The entire group of hospitals from seven states was then used to apply the index to the HSA level. Numerators and denominators for each hospital in an HSA were added to calculate an HSA-level proportion. Thus, the HSA level index value, though unweighted, is dominated by hospitals with larger numbers of observations. For each of the Dartmouth metrics, the adjusted odds ratio of being in a higher diagnostic overuse index tertile given being in a certain Dartmouth Atlas metric tertile was calculated using ordinal logistic regression. This model controlled for the mean number of beds of hospitals in the HSA (continuous variable), mean Elixhauser Comorbidity Index (ECI) score (continuous variable; unweighted average among hospitals in an HSA), whether the HSA had a major or minor teaching hospital (yes/no) or was a critical access hospital (yes/no), and state fixed effects. The ECI score is a validated score that uses the presence or absence of 29 comorbidities to predict in-hospital mortality.¹⁹ For discriminant validity, we also tested two variables not expected to be associated with overuse—hospital ownership and affiliation with the Catholic Church.

For the county-level analysis, ordinal logistic regression was used to predict the adjusted odds ratio of being in a higher diagnostic overuse index tertile given being in a certain tertile of a given county-level spending metric. This model controlled for hospital bed size (continuous variable), hospital ECI score (continuous variable), teaching status (major, minor, nonteaching), critical access hospital status (yes/no), and state fixed effects.

RESULTS

Descriptive Statistics for Metrics

A total of 620 acute care hospitals were included in the index. Thirteen hospitals were excluded because their denominator was five or fewer. The vast majority of HSAs (85.9%) had only one hospital, 8.2% had two hospitals, and 2.4% had three hospitals. Similarly, the majority of counties (68.7%) had only one hospital, 15.1% had two hospitals, and 6.6% had three hospitals (Appendix Tables 1.1 and 1.2). Nonspecific chest pain was the metric with largest denominator mean (650), SD (1,012), and range (0-10,725) (Appendix Table 2). Overall, the metric denominators were a small fraction of total hospital discharges, with means at the hospital level ranging from 0.69% for nausea and vomiting to 5.81% for nonspecific chest pain, suggesting that our index relies on a relatively small fraction of discharges.

Tests for Temporal Stability and Internal Consistency by Derivation and Validation Strategy

Overall, the ICCs for the derivation, validation, and entire cohort suggested strong temporal stability (Appendix Table 3). The EFA of the derivation, validation, and entire cohort showed high Eigenvalues for one principal component, with no other factors close to 1, indicating strong internal consistency (Appendix Table 4). The Cronbach’s alpha analysis also suggested strong internal consistency, with alpha values ranging from 0.73 for the validation cohort to 0.80 for the derivation cohort (Table 2).

Correlation With External Validation Measures

For the entire cohort, the Spearman’s rho for correlation between our overuse index and inpatient rate of coronary angiography at the HSA level was 0.186 (95% CI, 0.089-0.283), Medicare reimbursement at the HSA level was 0.355 (95% CI, 0.272-0.437), and Medicare spending during the last 6 months of life at the HSA level was 0.149 (95% CI, 0.061-0.236) (Appendix Figures 5.1-5.3). The Spearman’s rho for correlation between our overuse index and county level standardized procedure cost was 0.284 (95% CI, 0.210-0.358), imaging cost was 0.268 (95% CI, 0.195-0.342), and testing cost was 0.226 (95% CI, 0.152-0.300) (Appendix Figures 6.1-6.3).

Overall Index Values and Change Over Time

The mean hospital index value was 0.541 (SD, 0.178) (Appendix Table 7). There was a slight but statistically significant annual increase in the overall mean index value over the study period, suggesting a small rise in overuse of diagnostic testing (coefficient 0.011; P <.001) (Appendix Figure 8).

Diagnostic Overuse Index Tertiles

Hospitals in the lowest tertile of the index tended to be smaller (based on number of beds) (P < .0001) and were more likely to be critical access hospitals (P <.0001). There was a significant difference in the proportion of nonteaching, minor teaching, and major teaching hospitals, with more nonteaching hospitals in tertile 1 (P = .001) (Table 3). The median ECI score was not significantly different among tertiles. Neither of the variables tested for discriminant validity (hospital ownership and Catholic Church affiliation) was associated with our index.

Adjusted Multilevel Mixed-Effects Ordinal Logistic Regression

Our overuse index correlated most closely with physician reimbursement, with an odds ratio of 2.02 (95% CI, 1.11-3.66) of being in a higher tertile of the overuse index when comparing tertiles 3 and 1 of this Dartmouth metric. Of the Medicare county-level metrics, our index correlated most closely with cost of procedures, with an odds ratio of 2.03 (95% CI, 1.21-3.39) of being in a higher overuse index tertile when comparing tertiles 3 and 1 of the cost per procedure metric (Figure 1).

DISCUSSION

Previous research shows variation among hospitals for overall physician spending,²⁰ noninvasive cardiac imaging,²¹ and the rate of finding obstructive lesions during elective coronary angiography.²² However, there is a lack of standardized methods to study a broad range of diagnostic overuse at the hospital level. To our knowledge, no studies have attempted to develop a diagnostic overuse index at the hospital level. We used a derivation-validation approach to achieve our goal. Although the five metrics represent a range of conditions, the EFA and Cronbach’s alpha tests suggest that they measure a common phenomenon. To avoid systematically excluding smaller hospitals, we limited the extent to which we eliminated hospitals with few observations. Our findings suggest that it may be reasonable to make generalizations on the diagnostic intensity of a hospital based on a relatively small number of discharges. Moreover, our index is a proof of concept that rates of negative diagnostic testing can serve as a proxy for estimating diagnostic overuse.

Our hospital-level index values extrapolated to the HSA level weakly correlated with prespecified Dartmouth Atlas metrics. In a multivariate ordinal regression, there was a significant though weak association between hospitals in higher tertiles of the Dartmouth Atlas metrics and categorization in higher tertiles of our diagnostic overuse index. Similarly, our hospital-level index correlated with two of the three county-level metrics in a multivariate ordinal regression.

We do not assume that all of the metrics in our index track together. However, our results, including the wide dispersion of index values among the tertiles (Table 3), suggest that at least some hospitals are outliers in multiple metrics. We did not assume ex ante that our index should correlate with Dartmouth overuse metrics or Medicare county-level spending; however, we did believe that an association with these measures would assist in validating our index. Given that our index utilizes four common diagnoses, while the Dartmouth and Medicare cost metrics are based on a much broader range of conditions, we would not expect more than a weak correlation even if our index is a valid way to measure overuse.

All of the metrics were based on the concept that hospitals with high rates of negative testing are likely providing large amounts of low-value care. Prior studies on diagnostic yield of CT scans in the emergency department for pulmonary embolus (PE) found an increase in testing and decrease in yield over time; these studies also showed that physicians with more experience ordered fewer CT scans and had a higher yield.²³ A review of electronic health records and billing data also showed that hospitals with higher rates of D-dimer testing had higher yields on CT scans ordered to test for PE.²⁴

We took advantage of the coding convention that certain diagnoses only be listed as the primary discharge diagnosis if no more specific diagnosis is made. This allowed us to identify hospitals that likely had high rates of negative tests without granular data. Of course, the metrics are not measuring rates of negative testing per se, but a proxy for this, based instead on the proportion of patients with a symptom-based primary discharge diagnosis who underwent diagnostic testing.

Measuring diagnostic overuse at the hospital level may help to understand factors that drive overuse, given that institutional incentives and culture likely play important roles in ordering tests. There is evidence that financial incentives drive physicians’ decisions,^25-27 and there is also evidence that institutional culture impacts outcomes.²⁸ Further, quality improvement projects are typically designed at the hospital level and may be an effective way to curb overuse.^29,30

Previous studies have focused on measuring variation among providers and identifying outlier physicians.^9,10,20 Providing feedback to underperforming physicians has been shown to change practice habits.^31,32 Efforts to improve the practice habits of outlier hospitals may have a number of advantages, including economies of scale and scope and the added benefit of improving the habits of all providers—not just those who are underperforming.

Ordering expensive diagnostic tests on patients with a low pretest probability of having an organic etiology for their symptoms contributes to high healthcare costs. Of course, we do not believe that the ideal rate of negative testing is zero. However, hospitals with high rates of negative diagnostic testing are more likely to be those with clinicians who use expensive tests as a substitute for clinical judgment or less-expensive tests (eg, D-dimer testing to rule out PE).

One challenge we faced is that there is no gold standard of hospital-level overuse with which to validate our index. Our index is weakly correlated with a number of regional metrics that may be proxies for overuse. We are reassured that there is a statistically significant correlation with measures at both HSA and county levels. These correlations are weak, but these regional metrics are themselves imperfect surrogates for overuse. Furthermore, our index is preliminary and will need refinement in future studies.

Limitations

Our analysis has multiple limitations. First, since it relies heavily on primary ICD discharge diagnosis codes, biases could exist due to variations in coding practices. Second, the SID does not include observation stays or tests conducted in the ED, so differential use of observation stays among hospitals might impact results. Finally, based on utilization data, we were not able to distinguish between CT scans of the chest, abdomen, and pelvis because the SID labels each of these as body CT.

CONCLUSION

We developed a novel index to measure diagnostic intensity at the hospital level. This index relies on the concept that high rates of negative diagnostic testing likely indicate some degree of overuse. Our index is parsimonious, does not require granular claims data, and measures a range of potentially overused tests for common clinical scenarios. Our next steps include further refining the index, testing it with granular data, and validating it with other datasets. Thereafter, this index may be useful at identifying positive and negative outliers to understand what processes of care contribute to outlier high and low levels of diagnostic testing. We suspect our index is more useful at identifying extremes than comparing hospitals in the middle of the utilization curve. Additionally, exploring the relationship among individual metrics and the relationship between our index and quality measures like mortality and readmissions may be informative.

There is substantial geographic variation in intensity of healthcare use in the United States,¹ yet areas with higher healthcare utilization do not demonstrate superior clinical outcomes.² Low-value care exposes patients to unnecessary anxiety, radiation, and risk for adverse events.

Previous research has focused on measuring low-value care at the level of hospital referral regions,^3-6 metropolitan statistical areas,⁷ provider organizations,⁸ and individual physicians.^9,10 Hospital referral regions designate regional healthcare markets for tertiary care and generally include at least one major referral center.¹¹ Well-calibrated and validated hospital-level measures of diagnostic overuse are lacking.

We sought to construct a novel index to measure hospital level overuse of diagnostic testing. We focused on diagnostic intensity rather than other forms of overuse such as screening or treatment intensity. Moreover, we aimed to create a parsimonious index—one that is simple, relies on a small number of inputs, is derived from readily available administrative data without the need for chart review or complex logic, and does not require exclusion criteria.

METHODS

Conceptual Framework for Choosing Index Components

To create our overuse index, we took advantage of the requirements for International Classification of Diseases, 9th Revision-Clinical Modification (ICD-9-CM) billing codes 780-796; these codes are based on “symptoms, signs, and ill-defined conditions” and can only be listed as the primary discharge diagnosis if no more specific diagnosis is made.¹² As such, when coupled with expensive tests, a high prevalence of these symptom-based diagnosis codes at discharge may serve as a proxy for low-value care. One of the candidate metrics we selected was based on Choosing Wisely^® recommendations.¹³ The other candidate metrics were based on clinical experience and consensus of the study team.

Data Sources

We used hospital-level data on primary discharge diagnosis codes and utilization of testing data from the State Inpatient Databases (SID), which are part of the Agency for Healthcare Research and Quality Healthcare Cost and Utilization Project (HCUP). Our derivation cohort used data from acute care hospitals in Maryland, New Jersey, and Washington state. Our validation cohort used data from acute care hospitals in Kentucky, North Carolina, New York, and West Virginia. States were selected based on availability of data (certain states lacked complete testing utilization data) and cost of data acquisition. The SID contains hospital-level utilization of computed tomography (CT) scans (CT of the body and head) and diagnostic testing, including stress testing and esophagogastroduodenoscopy (EGD).

Data on three prespecified Dartmouth Atlas of Health Care metrics at the hospital service area (HSA) level were obtained from the Dartmouth Atlas website.¹⁴ These metrics were (1) rate of inpatient coronary angiograms per 1,000 Medicare enrollees, (2) price-adjusted physician reimbursement per fee-for-service Medicare enrollee per year (adjusted for patient sex, race, and age), and (3) mean inpatient spending per decedent in the last 6 months of life.¹⁵ Data on three prespecified Medicare metrics at the county level were obtained from the Centers for Medicare & Medicaid Services (CMS) website.¹⁶ These metrics were standardized per capita cost per (1) procedure, (2) imaging, and (3) test of Medicare fee-for-service patients. The CMS uses the Berenson-Eggers Type of Service Codes to classify fee-generating interventions into a number of categories, including procedure, imaging, and test.¹⁷

Components of the Overuse Index

We tested five candidate metrics for index inclusion (Table 1). We utilized Clinical Classifications Software (CCS) codes provided by HCUP, which combine several ICD-9-CM codes into a single primary CCS discharge code for ease of use. The components were (1) primary CCS diagnosis of “nausea and vomiting” coupled with body CT scan or EGD, (2) primary CCS diagnosis of abdominal pain and body CT scan or EGD, (3) primary CCS diagnosis of “nonspecific chest pain” and body CT scan or stress test, (4) primary CCS diagnosis of syncope and stress test, and (5) primary CCS diagnosis for syncope and CT of the brain. For a given metric, the denominator was all patients with the particular primary CCS discharge diagnosis code. The numerator was patients with the diagnostic code who also had the specific test or procedure. We characterized the denominators of each metric in terms of mean, SD, and range.

Index Inclusion Criteria and Construction

Specialty, pediatric, rehabilitation, and long-term care hospitals were excluded. Moreover, any hospital with an overall denominator (for the entire index, not an individual metric) of five or fewer observations was excluded. Admissions to acute care hospitals between January 2011 and September 2015 (time of transition from ICD-9-CM to ICD-10-CM) that had one of the specified diagnosis codes were included. For a given hospital, the value of each of the five candidate metrics was defined as the ratio of all admissions that had the given testing and all admissions during the observation period with inclusion CCS diagnosis codes.

Derivation and Validation of the Index

In our derivation cohort (hospitals in Maryland, New Jersey, and Washington state), we tested the temporal stability of each candidate metric by year using the intraclass correlation coefficient (ICC). Using exploratory factor analysis (EFA) and Cronbach’s alpha, we then tested internal consistency of the index candidate components to ensure that all measured a common underlying factor (ie, diagnostic overuse). To standardize data, test rates for both of these analyses were converted to z-scores. For the EFA, we expected that if the index was reflecting only a single underlying factor, the Eigenvalue for one factor should be much higher (typically above 1.0) than that for multiple factors. We calculated item-test correlation for each candidate metric and Cronbach’s alpha for the entire index. A high and stable value for item-test correlation for each index component, as well as a high Cronbach’s alpha, suggests that index components measure a single common factor. Given the small number of test items, we considered a Cronbach’s alpha above 0.6 to be satisfactory.

This analysis showed satisfactory temporal stability of each candidate metric and good internal consistency of the candidate metrics in the derivation cohort. Therefore, we decided to keep all metrics rather than discard any of them. This same process was repeated with the validation cohort (Kentucky, New York, North Carolina, and West Virginia) and then with the combined group of seven states. Tests on the validation and entire cohort further supported our decision to keep all five metrics.

To determine the overall index value for a hospital, all of its metric numerators and denominators were added to calculate one fraction. In this way for a given hospital, a metric for which there were no observations was effectively excluded from the index. This essentially weights each index component by frequency. We chose to count syncope admissions only once in the denominator to avoid the index being unduly influenced by this diagnosis. The hospital index values were combined into their HSAs by adding numerators and denominators from each hospital to calculate HSA index values, effectively giving higher weight to hospitals with more observations. Spearman’s correlation coefficients were measured for these Dartmouth Atlas metrics, also at the HSA level. For the county level analysis, we used a hospital-county crosswalk (available from the American Hospital Association [AHA] Annual Survey; https://www.ahadata.com/aha-annual-survey-database) to link a hospital overuse index value to a county level cost value rather than aggregating data at the county level. We felt this was appropriate, as HSAs were constructed to represent a local healthcare market, whereas counties are less likely to be homogenous from a healthcare perspective.

Analysis of Entire Hospital Sample

The mean index value and SD were calculated for the entire sample of hospitals and for each state. The mean index value for each year of data was calculated to measure the temporal change of the index (representing a change in diagnostic intensity over the study period) using linear regression. We divided the cohort of hospitals into tertiles based on their index value. This is consistent with the CMS categorization of hospital payments and value of care as being “at,” “significantly above,” or “significantly below” a mean value.¹⁸ The characteristics of hospitals by tertile were described by mean total hospital beds, mean annual admissions, teaching status (nonteaching hospital, minor teaching hospital, major teaching hospital), and critical access hospital (yes/no). We utilized the AHA Annual Survey for data on hospital characteristics. We calculated P values using analysis of variance for hospital bed size and a chi-square test for teaching status and critical access hospital.

The entire group of hospitals from seven states was then used to apply the index to the HSA level. Numerators and denominators for each hospital in an HSA were added to calculate an HSA-level proportion. Thus, the HSA level index value, though unweighted, is dominated by hospitals with larger numbers of observations. For each of the Dartmouth metrics, the adjusted odds ratio of being in a higher diagnostic overuse index tertile given being in a certain Dartmouth Atlas metric tertile was calculated using ordinal logistic regression. This model controlled for the mean number of beds of hospitals in the HSA (continuous variable), mean Elixhauser Comorbidity Index (ECI) score (continuous variable; unweighted average among hospitals in an HSA), whether the HSA had a major or minor teaching hospital (yes/no) or was a critical access hospital (yes/no), and state fixed effects. The ECI score is a validated score that uses the presence or absence of 29 comorbidities to predict in-hospital mortality.¹⁹ For discriminant validity, we also tested two variables not expected to be associated with overuse—hospital ownership and affiliation with the Catholic Church.

For the county-level analysis, ordinal logistic regression was used to predict the adjusted odds ratio of being in a higher diagnostic overuse index tertile given being in a certain tertile of a given county-level spending metric. This model controlled for hospital bed size (continuous variable), hospital ECI score (continuous variable), teaching status (major, minor, nonteaching), critical access hospital status (yes/no), and state fixed effects.

RESULTS

Descriptive Statistics for Metrics

A total of 620 acute care hospitals were included in the index. Thirteen hospitals were excluded because their denominator was five or fewer. The vast majority of HSAs (85.9%) had only one hospital, 8.2% had two hospitals, and 2.4% had three hospitals. Similarly, the majority of counties (68.7%) had only one hospital, 15.1% had two hospitals, and 6.6% had three hospitals (Appendix Tables 1.1 and 1.2). Nonspecific chest pain was the metric with largest denominator mean (650), SD (1,012), and range (0-10,725) (Appendix Table 2). Overall, the metric denominators were a small fraction of total hospital discharges, with means at the hospital level ranging from 0.69% for nausea and vomiting to 5.81% for nonspecific chest pain, suggesting that our index relies on a relatively small fraction of discharges.

Tests for Temporal Stability and Internal Consistency by Derivation and Validation Strategy

Overall, the ICCs for the derivation, validation, and entire cohort suggested strong temporal stability (Appendix Table 3). The EFA of the derivation, validation, and entire cohort showed high Eigenvalues for one principal component, with no other factors close to 1, indicating strong internal consistency (Appendix Table 4). The Cronbach’s alpha analysis also suggested strong internal consistency, with alpha values ranging from 0.73 for the validation cohort to 0.80 for the derivation cohort (Table 2).

Correlation With External Validation Measures

For the entire cohort, the Spearman’s rho for correlation between our overuse index and inpatient rate of coronary angiography at the HSA level was 0.186 (95% CI, 0.089-0.283), Medicare reimbursement at the HSA level was 0.355 (95% CI, 0.272-0.437), and Medicare spending during the last 6 months of life at the HSA level was 0.149 (95% CI, 0.061-0.236) (Appendix Figures 5.1-5.3). The Spearman’s rho for correlation between our overuse index and county level standardized procedure cost was 0.284 (95% CI, 0.210-0.358), imaging cost was 0.268 (95% CI, 0.195-0.342), and testing cost was 0.226 (95% CI, 0.152-0.300) (Appendix Figures 6.1-6.3).

Overall Index Values and Change Over Time

The mean hospital index value was 0.541 (SD, 0.178) (Appendix Table 7). There was a slight but statistically significant annual increase in the overall mean index value over the study period, suggesting a small rise in overuse of diagnostic testing (coefficient 0.011; P <.001) (Appendix Figure 8).

Diagnostic Overuse Index Tertiles

Hospitals in the lowest tertile of the index tended to be smaller (based on number of beds) (P < .0001) and were more likely to be critical access hospitals (P <.0001). There was a significant difference in the proportion of nonteaching, minor teaching, and major teaching hospitals, with more nonteaching hospitals in tertile 1 (P = .001) (Table 3). The median ECI score was not significantly different among tertiles. Neither of the variables tested for discriminant validity (hospital ownership and Catholic Church affiliation) was associated with our index.

Adjusted Multilevel Mixed-Effects Ordinal Logistic Regression

Our overuse index correlated most closely with physician reimbursement, with an odds ratio of 2.02 (95% CI, 1.11-3.66) of being in a higher tertile of the overuse index when comparing tertiles 3 and 1 of this Dartmouth metric. Of the Medicare county-level metrics, our index correlated most closely with cost of procedures, with an odds ratio of 2.03 (95% CI, 1.21-3.39) of being in a higher overuse index tertile when comparing tertiles 3 and 1 of the cost per procedure metric (Figure 1).

DISCUSSION

Previous research shows variation among hospitals for overall physician spending,²⁰ noninvasive cardiac imaging,²¹ and the rate of finding obstructive lesions during elective coronary angiography.²² However, there is a lack of standardized methods to study a broad range of diagnostic overuse at the hospital level. To our knowledge, no studies have attempted to develop a diagnostic overuse index at the hospital level. We used a derivation-validation approach to achieve our goal. Although the five metrics represent a range of conditions, the EFA and Cronbach’s alpha tests suggest that they measure a common phenomenon. To avoid systematically excluding smaller hospitals, we limited the extent to which we eliminated hospitals with few observations. Our findings suggest that it may be reasonable to make generalizations on the diagnostic intensity of a hospital based on a relatively small number of discharges. Moreover, our index is a proof of concept that rates of negative diagnostic testing can serve as a proxy for estimating diagnostic overuse.

Our hospital-level index values extrapolated to the HSA level weakly correlated with prespecified Dartmouth Atlas metrics. In a multivariate ordinal regression, there was a significant though weak association between hospitals in higher tertiles of the Dartmouth Atlas metrics and categorization in higher tertiles of our diagnostic overuse index. Similarly, our hospital-level index correlated with two of the three county-level metrics in a multivariate ordinal regression.

We do not assume that all of the metrics in our index track together. However, our results, including the wide dispersion of index values among the tertiles (Table 3), suggest that at least some hospitals are outliers in multiple metrics. We did not assume ex ante that our index should correlate with Dartmouth overuse metrics or Medicare county-level spending; however, we did believe that an association with these measures would assist in validating our index. Given that our index utilizes four common diagnoses, while the Dartmouth and Medicare cost metrics are based on a much broader range of conditions, we would not expect more than a weak correlation even if our index is a valid way to measure overuse.

All of the metrics were based on the concept that hospitals with high rates of negative testing are likely providing large amounts of low-value care. Prior studies on diagnostic yield of CT scans in the emergency department for pulmonary embolus (PE) found an increase in testing and decrease in yield over time; these studies also showed that physicians with more experience ordered fewer CT scans and had a higher yield.²³ A review of electronic health records and billing data also showed that hospitals with higher rates of D-dimer testing had higher yields on CT scans ordered to test for PE.²⁴

We took advantage of the coding convention that certain diagnoses only be listed as the primary discharge diagnosis if no more specific diagnosis is made. This allowed us to identify hospitals that likely had high rates of negative tests without granular data. Of course, the metrics are not measuring rates of negative testing per se, but a proxy for this, based instead on the proportion of patients with a symptom-based primary discharge diagnosis who underwent diagnostic testing.

Measuring diagnostic overuse at the hospital level may help to understand factors that drive overuse, given that institutional incentives and culture likely play important roles in ordering tests. There is evidence that financial incentives drive physicians’ decisions,^25-27 and there is also evidence that institutional culture impacts outcomes.²⁸ Further, quality improvement projects are typically designed at the hospital level and may be an effective way to curb overuse.^29,30

Previous studies have focused on measuring variation among providers and identifying outlier physicians.^9,10,20 Providing feedback to underperforming physicians has been shown to change practice habits.^31,32 Efforts to improve the practice habits of outlier hospitals may have a number of advantages, including economies of scale and scope and the added benefit of improving the habits of all providers—not just those who are underperforming.

Ordering expensive diagnostic tests on patients with a low pretest probability of having an organic etiology for their symptoms contributes to high healthcare costs. Of course, we do not believe that the ideal rate of negative testing is zero. However, hospitals with high rates of negative diagnostic testing are more likely to be those with clinicians who use expensive tests as a substitute for clinical judgment or less-expensive tests (eg, D-dimer testing to rule out PE).

One challenge we faced is that there is no gold standard of hospital-level overuse with which to validate our index. Our index is weakly correlated with a number of regional metrics that may be proxies for overuse. We are reassured that there is a statistically significant correlation with measures at both HSA and county levels. These correlations are weak, but these regional metrics are themselves imperfect surrogates for overuse. Furthermore, our index is preliminary and will need refinement in future studies.

Limitations

Our analysis has multiple limitations. First, since it relies heavily on primary ICD discharge diagnosis codes, biases could exist due to variations in coding practices. Second, the SID does not include observation stays or tests conducted in the ED, so differential use of observation stays among hospitals might impact results. Finally, based on utilization data, we were not able to distinguish between CT scans of the chest, abdomen, and pelvis because the SID labels each of these as body CT.

CONCLUSION

We developed a novel index to measure diagnostic intensity at the hospital level. This index relies on the concept that high rates of negative diagnostic testing likely indicate some degree of overuse. Our index is parsimonious, does not require granular claims data, and measures a range of potentially overused tests for common clinical scenarios. Our next steps include further refining the index, testing it with granular data, and validating it with other datasets. Thereafter, this index may be useful at identifying positive and negative outliers to understand what processes of care contribute to outlier high and low levels of diagnostic testing. We suspect our index is more useful at identifying extremes than comparing hospitals in the middle of the utilization curve. Additionally, exploring the relationship among individual metrics and the relationship between our index and quality measures like mortality and readmissions may be informative.

Gender-based discrimination refers to “any distinction, exclusion or restriction made on the basis of socially constructed gender roles and norms which prevents a person from enjoying full human rights.”¹ Similarly, sexual harassment encompasses a spectrum of sexual conduct that includes “unwelcome sexual advances, requests for sexual favors, and other verbal or physical harassment of a sexual nature,” as defined by the US Equal Employment Opportunity Commission.² Gender-based discrimination and sexual harassment can be “overt,” which includes explicitly recognizable behaviors, or they can be “implicit,” which includes verbal and nonverbal behaviors that often go unrecognized but convey hostility, objectification, or exclusion of another person. Gender-based discrimination and sexual harassment are commonly described and likely more prevalent in academic settings.^3-6 Female physicians are disproportionately affected by gender-based discrimination and sexual harassment, compared with their male peers.^4,7

Female physicians face workplace harassment from both patients and coworkers. A study in Canada reported that more than 75% of female physicians experienced sexual harassment from their patients.⁸ A recent study showed almost half of the physicians who reported harassment, which was three times more often among female physicians, described other physician colleagues as perpetrators.⁹ In a study among clinician-researchers in the field of academic medicine, 30% of females reported having experienced sexual harassment, compared with 4% of males.⁷ Among females who reported harassment in this study, 47% stated that these experiences adversely affected their opportunities for career advancement. Career stage may also affect experiences or perceptions of gender-based discrimination and sexual harassment, with females in earlier career stages reporting a less favorable environment of gender equity.¹⁰

Hospital medicine is a young and evolving specialty, and the number of hospitalists has increased substantially from a few hundred at the time of inception to over 50,000 as of 2016.¹¹ The proportion of female hospitalists increased from 31% in 2012 to 52% in 2014, reflecting equal gender representation in hospital medicine.¹² Available evidence shows gender disparities exist in hospital medicine disproportionately affecting female hospitalists in their career advancement, including leadership and scholarship opportunities.¹³ However, there remains a knowledge gap regarding the prevalence of gender-based discrimination and sexual harassment experienced by hospitalists.

Our study examines the experiences of academic hospitalists regarding gender-based discrimination and sexual harassment and the impact of gender on career satisfaction and advancement.

METHODS

Study Design and Participants

An online survey was developed and approved by the institutional board review (IRB) at the Medical College of Wisconsin in Milwaukee. University-based academic centers with hospitalist programs, identified through personal connections, from across the continental United Stated were identified as potential study sites, and leaders at each institution were contacted to ascertain potential participation in the survey. The survey was distributed to Internal Medicine hospitalists at 18 participating academic institutions from January 2019 to June 2019. Participation was voluntary. The cover letter explained the purpose of the study and provided a link to the survey. To maintain anonymity, none of the questionnaires requested identifying information from participants. A web-based Qualtrics online-based survey platform was used.

Survey Elements

The survey aimed to assess several elements of gender-based discrimination and sexual harassment. All questions about these experiences distinguished encounters with patients from those with colleagues, and questions about occurrences either over a career or in the last 30 days were intended to capture both distant and recent timeframes. The theme for the questions for the survey was based on previous studies.^4,7,8 The wording of questions was simplified to make them easily understandable, and the brevity of the survey was maintained to prevent possible nonresponses.¹⁴ Additional questions (mistaken for a healthcare provider other than a physician, feeling respected by patients and colleagues, referred to by terms such as “honey,” “dear,” “sweetheart,” “sugar,” or equivalent), which were deemed relevant in day-to-day clinical practice through consensus among study investigators and discussions among peer hospitalists, were incorporated into the final survey (Appendix). Survey questions were intended to capture several elements regarding interactions with patients and with colleagues or other healthcare providers (HCPs).

Questions on gender-based discrimination included:

• Has a patient [colleague or other healthcare provider] mistaken you for a healthcare provider other than a physician?
• Has a patient [colleague or other healthcare provider] asked you to do something not at your level of training?
• Do you feel respected?
Do you perceive your gender has impacted opportunities for your career advancement?

Questions on sexual harassment included:

• Has a patient [colleague or other healthcare provider] touched you inappropriately, made sexual remarks or gestures, or made suggestive looks?
• Has a patient [colleague or other healthcare provider] referred to you by terms such as “honey,” “dear,” “sweetheart,” “sugar,” or equivalent?

In addition, the survey sought demographic information of the participants (age, gender, and race/ethnicity) and information about their individual institutions (names and locations) (Appendix). The geographical locations of the institutions were further categorized into four different regions according to the United States Census Bureau (Northwest, Midwest, South, and West). At the end of the survey, participants were given an opportunity to provide any additional comments.

Statistical Analysis

Standard descriptive summary statistics were used for demographic data. Associations between the variables were analyzed using chi-square test or Fischer’s exact test, as appropriate, for categorical variables and t test for continuous variables. The variations among institution-based responses were presented in the form of inter-quartile range (IQR). All tests were 2-sided, and P values less than .05 were considered statistically significant. All analyses were performed using IBM^® SPSS^® Statistics software version 24. Relevant responses representative of the overall respondents’ sentiments as provided under additional comment section were discussed and cited.

RESULTS

Eighteen different academic institutions across the United States participated in the study, with 336 individual responses. The majority of respondents were females (57%), in younger age categories (58% were 30-39 years old), Caucasian (59%), and early-career hospitalists (>50% working as hospitalists for ≤5 years) (Table 1). Regarding the overall geographic distribution, the largest number of responses were from the Midwest (n = 115; 35.6%) (Table 1 and Appendix).

Gender Discrimination

Interactions With Patients

Over their careers, 69% of hospitalists reported being mistaken for an HCP other than a physician by patients. This was more common among females than among males (99% vs 29%, respectively; P < .001) (Table 2). Almost half (48%) reported this had occurred in the last 30 days, more frequently by females (76% vs 10%; P < .001).

Of responding hospitalists, 96% stated that, over their careers, they have been asked by patients to do something they did not consider to be at their level of training (eg, help get food or water, help with a bed pan), with a higher prevalence of such experiences among females than males (99% vs 93%, respectively; P = .004) (Table 2). Most (71%) said they had experienced this in the last 30 days, which was again more frequently reported by females (78% vs 62%; P = .001).

The responses from female hospitalists regarding their career-long experiences of being mistaken for an HCP or asked to do something not at their level of training by their patients had both the highest number of positive responses across institutions (median of hospital proportions, 100%) and the least institutional variation since both had the narrowest IQR) (Table 2).

Interactions With Colleagues or Other HCPs

Among hospitalists responding to the survey, 46% felt that, over their careers, they had been mistaken for nonphysician HCPs by colleagues or other HCPs. This was more prevalent among females than among males (65% vs 20%; P < .001) (Table 2). Among respondents, 14% reported these events had occurred in the last 30 days, which was again more common among females (21% vs 5%; P < .001).

Over their careers, 26% of hospitalists reported they have been asked by a colleague or HCP to do something not at their level of training (eg, clean up the physician’s work room, make coffee, take notes in a meeting), with similar prevalence among females and males (29% vs 23%; P = .228). Ten percent reported these occurrences in the last 30 days, which was similar between females and males (12% vs 9%; P = .330).

Feelings of Respect and Opportunities for Career Advancement

When asked to rate the statement “I feel respected by patients” on a 5-point Likert scale, female hospitalists overall scored significantly lower as compared with their male counterparts (mean score, 3.73 vs 4.04; P < .001) (Table 3); this was also true when asked about feelings of respect by physicians (mean score, 3.84 vs 4.15; P < .001). Female hospitalists were more likely than males to report that their gender has more negatively impacted their opportunities for career advancement (mean score, 2.73 vs 3.34; P < .001).

Sexual Harassment

Interactions With Patients

Over half (57%) of hospitalists reported career-long experiences of patient(s) touching them inappropriately, making sexual remarks or gestures, or making suggestive looks. These experiences were more prevalent among females than among males (72% vs 36%, respectively; P < .001) (Table 2). Fifteen percent said they had such experience in rhe last 30 days, which was also more common among females (22% vs 6%; P < .001).

Most hospitalists (84%) reported that patients have referred to them by inappropriately familiar terms such as “honey,” “dear,” “sweetheart,” “sugar,” or equivalent over their careers, with females more frequently reporting these behaviors (95% vs 68%; P < .001). Experiencing this during the last 30 days was reported by 48%, which was again more common among females (68% vs 23%; P < .001).

Interactions With Colleagues or Other HCPs

Within their careers, 15% of hospitalists reported at least one experience of a colleague or HCP touching them inappropriately or making sexual remarks, gestures, or suggestive looks. This was more prevalent for females than males (18% vs 10%, respectively; P = .033). Only 2% of both females and males reported these experiences in the last 30 days (2% vs 2%; P = .981).

Almost one-third of participants (32%) affirmed that another HCP has referred to them by terms such as “honey,” “dear,” “sweetheart,” “sugar,” or equivalent in their career, with a higher proportion of females than males reporting these events (39% vs 23%; P = .002) (Table 2). Experiencing this during the last 30 days was reported by 10%, which was similar between females and males (12% vs 7%; P = .112).

Additional Comments From Respondents

• “Throughout my training and now into my professional career, there are nearly weekly incidents of elderly male patients referring to me as “honey/dear/sweetie” or even by my first name, even though I introduce myself as their physician and politely correct them. They will often refer to me as a nurse and ask me to do something not at my level of training. Sometimes even despite correcting the patient, they continue to refer to me as such. Throughout the years, other female MDs and I have discussed that this is ‘status quo’ for female physicians and observe that this is not an experience that male MDs share.”
• “I frequently round with a male nurse practitioner and the patients almost always, despite introducing ourselves and our roles, turn to him and ask him questions instead of addressing them to me.”
• “Our institution allows female faculty to be interviewed about childcare, household labor division, plans for pregnancy. One professor asks women private details about their private relationships such as what they do with spouse on date night or weekends away.”
• “It’s hard to answer questions related to my level of training. I don’t think it’s unreasonable for people to ask me to do things, no matter my level of training. . . . I don’t think being a doctor means that I am above this, or that it is inappropriate to be asked to do this.”

DISCUSSION

This survey demonstrated that gender-based discrimination and sexual harassment in the academic hospitalist healthcare environment are common, both in more distant and recent time frames. Notably, these experiences are shared by female and male physicians in interactions with both patients and colleagues, though male hospitalists report most of these experiences at significantly lower frequencies than females. These results support past work showing that female physicians are significantly more likely to be subjected to gender-based discrimination and sexual harassment, but also challenges the perception that gender-based discrimination and sexual harassment are uniquely experienced by females.

A startling number of females and males in the study reported sexual harassment (inappropriate touching, remarks, gestures, and looks) when interacting with patients throughout their careers and in last 30 days. Many males and females reported that patients had referred to them with inappropriately familiar, and potentially demeaning, terms of endearment. For both overt and implicit sexual harassment, females were significantly more likely than males to report experiencing these behaviors when interacting with patients. Although some of these experiences may seem less harmful than others, a meta-analysis demonstrated that frequent, less intense experiences of gender-based discrimination and sexual harassment have a similar impact on female’s well-being as do less frequent, more intense experiences.¹⁵ Although the person using the terms of endearment like “honey,” “sugar,” or “sweetheart” may feel the terms are harmless, such expressions can be inappropriate and constitute sexual harassment according to the U.S. Department of the Interior’s Office of Civil Rights.¹⁶ The Sexual Harassment/Assault Response and Prevention Program (SHARP) also classifies such terms into verbal categories of sexual harrassment.¹⁷

Of female physicians surveyed, 99% reported that they had been mistaken for HCPs other than physicians by their patients over their careers. Although this was also reported by male physicians, the experience was 3.4 times as likely for female physicians. Misidentification by patients may represent a disconnect between the growing female representation in the physician workforce and patients’ conceptions of the traditional image of a physician.

In parallel with this finding of misidentification, an interesting area of the study was the question regarding being asked to do “something not at your level of training.” A recurring theme in the comments was a rejection of the notion that certain tasks were “beneath a level of training,” suggesting a common view that acts of caregiving are not bounded by hierarchy. Analysis of qualitative responses showed that 40% of these responses had comments regarding this question. An example was “It’s hard to answer questions related to my level of training. . . . I don’t think being a doctor means that I am above this, or that it is inappropriate to be asked to do this.” Notably, however, a larger number of female than male physicians responded yes to this question in both study time frames. This points to a differential in how female physicians are viewed by patients, both in frequent misidentification and in behaviors more frequently asked of female physicians than their male counterparts. Given the comments, it may also suggest a difference in how female and male physicians perceive the fluidity of bounds on their care-taking roles set by their “level of training.”

A large number of study participants were early-career hospitalists, which may in part explain some of the study results. In a previous study of gender equity in an Internal Medicine department, physicians practicing medicine for more than 15 years perceived the departmental culture as more favorable than physicians with shorter careers.¹⁰ Additionally, the perception of cultures was most discordant between senior male physicians and junior female physicians.¹⁰ Because many hospitalists are early-career physicians, they may have trained in an environment that had heightened awareness surrounding gender-based discrimination and sexual harassment, which affects the overall study results.

Multiple qualitative comments, mentioned above, were submitted by participants describing their experiences in all categories. Such comments paint a picture of insidious bias and cultural norms affecting the quality of female physicians’ work lives.

Two questions focused on career satisfaction and the sense of respect from patients and colleagues. In both responses, there was a statistically different response between males and females, with females less likely to report that they felt respected and that their gender adversely impacted their opportunities for career advancement. This is disturbing information and warrants more investigation.

The reasons for the observed prevalence of gender-based discrimination and sexual harassment in this broad survey of academic hospitalists are uncertain. Multiple studies to date have demonstrated that gender-based discrimination and sexual harassment have historically existed in medicine and continue to even today. Unlike physicians with long-term relationships with patients, hospitalists may face more exposure due to a lack of long-term continuity with patients. The absence of an established trust in the relationship also may make them more vulnerable to inappropriate behaviors when interacting with patients. Hospital medicine, however, is a young specialty with equal gender representation and should be at the forefront of addressing and solving these issues of gender-based discrimination and sexual harassment.

The survey had a good distribution between female and male participants. Additionally, the survey reflected the general distribution of the national hospitalist workforce in gender, age, and ethnic/racial distribution, as well as number of years in practice.¹² The study surveyed respondents regarding experiences in both long- and short-term time frames, as well as experiences with patients and colleagues.

Our study reflects a cross-sectional snapshot of hospitalists’ perceptions with no longitudinal follow-up. Since the survey was limited to academic medical centers, it may not reflect experiences in community/private practice settings. The small number of participants limited the ability to perform subgroup analyses by age, race, or years in practice, which may play a role in interactions with patients and colleagues. Since the number of respondents varied greatly by institution, a minority of institutions could have influenced some of the findings. Narrow IQRs of the hospital proportions as shown in Table 2 would suggest similar responses across institutions, whereas wide IQRs would suggest that a smaller number of institutions were possibly driving the findings. Because of the survey distribution method, it is unknown how many physicians received the survey and a response rate could not be calculated. Further, selection, recall, and detection biases cannot be ruled out.

CONCLUSION

This survey shows that gender-based discrimination and sexual harassment in the academic hospitalist healthcare environment are common and more frequently experienced by female physicians, both in interactions with patients and colleagues. Our study highlights the need to address this prevalent issue among academic hospitalists.

Gender-based discrimination refers to “any distinction, exclusion or restriction made on the basis of socially constructed gender roles and norms which prevents a person from enjoying full human rights.”¹ Similarly, sexual harassment encompasses a spectrum of sexual conduct that includes “unwelcome sexual advances, requests for sexual favors, and other verbal or physical harassment of a sexual nature,” as defined by the US Equal Employment Opportunity Commission.² Gender-based discrimination and sexual harassment can be “overt,” which includes explicitly recognizable behaviors, or they can be “implicit,” which includes verbal and nonverbal behaviors that often go unrecognized but convey hostility, objectification, or exclusion of another person. Gender-based discrimination and sexual harassment are commonly described and likely more prevalent in academic settings.^3-6 Female physicians are disproportionately affected by gender-based discrimination and sexual harassment, compared with their male peers.^4,7

Female physicians face workplace harassment from both patients and coworkers. A study in Canada reported that more than 75% of female physicians experienced sexual harassment from their patients.⁸ A recent study showed almost half of the physicians who reported harassment, which was three times more often among female physicians, described other physician colleagues as perpetrators.⁹ In a study among clinician-researchers in the field of academic medicine, 30% of females reported having experienced sexual harassment, compared with 4% of males.⁷ Among females who reported harassment in this study, 47% stated that these experiences adversely affected their opportunities for career advancement. Career stage may also affect experiences or perceptions of gender-based discrimination and sexual harassment, with females in earlier career stages reporting a less favorable environment of gender equity.¹⁰

Hospital medicine is a young and evolving specialty, and the number of hospitalists has increased substantially from a few hundred at the time of inception to over 50,000 as of 2016.¹¹ The proportion of female hospitalists increased from 31% in 2012 to 52% in 2014, reflecting equal gender representation in hospital medicine.¹² Available evidence shows gender disparities exist in hospital medicine disproportionately affecting female hospitalists in their career advancement, including leadership and scholarship opportunities.¹³ However, there remains a knowledge gap regarding the prevalence of gender-based discrimination and sexual harassment experienced by hospitalists.

Our study examines the experiences of academic hospitalists regarding gender-based discrimination and sexual harassment and the impact of gender on career satisfaction and advancement.

METHODS

Study Design and Participants

An online survey was developed and approved by the institutional board review (IRB) at the Medical College of Wisconsin in Milwaukee. University-based academic centers with hospitalist programs, identified through personal connections, from across the continental United Stated were identified as potential study sites, and leaders at each institution were contacted to ascertain potential participation in the survey. The survey was distributed to Internal Medicine hospitalists at 18 participating academic institutions from January 2019 to June 2019. Participation was voluntary. The cover letter explained the purpose of the study and provided a link to the survey. To maintain anonymity, none of the questionnaires requested identifying information from participants. A web-based Qualtrics online-based survey platform was used.

Survey Elements

The survey aimed to assess several elements of gender-based discrimination and sexual harassment. All questions about these experiences distinguished encounters with patients from those with colleagues, and questions about occurrences either over a career or in the last 30 days were intended to capture both distant and recent timeframes. The theme for the questions for the survey was based on previous studies.^4,7,8 The wording of questions was simplified to make them easily understandable, and the brevity of the survey was maintained to prevent possible nonresponses.¹⁴ Additional questions (mistaken for a healthcare provider other than a physician, feeling respected by patients and colleagues, referred to by terms such as “honey,” “dear,” “sweetheart,” “sugar,” or equivalent), which were deemed relevant in day-to-day clinical practice through consensus among study investigators and discussions among peer hospitalists, were incorporated into the final survey (Appendix). Survey questions were intended to capture several elements regarding interactions with patients and with colleagues or other healthcare providers (HCPs).

Questions on gender-based discrimination included:

• Has a patient [colleague or other healthcare provider] mistaken you for a healthcare provider other than a physician?
• Has a patient [colleague or other healthcare provider] asked you to do something not at your level of training?
• Do you feel respected?
Do you perceive your gender has impacted opportunities for your career advancement?

Questions on sexual harassment included:

• Has a patient [colleague or other healthcare provider] touched you inappropriately, made sexual remarks or gestures, or made suggestive looks?
• Has a patient [colleague or other healthcare provider] referred to you by terms such as “honey,” “dear,” “sweetheart,” “sugar,” or equivalent?

In addition, the survey sought demographic information of the participants (age, gender, and race/ethnicity) and information about their individual institutions (names and locations) (Appendix). The geographical locations of the institutions were further categorized into four different regions according to the United States Census Bureau (Northwest, Midwest, South, and West). At the end of the survey, participants were given an opportunity to provide any additional comments.

Statistical Analysis

Standard descriptive summary statistics were used for demographic data. Associations between the variables were analyzed using chi-square test or Fischer’s exact test, as appropriate, for categorical variables and t test for continuous variables. The variations among institution-based responses were presented in the form of inter-quartile range (IQR). All tests were 2-sided, and P values less than .05 were considered statistically significant. All analyses were performed using IBM^® SPSS^® Statistics software version 24. Relevant responses representative of the overall respondents’ sentiments as provided under additional comment section were discussed and cited.

RESULTS

Eighteen different academic institutions across the United States participated in the study, with 336 individual responses. The majority of respondents were females (57%), in younger age categories (58% were 30-39 years old), Caucasian (59%), and early-career hospitalists (>50% working as hospitalists for ≤5 years) (Table 1). Regarding the overall geographic distribution, the largest number of responses were from the Midwest (n = 115; 35.6%) (Table 1 and Appendix).

Gender Discrimination

Interactions With Patients

Over their careers, 69% of hospitalists reported being mistaken for an HCP other than a physician by patients. This was more common among females than among males (99% vs 29%, respectively; P < .001) (Table 2). Almost half (48%) reported this had occurred in the last 30 days, more frequently by females (76% vs 10%; P < .001).

Of responding hospitalists, 96% stated that, over their careers, they have been asked by patients to do something they did not consider to be at their level of training (eg, help get food or water, help with a bed pan), with a higher prevalence of such experiences among females than males (99% vs 93%, respectively; P = .004) (Table 2). Most (71%) said they had experienced this in the last 30 days, which was again more frequently reported by females (78% vs 62%; P = .001).

The responses from female hospitalists regarding their career-long experiences of being mistaken for an HCP or asked to do something not at their level of training by their patients had both the highest number of positive responses across institutions (median of hospital proportions, 100%) and the least institutional variation since both had the narrowest IQR) (Table 2).

Interactions With Colleagues or Other HCPs

Among hospitalists responding to the survey, 46% felt that, over their careers, they had been mistaken for nonphysician HCPs by colleagues or other HCPs. This was more prevalent among females than among males (65% vs 20%; P < .001) (Table 2). Among respondents, 14% reported these events had occurred in the last 30 days, which was again more common among females (21% vs 5%; P < .001).

Over their careers, 26% of hospitalists reported they have been asked by a colleague or HCP to do something not at their level of training (eg, clean up the physician’s work room, make coffee, take notes in a meeting), with similar prevalence among females and males (29% vs 23%; P = .228). Ten percent reported these occurrences in the last 30 days, which was similar between females and males (12% vs 9%; P = .330).

Feelings of Respect and Opportunities for Career Advancement

When asked to rate the statement “I feel respected by patients” on a 5-point Likert scale, female hospitalists overall scored significantly lower as compared with their male counterparts (mean score, 3.73 vs 4.04; P < .001) (Table 3); this was also true when asked about feelings of respect by physicians (mean score, 3.84 vs 4.15; P < .001). Female hospitalists were more likely than males to report that their gender has more negatively impacted their opportunities for career advancement (mean score, 2.73 vs 3.34; P < .001).

Sexual Harassment

Interactions With Patients

Over half (57%) of hospitalists reported career-long experiences of patient(s) touching them inappropriately, making sexual remarks or gestures, or making suggestive looks. These experiences were more prevalent among females than among males (72% vs 36%, respectively; P < .001) (Table 2). Fifteen percent said they had such experience in rhe last 30 days, which was also more common among females (22% vs 6%; P < .001).

Most hospitalists (84%) reported that patients have referred to them by inappropriately familiar terms such as “honey,” “dear,” “sweetheart,” “sugar,” or equivalent over their careers, with females more frequently reporting these behaviors (95% vs 68%; P < .001). Experiencing this during the last 30 days was reported by 48%, which was again more common among females (68% vs 23%; P < .001).

Interactions With Colleagues or Other HCPs

Within their careers, 15% of hospitalists reported at least one experience of a colleague or HCP touching them inappropriately or making sexual remarks, gestures, or suggestive looks. This was more prevalent for females than males (18% vs 10%, respectively; P = .033). Only 2% of both females and males reported these experiences in the last 30 days (2% vs 2%; P = .981).

Almost one-third of participants (32%) affirmed that another HCP has referred to them by terms such as “honey,” “dear,” “sweetheart,” “sugar,” or equivalent in their career, with a higher proportion of females than males reporting these events (39% vs 23%; P = .002) (Table 2). Experiencing this during the last 30 days was reported by 10%, which was similar between females and males (12% vs 7%; P = .112).

Additional Comments From Respondents

• “Throughout my training and now into my professional career, there are nearly weekly incidents of elderly male patients referring to me as “honey/dear/sweetie” or even by my first name, even though I introduce myself as their physician and politely correct them. They will often refer to me as a nurse and ask me to do something not at my level of training. Sometimes even despite correcting the patient, they continue to refer to me as such. Throughout the years, other female MDs and I have discussed that this is ‘status quo’ for female physicians and observe that this is not an experience that male MDs share.”
• “I frequently round with a male nurse practitioner and the patients almost always, despite introducing ourselves and our roles, turn to him and ask him questions instead of addressing them to me.”
• “Our institution allows female faculty to be interviewed about childcare, household labor division, plans for pregnancy. One professor asks women private details about their private relationships such as what they do with spouse on date night or weekends away.”
• “It’s hard to answer questions related to my level of training. I don’t think it’s unreasonable for people to ask me to do things, no matter my level of training. . . . I don’t think being a doctor means that I am above this, or that it is inappropriate to be asked to do this.”

DISCUSSION

This survey demonstrated that gender-based discrimination and sexual harassment in the academic hospitalist healthcare environment are common, both in more distant and recent time frames. Notably, these experiences are shared by female and male physicians in interactions with both patients and colleagues, though male hospitalists report most of these experiences at significantly lower frequencies than females. These results support past work showing that female physicians are significantly more likely to be subjected to gender-based discrimination and sexual harassment, but also challenges the perception that gender-based discrimination and sexual harassment are uniquely experienced by females.

A startling number of females and males in the study reported sexual harassment (inappropriate touching, remarks, gestures, and looks) when interacting with patients throughout their careers and in last 30 days. Many males and females reported that patients had referred to them with inappropriately familiar, and potentially demeaning, terms of endearment. For both overt and implicit sexual harassment, females were significantly more likely than males to report experiencing these behaviors when interacting with patients. Although some of these experiences may seem less harmful than others, a meta-analysis demonstrated that frequent, less intense experiences of gender-based discrimination and sexual harassment have a similar impact on female’s well-being as do less frequent, more intense experiences.¹⁵ Although the person using the terms of endearment like “honey,” “sugar,” or “sweetheart” may feel the terms are harmless, such expressions can be inappropriate and constitute sexual harassment according to the U.S. Department of the Interior’s Office of Civil Rights.¹⁶ The Sexual Harassment/Assault Response and Prevention Program (SHARP) also classifies such terms into verbal categories of sexual harrassment.¹⁷

Of female physicians surveyed, 99% reported that they had been mistaken for HCPs other than physicians by their patients over their careers. Although this was also reported by male physicians, the experience was 3.4 times as likely for female physicians. Misidentification by patients may represent a disconnect between the growing female representation in the physician workforce and patients’ conceptions of the traditional image of a physician.

In parallel with this finding of misidentification, an interesting area of the study was the question regarding being asked to do “something not at your level of training.” A recurring theme in the comments was a rejection of the notion that certain tasks were “beneath a level of training,” suggesting a common view that acts of caregiving are not bounded by hierarchy. Analysis of qualitative responses showed that 40% of these responses had comments regarding this question. An example was “It’s hard to answer questions related to my level of training. . . . I don’t think being a doctor means that I am above this, or that it is inappropriate to be asked to do this.” Notably, however, a larger number of female than male physicians responded yes to this question in both study time frames. This points to a differential in how female physicians are viewed by patients, both in frequent misidentification and in behaviors more frequently asked of female physicians than their male counterparts. Given the comments, it may also suggest a difference in how female and male physicians perceive the fluidity of bounds on their care-taking roles set by their “level of training.”

A large number of study participants were early-career hospitalists, which may in part explain some of the study results. In a previous study of gender equity in an Internal Medicine department, physicians practicing medicine for more than 15 years perceived the departmental culture as more favorable than physicians with shorter careers.¹⁰ Additionally, the perception of cultures was most discordant between senior male physicians and junior female physicians.¹⁰ Because many hospitalists are early-career physicians, they may have trained in an environment that had heightened awareness surrounding gender-based discrimination and sexual harassment, which affects the overall study results.

Multiple qualitative comments, mentioned above, were submitted by participants describing their experiences in all categories. Such comments paint a picture of insidious bias and cultural norms affecting the quality of female physicians’ work lives.

Two questions focused on career satisfaction and the sense of respect from patients and colleagues. In both responses, there was a statistically different response between males and females, with females less likely to report that they felt respected and that their gender adversely impacted their opportunities for career advancement. This is disturbing information and warrants more investigation.

The reasons for the observed prevalence of gender-based discrimination and sexual harassment in this broad survey of academic hospitalists are uncertain. Multiple studies to date have demonstrated that gender-based discrimination and sexual harassment have historically existed in medicine and continue to even today. Unlike physicians with long-term relationships with patients, hospitalists may face more exposure due to a lack of long-term continuity with patients. The absence of an established trust in the relationship also may make them more vulnerable to inappropriate behaviors when interacting with patients. Hospital medicine, however, is a young specialty with equal gender representation and should be at the forefront of addressing and solving these issues of gender-based discrimination and sexual harassment.

The survey had a good distribution between female and male participants. Additionally, the survey reflected the general distribution of the national hospitalist workforce in gender, age, and ethnic/racial distribution, as well as number of years in practice.¹² The study surveyed respondents regarding experiences in both long- and short-term time frames, as well as experiences with patients and colleagues.

Our study reflects a cross-sectional snapshot of hospitalists’ perceptions with no longitudinal follow-up. Since the survey was limited to academic medical centers, it may not reflect experiences in community/private practice settings. The small number of participants limited the ability to perform subgroup analyses by age, race, or years in practice, which may play a role in interactions with patients and colleagues. Since the number of respondents varied greatly by institution, a minority of institutions could have influenced some of the findings. Narrow IQRs of the hospital proportions as shown in Table 2 would suggest similar responses across institutions, whereas wide IQRs would suggest that a smaller number of institutions were possibly driving the findings. Because of the survey distribution method, it is unknown how many physicians received the survey and a response rate could not be calculated. Further, selection, recall, and detection biases cannot be ruled out.

CONCLUSION

This survey shows that gender-based discrimination and sexual harassment in the academic hospitalist healthcare environment are common and more frequently experienced by female physicians, both in interactions with patients and colleagues. Our study highlights the need to address this prevalent issue among academic hospitalists.

Gender-based discrimination refers to “any distinction, exclusion or restriction made on the basis of socially constructed gender roles and norms which prevents a person from enjoying full human rights.”¹ Similarly, sexual harassment encompasses a spectrum of sexual conduct that includes “unwelcome sexual advances, requests for sexual favors, and other verbal or physical harassment of a sexual nature,” as defined by the US Equal Employment Opportunity Commission.² Gender-based discrimination and sexual harassment can be “overt,” which includes explicitly recognizable behaviors, or they can be “implicit,” which includes verbal and nonverbal behaviors that often go unrecognized but convey hostility, objectification, or exclusion of another person. Gender-based discrimination and sexual harassment are commonly described and likely more prevalent in academic settings.^3-6 Female physicians are disproportionately affected by gender-based discrimination and sexual harassment, compared with their male peers.^4,7

Female physicians face workplace harassment from both patients and coworkers. A study in Canada reported that more than 75% of female physicians experienced sexual harassment from their patients.⁸ A recent study showed almost half of the physicians who reported harassment, which was three times more often among female physicians, described other physician colleagues as perpetrators.⁹ In a study among clinician-researchers in the field of academic medicine, 30% of females reported having experienced sexual harassment, compared with 4% of males.⁷ Among females who reported harassment in this study, 47% stated that these experiences adversely affected their opportunities for career advancement. Career stage may also affect experiences or perceptions of gender-based discrimination and sexual harassment, with females in earlier career stages reporting a less favorable environment of gender equity.¹⁰

Hospital medicine is a young and evolving specialty, and the number of hospitalists has increased substantially from a few hundred at the time of inception to over 50,000 as of 2016.¹¹ The proportion of female hospitalists increased from 31% in 2012 to 52% in 2014, reflecting equal gender representation in hospital medicine.¹² Available evidence shows gender disparities exist in hospital medicine disproportionately affecting female hospitalists in their career advancement, including leadership and scholarship opportunities.¹³ However, there remains a knowledge gap regarding the prevalence of gender-based discrimination and sexual harassment experienced by hospitalists.

Our study examines the experiences of academic hospitalists regarding gender-based discrimination and sexual harassment and the impact of gender on career satisfaction and advancement.

METHODS

Study Design and Participants

An online survey was developed and approved by the institutional board review (IRB) at the Medical College of Wisconsin in Milwaukee. University-based academic centers with hospitalist programs, identified through personal connections, from across the continental United Stated were identified as potential study sites, and leaders at each institution were contacted to ascertain potential participation in the survey. The survey was distributed to Internal Medicine hospitalists at 18 participating academic institutions from January 2019 to June 2019. Participation was voluntary. The cover letter explained the purpose of the study and provided a link to the survey. To maintain anonymity, none of the questionnaires requested identifying information from participants. A web-based Qualtrics online-based survey platform was used.

Survey Elements

The survey aimed to assess several elements of gender-based discrimination and sexual harassment. All questions about these experiences distinguished encounters with patients from those with colleagues, and questions about occurrences either over a career or in the last 30 days were intended to capture both distant and recent timeframes. The theme for the questions for the survey was based on previous studies.^4,7,8 The wording of questions was simplified to make them easily understandable, and the brevity of the survey was maintained to prevent possible nonresponses.¹⁴ Additional questions (mistaken for a healthcare provider other than a physician, feeling respected by patients and colleagues, referred to by terms such as “honey,” “dear,” “sweetheart,” “sugar,” or equivalent), which were deemed relevant in day-to-day clinical practice through consensus among study investigators and discussions among peer hospitalists, were incorporated into the final survey (Appendix). Survey questions were intended to capture several elements regarding interactions with patients and with colleagues or other healthcare providers (HCPs).

Questions on gender-based discrimination included:

• Has a patient [colleague or other healthcare provider] mistaken you for a healthcare provider other than a physician?
• Has a patient [colleague or other healthcare provider] asked you to do something not at your level of training?
• Do you feel respected?
Do you perceive your gender has impacted opportunities for your career advancement?

Questions on sexual harassment included:

• Has a patient [colleague or other healthcare provider] touched you inappropriately, made sexual remarks or gestures, or made suggestive looks?
• Has a patient [colleague or other healthcare provider] referred to you by terms such as “honey,” “dear,” “sweetheart,” “sugar,” or equivalent?

In addition, the survey sought demographic information of the participants (age, gender, and race/ethnicity) and information about their individual institutions (names and locations) (Appendix). The geographical locations of the institutions were further categorized into four different regions according to the United States Census Bureau (Northwest, Midwest, South, and West). At the end of the survey, participants were given an opportunity to provide any additional comments.

Statistical Analysis

Standard descriptive summary statistics were used for demographic data. Associations between the variables were analyzed using chi-square test or Fischer’s exact test, as appropriate, for categorical variables and t test for continuous variables. The variations among institution-based responses were presented in the form of inter-quartile range (IQR). All tests were 2-sided, and P values less than .05 were considered statistically significant. All analyses were performed using IBM^® SPSS^® Statistics software version 24. Relevant responses representative of the overall respondents’ sentiments as provided under additional comment section were discussed and cited.

RESULTS

Eighteen different academic institutions across the United States participated in the study, with 336 individual responses. The majority of respondents were females (57%), in younger age categories (58% were 30-39 years old), Caucasian (59%), and early-career hospitalists (>50% working as hospitalists for ≤5 years) (Table 1). Regarding the overall geographic distribution, the largest number of responses were from the Midwest (n = 115; 35.6%) (Table 1 and Appendix).

Gender Discrimination

Interactions With Patients

Over their careers, 69% of hospitalists reported being mistaken for an HCP other than a physician by patients. This was more common among females than among males (99% vs 29%, respectively; P < .001) (Table 2). Almost half (48%) reported this had occurred in the last 30 days, more frequently by females (76% vs 10%; P < .001).

Of responding hospitalists, 96% stated that, over their careers, they have been asked by patients to do something they did not consider to be at their level of training (eg, help get food or water, help with a bed pan), with a higher prevalence of such experiences among females than males (99% vs 93%, respectively; P = .004) (Table 2). Most (71%) said they had experienced this in the last 30 days, which was again more frequently reported by females (78% vs 62%; P = .001).

The responses from female hospitalists regarding their career-long experiences of being mistaken for an HCP or asked to do something not at their level of training by their patients had both the highest number of positive responses across institutions (median of hospital proportions, 100%) and the least institutional variation since both had the narrowest IQR) (Table 2).

Interactions With Colleagues or Other HCPs

Among hospitalists responding to the survey, 46% felt that, over their careers, they had been mistaken for nonphysician HCPs by colleagues or other HCPs. This was more prevalent among females than among males (65% vs 20%; P < .001) (Table 2). Among respondents, 14% reported these events had occurred in the last 30 days, which was again more common among females (21% vs 5%; P < .001).

Over their careers, 26% of hospitalists reported they have been asked by a colleague or HCP to do something not at their level of training (eg, clean up the physician’s work room, make coffee, take notes in a meeting), with similar prevalence among females and males (29% vs 23%; P = .228). Ten percent reported these occurrences in the last 30 days, which was similar between females and males (12% vs 9%; P = .330).

Feelings of Respect and Opportunities for Career Advancement

When asked to rate the statement “I feel respected by patients” on a 5-point Likert scale, female hospitalists overall scored significantly lower as compared with their male counterparts (mean score, 3.73 vs 4.04; P < .001) (Table 3); this was also true when asked about feelings of respect by physicians (mean score, 3.84 vs 4.15; P < .001). Female hospitalists were more likely than males to report that their gender has more negatively impacted their opportunities for career advancement (mean score, 2.73 vs 3.34; P < .001).

Sexual Harassment

Interactions With Patients

Over half (57%) of hospitalists reported career-long experiences of patient(s) touching them inappropriately, making sexual remarks or gestures, or making suggestive looks. These experiences were more prevalent among females than among males (72% vs 36%, respectively; P < .001) (Table 2). Fifteen percent said they had such experience in rhe last 30 days, which was also more common among females (22% vs 6%; P < .001).

Most hospitalists (84%) reported that patients have referred to them by inappropriately familiar terms such as “honey,” “dear,” “sweetheart,” “sugar,” or equivalent over their careers, with females more frequently reporting these behaviors (95% vs 68%; P < .001). Experiencing this during the last 30 days was reported by 48%, which was again more common among females (68% vs 23%; P < .001).

Interactions With Colleagues or Other HCPs

Within their careers, 15% of hospitalists reported at least one experience of a colleague or HCP touching them inappropriately or making sexual remarks, gestures, or suggestive looks. This was more prevalent for females than males (18% vs 10%, respectively; P = .033). Only 2% of both females and males reported these experiences in the last 30 days (2% vs 2%; P = .981).

Almost one-third of participants (32%) affirmed that another HCP has referred to them by terms such as “honey,” “dear,” “sweetheart,” “sugar,” or equivalent in their career, with a higher proportion of females than males reporting these events (39% vs 23%; P = .002) (Table 2). Experiencing this during the last 30 days was reported by 10%, which was similar between females and males (12% vs 7%; P = .112).

Additional Comments From Respondents

• “Throughout my training and now into my professional career, there are nearly weekly incidents of elderly male patients referring to me as “honey/dear/sweetie” or even by my first name, even though I introduce myself as their physician and politely correct them. They will often refer to me as a nurse and ask me to do something not at my level of training. Sometimes even despite correcting the patient, they continue to refer to me as such. Throughout the years, other female MDs and I have discussed that this is ‘status quo’ for female physicians and observe that this is not an experience that male MDs share.”
• “I frequently round with a male nurse practitioner and the patients almost always, despite introducing ourselves and our roles, turn to him and ask him questions instead of addressing them to me.”
• “Our institution allows female faculty to be interviewed about childcare, household labor division, plans for pregnancy. One professor asks women private details about their private relationships such as what they do with spouse on date night or weekends away.”
• “It’s hard to answer questions related to my level of training. I don’t think it’s unreasonable for people to ask me to do things, no matter my level of training. . . . I don’t think being a doctor means that I am above this, or that it is inappropriate to be asked to do this.”

DISCUSSION

This survey demonstrated that gender-based discrimination and sexual harassment in the academic hospitalist healthcare environment are common, both in more distant and recent time frames. Notably, these experiences are shared by female and male physicians in interactions with both patients and colleagues, though male hospitalists report most of these experiences at significantly lower frequencies than females. These results support past work showing that female physicians are significantly more likely to be subjected to gender-based discrimination and sexual harassment, but also challenges the perception that gender-based discrimination and sexual harassment are uniquely experienced by females.

A startling number of females and males in the study reported sexual harassment (inappropriate touching, remarks, gestures, and looks) when interacting with patients throughout their careers and in last 30 days. Many males and females reported that patients had referred to them with inappropriately familiar, and potentially demeaning, terms of endearment. For both overt and implicit sexual harassment, females were significantly more likely than males to report experiencing these behaviors when interacting with patients. Although some of these experiences may seem less harmful than others, a meta-analysis demonstrated that frequent, less intense experiences of gender-based discrimination and sexual harassment have a similar impact on female’s well-being as do less frequent, more intense experiences.¹⁵ Although the person using the terms of endearment like “honey,” “sugar,” or “sweetheart” may feel the terms are harmless, such expressions can be inappropriate and constitute sexual harassment according to the U.S. Department of the Interior’s Office of Civil Rights.¹⁶ The Sexual Harassment/Assault Response and Prevention Program (SHARP) also classifies such terms into verbal categories of sexual harrassment.¹⁷

Of female physicians surveyed, 99% reported that they had been mistaken for HCPs other than physicians by their patients over their careers. Although this was also reported by male physicians, the experience was 3.4 times as likely for female physicians. Misidentification by patients may represent a disconnect between the growing female representation in the physician workforce and patients’ conceptions of the traditional image of a physician.

In parallel with this finding of misidentification, an interesting area of the study was the question regarding being asked to do “something not at your level of training.” A recurring theme in the comments was a rejection of the notion that certain tasks were “beneath a level of training,” suggesting a common view that acts of caregiving are not bounded by hierarchy. Analysis of qualitative responses showed that 40% of these responses had comments regarding this question. An example was “It’s hard to answer questions related to my level of training. . . . I don’t think being a doctor means that I am above this, or that it is inappropriate to be asked to do this.” Notably, however, a larger number of female than male physicians responded yes to this question in both study time frames. This points to a differential in how female physicians are viewed by patients, both in frequent misidentification and in behaviors more frequently asked of female physicians than their male counterparts. Given the comments, it may also suggest a difference in how female and male physicians perceive the fluidity of bounds on their care-taking roles set by their “level of training.”

A large number of study participants were early-career hospitalists, which may in part explain some of the study results. In a previous study of gender equity in an Internal Medicine department, physicians practicing medicine for more than 15 years perceived the departmental culture as more favorable than physicians with shorter careers.¹⁰ Additionally, the perception of cultures was most discordant between senior male physicians and junior female physicians.¹⁰ Because many hospitalists are early-career physicians, they may have trained in an environment that had heightened awareness surrounding gender-based discrimination and sexual harassment, which affects the overall study results.

Multiple qualitative comments, mentioned above, were submitted by participants describing their experiences in all categories. Such comments paint a picture of insidious bias and cultural norms affecting the quality of female physicians’ work lives.

Two questions focused on career satisfaction and the sense of respect from patients and colleagues. In both responses, there was a statistically different response between males and females, with females less likely to report that they felt respected and that their gender adversely impacted their opportunities for career advancement. This is disturbing information and warrants more investigation.

The reasons for the observed prevalence of gender-based discrimination and sexual harassment in this broad survey of academic hospitalists are uncertain. Multiple studies to date have demonstrated that gender-based discrimination and sexual harassment have historically existed in medicine and continue to even today. Unlike physicians with long-term relationships with patients, hospitalists may face more exposure due to a lack of long-term continuity with patients. The absence of an established trust in the relationship also may make them more vulnerable to inappropriate behaviors when interacting with patients. Hospital medicine, however, is a young specialty with equal gender representation and should be at the forefront of addressing and solving these issues of gender-based discrimination and sexual harassment.

The survey had a good distribution between female and male participants. Additionally, the survey reflected the general distribution of the national hospitalist workforce in gender, age, and ethnic/racial distribution, as well as number of years in practice.¹² The study surveyed respondents regarding experiences in both long- and short-term time frames, as well as experiences with patients and colleagues.

Our study reflects a cross-sectional snapshot of hospitalists’ perceptions with no longitudinal follow-up. Since the survey was limited to academic medical centers, it may not reflect experiences in community/private practice settings. The small number of participants limited the ability to perform subgroup analyses by age, race, or years in practice, which may play a role in interactions with patients and colleagues. Since the number of respondents varied greatly by institution, a minority of institutions could have influenced some of the findings. Narrow IQRs of the hospital proportions as shown in Table 2 would suggest similar responses across institutions, whereas wide IQRs would suggest that a smaller number of institutions were possibly driving the findings. Because of the survey distribution method, it is unknown how many physicians received the survey and a response rate could not be calculated. Further, selection, recall, and detection biases cannot be ruled out.

CONCLUSION

This survey shows that gender-based discrimination and sexual harassment in the academic hospitalist healthcare environment are common and more frequently experienced by female physicians, both in interactions with patients and colleagues. Our study highlights the need to address this prevalent issue among academic hospitalists.

The advent of liquid biopsies targeting genetic mutations in solid tumors is a major milestone in the field of precision oncology.¹ Conventional methods of obtaining tissue for molecular studies are limited by sample size and often do not represent the entire bulk of the tumor.² This newer minimally invasive, revolutionary technique analyzes circulating cell-free DNA carrying tumor-specific alterations (circulating tumor DNA [ctDNA]) in peripheral blood and detects signature genomic alterations.¹ Tp53 mutations have been reported in 25 to 40% of prostatic cancers and > 50% of non-small cell lung cancers (NSCLC), being more common in late-stage and hormone refractory prostate cancers.^3,4 Tp53 mutation has been found to be associated with poor prognosis and increased germline mutations.⁵

The veteran patient population has distinct demographic characteristics that make veterans more vulnerable to genetic mutations and malignancies, including risk of exposure to Agent Orange, smoking, substance abuse, and asbestos. This area is understudied and extremely sparse in the literature for frequency of genetic mutations, risk factors in solid malignancies occurring in the veteran patient population, and the clinical impact of these risk factors. We herein present a quality assurance study for the utility of liquid biopsies regarding the frequency of DNA damage repair (DDR) gene, Tp53, and androgen receptor (AR) mutations. The clinical impact in advanced lung and prostate cancers in the veteran patient population and frequency are the quality assurance observations that are the study endpoints.

Methods

We reviewed for quality assurance documentation from the Foundation Medicine (www.foundationmedicine.com) cancer biomarker tests on liquid biopsies performed at the Corporal Michael J. Crescenz Veteran Affairs Medical Center in Philadelphia, Pennsylvania from May 2019 to April 15, 2020. All biopsies were performed on cancers with biochemical, imaging or tissue evidence of advanced tumor progression. The testing was performed on advanced solid malignancies, including NSCLC, prostate adenocarcinoma, and metastatic colon cancer. Statistical data for adequacy; cases with notable mutations; frequency; and type of mutations of AR, DDR, and Tp53 were noted. General and specific risk factors associated with the veteran patient population were studied and matched with the type of mutations (Table 1).

Results

Thirty-one liquid biopsies were performed over this period—23 for prostate cancer, 7 for patients with lung cancer patients, and 1 for a patient with colon cancer. Of 31 cases, sensitivity/adequacy of liquid biopsy for genetic mutation was detected in 29 (93.5%) cases (Figure 1). Two inadequate biopsies (both from patients with prostate cancer) were excluded from the study, leaving 29 liquid biopsies with adequate ctDNA for analysis that were considered for further statistical purpose—21 prostate, 7 lung, and 1 colon cancer.

Multiple (common and different) genetic mutations were identified; however, our study subcategorized the mutations into the those that were related to prostate cancer, lung cancer, and some common mutations that occur in both cancers. Only the significant ones will be discussed in this review and equivocal result for AR is excluded from this study. Of the 21 prostate cancers, 4 (19.0%) had directed the targeted therapy to driver mutation (AR being most common in prostate cancer), while KRAS mutation, which was more common in lung cancer, was detected in 2/7 (28.6%) lung cancers. Mutations common to both cancer types were DDR gene mutations, which is a broad name for numerous genes including CDK12, ATM, and CHEK2.

Of all cases irrespective of the cancer type, 23/29 (79.3%) showed notable mutations. DDR gene mutations were found in 6 of 21 (28.5%) patients with prostate cancer and 8 of 23 (34.7%) patients with advanced prostate and lung cancers, indicating poor outcome and possible resistance to the current therapy. Of 23 patients showing mutations irrespective of the cancer type, 15 (65.2%) harbored Tp53 mutations, which is much more frequent in veteran patient population when compared with the literature. Fifteen of the 31 (48.4%) total patients were Vietnam War-era veterans who were potentially exposed to Agent Orange and 20 (64.5%) patients who were not Vietnam War-era veterans had a history that included smoking (Figure 2).

 

Discussion

The veteran patient population is a unique cohort due to its distinct demographic characteristics with a high volume of cancer cases diagnosed each year. According to data from VA Central Cancer Registry (VACCR), the most frequently diagnosed cancers are prostate (29%) and lung (18%).⁶

Liquid biopsy is a novel, promising technology that uses ctDNA and circulating tumor cells in peripheral blood for detecting genetic alterations through next generation sequencing.^7-9 The advent of this minimally invasive, revolutionary technology has been a breakthrough in the field of precision oncology for prognosis, to monitor treatment response or resistance to therapy and further personalize cancer therapy.^9,10

Comprehensive genomic profiling by liquid biopsy has many advantages over the molecular studies performed on tissue biopsy. Due to the tumor heterogeneity, tissue samples may not represent the full profile of the tumor genomics of cancer, while liquid biopsy has full presentation of the disease.^11,12 Many times, tissue biopsy may be limited by a sample size that precludes full genetic profiling in addition to higher total cost, potential technical issues during processing, and possible side effects of the biopsy procedure.^7,13 Additionally, as the tumor progresses, new driver mutations other than the ones previously detected on the primary tissue may emerge, which can confer resistance to the existing therapy.^7,13

Advanced prostatic and lung carcinomas with biochemical, distant organ, or bony progression harbor unique signature genetic mutations indicating poor prognosis, lack of response or resistance to the existing therapy, and high risk of relapse.^14,15 Some of the unique characteristics of the veteran patient population include a more aged patient population multiple comorbidities, higher frequency of > 1 type of cancer, advanced cancer stage at presentation, and specific risks factors such as exposure to Agent Orange in veterans who served during the Vietnam War era.^16,17 We studied the utility of liquid biopsy in cancer care, including type and incidence of genomic alterations associated with advanced prostate and lung cancers, in this unique patient population.

The amount of cell-free DNA (cfDNA), also known as ctDNA varies widely in cancer patients. Some of the factors associated with low concentration of cfDNA are disease stage, intervening therapy, proliferation rates, and tumor vascularization.^18,19 In the peripheral blood, of the total cfDNA, fractions of cfDNA varies from 0.01 to 90%.^18,19 All samples containing ≥ 20 ng cfDNA (20 - 100 ng) were subjected to the hybrid capture-based NGS FoundationACT assay.²⁰ In our study, 2 specimens did not meet the minimum criteria of adequacy (20 ng cfDNA); however, the overall adequacy rate for the detection of mutation, irrespective of the cancer type was 29 of 31 (93.5%) with only 2 inadequate samples. This rate is higher than the rate reported in the literature, which is about 70%.²⁰

Significant differences were encountered in the incidence of DNA damage repair genes including Tp53 mutations when compared with those in the general patient population (Table 2). According to recent National Comprehensive Cancer Network (NCCN) guidelines, all prostate cancers should be screened for DDR gene mutations as these genes are common in aggressive prostate cancers and strongly associated with poor outcomes and shortened survival. Due to relatively high frequency of DDR gene mutations in advanced prostatic cancers, liquid biopsy in patients with these advanced stage prostate cancers may be a useful tool in clinical decision making and exploring targeted therapy.²⁰

Mutations in BRCA2, ATM, CDK12, and CHEK2 (DDR gene family) are common. Incidence of ATM and CDK12 mutations in the literature is 3 to 6% of cases.²¹ Of 21 liquid biopsies of advanced prostate cancer patients, we found combined DDR gene mutation of ATM, CHEK2, and CDK12 genes in 6 (28.5%) cases, which is substantially higher than the 3 to 6% rate reported in the literature.^21-24 Of the 23 patients who had notable mutations in our liquid biopsies, including both advanced prostate and lung cancer cases, 8 (34.7%) also showed mutation of the genes of DDR family. Our study did not show BRCA2 mutation, which is otherwise common in the literature.

We also evaluated the frequency of the most commonly occurring genetic mutations, Tp53 in advanced solid malignancies, especially advanced prostate and NSCLC. Previous studies have reported Tp53 mutation in association with risk factors (carcinogens) of cancer and have been a surrogate marker of poor survival or lack of response of therapy.²⁵ Knowledge of Tp53 mutation is crucial for closer disease monitoring, preparing the patient for rapid progression, and encouraging the physician to prepare future lines of therapy.^25-27 Although Tp53 mutation varies with histologic type and tissue of origin, Beltran and colleagues reported it in 30 to 40% of tumors, while Robles and colleagues reported about 40 to 42% incidence.^25,27

Our study showed notable mutations in 23 of 29 adequate cases. Further, our study showed a high frequency of mutated Tp53 in 65.2% of combined advanced prostate and NSCLC cases. We then correlated cases of Vietnam War-era veterans with risk potential of Agent Orange exposure and Tp53 mutation. We found 7 of 15 Vietnam War-era veterans were positive for Tp53 mutations irrespective of the cancer type. The high incidence of Tp53 mutations in advanced prostate and lung carcinomas in the veteran patient population makes this tumor marker an aspiration not only as a surrogate of aggressive disease and tumor progression, but also as a key marker for targeted therapy in advanced prostate and lung cancers with loss of Tp53 function (Figure 3).

Mutations and amplifications in the AR gene are fundamental to progression of prostate cancer associated with advanced, hormone-refractory prostate cancer with the potential for targeted therapy with AR inhibitors. In our study, AR amplification was detected in 4 of 21 (19%) advanced prostate cancer cases, which is significantly lower than the 30 to 50% previously reported in the literature.^28-32 Neither AR amplification or mutation was noted in advanced NSCLC in our study as previously reported in literature by Brennan and colleagues and Wang and colleagues.^33-35 This is significant as it provides a pathway for future studies to focus on additional driver mutations for targeted therapies in advanced prostate carcinoma. To date, AR gene mutation does not play a role for personalized therapy in advanced NSCLC. Perhaps, a large cohort study with longitudinal analysis is needed for absolutely ruling out the possibility of personalized medicine in advanced lung cancer using this biomarker.

 

Conclusions

Liquid biopsy successfully provides precision-based oncology and information for decision making in this unique population of veterans. Difference in frequency of the genetic mutations in this cohort can provide future insight into disease progression, lack of response, and mechanism of resistance to the implemented therapy. Future studies focused on this veteran patient population are needed for developing targeted therapies and patient tailored oncologic therapy. ctDNA has a high potential for monitoring clinically relevant cancer-related genetic and epigenetic modifications for discovering more detailed information on the tumor characterization. Although larger cohort trial with longitudinal analyses are needed, high prevalence of DDR gene and Tp53 mutation in our study instills promising hope for therapeutic interventions in this unique cohort.

The minimally invasive liquid biopsy shows a great promise as both diagnostic and prognostic tool in the personalized clinical management of advanced prostate, and NSCLC in the veteran patient population with unique demographic characteristics. De novo metastatic prostate cancer is more common in veterans when compared with the general population, and therefore veterans may benefit by liquid biopsy. Differences in the frequency of genetic mutations (DDR, TP53, AR) in this cohort provides valuable information for disease progression, lack of response, mechanism of resistance to the implemented therapy and clinical decision making. Precision oncology can be further tailored for this cohort by focusing on DNA repair genes and Tp53 mutations for future targeted therapy.

The advent of liquid biopsies targeting genetic mutations in solid tumors is a major milestone in the field of precision oncology.¹ Conventional methods of obtaining tissue for molecular studies are limited by sample size and often do not represent the entire bulk of the tumor.² This newer minimally invasive, revolutionary technique analyzes circulating cell-free DNA carrying tumor-specific alterations (circulating tumor DNA [ctDNA]) in peripheral blood and detects signature genomic alterations.¹ Tp53 mutations have been reported in 25 to 40% of prostatic cancers and > 50% of non-small cell lung cancers (NSCLC), being more common in late-stage and hormone refractory prostate cancers.^3,4 Tp53 mutation has been found to be associated with poor prognosis and increased germline mutations.⁵

The veteran patient population has distinct demographic characteristics that make veterans more vulnerable to genetic mutations and malignancies, including risk of exposure to Agent Orange, smoking, substance abuse, and asbestos. This area is understudied and extremely sparse in the literature for frequency of genetic mutations, risk factors in solid malignancies occurring in the veteran patient population, and the clinical impact of these risk factors. We herein present a quality assurance study for the utility of liquid biopsies regarding the frequency of DNA damage repair (DDR) gene, Tp53, and androgen receptor (AR) mutations. The clinical impact in advanced lung and prostate cancers in the veteran patient population and frequency are the quality assurance observations that are the study endpoints.

Methods

We reviewed for quality assurance documentation from the Foundation Medicine (www.foundationmedicine.com) cancer biomarker tests on liquid biopsies performed at the Corporal Michael J. Crescenz Veteran Affairs Medical Center in Philadelphia, Pennsylvania from May 2019 to April 15, 2020. All biopsies were performed on cancers with biochemical, imaging or tissue evidence of advanced tumor progression. The testing was performed on advanced solid malignancies, including NSCLC, prostate adenocarcinoma, and metastatic colon cancer. Statistical data for adequacy; cases with notable mutations; frequency; and type of mutations of AR, DDR, and Tp53 were noted. General and specific risk factors associated with the veteran patient population were studied and matched with the type of mutations (Table 1).

Results

Thirty-one liquid biopsies were performed over this period—23 for prostate cancer, 7 for patients with lung cancer patients, and 1 for a patient with colon cancer. Of 31 cases, sensitivity/adequacy of liquid biopsy for genetic mutation was detected in 29 (93.5%) cases (Figure 1). Two inadequate biopsies (both from patients with prostate cancer) were excluded from the study, leaving 29 liquid biopsies with adequate ctDNA for analysis that were considered for further statistical purpose—21 prostate, 7 lung, and 1 colon cancer.

Multiple (common and different) genetic mutations were identified; however, our study subcategorized the mutations into the those that were related to prostate cancer, lung cancer, and some common mutations that occur in both cancers. Only the significant ones will be discussed in this review and equivocal result for AR is excluded from this study. Of the 21 prostate cancers, 4 (19.0%) had directed the targeted therapy to driver mutation (AR being most common in prostate cancer), while KRAS mutation, which was more common in lung cancer, was detected in 2/7 (28.6%) lung cancers. Mutations common to both cancer types were DDR gene mutations, which is a broad name for numerous genes including CDK12, ATM, and CHEK2.

Of all cases irrespective of the cancer type, 23/29 (79.3%) showed notable mutations. DDR gene mutations were found in 6 of 21 (28.5%) patients with prostate cancer and 8 of 23 (34.7%) patients with advanced prostate and lung cancers, indicating poor outcome and possible resistance to the current therapy. Of 23 patients showing mutations irrespective of the cancer type, 15 (65.2%) harbored Tp53 mutations, which is much more frequent in veteran patient population when compared with the literature. Fifteen of the 31 (48.4%) total patients were Vietnam War-era veterans who were potentially exposed to Agent Orange and 20 (64.5%) patients who were not Vietnam War-era veterans had a history that included smoking (Figure 2).

 

Discussion

The veteran patient population is a unique cohort due to its distinct demographic characteristics with a high volume of cancer cases diagnosed each year. According to data from VA Central Cancer Registry (VACCR), the most frequently diagnosed cancers are prostate (29%) and lung (18%).⁶

Liquid biopsy is a novel, promising technology that uses ctDNA and circulating tumor cells in peripheral blood for detecting genetic alterations through next generation sequencing.^7-9 The advent of this minimally invasive, revolutionary technology has been a breakthrough in the field of precision oncology for prognosis, to monitor treatment response or resistance to therapy and further personalize cancer therapy.^9,10

Comprehensive genomic profiling by liquid biopsy has many advantages over the molecular studies performed on tissue biopsy. Due to the tumor heterogeneity, tissue samples may not represent the full profile of the tumor genomics of cancer, while liquid biopsy has full presentation of the disease.^11,12 Many times, tissue biopsy may be limited by a sample size that precludes full genetic profiling in addition to higher total cost, potential technical issues during processing, and possible side effects of the biopsy procedure.^7,13 Additionally, as the tumor progresses, new driver mutations other than the ones previously detected on the primary tissue may emerge, which can confer resistance to the existing therapy.^7,13

Advanced prostatic and lung carcinomas with biochemical, distant organ, or bony progression harbor unique signature genetic mutations indicating poor prognosis, lack of response or resistance to the existing therapy, and high risk of relapse.^14,15 Some of the unique characteristics of the veteran patient population include a more aged patient population multiple comorbidities, higher frequency of > 1 type of cancer, advanced cancer stage at presentation, and specific risks factors such as exposure to Agent Orange in veterans who served during the Vietnam War era.^16,17 We studied the utility of liquid biopsy in cancer care, including type and incidence of genomic alterations associated with advanced prostate and lung cancers, in this unique patient population.

The amount of cell-free DNA (cfDNA), also known as ctDNA varies widely in cancer patients. Some of the factors associated with low concentration of cfDNA are disease stage, intervening therapy, proliferation rates, and tumor vascularization.^18,19 In the peripheral blood, of the total cfDNA, fractions of cfDNA varies from 0.01 to 90%.^18,19 All samples containing ≥ 20 ng cfDNA (20 - 100 ng) were subjected to the hybrid capture-based NGS FoundationACT assay.²⁰ In our study, 2 specimens did not meet the minimum criteria of adequacy (20 ng cfDNA); however, the overall adequacy rate for the detection of mutation, irrespective of the cancer type was 29 of 31 (93.5%) with only 2 inadequate samples. This rate is higher than the rate reported in the literature, which is about 70%.²⁰

Significant differences were encountered in the incidence of DNA damage repair genes including Tp53 mutations when compared with those in the general patient population (Table 2). According to recent National Comprehensive Cancer Network (NCCN) guidelines, all prostate cancers should be screened for DDR gene mutations as these genes are common in aggressive prostate cancers and strongly associated with poor outcomes and shortened survival. Due to relatively high frequency of DDR gene mutations in advanced prostatic cancers, liquid biopsy in patients with these advanced stage prostate cancers may be a useful tool in clinical decision making and exploring targeted therapy.²⁰

Mutations in BRCA2, ATM, CDK12, and CHEK2 (DDR gene family) are common. Incidence of ATM and CDK12 mutations in the literature is 3 to 6% of cases.²¹ Of 21 liquid biopsies of advanced prostate cancer patients, we found combined DDR gene mutation of ATM, CHEK2, and CDK12 genes in 6 (28.5%) cases, which is substantially higher than the 3 to 6% rate reported in the literature.^21-24 Of the 23 patients who had notable mutations in our liquid biopsies, including both advanced prostate and lung cancer cases, 8 (34.7%) also showed mutation of the genes of DDR family. Our study did not show BRCA2 mutation, which is otherwise common in the literature.

We also evaluated the frequency of the most commonly occurring genetic mutations, Tp53 in advanced solid malignancies, especially advanced prostate and NSCLC. Previous studies have reported Tp53 mutation in association with risk factors (carcinogens) of cancer and have been a surrogate marker of poor survival or lack of response of therapy.²⁵ Knowledge of Tp53 mutation is crucial for closer disease monitoring, preparing the patient for rapid progression, and encouraging the physician to prepare future lines of therapy.^25-27 Although Tp53 mutation varies with histologic type and tissue of origin, Beltran and colleagues reported it in 30 to 40% of tumors, while Robles and colleagues reported about 40 to 42% incidence.^25,27

Our study showed notable mutations in 23 of 29 adequate cases. Further, our study showed a high frequency of mutated Tp53 in 65.2% of combined advanced prostate and NSCLC cases. We then correlated cases of Vietnam War-era veterans with risk potential of Agent Orange exposure and Tp53 mutation. We found 7 of 15 Vietnam War-era veterans were positive for Tp53 mutations irrespective of the cancer type. The high incidence of Tp53 mutations in advanced prostate and lung carcinomas in the veteran patient population makes this tumor marker an aspiration not only as a surrogate of aggressive disease and tumor progression, but also as a key marker for targeted therapy in advanced prostate and lung cancers with loss of Tp53 function (Figure 3).

Mutations and amplifications in the AR gene are fundamental to progression of prostate cancer associated with advanced, hormone-refractory prostate cancer with the potential for targeted therapy with AR inhibitors. In our study, AR amplification was detected in 4 of 21 (19%) advanced prostate cancer cases, which is significantly lower than the 30 to 50% previously reported in the literature.^28-32 Neither AR amplification or mutation was noted in advanced NSCLC in our study as previously reported in literature by Brennan and colleagues and Wang and colleagues.^33-35 This is significant as it provides a pathway for future studies to focus on additional driver mutations for targeted therapies in advanced prostate carcinoma. To date, AR gene mutation does not play a role for personalized therapy in advanced NSCLC. Perhaps, a large cohort study with longitudinal analysis is needed for absolutely ruling out the possibility of personalized medicine in advanced lung cancer using this biomarker.

 

Conclusions

Liquid biopsy successfully provides precision-based oncology and information for decision making in this unique population of veterans. Difference in frequency of the genetic mutations in this cohort can provide future insight into disease progression, lack of response, and mechanism of resistance to the implemented therapy. Future studies focused on this veteran patient population are needed for developing targeted therapies and patient tailored oncologic therapy. ctDNA has a high potential for monitoring clinically relevant cancer-related genetic and epigenetic modifications for discovering more detailed information on the tumor characterization. Although larger cohort trial with longitudinal analyses are needed, high prevalence of DDR gene and Tp53 mutation in our study instills promising hope for therapeutic interventions in this unique cohort.

The minimally invasive liquid biopsy shows a great promise as both diagnostic and prognostic tool in the personalized clinical management of advanced prostate, and NSCLC in the veteran patient population with unique demographic characteristics. De novo metastatic prostate cancer is more common in veterans when compared with the general population, and therefore veterans may benefit by liquid biopsy. Differences in the frequency of genetic mutations (DDR, TP53, AR) in this cohort provides valuable information for disease progression, lack of response, mechanism of resistance to the implemented therapy and clinical decision making. Precision oncology can be further tailored for this cohort by focusing on DNA repair genes and Tp53 mutations for future targeted therapy.

The advent of liquid biopsies targeting genetic mutations in solid tumors is a major milestone in the field of precision oncology.¹ Conventional methods of obtaining tissue for molecular studies are limited by sample size and often do not represent the entire bulk of the tumor.² This newer minimally invasive, revolutionary technique analyzes circulating cell-free DNA carrying tumor-specific alterations (circulating tumor DNA [ctDNA]) in peripheral blood and detects signature genomic alterations.¹ Tp53 mutations have been reported in 25 to 40% of prostatic cancers and > 50% of non-small cell lung cancers (NSCLC), being more common in late-stage and hormone refractory prostate cancers.^3,4 Tp53 mutation has been found to be associated with poor prognosis and increased germline mutations.⁵

The veteran patient population has distinct demographic characteristics that make veterans more vulnerable to genetic mutations and malignancies, including risk of exposure to Agent Orange, smoking, substance abuse, and asbestos. This area is understudied and extremely sparse in the literature for frequency of genetic mutations, risk factors in solid malignancies occurring in the veteran patient population, and the clinical impact of these risk factors. We herein present a quality assurance study for the utility of liquid biopsies regarding the frequency of DNA damage repair (DDR) gene, Tp53, and androgen receptor (AR) mutations. The clinical impact in advanced lung and prostate cancers in the veteran patient population and frequency are the quality assurance observations that are the study endpoints.

Methods

We reviewed for quality assurance documentation from the Foundation Medicine (www.foundationmedicine.com) cancer biomarker tests on liquid biopsies performed at the Corporal Michael J. Crescenz Veteran Affairs Medical Center in Philadelphia, Pennsylvania from May 2019 to April 15, 2020. All biopsies were performed on cancers with biochemical, imaging or tissue evidence of advanced tumor progression. The testing was performed on advanced solid malignancies, including NSCLC, prostate adenocarcinoma, and metastatic colon cancer. Statistical data for adequacy; cases with notable mutations; frequency; and type of mutations of AR, DDR, and Tp53 were noted. General and specific risk factors associated with the veteran patient population were studied and matched with the type of mutations (Table 1).

Results

Thirty-one liquid biopsies were performed over this period—23 for prostate cancer, 7 for patients with lung cancer patients, and 1 for a patient with colon cancer. Of 31 cases, sensitivity/adequacy of liquid biopsy for genetic mutation was detected in 29 (93.5%) cases (Figure 1). Two inadequate biopsies (both from patients with prostate cancer) were excluded from the study, leaving 29 liquid biopsies with adequate ctDNA for analysis that were considered for further statistical purpose—21 prostate, 7 lung, and 1 colon cancer.

Multiple (common and different) genetic mutations were identified; however, our study subcategorized the mutations into the those that were related to prostate cancer, lung cancer, and some common mutations that occur in both cancers. Only the significant ones will be discussed in this review and equivocal result for AR is excluded from this study. Of the 21 prostate cancers, 4 (19.0%) had directed the targeted therapy to driver mutation (AR being most common in prostate cancer), while KRAS mutation, which was more common in lung cancer, was detected in 2/7 (28.6%) lung cancers. Mutations common to both cancer types were DDR gene mutations, which is a broad name for numerous genes including CDK12, ATM, and CHEK2.

Of all cases irrespective of the cancer type, 23/29 (79.3%) showed notable mutations. DDR gene mutations were found in 6 of 21 (28.5%) patients with prostate cancer and 8 of 23 (34.7%) patients with advanced prostate and lung cancers, indicating poor outcome and possible resistance to the current therapy. Of 23 patients showing mutations irrespective of the cancer type, 15 (65.2%) harbored Tp53 mutations, which is much more frequent in veteran patient population when compared with the literature. Fifteen of the 31 (48.4%) total patients were Vietnam War-era veterans who were potentially exposed to Agent Orange and 20 (64.5%) patients who were not Vietnam War-era veterans had a history that included smoking (Figure 2).

 

Discussion

The veteran patient population is a unique cohort due to its distinct demographic characteristics with a high volume of cancer cases diagnosed each year. According to data from VA Central Cancer Registry (VACCR), the most frequently diagnosed cancers are prostate (29%) and lung (18%).⁶

Liquid biopsy is a novel, promising technology that uses ctDNA and circulating tumor cells in peripheral blood for detecting genetic alterations through next generation sequencing.^7-9 The advent of this minimally invasive, revolutionary technology has been a breakthrough in the field of precision oncology for prognosis, to monitor treatment response or resistance to therapy and further personalize cancer therapy.^9,10

Comprehensive genomic profiling by liquid biopsy has many advantages over the molecular studies performed on tissue biopsy. Due to the tumor heterogeneity, tissue samples may not represent the full profile of the tumor genomics of cancer, while liquid biopsy has full presentation of the disease.^11,12 Many times, tissue biopsy may be limited by a sample size that precludes full genetic profiling in addition to higher total cost, potential technical issues during processing, and possible side effects of the biopsy procedure.^7,13 Additionally, as the tumor progresses, new driver mutations other than the ones previously detected on the primary tissue may emerge, which can confer resistance to the existing therapy.^7,13

Advanced prostatic and lung carcinomas with biochemical, distant organ, or bony progression harbor unique signature genetic mutations indicating poor prognosis, lack of response or resistance to the existing therapy, and high risk of relapse.^14,15 Some of the unique characteristics of the veteran patient population include a more aged patient population multiple comorbidities, higher frequency of > 1 type of cancer, advanced cancer stage at presentation, and specific risks factors such as exposure to Agent Orange in veterans who served during the Vietnam War era.^16,17 We studied the utility of liquid biopsy in cancer care, including type and incidence of genomic alterations associated with advanced prostate and lung cancers, in this unique patient population.

The amount of cell-free DNA (cfDNA), also known as ctDNA varies widely in cancer patients. Some of the factors associated with low concentration of cfDNA are disease stage, intervening therapy, proliferation rates, and tumor vascularization.^18,19 In the peripheral blood, of the total cfDNA, fractions of cfDNA varies from 0.01 to 90%.^18,19 All samples containing ≥ 20 ng cfDNA (20 - 100 ng) were subjected to the hybrid capture-based NGS FoundationACT assay.²⁰ In our study, 2 specimens did not meet the minimum criteria of adequacy (20 ng cfDNA); however, the overall adequacy rate for the detection of mutation, irrespective of the cancer type was 29 of 31 (93.5%) with only 2 inadequate samples. This rate is higher than the rate reported in the literature, which is about 70%.²⁰

Significant differences were encountered in the incidence of DNA damage repair genes including Tp53 mutations when compared with those in the general patient population (Table 2). According to recent National Comprehensive Cancer Network (NCCN) guidelines, all prostate cancers should be screened for DDR gene mutations as these genes are common in aggressive prostate cancers and strongly associated with poor outcomes and shortened survival. Due to relatively high frequency of DDR gene mutations in advanced prostatic cancers, liquid biopsy in patients with these advanced stage prostate cancers may be a useful tool in clinical decision making and exploring targeted therapy.²⁰

Mutations in BRCA2, ATM, CDK12, and CHEK2 (DDR gene family) are common. Incidence of ATM and CDK12 mutations in the literature is 3 to 6% of cases.²¹ Of 21 liquid biopsies of advanced prostate cancer patients, we found combined DDR gene mutation of ATM, CHEK2, and CDK12 genes in 6 (28.5%) cases, which is substantially higher than the 3 to 6% rate reported in the literature.^21-24 Of the 23 patients who had notable mutations in our liquid biopsies, including both advanced prostate and lung cancer cases, 8 (34.7%) also showed mutation of the genes of DDR family. Our study did not show BRCA2 mutation, which is otherwise common in the literature.

We also evaluated the frequency of the most commonly occurring genetic mutations, Tp53 in advanced solid malignancies, especially advanced prostate and NSCLC. Previous studies have reported Tp53 mutation in association with risk factors (carcinogens) of cancer and have been a surrogate marker of poor survival or lack of response of therapy.²⁵ Knowledge of Tp53 mutation is crucial for closer disease monitoring, preparing the patient for rapid progression, and encouraging the physician to prepare future lines of therapy.^25-27 Although Tp53 mutation varies with histologic type and tissue of origin, Beltran and colleagues reported it in 30 to 40% of tumors, while Robles and colleagues reported about 40 to 42% incidence.^25,27

Our study showed notable mutations in 23 of 29 adequate cases. Further, our study showed a high frequency of mutated Tp53 in 65.2% of combined advanced prostate and NSCLC cases. We then correlated cases of Vietnam War-era veterans with risk potential of Agent Orange exposure and Tp53 mutation. We found 7 of 15 Vietnam War-era veterans were positive for Tp53 mutations irrespective of the cancer type. The high incidence of Tp53 mutations in advanced prostate and lung carcinomas in the veteran patient population makes this tumor marker an aspiration not only as a surrogate of aggressive disease and tumor progression, but also as a key marker for targeted therapy in advanced prostate and lung cancers with loss of Tp53 function (Figure 3).

Mutations and amplifications in the AR gene are fundamental to progression of prostate cancer associated with advanced, hormone-refractory prostate cancer with the potential for targeted therapy with AR inhibitors. In our study, AR amplification was detected in 4 of 21 (19%) advanced prostate cancer cases, which is significantly lower than the 30 to 50% previously reported in the literature.^28-32 Neither AR amplification or mutation was noted in advanced NSCLC in our study as previously reported in literature by Brennan and colleagues and Wang and colleagues.^33-35 This is significant as it provides a pathway for future studies to focus on additional driver mutations for targeted therapies in advanced prostate carcinoma. To date, AR gene mutation does not play a role for personalized therapy in advanced NSCLC. Perhaps, a large cohort study with longitudinal analysis is needed for absolutely ruling out the possibility of personalized medicine in advanced lung cancer using this biomarker.

 

Conclusions

Liquid biopsy successfully provides precision-based oncology and information for decision making in this unique population of veterans. Difference in frequency of the genetic mutations in this cohort can provide future insight into disease progression, lack of response, and mechanism of resistance to the implemented therapy. Future studies focused on this veteran patient population are needed for developing targeted therapies and patient tailored oncologic therapy. ctDNA has a high potential for monitoring clinically relevant cancer-related genetic and epigenetic modifications for discovering more detailed information on the tumor characterization. Although larger cohort trial with longitudinal analyses are needed, high prevalence of DDR gene and Tp53 mutation in our study instills promising hope for therapeutic interventions in this unique cohort.

The minimally invasive liquid biopsy shows a great promise as both diagnostic and prognostic tool in the personalized clinical management of advanced prostate, and NSCLC in the veteran patient population with unique demographic characteristics. De novo metastatic prostate cancer is more common in veterans when compared with the general population, and therefore veterans may benefit by liquid biopsy. Differences in the frequency of genetic mutations (DDR, TP53, AR) in this cohort provides valuable information for disease progression, lack of response, mechanism of resistance to the implemented therapy and clinical decision making. Precision oncology can be further tailored for this cohort by focusing on DNA repair genes and Tp53 mutations for future targeted therapy.

In the midst of an opioid overdose public health crisis, the US Department of Health and Human Services developed a 5-point strategy to combat this problem. One aspect of this strategy is improved pain management.¹ There is high demand for pain management services with a limited number of health care professionals appropriately trained to deliver care.² Pharmacists are integral members of the interdisciplinary pain team and meet this demand.

Background

For almost 50 years, pharmacists at the US Department of Veterans Affairs (VA) have been functioning as advanced practice providers (APP).³ Clinical pharmacy specialists (CPS) provide comprehensive medication management (CMM) and have a scope of practice (SOP). The SOP serves as the collaborating agreement and outlines the clinical duties permitted in delivering patient care. In addition, the SOP may indicate specific practice areas and are standardized across VA (Table 1).^4,5 Pharmacists apply for a SOP and must prove their competency in the practice area and provide documentation of their education, training, experience, knowledge, and skills.^5,6 Residency and/or board certification are not required though helpful. A pharmacist’s SOP is reviewed and approved by the facility executive committee.⁵ Pharmacists with a SOP undergo professional practice evaluation twice a year. Prescribing controlled substances is permissible in the SOP if approved by the facility and allowed by the state of licensure. According to the US Drug Enforcement Agency (DEA) as of February 10, 2020, 8 states (California, Washington, Idaho, Massachusetts, Montana, New Mexico, North Carolina, and Ohio) allow pharmacists to prescribe controlled substances.⁷

 

The VA developed the Pharmacists Achieve Results with Medications Documentation (PhARMD) tool that allows clinical pharmacists to document specific interventions made during clinical care and is included in their progress note. Data from fiscal year 2017 demonstrates that 136,041 pain management interventions were made by pharmacists across VA. The majority of these interventions were implemented by a CPS working autonomously as an APP.⁸

Several articles discuss the pharmacists role in the opioid crisis, although no outcomes data were provided. Chisholm-Burns and colleagues listed multiple potential ways that pharmacists can intervene, including managing pain in primary care clinic settings by using collaborative drug therapy agreements (CDTAs), using opioid exit plans and discharge planning in collaboration with other health care providers (HCPs), or making recommendations to the prescribers before writing prescriptions.⁹ Compton and colleagues similarly reviewed pharmacist roles in the opioid crisis. However, their focus was on dispensing pharmacists that provided education to patients about storage and disposal of opioids, identified opioid misuse, provided opioid overdose education and naloxone, and checked prescription drug monitoring programs (PDMPs).¹⁰ Missing from these articles was the role of the clinical pharmacist working as an APP delivering direct patient care and prescribing controlled substances.

Hammer and colleagues discussed the role of an oncology CPS with controlled substance prescriptive authority in pain management at an outpatient cancer center in Washington state.¹¹ Under a CDTA, pharmacists could prescribe medications, including controlled substances if they obtain DEA registration. The pharmacist completed a comprehensive in-person assessment. The attending physician conducted a physical examination. Then the pharmacist presented the patient and proposed regimen to the interprofessional team to determine a final plan. Ultimately, the pharmacist wrote any controlled substance prescriptions. The patient followed up every 1 to 4 weeks by telephone with a nurse, and in-person assessments occurred at least every 6 months. No outcomes data were provided.¹¹

Dole and colleagues reviewed the role of a pharmacist who had controlled substance prescriptive authority in a pain management clinic. The pharmacist provider saw up to 18 patients a day and then managed refill requests for 3 hours a day. The main outcome was change in visual analog scale (VAS) pain scores. Findings showed that reductions in VAS pain scores were statistically significant (P < .01). The pharmacist processed about 150 refills with an unclear number of controlled substances requests a day based on a medication-refill protocol. This was felt to improve access to physicians for acute needs, improve consistency in refills, and capture patients in need of follow-up. Additionally, the clinic saved $455,238 after 1 year.¹²

 

Study Aims

A review of the literature indicated sparse data on the impact of a pharmacist on opioid tapering, opioid dose, and opioid risk mitigation when the pharmacist is prescribing controlled substances. The purpose of this retrospective review was to characterize the controlled substance prescribing practices by the pharmacy pain clinic. The aim was to examine the pharmacist impact on morphine milligram equivalent (MME) and compliance with opioid risk mitigation strategies.

Methods

This project was a retrospective, single-center, chart review. The project was reviewed and approved by the University of Missouri-Columbia Institutional Review Board used by the Harry S. Truman Memorial Veterans’ Hospital (HSTMVH) as a quality improvement project. The author applied for controlled substance registration through the DEA and was issued registration April 30, 2018. The State of Ohio Board of Pharmacy was contacted as required by Ohio Administrative Code. The author's updated SOP to allow controlled substance prescribing was approved July 23, 2018. The CPS functions as an APP within an interdisciplinary pain management team that includes physicians, occupational and physical therapists, complementary and integrative health, and a psychologist. The reason for Pharmacy Pain Consult is required and it is primarily submitted through the electronic health record. The consult is reviewed for appropriateness and once approved is scheduled by support staff. Once the patient is stabilized, the patient is discharged back to their primary care provider (PCP) or referring provider for continued care. Patients were considered stabilized when their patient-specific goals were met, which varied from use of the lowest effective opioid dose to taper to discontinuation of opioids with no further medication changes needed. The taper strategy for each patient was individualized. Patients were generally tapered on their existing opioid medication unless they were new to the VA and on nonformulary medications or experiencing a significant adverse reaction. Numerous references are available through VA to assist with opioid tapering.^13,14 The CPS is able to refer patients to other services, including behavioral health for substance use disorder treatment and medication-assisted treatment if concerns were identified.

Initial data were collected from the Veterans Integrated Service Network (VISN) 15 Corporate Data Warehouse by the VISN pharmacy analytics program manager. The original report included patients prescribed a Schedule II to V controlled substance by the author from July 1, 2018 to January 31, 2020. Chart review was conducted on each patient to obtain additional data. At the time of consult and discharge the following data were collected: opioid medication; MME; use of opioid risk mitigation strategies, such as urine drug screens (UDS), informed consent, opioid overdose education and naloxone distribution program (OEND), risk assessment via stratification tool for opioid risk mitigation (STORM), PDMP checks; and nonopioid medication number and classes.

Patients were included in the review if they were prescribed an opioid Schedule II or III controlled substance between July 1, 2018 and January 31, 2020. Patient were excluded if they were prescribed an opioid Schedule II or III controlled substance primarily as coverage for another prescriber. Patients prescribed only pregabalin, tramadol, or a benzodiazepine also were excluded.

The primary endpoint was change in MME from baseline to discharge from clinic. Secondary endpoints included change in opioid risk mitigation strategies and change in opioid medications prescribed from baseline to discharge.

Descriptive statistics were used to analyze parts of the data. A 2-sided t test was used to compare baseline and discharge MME. The Fisher exact test was used to compare nominal data of opioid risk mitigation strategies.

Calculation of MME was performed using the conversion factors provided by the Centers Disease Control and Prevention (CDC) for opioid guideline.¹⁵ For buprenorphine, tapentadol, and levorphanol conversion ratios were obtained from other sources. The conversion ratios used, included 75:1 for oral morphine to transdermal buprenorphine, 1:3.3 for oral morphine to oral tapentadol, and 1:7.5 for oral levorphanol to oral morphine.^16,17 The Revised Standards for Quality Improvement Reporting Excellence (SQUIRE 2.0) was used to write the manuscript.¹⁸

 

Results

Seventy-five patients were included in this review. The average age of patients was 66 years; and 12% were female (n = 9) (Table 2). The largest number of consults came from PCPs (44%, n = 33) and the pain clinic (43%, n = 32). Nearly half (48%) of the consultations were for opioid tapering (n = 36), followed by 37% for opioid optimization or monitoring (n = 28), and 19% for nonopioid optimization (n = 14). The most common primary diagnoses at consultation were for chronic low back pain (56%), chronic neck pain (20%), and osteoarthritis (16%).

The average MME at time of consult was 93 MME compared with 31 MME at discharge which was statisticially significant (P < .01) (Figure 1). The mean percent change in MME was 46%, including methadone and 42% excluding methadone. There was a 26% change in UDS, 28% change in informed consent, 85% change in PDMP, 194% change in naloxone, and 357% change in STORM reviews from baseline to discharge with all demonstrating statistical significance (P < .01) (Figure 2). At discharge, the most common opioid prescribed was morphine SA (short acting) (n = 10, 13%, 44 average MME) and oxycodone/acetaminophen (n = 10, 13%, 28 average MME) (Table 3).

The average number of days from consult to initial visit was 23 days (Table 4). Face-to-face was the primary means of initial visit with 92% (n = 69) of visits, but phone was the primary mode of follow-up with 73% of visits (n = 55). The average number of follow-up visits was 7, representing 176 average days of time in the Pharmacy Pain Clinic. Consultation to the behavioral health performance program was the most common referral (n = 13, 17%).

Five patients were new opioid starts in the Pharmacy Pain Clinic. Two patients were on tramadol at time of consult. Of the 5 new opioid starts, 3 patients received oxycodone/acetaminophen, 1 received buprenorphine patch, and 1 received hydrocodone/acetaminophen. The new opioid start average was 25 MME. All 5 patients had a UDS for opioid risk mitigation, 4 used consent and STORM reviews, and 2 patients had PDMP checks and naloxone.

Discussion

There was a statistically significant decrease of the mean MME between the time of consult and the time of discharge. There also were statistically significant changes in use of opioid risk mitigation strategies. Since methadone has a high MME, the mean reduction of MME was calculated with methadone (46%) and without methadone (42%). These data are consistent with other published studies examining opioid tapers in the VA population. Harden and colleagues calculated a 46% mean reduction in MME over 12 months for 72 veterans from opioid tapers implemented by PCPs, pain service, or pharmacist-run clinics.¹⁹

There is controversy about equianalgesic doses and no established universal equianalgesic conversion calculator or dose. Numerous equianalgesic opioid dose calculators are available, but for this analysis the CDC MME conversion factors were used (available at: https://www.cdc.gov/drugoverdose/pdf/calculating_total_daily_dose-a.pdf). Previous literature compared existing calculators and found significant variances in calculated doses for methadone and fentanyl conversions.²⁰ Additionally, there have been concerns expressed with the safety of the CDC opioid calculator specifically surrounding the conversions for methadone and tapentadol.²¹ In the end, I chose the CDC calculator because it is established, readily available, and consistent.

Pharmacists in pain management can address access issues.^2,3,11,12 The average length of time from consult to initial visit was 23 days. Often patients may have seen a HCP who implemented a change at the time of consult and wanted the patient to be seen 1 month later. Many patients at the HSTMVH live far from the facility, making in-person visits difficult. A majority of the follow-up visits were conducted by telephone. Patients were offered all modalities available for follow-up, including telephone, in-person, or telemedicine, but patients most often picked telephone. Patients averaged 7 follow-up visits before discharge. This number of visits would have taken time from other health care team members who could have been addressing other veterans. Patients were seen in clinic for 176 days on average, which supports and follows recommendations for a slow, incremental taper.

The opioid medications prescribed changed over time in the clinic. Methadone prescriptions dropped from 20 to 6 at consult to discharge, and fentanyl prescriptions fell from 7 to 2, respectively. The CDC guideline suggests use of long-acting products with more predictable pharmacokinetics (eg, morphine SA or oxycodone SA) rather than fentanyl or methadone.¹⁵ Notably, the use of buprenorphine products with FDA approval for pain indications increased from consult to discharge. Many of the patients in this study had pulmonary comorbidities, placing them at higher risk for adverse outcomes. Buprenorphine is a partial μ-opioid receptor agonist with a ceiling on respiratory depression so is potentially less risky in those with pulmonary comorbidities.

The biggest changes in opioid risk mitigation occurred in PDMP, OEND program, and STORM reviews. An 85% increase in PDMP reviews occurred with referral to the clinic. Missouri is the only state without a state-run PDMP. However, the St. Louis County PDMP was developed based on city or county participation and encompasses 85% of the population of Missouri and 94% of HCPs in Missouri as of August 29, 2019.²² Because there is no state-level PDMP, a review of the St. Louis County PDMP was not required during the review period. Nevertheless, the Pharmacy Pain Clinic uses the St. Louis County PDMP at the initial visit and regularly during care. VA policy requires a specific note title be used to document each check of the PDMP.²³

There was a 194% increase in patients receiving naloxone with consultation to the Pharmacy Pain Clinic. Due to low coprescribing of naloxone for patients prescribed chronic opioid therapy, The author led an interdisciplinary team analysis of health care failure mode effects during the study period. This led to a process change with coprescribing of naloxone at refill in the primary care clinic.

The Comprehensive Addiction and Recovery Act of 2016 mandated that the VA review STORM on new start of opioids or patient identified as “very high-opioid prescription risk” category by an interdisciplinary opioid risk review team.²⁴ Thus many of the patients referred to clinic didn’t require STORM reviews since they were not new opioid starts or identified as high risk. However, in the standard review of all new patients to the Pharmacy Pain Clinic, a STORM review is conducted and documented to assess the patient’s level of risk.

Only 5 patients were started on opioid medications during the study period. This is consistent with both CDC and the joint VA/US Department of Defense opioid prescribing guidelines that recommend against initiation of opioids for chronic nonmalignant pain.^13,15 Two of the patients were prescribed tramadol for ineffective pain control at time of consult. Furthermore, 4 of the 5 patients were started on a short-acting opioid, which was supported by guidelines.^13,15 One patient was initiated on buprenorphine patches due to comorbid chronic kidney disease. The VA does not limit the quantity of new opioid prescriptions, although some states and private insurance plans are implementing limitations. Guidelines also recommend against exceeding 90 MME due to risk. The average MME in this project at discharge was 25 MME. Use of opioid risk mitigation for the new opioid starts was reasonable. The reason for the missing PDMP report is unknown based on chart review and atypical according to clinic practice.

Recently, efforts to expand pharmacist training and positions in pain management at VA facilities have been undertaken. In 2016, there were just 11 American Society of Health-System Pharmacists-accredited pharmacy postgraduate year 2 pain and palliative care residency programs, which has expanded to 26 sites in 2020.^2,3,25 In addition, the Clinical Pharmacy Practice Office and the VA Office of Rural Health have helped to hire 33 new pain management pharmacists.³

The role of pharmacists in prescribing controlled substances is limited mainly due to the small number of states that extend this authority.⁷ At the VA, a pharmacist can practice using any state of licensure. Therefore, a pharmacist working at a VA in a state that does not authorize controlled substance prescribing could obtain a license in a state that does permit it. However, the main barrier to obtaining other state licensures is the cost. At the time the author obtained controlled substance prescriptive authority, little direction was available on the process for advanced practice pharmacists at the VA. Since then, guidance has been developed to ease this process. Educational endeavors at VA have been implemented with the intent to increase the number of pharmacists with controlled substance prescriptive authority.

Barriers to pharmacists providing pain care extend beyond limited controlled substance prescriptive authority. Often pharmacists are still viewed in their traditional and operational role.^9,10 Other health care team members and patients may not be aware or familiar with the training, knowledge, and skills of pharmacist's and their suitability as an APP.^26,27 Most states permit pharmacists in establishing CDTA but not all. Additionally, some states recognize pharmacists as HCPs but many more do not. Furthermore, the Social Security Act does not include pharmacists as HCPs. This makes it challenging, though not impossible, for pharmacists to bill for their services.³

 

Strengths and Limitations

There were numerous strengths of the project. First, this addressed an unmet need in the literature with limited data discussing pharmacist prescribing controlled substances for pain management. There was 1 data reviewer who made the data collection process consistent. Since this retrospectively reviewed controlled substance prescribing in clinic, it captured real-world practice compared with that of experimental models. There were also several limitations in the project. The person collecting the data was also the person who conducted the clinic. The study was conducted retrospectively and based on documented information in the medical record. The population reviewed was primarily male and older, which fits the VA patient population but has less generalizability to other patient populations. This project was conducted at a single VA facility so may not be generalizable to other VA sites. It is unknown whether patients were again prescribed opioids if they left the VA for the community or another VA facility. The pain diagnoses or locations of pain were categorized to main groups and reliant on the referring provider. Another major weakness was the lack of comparison of pain scores or validated objective measure of function at baseline and at discharge. This consideration would be important for future work.

Conclusions

Pharmacists functioning as APP are key members of the pain management team. A review of a pharmacy-run pain clinic demonstrated statistically significant reduction in MME and improvement in opioid risk mitigation from consult to discharge. Patients enrolled in the pharmacy-managed clinic also had improvements in adherence to opioid risk mitigation strategies. Future attention should be focused on further expanding training and positions for pharmacists as APP in pain management.

Acknowledgments

The author thanks Chris Sedgwick for his assistance with data capture.

In the midst of an opioid overdose public health crisis, the US Department of Health and Human Services developed a 5-point strategy to combat this problem. One aspect of this strategy is improved pain management.¹ There is high demand for pain management services with a limited number of health care professionals appropriately trained to deliver care.² Pharmacists are integral members of the interdisciplinary pain team and meet this demand.

Background

For almost 50 years, pharmacists at the US Department of Veterans Affairs (VA) have been functioning as advanced practice providers (APP).³ Clinical pharmacy specialists (CPS) provide comprehensive medication management (CMM) and have a scope of practice (SOP). The SOP serves as the collaborating agreement and outlines the clinical duties permitted in delivering patient care. In addition, the SOP may indicate specific practice areas and are standardized across VA (Table 1).^4,5 Pharmacists apply for a SOP and must prove their competency in the practice area and provide documentation of their education, training, experience, knowledge, and skills.^5,6 Residency and/or board certification are not required though helpful. A pharmacist’s SOP is reviewed and approved by the facility executive committee.⁵ Pharmacists with a SOP undergo professional practice evaluation twice a year. Prescribing controlled substances is permissible in the SOP if approved by the facility and allowed by the state of licensure. According to the US Drug Enforcement Agency (DEA) as of February 10, 2020, 8 states (California, Washington, Idaho, Massachusetts, Montana, New Mexico, North Carolina, and Ohio) allow pharmacists to prescribe controlled substances.⁷

 

The VA developed the Pharmacists Achieve Results with Medications Documentation (PhARMD) tool that allows clinical pharmacists to document specific interventions made during clinical care and is included in their progress note. Data from fiscal year 2017 demonstrates that 136,041 pain management interventions were made by pharmacists across VA. The majority of these interventions were implemented by a CPS working autonomously as an APP.⁸

Several articles discuss the pharmacists role in the opioid crisis, although no outcomes data were provided. Chisholm-Burns and colleagues listed multiple potential ways that pharmacists can intervene, including managing pain in primary care clinic settings by using collaborative drug therapy agreements (CDTAs), using opioid exit plans and discharge planning in collaboration with other health care providers (HCPs), or making recommendations to the prescribers before writing prescriptions.⁹ Compton and colleagues similarly reviewed pharmacist roles in the opioid crisis. However, their focus was on dispensing pharmacists that provided education to patients about storage and disposal of opioids, identified opioid misuse, provided opioid overdose education and naloxone, and checked prescription drug monitoring programs (PDMPs).¹⁰ Missing from these articles was the role of the clinical pharmacist working as an APP delivering direct patient care and prescribing controlled substances.

Hammer and colleagues discussed the role of an oncology CPS with controlled substance prescriptive authority in pain management at an outpatient cancer center in Washington state.¹¹ Under a CDTA, pharmacists could prescribe medications, including controlled substances if they obtain DEA registration. The pharmacist completed a comprehensive in-person assessment. The attending physician conducted a physical examination. Then the pharmacist presented the patient and proposed regimen to the interprofessional team to determine a final plan. Ultimately, the pharmacist wrote any controlled substance prescriptions. The patient followed up every 1 to 4 weeks by telephone with a nurse, and in-person assessments occurred at least every 6 months. No outcomes data were provided.¹¹

Dole and colleagues reviewed the role of a pharmacist who had controlled substance prescriptive authority in a pain management clinic. The pharmacist provider saw up to 18 patients a day and then managed refill requests for 3 hours a day. The main outcome was change in visual analog scale (VAS) pain scores. Findings showed that reductions in VAS pain scores were statistically significant (P < .01). The pharmacist processed about 150 refills with an unclear number of controlled substances requests a day based on a medication-refill protocol. This was felt to improve access to physicians for acute needs, improve consistency in refills, and capture patients in need of follow-up. Additionally, the clinic saved $455,238 after 1 year.¹²

 

Study Aims

A review of the literature indicated sparse data on the impact of a pharmacist on opioid tapering, opioid dose, and opioid risk mitigation when the pharmacist is prescribing controlled substances. The purpose of this retrospective review was to characterize the controlled substance prescribing practices by the pharmacy pain clinic. The aim was to examine the pharmacist impact on morphine milligram equivalent (MME) and compliance with opioid risk mitigation strategies.

Methods

This project was a retrospective, single-center, chart review. The project was reviewed and approved by the University of Missouri-Columbia Institutional Review Board used by the Harry S. Truman Memorial Veterans’ Hospital (HSTMVH) as a quality improvement project. The author applied for controlled substance registration through the DEA and was issued registration April 30, 2018. The State of Ohio Board of Pharmacy was contacted as required by Ohio Administrative Code. The author's updated SOP to allow controlled substance prescribing was approved July 23, 2018. The CPS functions as an APP within an interdisciplinary pain management team that includes physicians, occupational and physical therapists, complementary and integrative health, and a psychologist. The reason for Pharmacy Pain Consult is required and it is primarily submitted through the electronic health record. The consult is reviewed for appropriateness and once approved is scheduled by support staff. Once the patient is stabilized, the patient is discharged back to their primary care provider (PCP) or referring provider for continued care. Patients were considered stabilized when their patient-specific goals were met, which varied from use of the lowest effective opioid dose to taper to discontinuation of opioids with no further medication changes needed. The taper strategy for each patient was individualized. Patients were generally tapered on their existing opioid medication unless they were new to the VA and on nonformulary medications or experiencing a significant adverse reaction. Numerous references are available through VA to assist with opioid tapering.^13,14 The CPS is able to refer patients to other services, including behavioral health for substance use disorder treatment and medication-assisted treatment if concerns were identified.

Initial data were collected from the Veterans Integrated Service Network (VISN) 15 Corporate Data Warehouse by the VISN pharmacy analytics program manager. The original report included patients prescribed a Schedule II to V controlled substance by the author from July 1, 2018 to January 31, 2020. Chart review was conducted on each patient to obtain additional data. At the time of consult and discharge the following data were collected: opioid medication; MME; use of opioid risk mitigation strategies, such as urine drug screens (UDS), informed consent, opioid overdose education and naloxone distribution program (OEND), risk assessment via stratification tool for opioid risk mitigation (STORM), PDMP checks; and nonopioid medication number and classes.

Patients were included in the review if they were prescribed an opioid Schedule II or III controlled substance between July 1, 2018 and January 31, 2020. Patient were excluded if they were prescribed an opioid Schedule II or III controlled substance primarily as coverage for another prescriber. Patients prescribed only pregabalin, tramadol, or a benzodiazepine also were excluded.

The primary endpoint was change in MME from baseline to discharge from clinic. Secondary endpoints included change in opioid risk mitigation strategies and change in opioid medications prescribed from baseline to discharge.

Descriptive statistics were used to analyze parts of the data. A 2-sided t test was used to compare baseline and discharge MME. The Fisher exact test was used to compare nominal data of opioid risk mitigation strategies.

Calculation of MME was performed using the conversion factors provided by the Centers Disease Control and Prevention (CDC) for opioid guideline.¹⁵ For buprenorphine, tapentadol, and levorphanol conversion ratios were obtained from other sources. The conversion ratios used, included 75:1 for oral morphine to transdermal buprenorphine, 1:3.3 for oral morphine to oral tapentadol, and 1:7.5 for oral levorphanol to oral morphine.^16,17 The Revised Standards for Quality Improvement Reporting Excellence (SQUIRE 2.0) was used to write the manuscript.¹⁸

 

Results

Seventy-five patients were included in this review. The average age of patients was 66 years; and 12% were female (n = 9) (Table 2). The largest number of consults came from PCPs (44%, n = 33) and the pain clinic (43%, n = 32). Nearly half (48%) of the consultations were for opioid tapering (n = 36), followed by 37% for opioid optimization or monitoring (n = 28), and 19% for nonopioid optimization (n = 14). The most common primary diagnoses at consultation were for chronic low back pain (56%), chronic neck pain (20%), and osteoarthritis (16%).

The average MME at time of consult was 93 MME compared with 31 MME at discharge which was statisticially significant (P < .01) (Figure 1). The mean percent change in MME was 46%, including methadone and 42% excluding methadone. There was a 26% change in UDS, 28% change in informed consent, 85% change in PDMP, 194% change in naloxone, and 357% change in STORM reviews from baseline to discharge with all demonstrating statistical significance (P < .01) (Figure 2). At discharge, the most common opioid prescribed was morphine SA (short acting) (n = 10, 13%, 44 average MME) and oxycodone/acetaminophen (n = 10, 13%, 28 average MME) (Table 3).

The average number of days from consult to initial visit was 23 days (Table 4). Face-to-face was the primary means of initial visit with 92% (n = 69) of visits, but phone was the primary mode of follow-up with 73% of visits (n = 55). The average number of follow-up visits was 7, representing 176 average days of time in the Pharmacy Pain Clinic. Consultation to the behavioral health performance program was the most common referral (n = 13, 17%).

Five patients were new opioid starts in the Pharmacy Pain Clinic. Two patients were on tramadol at time of consult. Of the 5 new opioid starts, 3 patients received oxycodone/acetaminophen, 1 received buprenorphine patch, and 1 received hydrocodone/acetaminophen. The new opioid start average was 25 MME. All 5 patients had a UDS for opioid risk mitigation, 4 used consent and STORM reviews, and 2 patients had PDMP checks and naloxone.

Discussion

There was a statistically significant decrease of the mean MME between the time of consult and the time of discharge. There also were statistically significant changes in use of opioid risk mitigation strategies. Since methadone has a high MME, the mean reduction of MME was calculated with methadone (46%) and without methadone (42%). These data are consistent with other published studies examining opioid tapers in the VA population. Harden and colleagues calculated a 46% mean reduction in MME over 12 months for 72 veterans from opioid tapers implemented by PCPs, pain service, or pharmacist-run clinics.¹⁹

There is controversy about equianalgesic doses and no established universal equianalgesic conversion calculator or dose. Numerous equianalgesic opioid dose calculators are available, but for this analysis the CDC MME conversion factors were used (available at: https://www.cdc.gov/drugoverdose/pdf/calculating_total_daily_dose-a.pdf). Previous literature compared existing calculators and found significant variances in calculated doses for methadone and fentanyl conversions.²⁰ Additionally, there have been concerns expressed with the safety of the CDC opioid calculator specifically surrounding the conversions for methadone and tapentadol.²¹ In the end, I chose the CDC calculator because it is established, readily available, and consistent.

Pharmacists in pain management can address access issues.^2,3,11,12 The average length of time from consult to initial visit was 23 days. Often patients may have seen a HCP who implemented a change at the time of consult and wanted the patient to be seen 1 month later. Many patients at the HSTMVH live far from the facility, making in-person visits difficult. A majority of the follow-up visits were conducted by telephone. Patients were offered all modalities available for follow-up, including telephone, in-person, or telemedicine, but patients most often picked telephone. Patients averaged 7 follow-up visits before discharge. This number of visits would have taken time from other health care team members who could have been addressing other veterans. Patients were seen in clinic for 176 days on average, which supports and follows recommendations for a slow, incremental taper.

The opioid medications prescribed changed over time in the clinic. Methadone prescriptions dropped from 20 to 6 at consult to discharge, and fentanyl prescriptions fell from 7 to 2, respectively. The CDC guideline suggests use of long-acting products with more predictable pharmacokinetics (eg, morphine SA or oxycodone SA) rather than fentanyl or methadone.¹⁵ Notably, the use of buprenorphine products with FDA approval for pain indications increased from consult to discharge. Many of the patients in this study had pulmonary comorbidities, placing them at higher risk for adverse outcomes. Buprenorphine is a partial μ-opioid receptor agonist with a ceiling on respiratory depression so is potentially less risky in those with pulmonary comorbidities.

The biggest changes in opioid risk mitigation occurred in PDMP, OEND program, and STORM reviews. An 85% increase in PDMP reviews occurred with referral to the clinic. Missouri is the only state without a state-run PDMP. However, the St. Louis County PDMP was developed based on city or county participation and encompasses 85% of the population of Missouri and 94% of HCPs in Missouri as of August 29, 2019.²² Because there is no state-level PDMP, a review of the St. Louis County PDMP was not required during the review period. Nevertheless, the Pharmacy Pain Clinic uses the St. Louis County PDMP at the initial visit and regularly during care. VA policy requires a specific note title be used to document each check of the PDMP.²³

There was a 194% increase in patients receiving naloxone with consultation to the Pharmacy Pain Clinic. Due to low coprescribing of naloxone for patients prescribed chronic opioid therapy, The author led an interdisciplinary team analysis of health care failure mode effects during the study period. This led to a process change with coprescribing of naloxone at refill in the primary care clinic.

The Comprehensive Addiction and Recovery Act of 2016 mandated that the VA review STORM on new start of opioids or patient identified as “very high-opioid prescription risk” category by an interdisciplinary opioid risk review team.²⁴ Thus many of the patients referred to clinic didn’t require STORM reviews since they were not new opioid starts or identified as high risk. However, in the standard review of all new patients to the Pharmacy Pain Clinic, a STORM review is conducted and documented to assess the patient’s level of risk.

Only 5 patients were started on opioid medications during the study period. This is consistent with both CDC and the joint VA/US Department of Defense opioid prescribing guidelines that recommend against initiation of opioids for chronic nonmalignant pain.^13,15 Two of the patients were prescribed tramadol for ineffective pain control at time of consult. Furthermore, 4 of the 5 patients were started on a short-acting opioid, which was supported by guidelines.^13,15 One patient was initiated on buprenorphine patches due to comorbid chronic kidney disease. The VA does not limit the quantity of new opioid prescriptions, although some states and private insurance plans are implementing limitations. Guidelines also recommend against exceeding 90 MME due to risk. The average MME in this project at discharge was 25 MME. Use of opioid risk mitigation for the new opioid starts was reasonable. The reason for the missing PDMP report is unknown based on chart review and atypical according to clinic practice.

Recently, efforts to expand pharmacist training and positions in pain management at VA facilities have been undertaken. In 2016, there were just 11 American Society of Health-System Pharmacists-accredited pharmacy postgraduate year 2 pain and palliative care residency programs, which has expanded to 26 sites in 2020.^2,3,25 In addition, the Clinical Pharmacy Practice Office and the VA Office of Rural Health have helped to hire 33 new pain management pharmacists.³

The role of pharmacists in prescribing controlled substances is limited mainly due to the small number of states that extend this authority.⁷ At the VA, a pharmacist can practice using any state of licensure. Therefore, a pharmacist working at a VA in a state that does not authorize controlled substance prescribing could obtain a license in a state that does permit it. However, the main barrier to obtaining other state licensures is the cost. At the time the author obtained controlled substance prescriptive authority, little direction was available on the process for advanced practice pharmacists at the VA. Since then, guidance has been developed to ease this process. Educational endeavors at VA have been implemented with the intent to increase the number of pharmacists with controlled substance prescriptive authority.

Barriers to pharmacists providing pain care extend beyond limited controlled substance prescriptive authority. Often pharmacists are still viewed in their traditional and operational role.^9,10 Other health care team members and patients may not be aware or familiar with the training, knowledge, and skills of pharmacist's and their suitability as an APP.^26,27 Most states permit pharmacists in establishing CDTA but not all. Additionally, some states recognize pharmacists as HCPs but many more do not. Furthermore, the Social Security Act does not include pharmacists as HCPs. This makes it challenging, though not impossible, for pharmacists to bill for their services.³

 

Strengths and Limitations

There were numerous strengths of the project. First, this addressed an unmet need in the literature with limited data discussing pharmacist prescribing controlled substances for pain management. There was 1 data reviewer who made the data collection process consistent. Since this retrospectively reviewed controlled substance prescribing in clinic, it captured real-world practice compared with that of experimental models. There were also several limitations in the project. The person collecting the data was also the person who conducted the clinic. The study was conducted retrospectively and based on documented information in the medical record. The population reviewed was primarily male and older, which fits the VA patient population but has less generalizability to other patient populations. This project was conducted at a single VA facility so may not be generalizable to other VA sites. It is unknown whether patients were again prescribed opioids if they left the VA for the community or another VA facility. The pain diagnoses or locations of pain were categorized to main groups and reliant on the referring provider. Another major weakness was the lack of comparison of pain scores or validated objective measure of function at baseline and at discharge. This consideration would be important for future work.

Conclusions

Pharmacists functioning as APP are key members of the pain management team. A review of a pharmacy-run pain clinic demonstrated statistically significant reduction in MME and improvement in opioid risk mitigation from consult to discharge. Patients enrolled in the pharmacy-managed clinic also had improvements in adherence to opioid risk mitigation strategies. Future attention should be focused on further expanding training and positions for pharmacists as APP in pain management.

Acknowledgments

The author thanks Chris Sedgwick for his assistance with data capture.

In the midst of an opioid overdose public health crisis, the US Department of Health and Human Services developed a 5-point strategy to combat this problem. One aspect of this strategy is improved pain management.¹ There is high demand for pain management services with a limited number of health care professionals appropriately trained to deliver care.² Pharmacists are integral members of the interdisciplinary pain team and meet this demand.

Background

For almost 50 years, pharmacists at the US Department of Veterans Affairs (VA) have been functioning as advanced practice providers (APP).³ Clinical pharmacy specialists (CPS) provide comprehensive medication management (CMM) and have a scope of practice (SOP). The SOP serves as the collaborating agreement and outlines the clinical duties permitted in delivering patient care. In addition, the SOP may indicate specific practice areas and are standardized across VA (Table 1).^4,5 Pharmacists apply for a SOP and must prove their competency in the practice area and provide documentation of their education, training, experience, knowledge, and skills.^5,6 Residency and/or board certification are not required though helpful. A pharmacist’s SOP is reviewed and approved by the facility executive committee.⁵ Pharmacists with a SOP undergo professional practice evaluation twice a year. Prescribing controlled substances is permissible in the SOP if approved by the facility and allowed by the state of licensure. According to the US Drug Enforcement Agency (DEA) as of February 10, 2020, 8 states (California, Washington, Idaho, Massachusetts, Montana, New Mexico, North Carolina, and Ohio) allow pharmacists to prescribe controlled substances.⁷

 

The VA developed the Pharmacists Achieve Results with Medications Documentation (PhARMD) tool that allows clinical pharmacists to document specific interventions made during clinical care and is included in their progress note. Data from fiscal year 2017 demonstrates that 136,041 pain management interventions were made by pharmacists across VA. The majority of these interventions were implemented by a CPS working autonomously as an APP.⁸

Several articles discuss the pharmacists role in the opioid crisis, although no outcomes data were provided. Chisholm-Burns and colleagues listed multiple potential ways that pharmacists can intervene, including managing pain in primary care clinic settings by using collaborative drug therapy agreements (CDTAs), using opioid exit plans and discharge planning in collaboration with other health care providers (HCPs), or making recommendations to the prescribers before writing prescriptions.⁹ Compton and colleagues similarly reviewed pharmacist roles in the opioid crisis. However, their focus was on dispensing pharmacists that provided education to patients about storage and disposal of opioids, identified opioid misuse, provided opioid overdose education and naloxone, and checked prescription drug monitoring programs (PDMPs).¹⁰ Missing from these articles was the role of the clinical pharmacist working as an APP delivering direct patient care and prescribing controlled substances.

Hammer and colleagues discussed the role of an oncology CPS with controlled substance prescriptive authority in pain management at an outpatient cancer center in Washington state.¹¹ Under a CDTA, pharmacists could prescribe medications, including controlled substances if they obtain DEA registration. The pharmacist completed a comprehensive in-person assessment. The attending physician conducted a physical examination. Then the pharmacist presented the patient and proposed regimen to the interprofessional team to determine a final plan. Ultimately, the pharmacist wrote any controlled substance prescriptions. The patient followed up every 1 to 4 weeks by telephone with a nurse, and in-person assessments occurred at least every 6 months. No outcomes data were provided.¹¹

Dole and colleagues reviewed the role of a pharmacist who had controlled substance prescriptive authority in a pain management clinic. The pharmacist provider saw up to 18 patients a day and then managed refill requests for 3 hours a day. The main outcome was change in visual analog scale (VAS) pain scores. Findings showed that reductions in VAS pain scores were statistically significant (P < .01). The pharmacist processed about 150 refills with an unclear number of controlled substances requests a day based on a medication-refill protocol. This was felt to improve access to physicians for acute needs, improve consistency in refills, and capture patients in need of follow-up. Additionally, the clinic saved $455,238 after 1 year.¹²

 

Study Aims

A review of the literature indicated sparse data on the impact of a pharmacist on opioid tapering, opioid dose, and opioid risk mitigation when the pharmacist is prescribing controlled substances. The purpose of this retrospective review was to characterize the controlled substance prescribing practices by the pharmacy pain clinic. The aim was to examine the pharmacist impact on morphine milligram equivalent (MME) and compliance with opioid risk mitigation strategies.

Methods

This project was a retrospective, single-center, chart review. The project was reviewed and approved by the University of Missouri-Columbia Institutional Review Board used by the Harry S. Truman Memorial Veterans’ Hospital (HSTMVH) as a quality improvement project. The author applied for controlled substance registration through the DEA and was issued registration April 30, 2018. The State of Ohio Board of Pharmacy was contacted as required by Ohio Administrative Code. The author's updated SOP to allow controlled substance prescribing was approved July 23, 2018. The CPS functions as an APP within an interdisciplinary pain management team that includes physicians, occupational and physical therapists, complementary and integrative health, and a psychologist. The reason for Pharmacy Pain Consult is required and it is primarily submitted through the electronic health record. The consult is reviewed for appropriateness and once approved is scheduled by support staff. Once the patient is stabilized, the patient is discharged back to their primary care provider (PCP) or referring provider for continued care. Patients were considered stabilized when their patient-specific goals were met, which varied from use of the lowest effective opioid dose to taper to discontinuation of opioids with no further medication changes needed. The taper strategy for each patient was individualized. Patients were generally tapered on their existing opioid medication unless they were new to the VA and on nonformulary medications or experiencing a significant adverse reaction. Numerous references are available through VA to assist with opioid tapering.^13,14 The CPS is able to refer patients to other services, including behavioral health for substance use disorder treatment and medication-assisted treatment if concerns were identified.

Initial data were collected from the Veterans Integrated Service Network (VISN) 15 Corporate Data Warehouse by the VISN pharmacy analytics program manager. The original report included patients prescribed a Schedule II to V controlled substance by the author from July 1, 2018 to January 31, 2020. Chart review was conducted on each patient to obtain additional data. At the time of consult and discharge the following data were collected: opioid medication; MME; use of opioid risk mitigation strategies, such as urine drug screens (UDS), informed consent, opioid overdose education and naloxone distribution program (OEND), risk assessment via stratification tool for opioid risk mitigation (STORM), PDMP checks; and nonopioid medication number and classes.

Patients were included in the review if they were prescribed an opioid Schedule II or III controlled substance between July 1, 2018 and January 31, 2020. Patient were excluded if they were prescribed an opioid Schedule II or III controlled substance primarily as coverage for another prescriber. Patients prescribed only pregabalin, tramadol, or a benzodiazepine also were excluded.

The primary endpoint was change in MME from baseline to discharge from clinic. Secondary endpoints included change in opioid risk mitigation strategies and change in opioid medications prescribed from baseline to discharge.

Descriptive statistics were used to analyze parts of the data. A 2-sided t test was used to compare baseline and discharge MME. The Fisher exact test was used to compare nominal data of opioid risk mitigation strategies.

Calculation of MME was performed using the conversion factors provided by the Centers Disease Control and Prevention (CDC) for opioid guideline.¹⁵ For buprenorphine, tapentadol, and levorphanol conversion ratios were obtained from other sources. The conversion ratios used, included 75:1 for oral morphine to transdermal buprenorphine, 1:3.3 for oral morphine to oral tapentadol, and 1:7.5 for oral levorphanol to oral morphine.^16,17 The Revised Standards for Quality Improvement Reporting Excellence (SQUIRE 2.0) was used to write the manuscript.¹⁸

 

Results

Seventy-five patients were included in this review. The average age of patients was 66 years; and 12% were female (n = 9) (Table 2). The largest number of consults came from PCPs (44%, n = 33) and the pain clinic (43%, n = 32). Nearly half (48%) of the consultations were for opioid tapering (n = 36), followed by 37% for opioid optimization or monitoring (n = 28), and 19% for nonopioid optimization (n = 14). The most common primary diagnoses at consultation were for chronic low back pain (56%), chronic neck pain (20%), and osteoarthritis (16%).

The average MME at time of consult was 93 MME compared with 31 MME at discharge which was statisticially significant (P < .01) (Figure 1). The mean percent change in MME was 46%, including methadone and 42% excluding methadone. There was a 26% change in UDS, 28% change in informed consent, 85% change in PDMP, 194% change in naloxone, and 357% change in STORM reviews from baseline to discharge with all demonstrating statistical significance (P < .01) (Figure 2). At discharge, the most common opioid prescribed was morphine SA (short acting) (n = 10, 13%, 44 average MME) and oxycodone/acetaminophen (n = 10, 13%, 28 average MME) (Table 3).

The average number of days from consult to initial visit was 23 days (Table 4). Face-to-face was the primary means of initial visit with 92% (n = 69) of visits, but phone was the primary mode of follow-up with 73% of visits (n = 55). The average number of follow-up visits was 7, representing 176 average days of time in the Pharmacy Pain Clinic. Consultation to the behavioral health performance program was the most common referral (n = 13, 17%).

Five patients were new opioid starts in the Pharmacy Pain Clinic. Two patients were on tramadol at time of consult. Of the 5 new opioid starts, 3 patients received oxycodone/acetaminophen, 1 received buprenorphine patch, and 1 received hydrocodone/acetaminophen. The new opioid start average was 25 MME. All 5 patients had a UDS for opioid risk mitigation, 4 used consent and STORM reviews, and 2 patients had PDMP checks and naloxone.

Discussion

There was a statistically significant decrease of the mean MME between the time of consult and the time of discharge. There also were statistically significant changes in use of opioid risk mitigation strategies. Since methadone has a high MME, the mean reduction of MME was calculated with methadone (46%) and without methadone (42%). These data are consistent with other published studies examining opioid tapers in the VA population. Harden and colleagues calculated a 46% mean reduction in MME over 12 months for 72 veterans from opioid tapers implemented by PCPs, pain service, or pharmacist-run clinics.¹⁹

There is controversy about equianalgesic doses and no established universal equianalgesic conversion calculator or dose. Numerous equianalgesic opioid dose calculators are available, but for this analysis the CDC MME conversion factors were used (available at: https://www.cdc.gov/drugoverdose/pdf/calculating_total_daily_dose-a.pdf). Previous literature compared existing calculators and found significant variances in calculated doses for methadone and fentanyl conversions.²⁰ Additionally, there have been concerns expressed with the safety of the CDC opioid calculator specifically surrounding the conversions for methadone and tapentadol.²¹ In the end, I chose the CDC calculator because it is established, readily available, and consistent.

Pharmacists in pain management can address access issues.^2,3,11,12 The average length of time from consult to initial visit was 23 days. Often patients may have seen a HCP who implemented a change at the time of consult and wanted the patient to be seen 1 month later. Many patients at the HSTMVH live far from the facility, making in-person visits difficult. A majority of the follow-up visits were conducted by telephone. Patients were offered all modalities available for follow-up, including telephone, in-person, or telemedicine, but patients most often picked telephone. Patients averaged 7 follow-up visits before discharge. This number of visits would have taken time from other health care team members who could have been addressing other veterans. Patients were seen in clinic for 176 days on average, which supports and follows recommendations for a slow, incremental taper.

The opioid medications prescribed changed over time in the clinic. Methadone prescriptions dropped from 20 to 6 at consult to discharge, and fentanyl prescriptions fell from 7 to 2, respectively. The CDC guideline suggests use of long-acting products with more predictable pharmacokinetics (eg, morphine SA or oxycodone SA) rather than fentanyl or methadone.¹⁵ Notably, the use of buprenorphine products with FDA approval for pain indications increased from consult to discharge. Many of the patients in this study had pulmonary comorbidities, placing them at higher risk for adverse outcomes. Buprenorphine is a partial μ-opioid receptor agonist with a ceiling on respiratory depression so is potentially less risky in those with pulmonary comorbidities.

The biggest changes in opioid risk mitigation occurred in PDMP, OEND program, and STORM reviews. An 85% increase in PDMP reviews occurred with referral to the clinic. Missouri is the only state without a state-run PDMP. However, the St. Louis County PDMP was developed based on city or county participation and encompasses 85% of the population of Missouri and 94% of HCPs in Missouri as of August 29, 2019.²² Because there is no state-level PDMP, a review of the St. Louis County PDMP was not required during the review period. Nevertheless, the Pharmacy Pain Clinic uses the St. Louis County PDMP at the initial visit and regularly during care. VA policy requires a specific note title be used to document each check of the PDMP.²³

There was a 194% increase in patients receiving naloxone with consultation to the Pharmacy Pain Clinic. Due to low coprescribing of naloxone for patients prescribed chronic opioid therapy, The author led an interdisciplinary team analysis of health care failure mode effects during the study period. This led to a process change with coprescribing of naloxone at refill in the primary care clinic.

The Comprehensive Addiction and Recovery Act of 2016 mandated that the VA review STORM on new start of opioids or patient identified as “very high-opioid prescription risk” category by an interdisciplinary opioid risk review team.²⁴ Thus many of the patients referred to clinic didn’t require STORM reviews since they were not new opioid starts or identified as high risk. However, in the standard review of all new patients to the Pharmacy Pain Clinic, a STORM review is conducted and documented to assess the patient’s level of risk.

Only 5 patients were started on opioid medications during the study period. This is consistent with both CDC and the joint VA/US Department of Defense opioid prescribing guidelines that recommend against initiation of opioids for chronic nonmalignant pain.^13,15 Two of the patients were prescribed tramadol for ineffective pain control at time of consult. Furthermore, 4 of the 5 patients were started on a short-acting opioid, which was supported by guidelines.^13,15 One patient was initiated on buprenorphine patches due to comorbid chronic kidney disease. The VA does not limit the quantity of new opioid prescriptions, although some states and private insurance plans are implementing limitations. Guidelines also recommend against exceeding 90 MME due to risk. The average MME in this project at discharge was 25 MME. Use of opioid risk mitigation for the new opioid starts was reasonable. The reason for the missing PDMP report is unknown based on chart review and atypical according to clinic practice.

Recently, efforts to expand pharmacist training and positions in pain management at VA facilities have been undertaken. In 2016, there were just 11 American Society of Health-System Pharmacists-accredited pharmacy postgraduate year 2 pain and palliative care residency programs, which has expanded to 26 sites in 2020.^2,3,25 In addition, the Clinical Pharmacy Practice Office and the VA Office of Rural Health have helped to hire 33 new pain management pharmacists.³

The role of pharmacists in prescribing controlled substances is limited mainly due to the small number of states that extend this authority.⁷ At the VA, a pharmacist can practice using any state of licensure. Therefore, a pharmacist working at a VA in a state that does not authorize controlled substance prescribing could obtain a license in a state that does permit it. However, the main barrier to obtaining other state licensures is the cost. At the time the author obtained controlled substance prescriptive authority, little direction was available on the process for advanced practice pharmacists at the VA. Since then, guidance has been developed to ease this process. Educational endeavors at VA have been implemented with the intent to increase the number of pharmacists with controlled substance prescriptive authority.

Barriers to pharmacists providing pain care extend beyond limited controlled substance prescriptive authority. Often pharmacists are still viewed in their traditional and operational role.^9,10 Other health care team members and patients may not be aware or familiar with the training, knowledge, and skills of pharmacist's and their suitability as an APP.^26,27 Most states permit pharmacists in establishing CDTA but not all. Additionally, some states recognize pharmacists as HCPs but many more do not. Furthermore, the Social Security Act does not include pharmacists as HCPs. This makes it challenging, though not impossible, for pharmacists to bill for their services.³

 

Strengths and Limitations

There were numerous strengths of the project. First, this addressed an unmet need in the literature with limited data discussing pharmacist prescribing controlled substances for pain management. There was 1 data reviewer who made the data collection process consistent. Since this retrospectively reviewed controlled substance prescribing in clinic, it captured real-world practice compared with that of experimental models. There were also several limitations in the project. The person collecting the data was also the person who conducted the clinic. The study was conducted retrospectively and based on documented information in the medical record. The population reviewed was primarily male and older, which fits the VA patient population but has less generalizability to other patient populations. This project was conducted at a single VA facility so may not be generalizable to other VA sites. It is unknown whether patients were again prescribed opioids if they left the VA for the community or another VA facility. The pain diagnoses or locations of pain were categorized to main groups and reliant on the referring provider. Another major weakness was the lack of comparison of pain scores or validated objective measure of function at baseline and at discharge. This consideration would be important for future work.

Conclusions

Pharmacists functioning as APP are key members of the pain management team. A review of a pharmacy-run pain clinic demonstrated statistically significant reduction in MME and improvement in opioid risk mitigation from consult to discharge. Patients enrolled in the pharmacy-managed clinic also had improvements in adherence to opioid risk mitigation strategies. Future attention should be focused on further expanding training and positions for pharmacists as APP in pain management.

Acknowledgments

The author thanks Chris Sedgwick for his assistance with data capture.