Original Research

With today’s technology, it is easier than ever to access web-based tools that enrich traditional dermatology education. The literature supports the use of these innovative platforms to enhance learning at the student and trainee levels. A controlled study of pediatric residents showed that online modules effectively supplemented clinical experience with atopic dermatitis.¹ In a randomized diagnostic study of medical students, practice with an image-based web application (app) that teaches rapid recognition of melanoma proved more effective than learning a rule-based algorithm.² Given the visual nature of dermatology, pattern recognition is an essential skill that is fostered through experience and is only made more accessible with technology.

With the added benefit of convenience and accessibility, mobile apps can supplement experiential learning. Mirroring the overall growth of mobile apps, the number of available dermatology apps has increased.³ Dermatology mobile apps serve purposes ranging from quick reference tools to comprehensive modules, journals, and question banks. At an academic hospital in Taiwan, both nondermatology and dermatology trainees’ examination performance improved after 3 weeks of using a smartphone-based wallpaper learning module displaying morphologic characteristics of fungi.⁴ With the expansion of virtual microscopy, mobile apps also have been created as a learning tool for dermatopathology, giving trainees the flexibility and autonomy to view slides on their own time.⁵ Nevertheless, the literature on dermatology mobile apps designed for the education of medical students and trainees is limited, demonstrating a need for further investigation.

Prior studies have reviewed dermatology apps for patients and practicing dermatologists.^6-8 Herein, we focus on mobile apps targeting students and residents learning dermatology. General dermatology reference apps and educational aid apps have grown by 33% and 32%, respectively, from 2014 to 2017.³ As with any resource meant to educate future and current medical providers, there must be an objective review process in place to ensure accurate, unbiased, evidence-based teaching.

Well-organized, comprehensive information and a user-friendly interface are additional factors of importance when selecting an educational mobile app. When discussing supplemental resources, accessibility and affordability also are priorities given the high cost of a medical education at baseline. Overall, there is a need for a standardized method to evaluate the key factors of an educational mobile app that make it appropriate for this demographic. We conducted a search of mobile apps relating to dermatology education for students and residents.

Methods

We searched for publicly available mobile apps relating to dermatology education in the App Store (Apple Inc) from September to November 2019 using the search terms dermatology education, dermoscopy education, melanoma education, skin cancer education, psoriasis education, rosacea education, acne education, eczema education, dermal fillers education, and Mohs surgery education. We excluded apps that were not in English, were created for a conference, cost more than $5 to download, or did not include a specific dermatology education section. In this way, we hoped to evaluate apps that were relevant, accessible, and affordable.

We modeled our study after a review of patient education apps performed by Masud et al⁶ and utilized their quantified grading rubric (scale of 1 to 4). We found their established criteria—educational objectives, content, accuracy, design, and conflict of interest—to be equally applicable for evaluating apps designed for professional education.⁶ Each app earned a minimum of 1 point and a maximum of 4 points per criterion. One point was given if the app did not fulfill the criterion, 2 points for minimally fulfilling the criterion, 3 points for mostly fulfilling the criterion, and 4 points if the criterion was completely fulfilled. Two medical students (E.H. and N.C.)—one at the preclinical stage and the other at the clinical stage of medical education—reviewed the apps using the given rubric, then discussed and resolved any discrepancies in points assigned. A dermatology resident (M.A.) independently reviewed the apps using the given rubric.

The mean of the student score and the resident score was calculated for each category. The sum of the averages for each category was considered the final score for an app, determining its overall quality. Apps with a total score of 5 to 10 were considered poor and inadequate for education. A total score of 10.5 to 15 indicated that an app was somewhat adequate (ie, useful for education in some aspects but falling short in others). Apps that were considered adequate for education, across all or most criteria, received a total score ranging from 15.5 to 20.

Results

Our search generated 130 apps. After applying exclusion criteria, 42 apps were eligible for review. At the time of publication, 36 of these apps were still available. The possible range of scores based on the rubric was 5 to 20. The actual range of scores was 7 to 20. Of the 36 apps, 2 (5.6%) were poor, 16 (44.4%) were somewhat adequate, and 18 (50%) were adequate. Formats included primary resources, such as clinical decision support tools, journals, references, and a podcast (Table 1). Additionally, interactive learning tools included games, learning modules, and apps for self-evaluation (Table 2). Thirty apps covered general dermatology; others focused on skin cancer (n=5) and cosmetic dermatology (n=1). Regarding cost, 29 apps were free to download, whereas 7 charged a fee (mean price, $2.56).

 

 

Comment

In addition to the convenience of having an educational tool in their white-coat pocket, learners of dermatology have been shown to benefit from supplementing their curriculum with mobile apps, which sets the stage for formal integration of mobile apps into dermatology teaching in the future.⁸ Prior to widespread adoption, mobile apps must be evaluated for content and utility, starting with an objective rubric.

Without official scientific standards in place, it was unsurprising that only half of the dermatology education applications were classified as adequate in this study. Among the types of apps offered—clinical decision support tools, journals, references, podcast, games, learning modules, and self-evaluation—certain categories scored higher than others. App formats with the highest average score (16.5 out of 20) were journals and podcast.

One barrier to utilization of these apps was that a subscription to the journals and podcast was required to obtain access to all available content. Students and trainees can seek out library resources at their academic institutions to take advantage of journal subscriptions available to them at no additional cost. Dermatology residents can take advantage of their complimentary membership in the American Academy of Dermatology for a free subscription to AAD Dialogues in Dermatology (otherwise $179 annually for nonresident members and $320 annually for nonmembers).

On the other hand, learning module was the lowest-rated format (average score, 11.3 out of 20), with only Medical Student: Dermatology qualifying as adequate (total score, 16). This finding is worrisome given that students and residents might look to learning modules for quick targeted lessons on specific topics.

The lowest-scoring app, a clinical decision support tool called Naturelize, received a total score of 7. Although it listed the indications and contraindications for dermal filler types to be used in different locations on the face, there was a clear conflict of interest, oversimplified design, and little evidence-based education, mirroring the current state of cosmetic dermatology training in residency, in which trainees think they are inadequately prepared for aesthetic procedures and comparative effectiveness research is lacking.^9-11

At the opposite end of the spectrum, MyDermPath+ was a reference app with a total score of 20. The app cited credible authors with a medical degree (MD) and had an easy-to-use, well-designed interface, including a reference guide, differential builder, and quiz for a range of topics within dermatology. As a free download without in-app purchases or advertisements, there was no evidence of conflict of interest. The position of a dermatopathology app as the top dermatology education mobile app might reflect an increased emphasis on dermatopathology education in residency as well as a transition to digitization of slides.⁵

The second-highest scoring apps (total score of 19 points) were Dermatology Database and VisualDx. Both were references covering a wide range of dermatology topics. Dermatology Database was a comprehensive search tool for diseases, drugs, procedures, and terms that was simple and entirely free to use but did not cite references. VisualDx, as its name suggests, offered quality clinical images, complete guides with references, and a unique differential builder. An annual subscription is $399.99, but the process to gain free access through a participating academic institution was simple.

Games were a unique mobile app format; however, 2 of 3 games scored in the somewhat adequate range. The game DiagnosUs, which tested users’ ability to differentiate skin cancer and psoriasis from dermatitis on clinical images, would benefit from more comprehensive content as well as professional verification of true diagnoses, which earned the app 2 points in both the content and accuracy categories. The Unusual Suspects tested the ABCDE algorithm in a short learning module, followed by a simple game that involved identification of melanoma in a timed setting. Although the design was novel and interactive, the game was limited to the same 5 melanoma tumors overlaid on pictures of normal skin. The narrow scope earned 1 point for content, the redundancy in the game earned 3 points for design, and the lack of real clinical images earned 2 points for educational objectives. Although game-format mobile apps have the capability to challenge the user’s knowledge with a built-in feedback or reward system, improvements should be made to ensure that apps are equally educational as they are engaging.

AAD Dialogues in Dermatology was the only app in the form of a podcast and provided expert interviews along with disclosures, transcripts, commentary, and references. More than half the content in the app could not be accessed without a subscription, earning 2.5 points in the conflict of interest category. Additionally, several flaws resulted in a design score of 2.5, including inconsistent availability of transcripts, poor quality of sound on some episodes, difficulty distinguishing new episodes from those already played, and a glitch that removed the episode duration. Still, the app was a valuable and comprehensive resource, with clear objectives and cited references. With improvements in content, affordability, and user experience, apps in unique formats such as games and podcasts might appeal to kinesthetic and auditory learners.

An important factor to consider when discussing mobile apps for students and residents is cost. With rising prices of board examinations and preparation materials, supplementary study tools should not come with an exorbitant price tag. Therefore, we limited our evaluation to apps that were free or cost less than $5 to download. Even so, subscriptions and other in-app purchases were an obstacle in one-third of apps, ranging from $4.99 to unlock additional content in Rash Decisions to $69.99 to access most topics in Fitzpatrick’s Color Atlas. The highest-rated app in our study, MyDermPath+, historically cost $19.99 to download but became free with a grant from the Sulzberger Foundation.¹² An initial investment to develop quality apps for the purpose of dermatology education might pay off in the end.

To evaluate the apps from the perspective of the target demographic of this study, 2 medical students—one in the preclinical stage and the other in the clinical stage of medical education—and a dermatology resident graded the apps. Certain limitations exist in this type of study, including differing learning styles, which might influence the types of apps that evaluators found most impactful to their education. Interestingly, some apps earned a higher resident score than student score. In particular, RightSite (a reference that helps with anatomically correct labeling) and Mohs Surgery Appropriate Use Criteria (a clinical decision support tool to determine whether to perform Mohs surgery) each had a 3-point discrepancy (data not shown). A resident might benefit from these practical apps in day-to-day practice, but a student would be less likely to find them useful as a learning tool.

Still, by defining adequate teaching value using specific categories of educational objectives, content, accuracy, design, and conflict of interest, we attempted to minimize the effect of personal preference on the grading process. Although we acknowledge a degree of subjectivity, we found that utilizing a previously published rubric with defined criteria was crucial in remaining unbiased.

Conclusion

Further studies should evaluate additional apps available on Apple’s iPad (tablet), as well as those on other operating systems, including Google’s Android. To ensure the existence of mobile apps as adequate education tools, they should be peer reviewed prior to publication or before widespread use by future and current providers at the minimum. To maximize free access to highly valuable resources available in the palm of their hand, students and trainees should contact the library at their academic institution.

With today’s technology, it is easier than ever to access web-based tools that enrich traditional dermatology education. The literature supports the use of these innovative platforms to enhance learning at the student and trainee levels. A controlled study of pediatric residents showed that online modules effectively supplemented clinical experience with atopic dermatitis.¹ In a randomized diagnostic study of medical students, practice with an image-based web application (app) that teaches rapid recognition of melanoma proved more effective than learning a rule-based algorithm.² Given the visual nature of dermatology, pattern recognition is an essential skill that is fostered through experience and is only made more accessible with technology.

With the added benefit of convenience and accessibility, mobile apps can supplement experiential learning. Mirroring the overall growth of mobile apps, the number of available dermatology apps has increased.³ Dermatology mobile apps serve purposes ranging from quick reference tools to comprehensive modules, journals, and question banks. At an academic hospital in Taiwan, both nondermatology and dermatology trainees’ examination performance improved after 3 weeks of using a smartphone-based wallpaper learning module displaying morphologic characteristics of fungi.⁴ With the expansion of virtual microscopy, mobile apps also have been created as a learning tool for dermatopathology, giving trainees the flexibility and autonomy to view slides on their own time.⁵ Nevertheless, the literature on dermatology mobile apps designed for the education of medical students and trainees is limited, demonstrating a need for further investigation.

Prior studies have reviewed dermatology apps for patients and practicing dermatologists.^6-8 Herein, we focus on mobile apps targeting students and residents learning dermatology. General dermatology reference apps and educational aid apps have grown by 33% and 32%, respectively, from 2014 to 2017.³ As with any resource meant to educate future and current medical providers, there must be an objective review process in place to ensure accurate, unbiased, evidence-based teaching.

Well-organized, comprehensive information and a user-friendly interface are additional factors of importance when selecting an educational mobile app. When discussing supplemental resources, accessibility and affordability also are priorities given the high cost of a medical education at baseline. Overall, there is a need for a standardized method to evaluate the key factors of an educational mobile app that make it appropriate for this demographic. We conducted a search of mobile apps relating to dermatology education for students and residents.

Methods

We searched for publicly available mobile apps relating to dermatology education in the App Store (Apple Inc) from September to November 2019 using the search terms dermatology education, dermoscopy education, melanoma education, skin cancer education, psoriasis education, rosacea education, acne education, eczema education, dermal fillers education, and Mohs surgery education. We excluded apps that were not in English, were created for a conference, cost more than $5 to download, or did not include a specific dermatology education section. In this way, we hoped to evaluate apps that were relevant, accessible, and affordable.

We modeled our study after a review of patient education apps performed by Masud et al⁶ and utilized their quantified grading rubric (scale of 1 to 4). We found their established criteria—educational objectives, content, accuracy, design, and conflict of interest—to be equally applicable for evaluating apps designed for professional education.⁶ Each app earned a minimum of 1 point and a maximum of 4 points per criterion. One point was given if the app did not fulfill the criterion, 2 points for minimally fulfilling the criterion, 3 points for mostly fulfilling the criterion, and 4 points if the criterion was completely fulfilled. Two medical students (E.H. and N.C.)—one at the preclinical stage and the other at the clinical stage of medical education—reviewed the apps using the given rubric, then discussed and resolved any discrepancies in points assigned. A dermatology resident (M.A.) independently reviewed the apps using the given rubric.

The mean of the student score and the resident score was calculated for each category. The sum of the averages for each category was considered the final score for an app, determining its overall quality. Apps with a total score of 5 to 10 were considered poor and inadequate for education. A total score of 10.5 to 15 indicated that an app was somewhat adequate (ie, useful for education in some aspects but falling short in others). Apps that were considered adequate for education, across all or most criteria, received a total score ranging from 15.5 to 20.

Results

Our search generated 130 apps. After applying exclusion criteria, 42 apps were eligible for review. At the time of publication, 36 of these apps were still available. The possible range of scores based on the rubric was 5 to 20. The actual range of scores was 7 to 20. Of the 36 apps, 2 (5.6%) were poor, 16 (44.4%) were somewhat adequate, and 18 (50%) were adequate. Formats included primary resources, such as clinical decision support tools, journals, references, and a podcast (Table 1). Additionally, interactive learning tools included games, learning modules, and apps for self-evaluation (Table 2). Thirty apps covered general dermatology; others focused on skin cancer (n=5) and cosmetic dermatology (n=1). Regarding cost, 29 apps were free to download, whereas 7 charged a fee (mean price, $2.56).

 

 

Comment

In addition to the convenience of having an educational tool in their white-coat pocket, learners of dermatology have been shown to benefit from supplementing their curriculum with mobile apps, which sets the stage for formal integration of mobile apps into dermatology teaching in the future.⁸ Prior to widespread adoption, mobile apps must be evaluated for content and utility, starting with an objective rubric.

Without official scientific standards in place, it was unsurprising that only half of the dermatology education applications were classified as adequate in this study. Among the types of apps offered—clinical decision support tools, journals, references, podcast, games, learning modules, and self-evaluation—certain categories scored higher than others. App formats with the highest average score (16.5 out of 20) were journals and podcast.

One barrier to utilization of these apps was that a subscription to the journals and podcast was required to obtain access to all available content. Students and trainees can seek out library resources at their academic institutions to take advantage of journal subscriptions available to them at no additional cost. Dermatology residents can take advantage of their complimentary membership in the American Academy of Dermatology for a free subscription to AAD Dialogues in Dermatology (otherwise $179 annually for nonresident members and $320 annually for nonmembers).

On the other hand, learning module was the lowest-rated format (average score, 11.3 out of 20), with only Medical Student: Dermatology qualifying as adequate (total score, 16). This finding is worrisome given that students and residents might look to learning modules for quick targeted lessons on specific topics.

The lowest-scoring app, a clinical decision support tool called Naturelize, received a total score of 7. Although it listed the indications and contraindications for dermal filler types to be used in different locations on the face, there was a clear conflict of interest, oversimplified design, and little evidence-based education, mirroring the current state of cosmetic dermatology training in residency, in which trainees think they are inadequately prepared for aesthetic procedures and comparative effectiveness research is lacking.^9-11

At the opposite end of the spectrum, MyDermPath+ was a reference app with a total score of 20. The app cited credible authors with a medical degree (MD) and had an easy-to-use, well-designed interface, including a reference guide, differential builder, and quiz for a range of topics within dermatology. As a free download without in-app purchases or advertisements, there was no evidence of conflict of interest. The position of a dermatopathology app as the top dermatology education mobile app might reflect an increased emphasis on dermatopathology education in residency as well as a transition to digitization of slides.⁵

The second-highest scoring apps (total score of 19 points) were Dermatology Database and VisualDx. Both were references covering a wide range of dermatology topics. Dermatology Database was a comprehensive search tool for diseases, drugs, procedures, and terms that was simple and entirely free to use but did not cite references. VisualDx, as its name suggests, offered quality clinical images, complete guides with references, and a unique differential builder. An annual subscription is $399.99, but the process to gain free access through a participating academic institution was simple.

Games were a unique mobile app format; however, 2 of 3 games scored in the somewhat adequate range. The game DiagnosUs, which tested users’ ability to differentiate skin cancer and psoriasis from dermatitis on clinical images, would benefit from more comprehensive content as well as professional verification of true diagnoses, which earned the app 2 points in both the content and accuracy categories. The Unusual Suspects tested the ABCDE algorithm in a short learning module, followed by a simple game that involved identification of melanoma in a timed setting. Although the design was novel and interactive, the game was limited to the same 5 melanoma tumors overlaid on pictures of normal skin. The narrow scope earned 1 point for content, the redundancy in the game earned 3 points for design, and the lack of real clinical images earned 2 points for educational objectives. Although game-format mobile apps have the capability to challenge the user’s knowledge with a built-in feedback or reward system, improvements should be made to ensure that apps are equally educational as they are engaging.

AAD Dialogues in Dermatology was the only app in the form of a podcast and provided expert interviews along with disclosures, transcripts, commentary, and references. More than half the content in the app could not be accessed without a subscription, earning 2.5 points in the conflict of interest category. Additionally, several flaws resulted in a design score of 2.5, including inconsistent availability of transcripts, poor quality of sound on some episodes, difficulty distinguishing new episodes from those already played, and a glitch that removed the episode duration. Still, the app was a valuable and comprehensive resource, with clear objectives and cited references. With improvements in content, affordability, and user experience, apps in unique formats such as games and podcasts might appeal to kinesthetic and auditory learners.

An important factor to consider when discussing mobile apps for students and residents is cost. With rising prices of board examinations and preparation materials, supplementary study tools should not come with an exorbitant price tag. Therefore, we limited our evaluation to apps that were free or cost less than $5 to download. Even so, subscriptions and other in-app purchases were an obstacle in one-third of apps, ranging from $4.99 to unlock additional content in Rash Decisions to $69.99 to access most topics in Fitzpatrick’s Color Atlas. The highest-rated app in our study, MyDermPath+, historically cost $19.99 to download but became free with a grant from the Sulzberger Foundation.¹² An initial investment to develop quality apps for the purpose of dermatology education might pay off in the end.

To evaluate the apps from the perspective of the target demographic of this study, 2 medical students—one in the preclinical stage and the other in the clinical stage of medical education—and a dermatology resident graded the apps. Certain limitations exist in this type of study, including differing learning styles, which might influence the types of apps that evaluators found most impactful to their education. Interestingly, some apps earned a higher resident score than student score. In particular, RightSite (a reference that helps with anatomically correct labeling) and Mohs Surgery Appropriate Use Criteria (a clinical decision support tool to determine whether to perform Mohs surgery) each had a 3-point discrepancy (data not shown). A resident might benefit from these practical apps in day-to-day practice, but a student would be less likely to find them useful as a learning tool.

Still, by defining adequate teaching value using specific categories of educational objectives, content, accuracy, design, and conflict of interest, we attempted to minimize the effect of personal preference on the grading process. Although we acknowledge a degree of subjectivity, we found that utilizing a previously published rubric with defined criteria was crucial in remaining unbiased.

Conclusion

Further studies should evaluate additional apps available on Apple’s iPad (tablet), as well as those on other operating systems, including Google’s Android. To ensure the existence of mobile apps as adequate education tools, they should be peer reviewed prior to publication or before widespread use by future and current providers at the minimum. To maximize free access to highly valuable resources available in the palm of their hand, students and trainees should contact the library at their academic institution.

With today’s technology, it is easier than ever to access web-based tools that enrich traditional dermatology education. The literature supports the use of these innovative platforms to enhance learning at the student and trainee levels. A controlled study of pediatric residents showed that online modules effectively supplemented clinical experience with atopic dermatitis.¹ In a randomized diagnostic study of medical students, practice with an image-based web application (app) that teaches rapid recognition of melanoma proved more effective than learning a rule-based algorithm.² Given the visual nature of dermatology, pattern recognition is an essential skill that is fostered through experience and is only made more accessible with technology.

With the added benefit of convenience and accessibility, mobile apps can supplement experiential learning. Mirroring the overall growth of mobile apps, the number of available dermatology apps has increased.³ Dermatology mobile apps serve purposes ranging from quick reference tools to comprehensive modules, journals, and question banks. At an academic hospital in Taiwan, both nondermatology and dermatology trainees’ examination performance improved after 3 weeks of using a smartphone-based wallpaper learning module displaying morphologic characteristics of fungi.⁴ With the expansion of virtual microscopy, mobile apps also have been created as a learning tool for dermatopathology, giving trainees the flexibility and autonomy to view slides on their own time.⁵ Nevertheless, the literature on dermatology mobile apps designed for the education of medical students and trainees is limited, demonstrating a need for further investigation.

Prior studies have reviewed dermatology apps for patients and practicing dermatologists.^6-8 Herein, we focus on mobile apps targeting students and residents learning dermatology. General dermatology reference apps and educational aid apps have grown by 33% and 32%, respectively, from 2014 to 2017.³ As with any resource meant to educate future and current medical providers, there must be an objective review process in place to ensure accurate, unbiased, evidence-based teaching.

Well-organized, comprehensive information and a user-friendly interface are additional factors of importance when selecting an educational mobile app. When discussing supplemental resources, accessibility and affordability also are priorities given the high cost of a medical education at baseline. Overall, there is a need for a standardized method to evaluate the key factors of an educational mobile app that make it appropriate for this demographic. We conducted a search of mobile apps relating to dermatology education for students and residents.

Methods

We searched for publicly available mobile apps relating to dermatology education in the App Store (Apple Inc) from September to November 2019 using the search terms dermatology education, dermoscopy education, melanoma education, skin cancer education, psoriasis education, rosacea education, acne education, eczema education, dermal fillers education, and Mohs surgery education. We excluded apps that were not in English, were created for a conference, cost more than $5 to download, or did not include a specific dermatology education section. In this way, we hoped to evaluate apps that were relevant, accessible, and affordable.

We modeled our study after a review of patient education apps performed by Masud et al⁶ and utilized their quantified grading rubric (scale of 1 to 4). We found their established criteria—educational objectives, content, accuracy, design, and conflict of interest—to be equally applicable for evaluating apps designed for professional education.⁶ Each app earned a minimum of 1 point and a maximum of 4 points per criterion. One point was given if the app did not fulfill the criterion, 2 points for minimally fulfilling the criterion, 3 points for mostly fulfilling the criterion, and 4 points if the criterion was completely fulfilled. Two medical students (E.H. and N.C.)—one at the preclinical stage and the other at the clinical stage of medical education—reviewed the apps using the given rubric, then discussed and resolved any discrepancies in points assigned. A dermatology resident (M.A.) independently reviewed the apps using the given rubric.

The mean of the student score and the resident score was calculated for each category. The sum of the averages for each category was considered the final score for an app, determining its overall quality. Apps with a total score of 5 to 10 were considered poor and inadequate for education. A total score of 10.5 to 15 indicated that an app was somewhat adequate (ie, useful for education in some aspects but falling short in others). Apps that were considered adequate for education, across all or most criteria, received a total score ranging from 15.5 to 20.

Results

Our search generated 130 apps. After applying exclusion criteria, 42 apps were eligible for review. At the time of publication, 36 of these apps were still available. The possible range of scores based on the rubric was 5 to 20. The actual range of scores was 7 to 20. Of the 36 apps, 2 (5.6%) were poor, 16 (44.4%) were somewhat adequate, and 18 (50%) were adequate. Formats included primary resources, such as clinical decision support tools, journals, references, and a podcast (Table 1). Additionally, interactive learning tools included games, learning modules, and apps for self-evaluation (Table 2). Thirty apps covered general dermatology; others focused on skin cancer (n=5) and cosmetic dermatology (n=1). Regarding cost, 29 apps were free to download, whereas 7 charged a fee (mean price, $2.56).

 

 

Comment

In addition to the convenience of having an educational tool in their white-coat pocket, learners of dermatology have been shown to benefit from supplementing their curriculum with mobile apps, which sets the stage for formal integration of mobile apps into dermatology teaching in the future.⁸ Prior to widespread adoption, mobile apps must be evaluated for content and utility, starting with an objective rubric.

Without official scientific standards in place, it was unsurprising that only half of the dermatology education applications were classified as adequate in this study. Among the types of apps offered—clinical decision support tools, journals, references, podcast, games, learning modules, and self-evaluation—certain categories scored higher than others. App formats with the highest average score (16.5 out of 20) were journals and podcast.

One barrier to utilization of these apps was that a subscription to the journals and podcast was required to obtain access to all available content. Students and trainees can seek out library resources at their academic institutions to take advantage of journal subscriptions available to them at no additional cost. Dermatology residents can take advantage of their complimentary membership in the American Academy of Dermatology for a free subscription to AAD Dialogues in Dermatology (otherwise $179 annually for nonresident members and $320 annually for nonmembers).

On the other hand, learning module was the lowest-rated format (average score, 11.3 out of 20), with only Medical Student: Dermatology qualifying as adequate (total score, 16). This finding is worrisome given that students and residents might look to learning modules for quick targeted lessons on specific topics.

The lowest-scoring app, a clinical decision support tool called Naturelize, received a total score of 7. Although it listed the indications and contraindications for dermal filler types to be used in different locations on the face, there was a clear conflict of interest, oversimplified design, and little evidence-based education, mirroring the current state of cosmetic dermatology training in residency, in which trainees think they are inadequately prepared for aesthetic procedures and comparative effectiveness research is lacking.^9-11

At the opposite end of the spectrum, MyDermPath+ was a reference app with a total score of 20. The app cited credible authors with a medical degree (MD) and had an easy-to-use, well-designed interface, including a reference guide, differential builder, and quiz for a range of topics within dermatology. As a free download without in-app purchases or advertisements, there was no evidence of conflict of interest. The position of a dermatopathology app as the top dermatology education mobile app might reflect an increased emphasis on dermatopathology education in residency as well as a transition to digitization of slides.⁵

The second-highest scoring apps (total score of 19 points) were Dermatology Database and VisualDx. Both were references covering a wide range of dermatology topics. Dermatology Database was a comprehensive search tool for diseases, drugs, procedures, and terms that was simple and entirely free to use but did not cite references. VisualDx, as its name suggests, offered quality clinical images, complete guides with references, and a unique differential builder. An annual subscription is $399.99, but the process to gain free access through a participating academic institution was simple.

Games were a unique mobile app format; however, 2 of 3 games scored in the somewhat adequate range. The game DiagnosUs, which tested users’ ability to differentiate skin cancer and psoriasis from dermatitis on clinical images, would benefit from more comprehensive content as well as professional verification of true diagnoses, which earned the app 2 points in both the content and accuracy categories. The Unusual Suspects tested the ABCDE algorithm in a short learning module, followed by a simple game that involved identification of melanoma in a timed setting. Although the design was novel and interactive, the game was limited to the same 5 melanoma tumors overlaid on pictures of normal skin. The narrow scope earned 1 point for content, the redundancy in the game earned 3 points for design, and the lack of real clinical images earned 2 points for educational objectives. Although game-format mobile apps have the capability to challenge the user’s knowledge with a built-in feedback or reward system, improvements should be made to ensure that apps are equally educational as they are engaging.

AAD Dialogues in Dermatology was the only app in the form of a podcast and provided expert interviews along with disclosures, transcripts, commentary, and references. More than half the content in the app could not be accessed without a subscription, earning 2.5 points in the conflict of interest category. Additionally, several flaws resulted in a design score of 2.5, including inconsistent availability of transcripts, poor quality of sound on some episodes, difficulty distinguishing new episodes from those already played, and a glitch that removed the episode duration. Still, the app was a valuable and comprehensive resource, with clear objectives and cited references. With improvements in content, affordability, and user experience, apps in unique formats such as games and podcasts might appeal to kinesthetic and auditory learners.

An important factor to consider when discussing mobile apps for students and residents is cost. With rising prices of board examinations and preparation materials, supplementary study tools should not come with an exorbitant price tag. Therefore, we limited our evaluation to apps that were free or cost less than $5 to download. Even so, subscriptions and other in-app purchases were an obstacle in one-third of apps, ranging from $4.99 to unlock additional content in Rash Decisions to $69.99 to access most topics in Fitzpatrick’s Color Atlas. The highest-rated app in our study, MyDermPath+, historically cost $19.99 to download but became free with a grant from the Sulzberger Foundation.¹² An initial investment to develop quality apps for the purpose of dermatology education might pay off in the end.

To evaluate the apps from the perspective of the target demographic of this study, 2 medical students—one in the preclinical stage and the other in the clinical stage of medical education—and a dermatology resident graded the apps. Certain limitations exist in this type of study, including differing learning styles, which might influence the types of apps that evaluators found most impactful to their education. Interestingly, some apps earned a higher resident score than student score. In particular, RightSite (a reference that helps with anatomically correct labeling) and Mohs Surgery Appropriate Use Criteria (a clinical decision support tool to determine whether to perform Mohs surgery) each had a 3-point discrepancy (data not shown). A resident might benefit from these practical apps in day-to-day practice, but a student would be less likely to find them useful as a learning tool.

Still, by defining adequate teaching value using specific categories of educational objectives, content, accuracy, design, and conflict of interest, we attempted to minimize the effect of personal preference on the grading process. Although we acknowledge a degree of subjectivity, we found that utilizing a previously published rubric with defined criteria was crucial in remaining unbiased.

Conclusion

Further studies should evaluate additional apps available on Apple’s iPad (tablet), as well as those on other operating systems, including Google’s Android. To ensure the existence of mobile apps as adequate education tools, they should be peer reviewed prior to publication or before widespread use by future and current providers at the minimum. To maximize free access to highly valuable resources available in the palm of their hand, students and trainees should contact the library at their academic institution.

Basal cell carcinoma (BCC) is the most common type of skin cancer frequently encountered in both dermatology and primary care settings.¹ When biopsies of these neoplasms are performed to confirm the diagnosis, pathology reports may indicate positive or negative margin status. No guidelines exist for reporting biopsy margin status for BCC, resulting in varied reporting practices among dermatopathologists. Furthermore, the terminology used to describe margin status can be ambiguous and differs among pathologists; language such as “approaches the margin” or “margins appear free” may be used, with nonuniform interpretation between pathologists and providers, leading to variability in patient management.²

When interpreting a negative margin status on a pathology report, one must question if the BCC extends beyond the margin in unexamined sections of the specimen, which could be the result of an irregular tumor growth pattern or tissue processing. It has been estimated that less than 2% of the peripheral surgical margin is ultimately examined when serial cross-sections are prepared histologically (the bread loaf technique). However, this estimation would depend on several variables, including the number and thickness of sections and the amount of tissue discarded during processing.³ Importantly, reports of a false-negative margin could lead both the clinician and patient to believe that the neoplasm has been completely removed, which could have serious consequences.

Our study sought to determine the reliability of negative biopsy margin status for BCC. We examined BCC biopsy specimens initially determined to have uninvolved margins on routine tissue processing and determined the proportion with truly negative margins after complete tissue block sectioning of the initial biopsy specimen. We felt this technique was a more accurate measurement of true margin status than examination of a re-excision specimen. We also identified any factors that were predictive of positive true margins.

Methods

We conducted a retrospective study evaluating tissue samples collected at Geisinger Health System (Danville, Pennsylvania) from January to December 2016. Specimens were queried via the electronic database system at our institution (CoPath). We included BCC biopsy specimens with negative histologic margins on initial assessment that subsequently had block exhaust levels routinely ordered. These levels are cut every 100 to 150 µm, generating approximately 8 glass slides. We excluded all tumors that did not fit these criteria as well as those in patients younger than 18 years. Data collection was performed utilizing specimen pathology reports in addition to the note from the corresponding clinician office visit from the institution’s electronic medical record (Epic). Appropriate statistical calculations were performed. This study was approved by an institutional review board at our institution, which is required for all research involving human participants. This served to ensure the proper review and storage of patients’ protected health information.

 

Results

The search yielded a total of 122 specimens from 104 patients after appropriate exclusions. We examined a total of 122 BCC biopsy specimens with negative initial margins: 121 (99.2%) shave biopsies and 1 (0.8%) punch biopsy. Of 122 specimens with negative initial margins, 53 (43.4%) were found to have a truly positive margin based on the presence of either tumor or stroma at the lateral or deep tissue edge after complete tissue block sectioning. Sixty-nine (56.6%) specimens had clear margins and were categorized as truly negative after complete tissue block sectioning. Specimens with positive and negative final margin status did not differ significantly with respect to patient age; gender; biopsy technique; number of gross specimen sections; or tumor characteristics, including location, size, and subtype (Table)(P>.05).

We also examined the type of treatment performed, which varied and included curettage, electrodesiccation and curettage, excision, and Mohs micrographic surgery. Clinicians, who were not made aware of the exhaust level protocol, chose not to pursue further treatment in 6 (4.9%) of the cases because of negative biopsy margins. Four (66.7%) of the 6 providers were physicians, and 2 (33.3%) were advanced practitioners. All of the providers practiced within the Department of Dermatology.

Comment

Our findings support prior smaller studies investigating this topic. A prospective study by Schnebelen et al⁴ examined 27 BCC biopsy specimens and found that 8 (30%) were erroneously classified as negative on routine examination. This study similarly determined true margin status by assessing the margins at complete tissue block exhaustion.⁴ Willardson et al⁵ also demonstrated the poor predictive value of margin status based on the presence of residual BCC in subsequent excisions. They found that 34 (24%) of 143 cases with negative biopsy margins contained residual tumor in the corresponding excision.⁵

Our study revealed that almost half of BCC biopsy specimens that had negative histologic margins with routine sectioning had truly positive margins on complete block exhaustion. This finding was independent of multiple factors, including tumor subtype, indicating that even nonaggressive tumors are prone to false-negative margin reports. We also found that reports of negative margins persuaded some clinicians to forgo definitive treatment. This study serves to remind clinicians of the limitations of margin assessment and provides impetus for dermatopathologists to consider modifying how margin status is reported.

Limitations of this study include a small number of cases and limited generalizability. Institutions that routinely examine more levels of each biopsy specimen may be less likely to erroneously categorize a positive margin as negative. Furthermore, despite exhausting the tissue block, we still may have underestimated the number of cases with truly positive margins, as this method inherently does not allow for complete margin examination.

Acknowledgments
We thank the Geisinger Department of Dermatopathology and the Geisinger Biostatistics & Research Data Core (Danville, Pennsylvania) for their assistance with our project.

Basal cell carcinoma (BCC) is the most common type of skin cancer frequently encountered in both dermatology and primary care settings.¹ When biopsies of these neoplasms are performed to confirm the diagnosis, pathology reports may indicate positive or negative margin status. No guidelines exist for reporting biopsy margin status for BCC, resulting in varied reporting practices among dermatopathologists. Furthermore, the terminology used to describe margin status can be ambiguous and differs among pathologists; language such as “approaches the margin” or “margins appear free” may be used, with nonuniform interpretation between pathologists and providers, leading to variability in patient management.²

When interpreting a negative margin status on a pathology report, one must question if the BCC extends beyond the margin in unexamined sections of the specimen, which could be the result of an irregular tumor growth pattern or tissue processing. It has been estimated that less than 2% of the peripheral surgical margin is ultimately examined when serial cross-sections are prepared histologically (the bread loaf technique). However, this estimation would depend on several variables, including the number and thickness of sections and the amount of tissue discarded during processing.³ Importantly, reports of a false-negative margin could lead both the clinician and patient to believe that the neoplasm has been completely removed, which could have serious consequences.

Our study sought to determine the reliability of negative biopsy margin status for BCC. We examined BCC biopsy specimens initially determined to have uninvolved margins on routine tissue processing and determined the proportion with truly negative margins after complete tissue block sectioning of the initial biopsy specimen. We felt this technique was a more accurate measurement of true margin status than examination of a re-excision specimen. We also identified any factors that were predictive of positive true margins.

Methods

We conducted a retrospective study evaluating tissue samples collected at Geisinger Health System (Danville, Pennsylvania) from January to December 2016. Specimens were queried via the electronic database system at our institution (CoPath). We included BCC biopsy specimens with negative histologic margins on initial assessment that subsequently had block exhaust levels routinely ordered. These levels are cut every 100 to 150 µm, generating approximately 8 glass slides. We excluded all tumors that did not fit these criteria as well as those in patients younger than 18 years. Data collection was performed utilizing specimen pathology reports in addition to the note from the corresponding clinician office visit from the institution’s electronic medical record (Epic). Appropriate statistical calculations were performed. This study was approved by an institutional review board at our institution, which is required for all research involving human participants. This served to ensure the proper review and storage of patients’ protected health information.

 

Results

The search yielded a total of 122 specimens from 104 patients after appropriate exclusions. We examined a total of 122 BCC biopsy specimens with negative initial margins: 121 (99.2%) shave biopsies and 1 (0.8%) punch biopsy. Of 122 specimens with negative initial margins, 53 (43.4%) were found to have a truly positive margin based on the presence of either tumor or stroma at the lateral or deep tissue edge after complete tissue block sectioning. Sixty-nine (56.6%) specimens had clear margins and were categorized as truly negative after complete tissue block sectioning. Specimens with positive and negative final margin status did not differ significantly with respect to patient age; gender; biopsy technique; number of gross specimen sections; or tumor characteristics, including location, size, and subtype (Table)(P>.05).

We also examined the type of treatment performed, which varied and included curettage, electrodesiccation and curettage, excision, and Mohs micrographic surgery. Clinicians, who were not made aware of the exhaust level protocol, chose not to pursue further treatment in 6 (4.9%) of the cases because of negative biopsy margins. Four (66.7%) of the 6 providers were physicians, and 2 (33.3%) were advanced practitioners. All of the providers practiced within the Department of Dermatology.

Comment

Our findings support prior smaller studies investigating this topic. A prospective study by Schnebelen et al⁴ examined 27 BCC biopsy specimens and found that 8 (30%) were erroneously classified as negative on routine examination. This study similarly determined true margin status by assessing the margins at complete tissue block exhaustion.⁴ Willardson et al⁵ also demonstrated the poor predictive value of margin status based on the presence of residual BCC in subsequent excisions. They found that 34 (24%) of 143 cases with negative biopsy margins contained residual tumor in the corresponding excision.⁵

Our study revealed that almost half of BCC biopsy specimens that had negative histologic margins with routine sectioning had truly positive margins on complete block exhaustion. This finding was independent of multiple factors, including tumor subtype, indicating that even nonaggressive tumors are prone to false-negative margin reports. We also found that reports of negative margins persuaded some clinicians to forgo definitive treatment. This study serves to remind clinicians of the limitations of margin assessment and provides impetus for dermatopathologists to consider modifying how margin status is reported.

Limitations of this study include a small number of cases and limited generalizability. Institutions that routinely examine more levels of each biopsy specimen may be less likely to erroneously categorize a positive margin as negative. Furthermore, despite exhausting the tissue block, we still may have underestimated the number of cases with truly positive margins, as this method inherently does not allow for complete margin examination.

Acknowledgments
We thank the Geisinger Department of Dermatopathology and the Geisinger Biostatistics & Research Data Core (Danville, Pennsylvania) for their assistance with our project.

Basal cell carcinoma (BCC) is the most common type of skin cancer frequently encountered in both dermatology and primary care settings.¹ When biopsies of these neoplasms are performed to confirm the diagnosis, pathology reports may indicate positive or negative margin status. No guidelines exist for reporting biopsy margin status for BCC, resulting in varied reporting practices among dermatopathologists. Furthermore, the terminology used to describe margin status can be ambiguous and differs among pathologists; language such as “approaches the margin” or “margins appear free” may be used, with nonuniform interpretation between pathologists and providers, leading to variability in patient management.²

When interpreting a negative margin status on a pathology report, one must question if the BCC extends beyond the margin in unexamined sections of the specimen, which could be the result of an irregular tumor growth pattern or tissue processing. It has been estimated that less than 2% of the peripheral surgical margin is ultimately examined when serial cross-sections are prepared histologically (the bread loaf technique). However, this estimation would depend on several variables, including the number and thickness of sections and the amount of tissue discarded during processing.³ Importantly, reports of a false-negative margin could lead both the clinician and patient to believe that the neoplasm has been completely removed, which could have serious consequences.

Our study sought to determine the reliability of negative biopsy margin status for BCC. We examined BCC biopsy specimens initially determined to have uninvolved margins on routine tissue processing and determined the proportion with truly negative margins after complete tissue block sectioning of the initial biopsy specimen. We felt this technique was a more accurate measurement of true margin status than examination of a re-excision specimen. We also identified any factors that were predictive of positive true margins.

Methods

We conducted a retrospective study evaluating tissue samples collected at Geisinger Health System (Danville, Pennsylvania) from January to December 2016. Specimens were queried via the electronic database system at our institution (CoPath). We included BCC biopsy specimens with negative histologic margins on initial assessment that subsequently had block exhaust levels routinely ordered. These levels are cut every 100 to 150 µm, generating approximately 8 glass slides. We excluded all tumors that did not fit these criteria as well as those in patients younger than 18 years. Data collection was performed utilizing specimen pathology reports in addition to the note from the corresponding clinician office visit from the institution’s electronic medical record (Epic). Appropriate statistical calculations were performed. This study was approved by an institutional review board at our institution, which is required for all research involving human participants. This served to ensure the proper review and storage of patients’ protected health information.

 

Results

The search yielded a total of 122 specimens from 104 patients after appropriate exclusions. We examined a total of 122 BCC biopsy specimens with negative initial margins: 121 (99.2%) shave biopsies and 1 (0.8%) punch biopsy. Of 122 specimens with negative initial margins, 53 (43.4%) were found to have a truly positive margin based on the presence of either tumor or stroma at the lateral or deep tissue edge after complete tissue block sectioning. Sixty-nine (56.6%) specimens had clear margins and were categorized as truly negative after complete tissue block sectioning. Specimens with positive and negative final margin status did not differ significantly with respect to patient age; gender; biopsy technique; number of gross specimen sections; or tumor characteristics, including location, size, and subtype (Table)(P>.05).

We also examined the type of treatment performed, which varied and included curettage, electrodesiccation and curettage, excision, and Mohs micrographic surgery. Clinicians, who were not made aware of the exhaust level protocol, chose not to pursue further treatment in 6 (4.9%) of the cases because of negative biopsy margins. Four (66.7%) of the 6 providers were physicians, and 2 (33.3%) were advanced practitioners. All of the providers practiced within the Department of Dermatology.

Comment

Our findings support prior smaller studies investigating this topic. A prospective study by Schnebelen et al⁴ examined 27 BCC biopsy specimens and found that 8 (30%) were erroneously classified as negative on routine examination. This study similarly determined true margin status by assessing the margins at complete tissue block exhaustion.⁴ Willardson et al⁵ also demonstrated the poor predictive value of margin status based on the presence of residual BCC in subsequent excisions. They found that 34 (24%) of 143 cases with negative biopsy margins contained residual tumor in the corresponding excision.⁵

Our study revealed that almost half of BCC biopsy specimens that had negative histologic margins with routine sectioning had truly positive margins on complete block exhaustion. This finding was independent of multiple factors, including tumor subtype, indicating that even nonaggressive tumors are prone to false-negative margin reports. We also found that reports of negative margins persuaded some clinicians to forgo definitive treatment. This study serves to remind clinicians of the limitations of margin assessment and provides impetus for dermatopathologists to consider modifying how margin status is reported.

Limitations of this study include a small number of cases and limited generalizability. Institutions that routinely examine more levels of each biopsy specimen may be less likely to erroneously categorize a positive margin as negative. Furthermore, despite exhausting the tissue block, we still may have underestimated the number of cases with truly positive margins, as this method inherently does not allow for complete margin examination.

Acknowledgments
We thank the Geisinger Department of Dermatopathology and the Geisinger Biostatistics & Research Data Core (Danville, Pennsylvania) for their assistance with our project.

From the University of Toronto (Dr. Basuita, Corey L. Kamen, and Dr. Soong) and Sinai Health System (Corey L. Kamen, Cheryl Ethier, and Dr. Soong), Toronto, Ontario, Canada. Co-first authors are Manpreet Basuita, MD, and Corey L. Kamen, BSc.

Abstract

• Objective: Routine laboratory testing is common among medical inpatients; however, when ordered inappropriately testing can represent low-value care. We examined the impact of an evidence-based intervention bundle on utilization.
• Participants/setting: This prospective cohort study took place at a tertiary academic medical center and included 6424 patients admitted to the general internal medicine service between April 2016 and March 2018.
• Intervention: An intervention bundle, whose first components were implemented in July 2016, included computer order entry restrictions on repetitive laboratory testing, education, and audit-feedback.
• Measures: Data were extracted from the hospital electronic health record. The primary outcome was the number of routine blood tests (complete blood count, creatinine, and electrolytes) ordered per inpatient day.
• Analysis: Descriptive statistics were calculated for demographic variables. We used statistical process control charts to compare the baseline period (April 2016-June 2017) and the intervention period (July 2017-March 2018) for the primary outcome.
• Results: The mean number of combined routine laboratory tests ordered per inpatient day decreased from 1.19 (SD, 0.21) tests to 1.11 (SD, 0.05), a relative reduction of 6.7% (P < 0.0001). Mean cost per case related to laboratory tests decreased from $17.24 in the pre-intervention period to $16.17 in the post-intervention period (relative reduction of 6.2%). This resulted in savings of $26,851 in the intervention year.
• Conclusion: A laboratory intervention bundle was associated with small reductions in testing and costs. A routine test performed less than once per inpatient day may not be clinically appropriate or possible.

Keywords: utilization; clinical costs; quality improvement; QI intervention; internal medicine; inpatient.

Routine laboratory blood testing is a commonly used diagnostic tool that physicians rely on to provide patient care. Although routine blood testing represents less than 5% of most hospital budgets, routine use and over-reliance on testing among physicians makes it a target of cost-reduction efforts.^1-3 A variety of interventions have been proposed to reduce inappropriate laboratory tests, with varying results.^1,4-6 Successful interventions include providing physicians with fee data associated with ordered laboratory tests, unbundling panels of tests, and multicomponent interventions.⁶ We conducted a multifaceted quality improvement study to promote and develop interventions to adopt appropriate blood test ordering practices.

Methods

Setting

This prospective cohort study took place at Mount Sinai Hospital, a 443-bed academic hospital affiliated with the University of Toronto, where more than 2400 learners rotate through annually. The study was approved by the Mount Sinai Hospital Research Ethics Board.

Participants

We included all inpatient admissions to the general internal medicine service between April 2016 and March 2018. Exclusion criteria included a length of stay (LOS) longer than 365 days and admission to a critical care unit. Patients with more than 1 admission were counted as separate hospital inpatient visits.

 

Intervention

Based on internal data, we targeted the top 3 most frequently ordered routine blood tests: complete blood count (CBC), creatinine, and electrolytes. Trainee interviews revealed that habit, bundled order sets, and fear of “missing something” contributed to inappropriate routine blood test ordering. Based on these root causes, we used the Model for Improvement to iteratively develop a multimodal intervention that began in July 2016.^7,8 This included a change to the computerized provider order entry (CPOE) to nudge clinicians to a restrictive ordering strategy by substituting the “Daily x3” frequency of blood test ordering with a “Daily x1” option on a pick list of order options. Clinicians could still order daily routine blood tests for any specified duration, but would have to do so by manually changing the default setting within the CPOE.

From July 2017 to March 2018, the research team educated residents on appropriate laboratory test ordering and provided audit and feedback data to the clinicians. Diagnostic uncertainty was addressed in teaching sessions. Attending physicians were surveyed on appropriate indications for daily laboratory testing for each of CBC, electrolytes, and creatinine. Appropriate indications (Figure 1) were displayed in visible clinical areas and incorporated into teaching sessions.⁹

Clinician teams received real-time performance data on their routine blood test ordering patterns compared with an institutional benchmark. Bar graphs of blood work ordering rates (sum of CBCs, creatinine, and electrolytes ordered for all patients on a given team divided by the total LOS for all patients) were distributed to each internal medicine team via email every 2 weeks (Figure 2).^1,10-12

 

Data Collection and Analysis

Data were extracted from the hospital electronic health record (EHR). The primary outcome was the number of routine blood tests (CBC, creatinine, and electrolytes) ordered per inpatient day. Descriptive statistics were calculated for demographic variables. We used statistical process control (SPC) charts to compare the baseline period (April 2016-June 2017) and the intervention period (July 2017-March 2018) for the primary outcome. SPC charts display process changes over time. Data are plotted in chronological order, with the central line representing the outcome mean, an upper line representing the upper control limit, and a lower line representing the lower control limit. The upper and lower limits were set at 3δ, which correspond to 3 standard deviations above and below the mean. Six successive points above or beyond the mean suggests “special cause variation,” indicating that observed results are unlikely due to secular trends. SPC charts are commonly used quality tools for process improvement as well as research.^13-16 These charts were created using QI Macros SPC software for Excel V. 2012.07 (KnowWare International, Denver, CO).

The direct cost of each laboratory test was acquired from the hospital laboratory department. The cost of each laboratory test (CBC = $7.54/test, electrolytes = $2.04/test, creatinine = $1.28/test, in Canadian dollars) was subsequently added together and multiplied by the pre- and post-intervention difference of total blood tests saved per inpatient day and then multiplied by 365 to arrive at an estimated cost savings per year.

 

Results

Over the study period, there were 6424 unique patient admissions on the general internal medicine service, with a median LOS of 3.5 days (Table).

The majority of inpatient visits had at least 1 test of CBC (80%; mean, 3.6 tests/visit), creatinine (79.3%; mean, 3.5 tests/visit), or electrolytes (81.6%; mean, 3.9 tests/visit) completed. In total, 56,767 laboratory tests were ordered.

Following the intervention, there was a reduction in both rates of routine blood test orders and their associated costs, with a shift below the mean. The mean number of tests ordered (combined CBC, creatinine, and electrolytes) per inpatient day decreased from 1.19 (SD, 0.21) in the pre-intervention period to 1.11 (SD, 0.05) in the post-intervention period (P < 0.0001), representing a 6.7% relative reduction (Figure 3). We observed a 6.2% relative reduction in costs per inpatient day, translating to a total savings of $26,851 over 1 year for the intervention period.

Discussion

Our study suggests that a multimodal intervention, including CPOE restrictions, resident education with posters, and audit and feedback strategies, can reduce lab test ordering on general internal medicine wards. This finding is similar to those of previous studies using a similar intervention, although different laboratory tests were targeted.^{1,2,5,6,10,17}

Our study found lower test result reductions than those reported by a previous study, which reported a relative reduction of 17% to 30%,¹⁸ and by another investigation that was conducted recently in a similar setting.¹⁷ In the latter study, reductions in laboratory testing were mostly found in nonroutine tests, and no significant improvements were noted in CBC, electrolytes, and creatine, the 3 tests we studied over the same duration.¹⁷ This may represent a ceiling effect to reducing laboratory testing, and efforts to reduce CBC, electrolytes, and creatinine testing beyond 0.3 to 0.4 tests per inpatient day (or combined 1.16 tests per inpatient day) may not be clinically appropriate or possible. This information can guide institutions to include other areas of overuse based on rates of utilization in order to maximize the benefits from a resource intensive intervention.

There are a number of limitations that merit discussion. First, observational studies do not demonstrate causation; however, to our knowledge, there were no other co-interventions that were being conducted during the study duration. One important note is that our project’s intervention began in July, at which point there are new internal medicine residents beginning their training. As the concept of resource allocation becomes more important, medical schools are spending more time educating students about Choosing Wisely, and, therefore, newer cohorts of residents may be more cognizant of appropriate blood testing. Second, this is a single-center study, limiting generalizability; however, we note that many other centers have reported similar findings. Another limitation is that we do not know whether there were any adverse clinical events associated with blood work ordering that was too restrictive, although informal tracking of STAT laboratory testing remained stable throughout the study period. It is important to ensure that blood work is ordered in moderation and tailored to patients using one’s clinical judgment.

Future Directions

We observed modest reductions in the quantity and costs associated with a quality improvement intervention aimed at reducing routine blood testing. A baseline rate of laboratory testing of less than 1 test per inpatient day may require including other target tests to drive down absolute utilization.

Corresponding author: Christine Soong, MD, MSc, 433-600 University Avenue, Toronto, Ontario, Canada M5G 1X5; Christine.soong@utoronto.ca.

Financial disclosures: None.

From the University of Toronto (Dr. Basuita, Corey L. Kamen, and Dr. Soong) and Sinai Health System (Corey L. Kamen, Cheryl Ethier, and Dr. Soong), Toronto, Ontario, Canada. Co-first authors are Manpreet Basuita, MD, and Corey L. Kamen, BSc.

Abstract

• Objective: Routine laboratory testing is common among medical inpatients; however, when ordered inappropriately testing can represent low-value care. We examined the impact of an evidence-based intervention bundle on utilization.
• Participants/setting: This prospective cohort study took place at a tertiary academic medical center and included 6424 patients admitted to the general internal medicine service between April 2016 and March 2018.
• Intervention: An intervention bundle, whose first components were implemented in July 2016, included computer order entry restrictions on repetitive laboratory testing, education, and audit-feedback.
• Measures: Data were extracted from the hospital electronic health record. The primary outcome was the number of routine blood tests (complete blood count, creatinine, and electrolytes) ordered per inpatient day.
• Analysis: Descriptive statistics were calculated for demographic variables. We used statistical process control charts to compare the baseline period (April 2016-June 2017) and the intervention period (July 2017-March 2018) for the primary outcome.
• Results: The mean number of combined routine laboratory tests ordered per inpatient day decreased from 1.19 (SD, 0.21) tests to 1.11 (SD, 0.05), a relative reduction of 6.7% (P < 0.0001). Mean cost per case related to laboratory tests decreased from $17.24 in the pre-intervention period to $16.17 in the post-intervention period (relative reduction of 6.2%). This resulted in savings of $26,851 in the intervention year.
• Conclusion: A laboratory intervention bundle was associated with small reductions in testing and costs. A routine test performed less than once per inpatient day may not be clinically appropriate or possible.

Keywords: utilization; clinical costs; quality improvement; QI intervention; internal medicine; inpatient.

Routine laboratory blood testing is a commonly used diagnostic tool that physicians rely on to provide patient care. Although routine blood testing represents less than 5% of most hospital budgets, routine use and over-reliance on testing among physicians makes it a target of cost-reduction efforts.^1-3 A variety of interventions have been proposed to reduce inappropriate laboratory tests, with varying results.^1,4-6 Successful interventions include providing physicians with fee data associated with ordered laboratory tests, unbundling panels of tests, and multicomponent interventions.⁶ We conducted a multifaceted quality improvement study to promote and develop interventions to adopt appropriate blood test ordering practices.

Methods

Setting

This prospective cohort study took place at Mount Sinai Hospital, a 443-bed academic hospital affiliated with the University of Toronto, where more than 2400 learners rotate through annually. The study was approved by the Mount Sinai Hospital Research Ethics Board.

Participants

We included all inpatient admissions to the general internal medicine service between April 2016 and March 2018. Exclusion criteria included a length of stay (LOS) longer than 365 days and admission to a critical care unit. Patients with more than 1 admission were counted as separate hospital inpatient visits.

 

Intervention

Based on internal data, we targeted the top 3 most frequently ordered routine blood tests: complete blood count (CBC), creatinine, and electrolytes. Trainee interviews revealed that habit, bundled order sets, and fear of “missing something” contributed to inappropriate routine blood test ordering. Based on these root causes, we used the Model for Improvement to iteratively develop a multimodal intervention that began in July 2016.^7,8 This included a change to the computerized provider order entry (CPOE) to nudge clinicians to a restrictive ordering strategy by substituting the “Daily x3” frequency of blood test ordering with a “Daily x1” option on a pick list of order options. Clinicians could still order daily routine blood tests for any specified duration, but would have to do so by manually changing the default setting within the CPOE.

From July 2017 to March 2018, the research team educated residents on appropriate laboratory test ordering and provided audit and feedback data to the clinicians. Diagnostic uncertainty was addressed in teaching sessions. Attending physicians were surveyed on appropriate indications for daily laboratory testing for each of CBC, electrolytes, and creatinine. Appropriate indications (Figure 1) were displayed in visible clinical areas and incorporated into teaching sessions.⁹

Clinician teams received real-time performance data on their routine blood test ordering patterns compared with an institutional benchmark. Bar graphs of blood work ordering rates (sum of CBCs, creatinine, and electrolytes ordered for all patients on a given team divided by the total LOS for all patients) were distributed to each internal medicine team via email every 2 weeks (Figure 2).^1,10-12

 

Data Collection and Analysis

Data were extracted from the hospital electronic health record (EHR). The primary outcome was the number of routine blood tests (CBC, creatinine, and electrolytes) ordered per inpatient day. Descriptive statistics were calculated for demographic variables. We used statistical process control (SPC) charts to compare the baseline period (April 2016-June 2017) and the intervention period (July 2017-March 2018) for the primary outcome. SPC charts display process changes over time. Data are plotted in chronological order, with the central line representing the outcome mean, an upper line representing the upper control limit, and a lower line representing the lower control limit. The upper and lower limits were set at 3δ, which correspond to 3 standard deviations above and below the mean. Six successive points above or beyond the mean suggests “special cause variation,” indicating that observed results are unlikely due to secular trends. SPC charts are commonly used quality tools for process improvement as well as research.^13-16 These charts were created using QI Macros SPC software for Excel V. 2012.07 (KnowWare International, Denver, CO).

The direct cost of each laboratory test was acquired from the hospital laboratory department. The cost of each laboratory test (CBC = $7.54/test, electrolytes = $2.04/test, creatinine = $1.28/test, in Canadian dollars) was subsequently added together and multiplied by the pre- and post-intervention difference of total blood tests saved per inpatient day and then multiplied by 365 to arrive at an estimated cost savings per year.

 

Results

Over the study period, there were 6424 unique patient admissions on the general internal medicine service, with a median LOS of 3.5 days (Table).

The majority of inpatient visits had at least 1 test of CBC (80%; mean, 3.6 tests/visit), creatinine (79.3%; mean, 3.5 tests/visit), or electrolytes (81.6%; mean, 3.9 tests/visit) completed. In total, 56,767 laboratory tests were ordered.

Following the intervention, there was a reduction in both rates of routine blood test orders and their associated costs, with a shift below the mean. The mean number of tests ordered (combined CBC, creatinine, and electrolytes) per inpatient day decreased from 1.19 (SD, 0.21) in the pre-intervention period to 1.11 (SD, 0.05) in the post-intervention period (P < 0.0001), representing a 6.7% relative reduction (Figure 3). We observed a 6.2% relative reduction in costs per inpatient day, translating to a total savings of $26,851 over 1 year for the intervention period.

Discussion

Our study suggests that a multimodal intervention, including CPOE restrictions, resident education with posters, and audit and feedback strategies, can reduce lab test ordering on general internal medicine wards. This finding is similar to those of previous studies using a similar intervention, although different laboratory tests were targeted.^{1,2,5,6,10,17}

Our study found lower test result reductions than those reported by a previous study, which reported a relative reduction of 17% to 30%,¹⁸ and by another investigation that was conducted recently in a similar setting.¹⁷ In the latter study, reductions in laboratory testing were mostly found in nonroutine tests, and no significant improvements were noted in CBC, electrolytes, and creatine, the 3 tests we studied over the same duration.¹⁷ This may represent a ceiling effect to reducing laboratory testing, and efforts to reduce CBC, electrolytes, and creatinine testing beyond 0.3 to 0.4 tests per inpatient day (or combined 1.16 tests per inpatient day) may not be clinically appropriate or possible. This information can guide institutions to include other areas of overuse based on rates of utilization in order to maximize the benefits from a resource intensive intervention.

There are a number of limitations that merit discussion. First, observational studies do not demonstrate causation; however, to our knowledge, there were no other co-interventions that were being conducted during the study duration. One important note is that our project’s intervention began in July, at which point there are new internal medicine residents beginning their training. As the concept of resource allocation becomes more important, medical schools are spending more time educating students about Choosing Wisely, and, therefore, newer cohorts of residents may be more cognizant of appropriate blood testing. Second, this is a single-center study, limiting generalizability; however, we note that many other centers have reported similar findings. Another limitation is that we do not know whether there were any adverse clinical events associated with blood work ordering that was too restrictive, although informal tracking of STAT laboratory testing remained stable throughout the study period. It is important to ensure that blood work is ordered in moderation and tailored to patients using one’s clinical judgment.

Future Directions

We observed modest reductions in the quantity and costs associated with a quality improvement intervention aimed at reducing routine blood testing. A baseline rate of laboratory testing of less than 1 test per inpatient day may require including other target tests to drive down absolute utilization.

Corresponding author: Christine Soong, MD, MSc, 433-600 University Avenue, Toronto, Ontario, Canada M5G 1X5; Christine.soong@utoronto.ca.

Financial disclosures: None.

From the University of Toronto (Dr. Basuita, Corey L. Kamen, and Dr. Soong) and Sinai Health System (Corey L. Kamen, Cheryl Ethier, and Dr. Soong), Toronto, Ontario, Canada. Co-first authors are Manpreet Basuita, MD, and Corey L. Kamen, BSc.

Abstract

• Objective: Routine laboratory testing is common among medical inpatients; however, when ordered inappropriately testing can represent low-value care. We examined the impact of an evidence-based intervention bundle on utilization.
• Participants/setting: This prospective cohort study took place at a tertiary academic medical center and included 6424 patients admitted to the general internal medicine service between April 2016 and March 2018.
• Intervention: An intervention bundle, whose first components were implemented in July 2016, included computer order entry restrictions on repetitive laboratory testing, education, and audit-feedback.
• Measures: Data were extracted from the hospital electronic health record. The primary outcome was the number of routine blood tests (complete blood count, creatinine, and electrolytes) ordered per inpatient day.
• Analysis: Descriptive statistics were calculated for demographic variables. We used statistical process control charts to compare the baseline period (April 2016-June 2017) and the intervention period (July 2017-March 2018) for the primary outcome.
• Results: The mean number of combined routine laboratory tests ordered per inpatient day decreased from 1.19 (SD, 0.21) tests to 1.11 (SD, 0.05), a relative reduction of 6.7% (P < 0.0001). Mean cost per case related to laboratory tests decreased from $17.24 in the pre-intervention period to $16.17 in the post-intervention period (relative reduction of 6.2%). This resulted in savings of $26,851 in the intervention year.
• Conclusion: A laboratory intervention bundle was associated with small reductions in testing and costs. A routine test performed less than once per inpatient day may not be clinically appropriate or possible.

Keywords: utilization; clinical costs; quality improvement; QI intervention; internal medicine; inpatient.

Routine laboratory blood testing is a commonly used diagnostic tool that physicians rely on to provide patient care. Although routine blood testing represents less than 5% of most hospital budgets, routine use and over-reliance on testing among physicians makes it a target of cost-reduction efforts.^1-3 A variety of interventions have been proposed to reduce inappropriate laboratory tests, with varying results.^1,4-6 Successful interventions include providing physicians with fee data associated with ordered laboratory tests, unbundling panels of tests, and multicomponent interventions.⁶ We conducted a multifaceted quality improvement study to promote and develop interventions to adopt appropriate blood test ordering practices.

Methods

Setting

This prospective cohort study took place at Mount Sinai Hospital, a 443-bed academic hospital affiliated with the University of Toronto, where more than 2400 learners rotate through annually. The study was approved by the Mount Sinai Hospital Research Ethics Board.

Participants

We included all inpatient admissions to the general internal medicine service between April 2016 and March 2018. Exclusion criteria included a length of stay (LOS) longer than 365 days and admission to a critical care unit. Patients with more than 1 admission were counted as separate hospital inpatient visits.

 

Intervention

Based on internal data, we targeted the top 3 most frequently ordered routine blood tests: complete blood count (CBC), creatinine, and electrolytes. Trainee interviews revealed that habit, bundled order sets, and fear of “missing something” contributed to inappropriate routine blood test ordering. Based on these root causes, we used the Model for Improvement to iteratively develop a multimodal intervention that began in July 2016.^7,8 This included a change to the computerized provider order entry (CPOE) to nudge clinicians to a restrictive ordering strategy by substituting the “Daily x3” frequency of blood test ordering with a “Daily x1” option on a pick list of order options. Clinicians could still order daily routine blood tests for any specified duration, but would have to do so by manually changing the default setting within the CPOE.

From July 2017 to March 2018, the research team educated residents on appropriate laboratory test ordering and provided audit and feedback data to the clinicians. Diagnostic uncertainty was addressed in teaching sessions. Attending physicians were surveyed on appropriate indications for daily laboratory testing for each of CBC, electrolytes, and creatinine. Appropriate indications (Figure 1) were displayed in visible clinical areas and incorporated into teaching sessions.⁹

Clinician teams received real-time performance data on their routine blood test ordering patterns compared with an institutional benchmark. Bar graphs of blood work ordering rates (sum of CBCs, creatinine, and electrolytes ordered for all patients on a given team divided by the total LOS for all patients) were distributed to each internal medicine team via email every 2 weeks (Figure 2).^1,10-12

 

Data Collection and Analysis

Data were extracted from the hospital electronic health record (EHR). The primary outcome was the number of routine blood tests (CBC, creatinine, and electrolytes) ordered per inpatient day. Descriptive statistics were calculated for demographic variables. We used statistical process control (SPC) charts to compare the baseline period (April 2016-June 2017) and the intervention period (July 2017-March 2018) for the primary outcome. SPC charts display process changes over time. Data are plotted in chronological order, with the central line representing the outcome mean, an upper line representing the upper control limit, and a lower line representing the lower control limit. The upper and lower limits were set at 3δ, which correspond to 3 standard deviations above and below the mean. Six successive points above or beyond the mean suggests “special cause variation,” indicating that observed results are unlikely due to secular trends. SPC charts are commonly used quality tools for process improvement as well as research.^13-16 These charts were created using QI Macros SPC software for Excel V. 2012.07 (KnowWare International, Denver, CO).

The direct cost of each laboratory test was acquired from the hospital laboratory department. The cost of each laboratory test (CBC = $7.54/test, electrolytes = $2.04/test, creatinine = $1.28/test, in Canadian dollars) was subsequently added together and multiplied by the pre- and post-intervention difference of total blood tests saved per inpatient day and then multiplied by 365 to arrive at an estimated cost savings per year.

 

Results

Over the study period, there were 6424 unique patient admissions on the general internal medicine service, with a median LOS of 3.5 days (Table).

The majority of inpatient visits had at least 1 test of CBC (80%; mean, 3.6 tests/visit), creatinine (79.3%; mean, 3.5 tests/visit), or electrolytes (81.6%; mean, 3.9 tests/visit) completed. In total, 56,767 laboratory tests were ordered.

Following the intervention, there was a reduction in both rates of routine blood test orders and their associated costs, with a shift below the mean. The mean number of tests ordered (combined CBC, creatinine, and electrolytes) per inpatient day decreased from 1.19 (SD, 0.21) in the pre-intervention period to 1.11 (SD, 0.05) in the post-intervention period (P < 0.0001), representing a 6.7% relative reduction (Figure 3). We observed a 6.2% relative reduction in costs per inpatient day, translating to a total savings of $26,851 over 1 year for the intervention period.

Discussion

Our study suggests that a multimodal intervention, including CPOE restrictions, resident education with posters, and audit and feedback strategies, can reduce lab test ordering on general internal medicine wards. This finding is similar to those of previous studies using a similar intervention, although different laboratory tests were targeted.^{1,2,5,6,10,17}

Our study found lower test result reductions than those reported by a previous study, which reported a relative reduction of 17% to 30%,¹⁸ and by another investigation that was conducted recently in a similar setting.¹⁷ In the latter study, reductions in laboratory testing were mostly found in nonroutine tests, and no significant improvements were noted in CBC, electrolytes, and creatine, the 3 tests we studied over the same duration.¹⁷ This may represent a ceiling effect to reducing laboratory testing, and efforts to reduce CBC, electrolytes, and creatinine testing beyond 0.3 to 0.4 tests per inpatient day (or combined 1.16 tests per inpatient day) may not be clinically appropriate or possible. This information can guide institutions to include other areas of overuse based on rates of utilization in order to maximize the benefits from a resource intensive intervention.

There are a number of limitations that merit discussion. First, observational studies do not demonstrate causation; however, to our knowledge, there were no other co-interventions that were being conducted during the study duration. One important note is that our project’s intervention began in July, at which point there are new internal medicine residents beginning their training. As the concept of resource allocation becomes more important, medical schools are spending more time educating students about Choosing Wisely, and, therefore, newer cohorts of residents may be more cognizant of appropriate blood testing. Second, this is a single-center study, limiting generalizability; however, we note that many other centers have reported similar findings. Another limitation is that we do not know whether there were any adverse clinical events associated with blood work ordering that was too restrictive, although informal tracking of STAT laboratory testing remained stable throughout the study period. It is important to ensure that blood work is ordered in moderation and tailored to patients using one’s clinical judgment.

Future Directions

We observed modest reductions in the quantity and costs associated with a quality improvement intervention aimed at reducing routine blood testing. A baseline rate of laboratory testing of less than 1 test per inpatient day may require including other target tests to drive down absolute utilization.

Corresponding author: Christine Soong, MD, MSc, 433-600 University Avenue, Toronto, Ontario, Canada M5G 1X5; Christine.soong@utoronto.ca.

Financial disclosures: None.

The Accreditation Council for Graduate Medical Education (ACGME) first mandated residency work hour restrictions in 2003.¹ In 2011, revised work hour requirements were issued, further limiting the maximum duration of a shift and extending the duration of time off between scheduled shifts.² Academic medical centers have been forced to adapt to work hour restrictions, and cuts in funding to research and educational missions have pressured institutions to restructure with a greater focus on high-quality, lower-cost care.^3,4 In response, many academic hospitals have added hospitalist teams, or incorporated advanced practice clinicians (APCs) (nurse practitioners [NPs] and physician assistants [PAs]) to accommodate resident physician duty hour restrictions on their inpatient general medicine services.^5,6 More recently, the COVID-19 pandemic has created unanticipated physician shortages forcing medical centers to rapidly expand and broaden the scope of their existing APC workforce.⁷

Several comparisons of clinical outcomes, cost, and patient satisfaction between different combinations of hospitalist-based, resident-based, or APC-based inpatient teams have been reported with conflicting observations.^6,8-14 Roy et al reported no significant differences in mortality, length of stay (LOS), or readmissions between PA and resident teams.⁶ Timmermans et al reported similar cost-effectiveness, LOS, and quality of care between PA and physician teams that included a hybrid of attending only and resident teams.^13,14 Alternatively, Singh et al and Iannuzzi et al reported increased LOS among PA teams,^10,12 whereas Chin et al observed an increased LOS and reduced 30-day readmissions among hospitalist teams.⁸ While these observed differences may be attributable to heterogeneous patient populations or institution-specific team structure, the exact reasons remain unknown. Furthermore, understanding the value of alternate staffing models is essential for medical centers to prepare for potential COVID-19 related physician shortages. To our knowledge, no study to date has directly compared outcomes between resident, APC, and hospitalist team structures within an academic medical center.

We believe our institution provides a unique environment to study the differences in inpatient general medicine team structure with respect to quality and efficiency of care delivery. The objective of our study is to directly compare clinical outcomes and resource utilization among three distinct team structures: APC, resident, and solo hospitalist. We hypothesize that clinical outcomes, cost, and utilization of consult services will be similar across all team structures and hospitalist teams will discharge patients earlier than resident and APC teams.

METHODS

Study Design and Setting

We conducted a retrospective observational cohort study at the University of Utah Medical Center, a 548-bed academic medical center in Salt Lake City. An electronic database query was used to identify all patients discharged from the inpatient general internal medicine service between July 1, 2015, and July 1, 2018. Baseline patient characteristics were collected including age, gender, and Charlson comorbidity index (CCI).¹⁵ Case-mix index was determined for admissions where a Medicare Severity Diagnosis Related Group (MS-DRG) and corresponding weight was assigned.^16,17 Source of admission was collected to identify patients transferred from an outside hospital, typically due to increased medical complexity or need for specialty care not available at the referring center. Time of admission was collected to classify whether a patient was admitted during the day or at night. Length of stay was calculated as the difference between discharge date/time and admission date/time. Discharge order time was collected as a measure of clinician efficiency. The number of consults per admission was determined by the number of different medical or surgical subspecialty services that wrote at least one consultation or progress note after the time of admission and were not the primary service at the time the note was written. The project was reviewed and deemed exempt by the University of Utah Institutional Review Board (IRB 00104884).

Inpatient Care Team Structure

Patients were assigned to one of three cohorts dependent on the assigned treatment team at the time of discharge. The three inpatient team structures were as follows: (1) a “resident team” composed of a senior resident (postgraduate year [PGY] 2 or PGY3) and one to two medical students or one senior resident, two interns (PGY1), and one to two medical students supervised by a hospitalist physician; (2) an “APC team” composed of one to two APCs supervised by a hospitalist physician; and (3) a “hospitalist team” composed of one attending hospitalist independently managing all patients.

Advanced Practice Clinicians

The APC service included 10 APCs (8 PAs and 2 NPs), with a combined workforce of nine APC full-time equivalents during the study period. Their experience ranged from new graduate to 11 years of clinical experience, with an average of 4.2 years. Among the 6 APCs with prior clinical experience, the majority (86%) of their years of clinical experience were within inpatient medicine, oncology, or cardiology. Recognizing the variability in clinical experience, we employed a rigorous onboarding program that entailed an average of 80 hours of didactic sessions including 1:1 teaching of the inpatient Society of Hospital Medicine core lecture series combined with initial intense clinical oversight.¹⁸ This program ranged from 2 weeks to 6 weeks depending on the individual APC’s clinical experience, progress, and comfort working independently. This onboarding program has subsequently been formalized into a 1-year APC fellowship that began after the study period concluded.

The degree of autonomy for each APC was individualized based on their clinical experience and ability to recognize limitations such as medical decision-making, clinical knowledge, and effective use of interprofessional team members (eg, peers, nursing, ancillary staff, consultants, and support personnel). Those APCs who demonstrated a sufficient level of clinical competence functioned with a high level of autonomy. During the day, APCs were expected to be the first point of contact for interprofessional team members, to respond to acute clinical changes in a patient’s condition, and to discuss active issues with the supervising attending, all with the majority of medical decision-making, direct patient communication, documentation, and care coordination performed by the APC. An experienced subset of the APC service was responsible for overnight coverage. Nocturnist APCs independently managed all cross-cover issues on patients assigned to APC and hospitalist teams and performed admissions with very little to no direct supervision of the overnight attending physician.

Patient Admission and Redistribution Process

During the study period, resident teams performed all daytime admissions (6 am to 6 pm) on a rotating basis. On any given day, three of four resident teams performed daytime admissions with the fourth team designated as “golden” and free from admitting duties. Patients admitted during the day remained assigned to the resident team for continuity. The APC and hospitalist teams did not accept new admissions during the day. Nighttime admissions (6 pm to 6 am) were performed by a separate team composed of two senior residents, two interns, one APC, occasional APC and medical students, and one supervising attending hospitalist. This team functioned as a single unit. Nighttime admissions were performed in a sequential and rotating fashion (eg, Intern A > Intern B > Resident A > Resident B > APC > student(s) > Intern A > Intern B, etc). Patients admitted overnight were randomly redistributed the following morning, with the majority reassigned to an APC team or hospitalist team in order to offset the workload of the resident teams performing daytime admissions. Following redistribution, a patient would remain assigned to the daytime APC or hospitalist team for the duration of their hospitalization. The redistribution decisions were based on individual team census, without systematic consideration of an individual patient’s diagnosis, medical complexity, socioeconomic status, or perceived quality of learning potential (eg, good teaching case).

Study Outcomes

We divided study outcomes into two categories, clinical outcomes and resource utilization. Clinical outcomes included LOS, unplanned readmission within 30-days, and inpatient mortality and were designed to measure patient-related outcomes as a reflection of the quality of care delivered by different team structures. Resource utilization included discharge order time, discharge time, consults per admission, and total direct cost, which were designed to measure provider-related differences in efficiency and cost of care.

Statistical analysis

Baseline characteristics and unadjusted outcomes are reported as frequency and percent, normally distributed variables as mean with SD, and nonnormally distributed variables as median with interquartile range (IQR). Baseline characteristics and unadjusted outcomes were compared using the chi-square test or the t test, where appropriate. Multivariable regression analysis using generalized linear models with a log link function and gamma distribution was used for continuous outcomes. Multivariable logistic regression was used for binary outcomes.¹⁰ Covariates included in regression models were age, gender, CCI, transfer from an outside hospital, and nighttime admission. In a sensitivity analysis, we included MS-DRG weight as a covariate for 85% of hospitalizations in our cohort exclusive of observation stays, and our findings were qualitatively similar (data not reported but available on request). Adjusted continuous outcomes were estimated using marginal effects at the means.¹⁹ Due to the sensitivity of cost data and an institutional policy against disclosing cost figures, total direct costs were normalized using the unadjusted median and adjusted mean total direct cost of an admission to an APC team as the normalizing value. A P value cutoff of .05 was used to determine statistical significance. Stata/IC version 16.1 (StataCorp) was used for all analyses.

RESULTS

Study Population

A total of 12,716 hospital admissions were identified during the study period. Of these, 7,943 (62.5%) admissions were assigned to a resident team, 3,519 (27.7%) admissions were assigned to an APC team, and the remaining 1,254 (9.9%) were assigned to a hospitalist team. Baseline patient characteristics are reported in Table 1. Patients admitted to resident teams (mean age [SD], 56.9 [19.1] years) were younger than those admitted to an APC team (58.0 [19.3] years; P = .004) or a hospitalist team (58.2 [19.3] years; P = .026). The case-mix index (mean MS-DRG weight [SD], 1.44 [0.87]) was slightly lower for resident teams than that for APC teams (1.49 [0.90]; P = .025).Resident teams had a significantly lower proportion of night admissions than did APC teams (32.0% vs 49.5%; P < .001) and hospitalist teams (48.6%; P < .001). APC teams were assigned more patients transferred from an outside hospital (19.1%), compared with resident teams (15.0%; P < .001) and hospitalist teams (16.0%; P = .015). No other significant differences were observed in baseline characteristics between cohorts.

Clinical Outcomes

Unadjusted analysis demonstrated the LOS was similar among resident, APC, and hospitalist teams with a median (IQR) LOS of 2.90 (1.86, 4.26) days, 2.93 (1.89, 4.66) days, and 2.86 (1.84, 4.67) days, respectively. No significant differences were observed in unadjusted 30-day readmissions or inpatient mortality among the team structures (Table 2). Following multivariable adjustment for differences in baseline characteristics, no significant differences were observed in LOS, 30-day readmission, or inpatient mortality among teams (Table 3).

Resource Utilization

In unadjusted comparisons, hospitalist teams were observed to place discharge orders more than 30 minutes earlier than APC teams (median hours after midnight [IQR], 11.20 [9.63, 13.60] vs 11.73 [10.00, 13.87]; P < .001) and 54 minutes earlier than resident teams (12.10 [10.38, 13.90]; P < .001) (Table 2). Consistent with the earlier placement of discharge orders, hospitalist patients were also discharged from the hospital 26 and 32 minutes earlier than APC and resident patients, respectively. APC teams also discharged patients slightly earlier (6 minutes) than resident teams (median hours after midnight [IQR], 14.97 [13.23, 16.72] vs 15.07 [13.42, 16.73]; P = .045). Median consultation use among teams was similar, although statistically significant differences were present. Normalized total direct cost was 8% higher (P < .001) for admissions to APC teams than that for resident teams and 7% higher (P = .008) than that for hospitalist teams in unadjusted analysis (Table 2).

Following multivariable adjustment, the mean differences in discharge order time and discharge time remained significant with hospitalist teams discharging patients an average of 20 to 30 minutes earlier than APC and resident teams (Table 3). Consultant utilization remained significantly different between teams, with APC teams utilizing consultants on average 15% more than hospitalist teams (P < .001) and 7% more than resident teams (P = .001). The differences in total direct costs were not significant after adjusted analysis.

DISCUSSION

Many academic medical centers have expanded their workforce with APC or nonteaching hospitalist teams to accommodate the increasing volume of hospital admissions, resident work hour restrictions,^1,2 and medical complexity of an aging population. Several hospitals have reported comparative outcomes between different care delivery models, with conflicting results.^6,8,10-12 In our study, we directly evaluated three inpatient care delivery models and found that hospitalist teams discharged patients more efficiently and utilized fewer consultants, compared with APC and resident teams. In spite of this improved efficiency, no significant differences were observed in cost or other clinical outcomes.

Our findings are important and further strengthen the evidence supporting the use of APCs on inpatient general medicine services and are of particular interest to academic centers struggling to expand staffing in order to offset the growth in patient volume and reduction in resident workforce. We believe several findings from our study warrant further discussion.

First, although hospitalist teams were able to discharge patients more efficiently, this observation may be influenced by factors of workflow rather than caused by significant disparities in efficiency between provider types (ie, APC vs hospitalist vs resident physician). As with most academic centers, patients assigned to resident teams are presented by house staff to an attending physician who is ultimately responsible for patient care decisions. Therefore, it is conceivable that delays in the discharge process are in part related to the convention of bedside rounding and discussing the care plan prior to discharge.²⁰ In fact, we recognized this as a bottleneck and changed our discharge process for resident teams in June 2017, with a measurable improvement in discharge times. In the absence of this intervention, our observed differences in discharge times among teams may have been even greater.

Second, no significant differences in clinical outcomes were observed in our adjusted analyses, which suggests that a similar quality of care is delivered to patients regardless of team structure, an important observation when considering different staffing models.

Third, we observed a significant increase in consultation use among resident and APC teams, compared with hospitalists. While we are not able to precisely identify the basis for this variation, we believe it could reflect differences in clinical experience, comfort with diagnostic uncertainty, or the unequal distribution of patients transferred from outside hospitals for tertiary care. Interestingly, the greater consultation use did not correlate with higher healthcare costs, a finding recently reported by Stevens et al.²¹

Fourth, we believe the lack of differences in cost and clinical outcomes among team structures may be of particular interest to academic centers when considering physician burnout, salaries, and clinical education. The relationship between clerical burden, such as completing clinical documentation and computerized physician order entry, has been implicated as a risk factor for physician burnout.²² Incorporating APCs into roles similar to those performed by resident physicians may reduce the clerical burden on hospitalists, thereby reducing the risk of physician burnout. The addition of APCs may also represent opportunities for cost savings for healthcare centers when comparing the median salary of an APC to that of an internal medicine hospitalist.^23,24 Moreover, academic hospitalists have been shown to be excellent medical educators and report increased job satisfaction with a variety of duties beyond direct patient care.^24,25 Unforeseen benefits of adding APC teams within our institution has been the added teaching opportunities for APCs and APC students, increased collegiality with the APCs, and the creation of an APC fellowship program with a focus on inpatient medicine. Similar postgraduate training programs have been reported and serve as effective models to train APCs for hospital-based practice.²⁶

Lastly, although this project was conceived and completed prior to the COVID-19 pandemic, our observations may be informative for medical centers experiencing a workforce shortage caused by a surge of COVID-19 patients. During a physician shortage we believe our APC team model could be rapidly expanded to accommodate a large influx of patients. This expansion could be accomplished through a single attending physician overseeing multiple APC teams. In this model, the supervising physician would only evaluate the most complex patients with most patients being managed solely by an APC from admission to discharge. Such changes may require temporary suspension of state laws restricting APC independent practice.^27,28

Our findings contrast those of previous reports in that we did not observe significant differences in clinical outcomes (ie, LOS, inpatient mortality, and 30-day readmissions) or total direct cost.^8,10,21 Other institutions have noted an increased LOS among APC teams and hospitalist teams, compared with resident teams.^8,10 Furthermore, Chin et al and Iannuzzi et al reported reductions in healthcare cost for resident teams, whereas our study did not identify significant cost differences among team structures. Although we cannot pinpoint the exact reason(s) for these dissimilarities, it is plausible that unmeasured factors such as institutional differences in APC training, direct physician supervision, admission processes, or inpatient team census may play a role.

Several study limitations should be recognized. First, the retrospective, nonrandomized design is one of the largest limitations of our study. Administrative data was obtained via an electronic query of our data warehouse, and although we aimed to identify as many patient characteristics as possible to adjust for cofounding effects, undetected differences among cohorts may exist. Second, our inpatient admission process may have placed undue burden on resident teams to perform all daytime admissions, inadvertently affecting study outcomes. It is possible the observed benefits of a solo hospitalist team are attributable to the lack of admitting duties rather than inherent advantages of the team structure. If this were the case, we would expect similar benefits among APC teams, which we did not note. Third, the study was performed at a single academic center, which may limit the generalizability of our results. Fourth, it is possible the outcomes are similar among teams because our hospitalist faculty rotate proportionately between the different teams. Lastly, the study was underpowered to detect a significant difference in mortality between hospitalist and APC teams. A post hoc power calculation based on our observed sample and effect sizes estimated 75% power to detect a mortality difference between hospitalists and APCs; other mortality comparisons were adequately powered.

CONCLUSION

We observed similar total direct costs, LOS, 30-day readmission, and inpatient mortality between hospitalist, APC, and resident teams. APC and resident teams utilized more consultants and discharged patient later than hospitalists. Our analysis suggests clinical outcomes are not significantly affected by inpatient team structure, and the addition of general medicine inpatient APC or hospitalist teams represent safe and efficient alternatives to traditional resident teams within an academic medical center.

Disclosures

All authors declare they have no conflicts of interest.

The Accreditation Council for Graduate Medical Education (ACGME) first mandated residency work hour restrictions in 2003.¹ In 2011, revised work hour requirements were issued, further limiting the maximum duration of a shift and extending the duration of time off between scheduled shifts.² Academic medical centers have been forced to adapt to work hour restrictions, and cuts in funding to research and educational missions have pressured institutions to restructure with a greater focus on high-quality, lower-cost care.^3,4 In response, many academic hospitals have added hospitalist teams, or incorporated advanced practice clinicians (APCs) (nurse practitioners [NPs] and physician assistants [PAs]) to accommodate resident physician duty hour restrictions on their inpatient general medicine services.^5,6 More recently, the COVID-19 pandemic has created unanticipated physician shortages forcing medical centers to rapidly expand and broaden the scope of their existing APC workforce.⁷

Several comparisons of clinical outcomes, cost, and patient satisfaction between different combinations of hospitalist-based, resident-based, or APC-based inpatient teams have been reported with conflicting observations.^6,8-14 Roy et al reported no significant differences in mortality, length of stay (LOS), or readmissions between PA and resident teams.⁶ Timmermans et al reported similar cost-effectiveness, LOS, and quality of care between PA and physician teams that included a hybrid of attending only and resident teams.^13,14 Alternatively, Singh et al and Iannuzzi et al reported increased LOS among PA teams,^10,12 whereas Chin et al observed an increased LOS and reduced 30-day readmissions among hospitalist teams.⁸ While these observed differences may be attributable to heterogeneous patient populations or institution-specific team structure, the exact reasons remain unknown. Furthermore, understanding the value of alternate staffing models is essential for medical centers to prepare for potential COVID-19 related physician shortages. To our knowledge, no study to date has directly compared outcomes between resident, APC, and hospitalist team structures within an academic medical center.

We believe our institution provides a unique environment to study the differences in inpatient general medicine team structure with respect to quality and efficiency of care delivery. The objective of our study is to directly compare clinical outcomes and resource utilization among three distinct team structures: APC, resident, and solo hospitalist. We hypothesize that clinical outcomes, cost, and utilization of consult services will be similar across all team structures and hospitalist teams will discharge patients earlier than resident and APC teams.

METHODS

Study Design and Setting

We conducted a retrospective observational cohort study at the University of Utah Medical Center, a 548-bed academic medical center in Salt Lake City. An electronic database query was used to identify all patients discharged from the inpatient general internal medicine service between July 1, 2015, and July 1, 2018. Baseline patient characteristics were collected including age, gender, and Charlson comorbidity index (CCI).¹⁵ Case-mix index was determined for admissions where a Medicare Severity Diagnosis Related Group (MS-DRG) and corresponding weight was assigned.^16,17 Source of admission was collected to identify patients transferred from an outside hospital, typically due to increased medical complexity or need for specialty care not available at the referring center. Time of admission was collected to classify whether a patient was admitted during the day or at night. Length of stay was calculated as the difference between discharge date/time and admission date/time. Discharge order time was collected as a measure of clinician efficiency. The number of consults per admission was determined by the number of different medical or surgical subspecialty services that wrote at least one consultation or progress note after the time of admission and were not the primary service at the time the note was written. The project was reviewed and deemed exempt by the University of Utah Institutional Review Board (IRB 00104884).

Inpatient Care Team Structure

Patients were assigned to one of three cohorts dependent on the assigned treatment team at the time of discharge. The three inpatient team structures were as follows: (1) a “resident team” composed of a senior resident (postgraduate year [PGY] 2 or PGY3) and one to two medical students or one senior resident, two interns (PGY1), and one to two medical students supervised by a hospitalist physician; (2) an “APC team” composed of one to two APCs supervised by a hospitalist physician; and (3) a “hospitalist team” composed of one attending hospitalist independently managing all patients.

Advanced Practice Clinicians

The APC service included 10 APCs (8 PAs and 2 NPs), with a combined workforce of nine APC full-time equivalents during the study period. Their experience ranged from new graduate to 11 years of clinical experience, with an average of 4.2 years. Among the 6 APCs with prior clinical experience, the majority (86%) of their years of clinical experience were within inpatient medicine, oncology, or cardiology. Recognizing the variability in clinical experience, we employed a rigorous onboarding program that entailed an average of 80 hours of didactic sessions including 1:1 teaching of the inpatient Society of Hospital Medicine core lecture series combined with initial intense clinical oversight.¹⁸ This program ranged from 2 weeks to 6 weeks depending on the individual APC’s clinical experience, progress, and comfort working independently. This onboarding program has subsequently been formalized into a 1-year APC fellowship that began after the study period concluded.

The degree of autonomy for each APC was individualized based on their clinical experience and ability to recognize limitations such as medical decision-making, clinical knowledge, and effective use of interprofessional team members (eg, peers, nursing, ancillary staff, consultants, and support personnel). Those APCs who demonstrated a sufficient level of clinical competence functioned with a high level of autonomy. During the day, APCs were expected to be the first point of contact for interprofessional team members, to respond to acute clinical changes in a patient’s condition, and to discuss active issues with the supervising attending, all with the majority of medical decision-making, direct patient communication, documentation, and care coordination performed by the APC. An experienced subset of the APC service was responsible for overnight coverage. Nocturnist APCs independently managed all cross-cover issues on patients assigned to APC and hospitalist teams and performed admissions with very little to no direct supervision of the overnight attending physician.

Patient Admission and Redistribution Process

During the study period, resident teams performed all daytime admissions (6 am to 6 pm) on a rotating basis. On any given day, three of four resident teams performed daytime admissions with the fourth team designated as “golden” and free from admitting duties. Patients admitted during the day remained assigned to the resident team for continuity. The APC and hospitalist teams did not accept new admissions during the day. Nighttime admissions (6 pm to 6 am) were performed by a separate team composed of two senior residents, two interns, one APC, occasional APC and medical students, and one supervising attending hospitalist. This team functioned as a single unit. Nighttime admissions were performed in a sequential and rotating fashion (eg, Intern A > Intern B > Resident A > Resident B > APC > student(s) > Intern A > Intern B, etc). Patients admitted overnight were randomly redistributed the following morning, with the majority reassigned to an APC team or hospitalist team in order to offset the workload of the resident teams performing daytime admissions. Following redistribution, a patient would remain assigned to the daytime APC or hospitalist team for the duration of their hospitalization. The redistribution decisions were based on individual team census, without systematic consideration of an individual patient’s diagnosis, medical complexity, socioeconomic status, or perceived quality of learning potential (eg, good teaching case).

Study Outcomes

We divided study outcomes into two categories, clinical outcomes and resource utilization. Clinical outcomes included LOS, unplanned readmission within 30-days, and inpatient mortality and were designed to measure patient-related outcomes as a reflection of the quality of care delivered by different team structures. Resource utilization included discharge order time, discharge time, consults per admission, and total direct cost, which were designed to measure provider-related differences in efficiency and cost of care.

Statistical analysis

Baseline characteristics and unadjusted outcomes are reported as frequency and percent, normally distributed variables as mean with SD, and nonnormally distributed variables as median with interquartile range (IQR). Baseline characteristics and unadjusted outcomes were compared using the chi-square test or the t test, where appropriate. Multivariable regression analysis using generalized linear models with a log link function and gamma distribution was used for continuous outcomes. Multivariable logistic regression was used for binary outcomes.¹⁰ Covariates included in regression models were age, gender, CCI, transfer from an outside hospital, and nighttime admission. In a sensitivity analysis, we included MS-DRG weight as a covariate for 85% of hospitalizations in our cohort exclusive of observation stays, and our findings were qualitatively similar (data not reported but available on request). Adjusted continuous outcomes were estimated using marginal effects at the means.¹⁹ Due to the sensitivity of cost data and an institutional policy against disclosing cost figures, total direct costs were normalized using the unadjusted median and adjusted mean total direct cost of an admission to an APC team as the normalizing value. A P value cutoff of .05 was used to determine statistical significance. Stata/IC version 16.1 (StataCorp) was used for all analyses.

RESULTS

Study Population

A total of 12,716 hospital admissions were identified during the study period. Of these, 7,943 (62.5%) admissions were assigned to a resident team, 3,519 (27.7%) admissions were assigned to an APC team, and the remaining 1,254 (9.9%) were assigned to a hospitalist team. Baseline patient characteristics are reported in Table 1. Patients admitted to resident teams (mean age [SD], 56.9 [19.1] years) were younger than those admitted to an APC team (58.0 [19.3] years; P = .004) or a hospitalist team (58.2 [19.3] years; P = .026). The case-mix index (mean MS-DRG weight [SD], 1.44 [0.87]) was slightly lower for resident teams than that for APC teams (1.49 [0.90]; P = .025).Resident teams had a significantly lower proportion of night admissions than did APC teams (32.0% vs 49.5%; P < .001) and hospitalist teams (48.6%; P < .001). APC teams were assigned more patients transferred from an outside hospital (19.1%), compared with resident teams (15.0%; P < .001) and hospitalist teams (16.0%; P = .015). No other significant differences were observed in baseline characteristics between cohorts.

Clinical Outcomes

Unadjusted analysis demonstrated the LOS was similar among resident, APC, and hospitalist teams with a median (IQR) LOS of 2.90 (1.86, 4.26) days, 2.93 (1.89, 4.66) days, and 2.86 (1.84, 4.67) days, respectively. No significant differences were observed in unadjusted 30-day readmissions or inpatient mortality among the team structures (Table 2). Following multivariable adjustment for differences in baseline characteristics, no significant differences were observed in LOS, 30-day readmission, or inpatient mortality among teams (Table 3).

Resource Utilization

In unadjusted comparisons, hospitalist teams were observed to place discharge orders more than 30 minutes earlier than APC teams (median hours after midnight [IQR], 11.20 [9.63, 13.60] vs 11.73 [10.00, 13.87]; P < .001) and 54 minutes earlier than resident teams (12.10 [10.38, 13.90]; P < .001) (Table 2). Consistent with the earlier placement of discharge orders, hospitalist patients were also discharged from the hospital 26 and 32 minutes earlier than APC and resident patients, respectively. APC teams also discharged patients slightly earlier (6 minutes) than resident teams (median hours after midnight [IQR], 14.97 [13.23, 16.72] vs 15.07 [13.42, 16.73]; P = .045). Median consultation use among teams was similar, although statistically significant differences were present. Normalized total direct cost was 8% higher (P < .001) for admissions to APC teams than that for resident teams and 7% higher (P = .008) than that for hospitalist teams in unadjusted analysis (Table 2).

Following multivariable adjustment, the mean differences in discharge order time and discharge time remained significant with hospitalist teams discharging patients an average of 20 to 30 minutes earlier than APC and resident teams (Table 3). Consultant utilization remained significantly different between teams, with APC teams utilizing consultants on average 15% more than hospitalist teams (P < .001) and 7% more than resident teams (P = .001). The differences in total direct costs were not significant after adjusted analysis.

DISCUSSION

Many academic medical centers have expanded their workforce with APC or nonteaching hospitalist teams to accommodate the increasing volume of hospital admissions, resident work hour restrictions,^1,2 and medical complexity of an aging population. Several hospitals have reported comparative outcomes between different care delivery models, with conflicting results.^6,8,10-12 In our study, we directly evaluated three inpatient care delivery models and found that hospitalist teams discharged patients more efficiently and utilized fewer consultants, compared with APC and resident teams. In spite of this improved efficiency, no significant differences were observed in cost or other clinical outcomes.

Our findings are important and further strengthen the evidence supporting the use of APCs on inpatient general medicine services and are of particular interest to academic centers struggling to expand staffing in order to offset the growth in patient volume and reduction in resident workforce. We believe several findings from our study warrant further discussion.

First, although hospitalist teams were able to discharge patients more efficiently, this observation may be influenced by factors of workflow rather than caused by significant disparities in efficiency between provider types (ie, APC vs hospitalist vs resident physician). As with most academic centers, patients assigned to resident teams are presented by house staff to an attending physician who is ultimately responsible for patient care decisions. Therefore, it is conceivable that delays in the discharge process are in part related to the convention of bedside rounding and discussing the care plan prior to discharge.²⁰ In fact, we recognized this as a bottleneck and changed our discharge process for resident teams in June 2017, with a measurable improvement in discharge times. In the absence of this intervention, our observed differences in discharge times among teams may have been even greater.

Second, no significant differences in clinical outcomes were observed in our adjusted analyses, which suggests that a similar quality of care is delivered to patients regardless of team structure, an important observation when considering different staffing models.

Third, we observed a significant increase in consultation use among resident and APC teams, compared with hospitalists. While we are not able to precisely identify the basis for this variation, we believe it could reflect differences in clinical experience, comfort with diagnostic uncertainty, or the unequal distribution of patients transferred from outside hospitals for tertiary care. Interestingly, the greater consultation use did not correlate with higher healthcare costs, a finding recently reported by Stevens et al.²¹

Fourth, we believe the lack of differences in cost and clinical outcomes among team structures may be of particular interest to academic centers when considering physician burnout, salaries, and clinical education. The relationship between clerical burden, such as completing clinical documentation and computerized physician order entry, has been implicated as a risk factor for physician burnout.²² Incorporating APCs into roles similar to those performed by resident physicians may reduce the clerical burden on hospitalists, thereby reducing the risk of physician burnout. The addition of APCs may also represent opportunities for cost savings for healthcare centers when comparing the median salary of an APC to that of an internal medicine hospitalist.^23,24 Moreover, academic hospitalists have been shown to be excellent medical educators and report increased job satisfaction with a variety of duties beyond direct patient care.^24,25 Unforeseen benefits of adding APC teams within our institution has been the added teaching opportunities for APCs and APC students, increased collegiality with the APCs, and the creation of an APC fellowship program with a focus on inpatient medicine. Similar postgraduate training programs have been reported and serve as effective models to train APCs for hospital-based practice.²⁶

Lastly, although this project was conceived and completed prior to the COVID-19 pandemic, our observations may be informative for medical centers experiencing a workforce shortage caused by a surge of COVID-19 patients. During a physician shortage we believe our APC team model could be rapidly expanded to accommodate a large influx of patients. This expansion could be accomplished through a single attending physician overseeing multiple APC teams. In this model, the supervising physician would only evaluate the most complex patients with most patients being managed solely by an APC from admission to discharge. Such changes may require temporary suspension of state laws restricting APC independent practice.^27,28

Our findings contrast those of previous reports in that we did not observe significant differences in clinical outcomes (ie, LOS, inpatient mortality, and 30-day readmissions) or total direct cost.^8,10,21 Other institutions have noted an increased LOS among APC teams and hospitalist teams, compared with resident teams.^8,10 Furthermore, Chin et al and Iannuzzi et al reported reductions in healthcare cost for resident teams, whereas our study did not identify significant cost differences among team structures. Although we cannot pinpoint the exact reason(s) for these dissimilarities, it is plausible that unmeasured factors such as institutional differences in APC training, direct physician supervision, admission processes, or inpatient team census may play a role.

Several study limitations should be recognized. First, the retrospective, nonrandomized design is one of the largest limitations of our study. Administrative data was obtained via an electronic query of our data warehouse, and although we aimed to identify as many patient characteristics as possible to adjust for cofounding effects, undetected differences among cohorts may exist. Second, our inpatient admission process may have placed undue burden on resident teams to perform all daytime admissions, inadvertently affecting study outcomes. It is possible the observed benefits of a solo hospitalist team are attributable to the lack of admitting duties rather than inherent advantages of the team structure. If this were the case, we would expect similar benefits among APC teams, which we did not note. Third, the study was performed at a single academic center, which may limit the generalizability of our results. Fourth, it is possible the outcomes are similar among teams because our hospitalist faculty rotate proportionately between the different teams. Lastly, the study was underpowered to detect a significant difference in mortality between hospitalist and APC teams. A post hoc power calculation based on our observed sample and effect sizes estimated 75% power to detect a mortality difference between hospitalists and APCs; other mortality comparisons were adequately powered.

CONCLUSION

We observed similar total direct costs, LOS, 30-day readmission, and inpatient mortality between hospitalist, APC, and resident teams. APC and resident teams utilized more consultants and discharged patient later than hospitalists. Our analysis suggests clinical outcomes are not significantly affected by inpatient team structure, and the addition of general medicine inpatient APC or hospitalist teams represent safe and efficient alternatives to traditional resident teams within an academic medical center.

Disclosures

All authors declare they have no conflicts of interest.

The Accreditation Council for Graduate Medical Education (ACGME) first mandated residency work hour restrictions in 2003.¹ In 2011, revised work hour requirements were issued, further limiting the maximum duration of a shift and extending the duration of time off between scheduled shifts.² Academic medical centers have been forced to adapt to work hour restrictions, and cuts in funding to research and educational missions have pressured institutions to restructure with a greater focus on high-quality, lower-cost care.^3,4 In response, many academic hospitals have added hospitalist teams, or incorporated advanced practice clinicians (APCs) (nurse practitioners [NPs] and physician assistants [PAs]) to accommodate resident physician duty hour restrictions on their inpatient general medicine services.^5,6 More recently, the COVID-19 pandemic has created unanticipated physician shortages forcing medical centers to rapidly expand and broaden the scope of their existing APC workforce.⁷

Several comparisons of clinical outcomes, cost, and patient satisfaction between different combinations of hospitalist-based, resident-based, or APC-based inpatient teams have been reported with conflicting observations.^6,8-14 Roy et al reported no significant differences in mortality, length of stay (LOS), or readmissions between PA and resident teams.⁶ Timmermans et al reported similar cost-effectiveness, LOS, and quality of care between PA and physician teams that included a hybrid of attending only and resident teams.^13,14 Alternatively, Singh et al and Iannuzzi et al reported increased LOS among PA teams,^10,12 whereas Chin et al observed an increased LOS and reduced 30-day readmissions among hospitalist teams.⁸ While these observed differences may be attributable to heterogeneous patient populations or institution-specific team structure, the exact reasons remain unknown. Furthermore, understanding the value of alternate staffing models is essential for medical centers to prepare for potential COVID-19 related physician shortages. To our knowledge, no study to date has directly compared outcomes between resident, APC, and hospitalist team structures within an academic medical center.

We believe our institution provides a unique environment to study the differences in inpatient general medicine team structure with respect to quality and efficiency of care delivery. The objective of our study is to directly compare clinical outcomes and resource utilization among three distinct team structures: APC, resident, and solo hospitalist. We hypothesize that clinical outcomes, cost, and utilization of consult services will be similar across all team structures and hospitalist teams will discharge patients earlier than resident and APC teams.

METHODS

Study Design and Setting

We conducted a retrospective observational cohort study at the University of Utah Medical Center, a 548-bed academic medical center in Salt Lake City. An electronic database query was used to identify all patients discharged from the inpatient general internal medicine service between July 1, 2015, and July 1, 2018. Baseline patient characteristics were collected including age, gender, and Charlson comorbidity index (CCI).¹⁵ Case-mix index was determined for admissions where a Medicare Severity Diagnosis Related Group (MS-DRG) and corresponding weight was assigned.^16,17 Source of admission was collected to identify patients transferred from an outside hospital, typically due to increased medical complexity or need for specialty care not available at the referring center. Time of admission was collected to classify whether a patient was admitted during the day or at night. Length of stay was calculated as the difference between discharge date/time and admission date/time. Discharge order time was collected as a measure of clinician efficiency. The number of consults per admission was determined by the number of different medical or surgical subspecialty services that wrote at least one consultation or progress note after the time of admission and were not the primary service at the time the note was written. The project was reviewed and deemed exempt by the University of Utah Institutional Review Board (IRB 00104884).

Inpatient Care Team Structure

Patients were assigned to one of three cohorts dependent on the assigned treatment team at the time of discharge. The three inpatient team structures were as follows: (1) a “resident team” composed of a senior resident (postgraduate year [PGY] 2 or PGY3) and one to two medical students or one senior resident, two interns (PGY1), and one to two medical students supervised by a hospitalist physician; (2) an “APC team” composed of one to two APCs supervised by a hospitalist physician; and (3) a “hospitalist team” composed of one attending hospitalist independently managing all patients.

Advanced Practice Clinicians

The APC service included 10 APCs (8 PAs and 2 NPs), with a combined workforce of nine APC full-time equivalents during the study period. Their experience ranged from new graduate to 11 years of clinical experience, with an average of 4.2 years. Among the 6 APCs with prior clinical experience, the majority (86%) of their years of clinical experience were within inpatient medicine, oncology, or cardiology. Recognizing the variability in clinical experience, we employed a rigorous onboarding program that entailed an average of 80 hours of didactic sessions including 1:1 teaching of the inpatient Society of Hospital Medicine core lecture series combined with initial intense clinical oversight.¹⁸ This program ranged from 2 weeks to 6 weeks depending on the individual APC’s clinical experience, progress, and comfort working independently. This onboarding program has subsequently been formalized into a 1-year APC fellowship that began after the study period concluded.

The degree of autonomy for each APC was individualized based on their clinical experience and ability to recognize limitations such as medical decision-making, clinical knowledge, and effective use of interprofessional team members (eg, peers, nursing, ancillary staff, consultants, and support personnel). Those APCs who demonstrated a sufficient level of clinical competence functioned with a high level of autonomy. During the day, APCs were expected to be the first point of contact for interprofessional team members, to respond to acute clinical changes in a patient’s condition, and to discuss active issues with the supervising attending, all with the majority of medical decision-making, direct patient communication, documentation, and care coordination performed by the APC. An experienced subset of the APC service was responsible for overnight coverage. Nocturnist APCs independently managed all cross-cover issues on patients assigned to APC and hospitalist teams and performed admissions with very little to no direct supervision of the overnight attending physician.

Patient Admission and Redistribution Process

During the study period, resident teams performed all daytime admissions (6 am to 6 pm) on a rotating basis. On any given day, three of four resident teams performed daytime admissions with the fourth team designated as “golden” and free from admitting duties. Patients admitted during the day remained assigned to the resident team for continuity. The APC and hospitalist teams did not accept new admissions during the day. Nighttime admissions (6 pm to 6 am) were performed by a separate team composed of two senior residents, two interns, one APC, occasional APC and medical students, and one supervising attending hospitalist. This team functioned as a single unit. Nighttime admissions were performed in a sequential and rotating fashion (eg, Intern A > Intern B > Resident A > Resident B > APC > student(s) > Intern A > Intern B, etc). Patients admitted overnight were randomly redistributed the following morning, with the majority reassigned to an APC team or hospitalist team in order to offset the workload of the resident teams performing daytime admissions. Following redistribution, a patient would remain assigned to the daytime APC or hospitalist team for the duration of their hospitalization. The redistribution decisions were based on individual team census, without systematic consideration of an individual patient’s diagnosis, medical complexity, socioeconomic status, or perceived quality of learning potential (eg, good teaching case).

Study Outcomes

We divided study outcomes into two categories, clinical outcomes and resource utilization. Clinical outcomes included LOS, unplanned readmission within 30-days, and inpatient mortality and were designed to measure patient-related outcomes as a reflection of the quality of care delivered by different team structures. Resource utilization included discharge order time, discharge time, consults per admission, and total direct cost, which were designed to measure provider-related differences in efficiency and cost of care.

Statistical analysis

Baseline characteristics and unadjusted outcomes are reported as frequency and percent, normally distributed variables as mean with SD, and nonnormally distributed variables as median with interquartile range (IQR). Baseline characteristics and unadjusted outcomes were compared using the chi-square test or the t test, where appropriate. Multivariable regression analysis using generalized linear models with a log link function and gamma distribution was used for continuous outcomes. Multivariable logistic regression was used for binary outcomes.¹⁰ Covariates included in regression models were age, gender, CCI, transfer from an outside hospital, and nighttime admission. In a sensitivity analysis, we included MS-DRG weight as a covariate for 85% of hospitalizations in our cohort exclusive of observation stays, and our findings were qualitatively similar (data not reported but available on request). Adjusted continuous outcomes were estimated using marginal effects at the means.¹⁹ Due to the sensitivity of cost data and an institutional policy against disclosing cost figures, total direct costs were normalized using the unadjusted median and adjusted mean total direct cost of an admission to an APC team as the normalizing value. A P value cutoff of .05 was used to determine statistical significance. Stata/IC version 16.1 (StataCorp) was used for all analyses.

RESULTS

Study Population

A total of 12,716 hospital admissions were identified during the study period. Of these, 7,943 (62.5%) admissions were assigned to a resident team, 3,519 (27.7%) admissions were assigned to an APC team, and the remaining 1,254 (9.9%) were assigned to a hospitalist team. Baseline patient characteristics are reported in Table 1. Patients admitted to resident teams (mean age [SD], 56.9 [19.1] years) were younger than those admitted to an APC team (58.0 [19.3] years; P = .004) or a hospitalist team (58.2 [19.3] years; P = .026). The case-mix index (mean MS-DRG weight [SD], 1.44 [0.87]) was slightly lower for resident teams than that for APC teams (1.49 [0.90]; P = .025).Resident teams had a significantly lower proportion of night admissions than did APC teams (32.0% vs 49.5%; P < .001) and hospitalist teams (48.6%; P < .001). APC teams were assigned more patients transferred from an outside hospital (19.1%), compared with resident teams (15.0%; P < .001) and hospitalist teams (16.0%; P = .015). No other significant differences were observed in baseline characteristics between cohorts.

Clinical Outcomes

Unadjusted analysis demonstrated the LOS was similar among resident, APC, and hospitalist teams with a median (IQR) LOS of 2.90 (1.86, 4.26) days, 2.93 (1.89, 4.66) days, and 2.86 (1.84, 4.67) days, respectively. No significant differences were observed in unadjusted 30-day readmissions or inpatient mortality among the team structures (Table 2). Following multivariable adjustment for differences in baseline characteristics, no significant differences were observed in LOS, 30-day readmission, or inpatient mortality among teams (Table 3).

Resource Utilization

In unadjusted comparisons, hospitalist teams were observed to place discharge orders more than 30 minutes earlier than APC teams (median hours after midnight [IQR], 11.20 [9.63, 13.60] vs 11.73 [10.00, 13.87]; P < .001) and 54 minutes earlier than resident teams (12.10 [10.38, 13.90]; P < .001) (Table 2). Consistent with the earlier placement of discharge orders, hospitalist patients were also discharged from the hospital 26 and 32 minutes earlier than APC and resident patients, respectively. APC teams also discharged patients slightly earlier (6 minutes) than resident teams (median hours after midnight [IQR], 14.97 [13.23, 16.72] vs 15.07 [13.42, 16.73]; P = .045). Median consultation use among teams was similar, although statistically significant differences were present. Normalized total direct cost was 8% higher (P < .001) for admissions to APC teams than that for resident teams and 7% higher (P = .008) than that for hospitalist teams in unadjusted analysis (Table 2).

Following multivariable adjustment, the mean differences in discharge order time and discharge time remained significant with hospitalist teams discharging patients an average of 20 to 30 minutes earlier than APC and resident teams (Table 3). Consultant utilization remained significantly different between teams, with APC teams utilizing consultants on average 15% more than hospitalist teams (P < .001) and 7% more than resident teams (P = .001). The differences in total direct costs were not significant after adjusted analysis.

DISCUSSION

Many academic medical centers have expanded their workforce with APC or nonteaching hospitalist teams to accommodate the increasing volume of hospital admissions, resident work hour restrictions,^1,2 and medical complexity of an aging population. Several hospitals have reported comparative outcomes between different care delivery models, with conflicting results.^6,8,10-12 In our study, we directly evaluated three inpatient care delivery models and found that hospitalist teams discharged patients more efficiently and utilized fewer consultants, compared with APC and resident teams. In spite of this improved efficiency, no significant differences were observed in cost or other clinical outcomes.

Our findings are important and further strengthen the evidence supporting the use of APCs on inpatient general medicine services and are of particular interest to academic centers struggling to expand staffing in order to offset the growth in patient volume and reduction in resident workforce. We believe several findings from our study warrant further discussion.

First, although hospitalist teams were able to discharge patients more efficiently, this observation may be influenced by factors of workflow rather than caused by significant disparities in efficiency between provider types (ie, APC vs hospitalist vs resident physician). As with most academic centers, patients assigned to resident teams are presented by house staff to an attending physician who is ultimately responsible for patient care decisions. Therefore, it is conceivable that delays in the discharge process are in part related to the convention of bedside rounding and discussing the care plan prior to discharge.²⁰ In fact, we recognized this as a bottleneck and changed our discharge process for resident teams in June 2017, with a measurable improvement in discharge times. In the absence of this intervention, our observed differences in discharge times among teams may have been even greater.

Second, no significant differences in clinical outcomes were observed in our adjusted analyses, which suggests that a similar quality of care is delivered to patients regardless of team structure, an important observation when considering different staffing models.

Third, we observed a significant increase in consultation use among resident and APC teams, compared with hospitalists. While we are not able to precisely identify the basis for this variation, we believe it could reflect differences in clinical experience, comfort with diagnostic uncertainty, or the unequal distribution of patients transferred from outside hospitals for tertiary care. Interestingly, the greater consultation use did not correlate with higher healthcare costs, a finding recently reported by Stevens et al.²¹

Fourth, we believe the lack of differences in cost and clinical outcomes among team structures may be of particular interest to academic centers when considering physician burnout, salaries, and clinical education. The relationship between clerical burden, such as completing clinical documentation and computerized physician order entry, has been implicated as a risk factor for physician burnout.²² Incorporating APCs into roles similar to those performed by resident physicians may reduce the clerical burden on hospitalists, thereby reducing the risk of physician burnout. The addition of APCs may also represent opportunities for cost savings for healthcare centers when comparing the median salary of an APC to that of an internal medicine hospitalist.^23,24 Moreover, academic hospitalists have been shown to be excellent medical educators and report increased job satisfaction with a variety of duties beyond direct patient care.^24,25 Unforeseen benefits of adding APC teams within our institution has been the added teaching opportunities for APCs and APC students, increased collegiality with the APCs, and the creation of an APC fellowship program with a focus on inpatient medicine. Similar postgraduate training programs have been reported and serve as effective models to train APCs for hospital-based practice.²⁶

Lastly, although this project was conceived and completed prior to the COVID-19 pandemic, our observations may be informative for medical centers experiencing a workforce shortage caused by a surge of COVID-19 patients. During a physician shortage we believe our APC team model could be rapidly expanded to accommodate a large influx of patients. This expansion could be accomplished through a single attending physician overseeing multiple APC teams. In this model, the supervising physician would only evaluate the most complex patients with most patients being managed solely by an APC from admission to discharge. Such changes may require temporary suspension of state laws restricting APC independent practice.^27,28

Our findings contrast those of previous reports in that we did not observe significant differences in clinical outcomes (ie, LOS, inpatient mortality, and 30-day readmissions) or total direct cost.^8,10,21 Other institutions have noted an increased LOS among APC teams and hospitalist teams, compared with resident teams.^8,10 Furthermore, Chin et al and Iannuzzi et al reported reductions in healthcare cost for resident teams, whereas our study did not identify significant cost differences among team structures. Although we cannot pinpoint the exact reason(s) for these dissimilarities, it is plausible that unmeasured factors such as institutional differences in APC training, direct physician supervision, admission processes, or inpatient team census may play a role.

Several study limitations should be recognized. First, the retrospective, nonrandomized design is one of the largest limitations of our study. Administrative data was obtained via an electronic query of our data warehouse, and although we aimed to identify as many patient characteristics as possible to adjust for cofounding effects, undetected differences among cohorts may exist. Second, our inpatient admission process may have placed undue burden on resident teams to perform all daytime admissions, inadvertently affecting study outcomes. It is possible the observed benefits of a solo hospitalist team are attributable to the lack of admitting duties rather than inherent advantages of the team structure. If this were the case, we would expect similar benefits among APC teams, which we did not note. Third, the study was performed at a single academic center, which may limit the generalizability of our results. Fourth, it is possible the outcomes are similar among teams because our hospitalist faculty rotate proportionately between the different teams. Lastly, the study was underpowered to detect a significant difference in mortality between hospitalist and APC teams. A post hoc power calculation based on our observed sample and effect sizes estimated 75% power to detect a mortality difference between hospitalists and APCs; other mortality comparisons were adequately powered.

CONCLUSION

We observed similar total direct costs, LOS, 30-day readmission, and inpatient mortality between hospitalist, APC, and resident teams. APC and resident teams utilized more consultants and discharged patient later than hospitalists. Our analysis suggests clinical outcomes are not significantly affected by inpatient team structure, and the addition of general medicine inpatient APC or hospitalist teams represent safe and efficient alternatives to traditional resident teams within an academic medical center.

Disclosures

All authors declare they have no conflicts of interest.

Readmission rates are frequently used as a hospital quality metric, with use including payment incentive at the hospital level,¹ specific condition quality measurement,² balancing measures for quality improvement projects,^3-5 transition success,^6,7 and use in public hospital rankings.⁸ Currently, four methods are commonly used to evaluate pediatric readmissions, each with strengths and limitations, including the following (Appendix Table 1):

1. All-cause readmissions: A measure of any readmission within a given time period regardless of the reason for readmission.⁹

2. Unplanned readmission/time flag: A measure intended to identify unplanned readmissions. This measure relies on time designations within the electronic health record. The time between hospital registration and admission is calculated, and if the readmission is registered more than 24 hours prior to admission, the readmission is considered planned.¹⁰ Hereafter, this measure will be referred to as the time flag measure.

3. Pediatric all-condition readmission (PACR): A measure intended to identify unplanned readmission through the exclusion of certain procedures and diagnoses.¹¹

4. Potentially preventable readmission (PPR): A method to identify preventable readmissions based on a proprietary algorithm developed by 3M Health Information Systems.^12,13

While all four of these measures are used to assess quality, there is little known about these measures’ ability to exclude planned readmissions and identify only preventable pediatric readmission, which conceptually is most relevant to the quality of care. However, many of these measures were not intended to capture preventability, but instead capture the related issue of whether the readmission was planned. Therefore, we sought to evaluate the four readmission measures as they relate to both preventability and unplanned status as determined through medical record review with multidisciplinary care provider input.

METHODS

As part of a hospital-wide readmission reduction quality improvement collaborative at a free-standing tertiary care children’s hospital, clinicians from hospital medicine, cardiology, neonatology, and neurology teams reviewed 30-day readmissions using a standardized abstraction tool. All readmission events (observation or inpatient encounter) after any discharge (observation or inpatient encounter) from eligible units were reviewed; therefore, each hospitalization was a potential index hospitalization. We classified the preventability of each readmission with use of a previously described Likert scale with high interrater reliability.¹⁴ For these analyses, readmissions were considered preventable if the reviewing team rated them as either “more likely preventable” or “preventable in most circumstances.” Each readmission was also evaluated as planned or unplanned. Methods for readmission review and classification are in the Appendix.

We included all readmissions between July 2014 and June 2016. We compared the medical record review classifications with the assessments from each of the four measures of pediatric readmission. We calculated sensitivity and specificity for both outcomes (planned/unplanned and preventable/not preventable) for all four measures. For standardization of discussion, we categorized description of measure performance as “very poor” as less than 50%, “poor” between 50%-75%, “fair” as 75%-85%, “good” as 85%-90%, “very good” as 90%-95% and excellent as greater than 95%. We also calculated positive and negative predictive value (PPV and NPV) over plausible ranges of prevalence using the sensitivity and specificity of each comparison (Appendix).

Of note, certain exclusions are outlined by the PACR and PPR algorithms. The PACR evaluates only readmission events that occur in children younger than 18 years. The PPR algorithm does not assign preventability if either the index or readmission event is classified as an observation stay or if it is part of a larger chain of readmissions.

RESULTS

Among 30-day readmissions considered, 1,643 were eligible for medical record review; 1,125 reviews were completed by the clinical teams (68.5%). The median time to readmission was 7 days (interquartile range [IQR], 4-18). Most children were non-Hispanic White (71%) or Black (20%). The median age at hospitalization was 2.3 years (IQR 0.4-12.1). Most children had Medicaid (56%) or private (41%) insurance. Most of the reviews were performed in cardiology (43%) and hospital medicine (37%) with patients in neurology (13%) and neonatology (7%) constituting the remaining reviews. Uncontrolled advancement of chronic disease was the most common readmission category on medical record review (25.1%), followed by unrelated readmission (20.7%), scheduled readmission (20.4%), and progression of acute disease (16.6%) (Appendix Table 2).

Assessment of Preventable and Unplanned Readmissions

On multidisciplinary medical record review, most readmissions were classified as not preventable (84.5%). Specifically, 64% were not preventable and unplanned; 20% were deemed not preventable and planned. Only 15% were classified as unplanned and preventable and 1% as planned and preventable (Appendix Figure: Population A/B).

Matching Chart Review to the Four Algorithms

All 1,125 readmissions were assessed by the all-cause and time flag readmission measures (Appendix Figure: Population A/B). After applying algorithm exclusions (details in Appendix), only 804 of the 1,125 (71.5%) reviewed readmissions matched for PACR readmission comparison (Appendix Figure: Population C); 487 of the 1,125 (43.3%) of the reviewed readmissions matched for PPR comparison (Appendix Figure: Population D).

All-Cause

Because all-cause determines only if a readmission occurs, the measure is by definition 100% sensitive and 0% specific in both assessment of preventability and unplanned readmission (Table: Section A).

Time Flag

The time flag measure identified 80% (866/1,112) of the readmissions as unplanned. This measure had very good sensitivity but very poor specificity in identifying preventable readmissions, which corresponded to very poor PPV and good to excellent NPV. In terms of identifying unplanned readmissions, the time flag measure had excellent sensitivity and very good specificity, which corresponded to very good to excellent PPV and good to very good NPV (Table: Section B).

PACR

The PACR algorithm identified 75% (599/796) of readmissions as unplanned. The PACR has good sensitivity but very poor specificity in identifying preventable readmissions, which corresponded to very poor PPV and fair to very good NPV. In terms of identifying unplanned readmissions, the PACR had fair sensitivity but poor specificity, which corresponded to fair PPV and poor NPV (Table: Section C).

PPR

The PPR algorithm identified 53% (257/487) of admissions as potentially preventable. The PPR algorithm had poor sensitivity and specificity in identifying preventable readmissions, which corresponded to very poor PPV and fair to very good NPV. In terms of identifying unplanned readmissions, the PPR algorithm had poor sensitivity and fair specificity in identifying unplanned readmissions, which corresponded to fair to good PPV and very poor to poor NPV (Table: Section D).

Evaluation of Excluded Readmission Events

Because both the PACR and PPR had large numbers of algorithm exclusions, we describe the preventability and unplanned assessment of the excluded readmission events. Both algorithms excluded preventable events. Of the 321 readmissions excluded by the PACR algorithm, 13.4% were classified as preventable by chart review. Likewise, 14.9% of 638 readmissions excluded by PPR were classified as preventable by chart review.

DISCUSSION

The ability to accurately capture preventable pediatric readmission is a goal for hospital quality experts and health policymakers alike. Of the four commonly used readmission measures to assess readmission, only PPR is designed to focus on preventability. Unfortunately, none of these four measures is adequately sensitive or specific to identify preventable readmissions; all measures had very poor PPV for preventability. Of the four measures, the time flag measure had the best sensitivity, specificity, PPV, and NPV for identifying unplanned readmissions.

The overall percentage of unplanned readmissions identified by both the time flag and by PACR measures match the overall percentage of unplanned readmissions identified in chart review: The time flag measure identified 80% of admissions as unplanned versus 79% identified by chart review (Appendix Figure: Population A/B); PACR classified 75% as unplanned versus 81% identified by chart review for PACR-eligible readmissions (Appendix Figure: Population C). In contrast, the PPR algorithm classified many more readmissions as potentially preventable (53%) than were identified by chart review at only 16% (Appendix Figure: Population D). The PACR and PPR algorithms also exclude a significant number of readmissions that are unplanned and a smaller, but not trivial, number of readmissions that are preventable; these exclusions limit their accuracy.

The ability to apply these four measures in real time during a hospitalization varies by metric. Two of the measures, the all-cause and time flag, can be applied during a readmission event, which is appealing for quality improvement initiatives. These measures allow for notification of providers that a current hospitalization is a readmission event, which allows providers the opportunity to learn from these events as they occur (Appendix Table 1). While “unplanned” is not the same as “potentially preventable,” almost all potentially preventable readmissions are unplanned; therefore, accurately identifying unplanned readmissions is more beneficial than all-cause. Additionally, a low all-cause readmission rate can be indicative of poor access to scheduled procedures. Nevertheless, all-cause readmission is sometimes used to measure quality.^1,8 While the time flag measure may be more useful for quality improvement initiatives and hospital providers, it relies on hospital registration time, which is not widely available in administrative data sources and, therefore, has limited usefulness to policymakers.

Both PACR and PPR require administrative claims analysis, which is appealing from a policy standpoint. However, the reliance on claims data means the inclusion/exclusion of events can occur only retrospectively, which limits the usefulness of these measures in learning and intervening in real time. When the two measures are compared, PACR offers better sensitivity and PPR offers better specificity with regard to identifying unplanned readmission. The PPR software overcalls preventable readmissions, identifying more readmissions as preventable than there actually are. Nevertheless, Medicaid in several states uses PPR for payment incentive.^1,15-17 Given the poor performance of PPR in assessing both preventable and unplanned pediatric readmission, the use of this measure as a quality metric should be limited.

This study should be considered in the context of several limitations. Because the assessment of preventability was determined as part of a learning quality improvement collaborative and not as a planned research endeavor, not all readmission reviews were completed nor were other existent tools¹⁸ that allow for preventability assessment via more structured medical record review used. Second, we reviewed cases only from certain clinical services, which would limit generalizability of these findings to all pediatric admissions. However, given the low sensitivity and specificity of some of the metrics, we would not anticipate that the addition of other types of admissions would improve the sensitivity and specificity enough to ensure reliability. Third, while we relied on an established method to determine preventability, prior work has demonstrated that additional information gathered from families may change preventability.¹⁹ Finally, due to the exclusions required by the PPR and PACR algorithms, not all readmission events were reviewed. However, these exclusions reflect the actual specifications of use for both measures.

CONCLUSION

The PPR software has poor fidelity in identifying preventable and unplanned pediatric readmission; this finding has broad policy implications given how widely it is used by state Medicaid offices to assess financial penalties. Among the four pediatric readmission measures used, the time flag metric best identifies unplanned readmissions.

Disclosures

The authors have no conflicts of interest or financial relationships relevant to this article to disclose.

Funding

Dr Auger’s research is supported by a grant from the Agency for Healthcare Research and Quality (1K08HS204735-01A1). The project described was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health, under Award Number 5UL1TR001425-04. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

Readmission rates are frequently used as a hospital quality metric, with use including payment incentive at the hospital level,¹ specific condition quality measurement,² balancing measures for quality improvement projects,^3-5 transition success,^6,7 and use in public hospital rankings.⁸ Currently, four methods are commonly used to evaluate pediatric readmissions, each with strengths and limitations, including the following (Appendix Table 1):

1. All-cause readmissions: A measure of any readmission within a given time period regardless of the reason for readmission.⁹

2. Unplanned readmission/time flag: A measure intended to identify unplanned readmissions. This measure relies on time designations within the electronic health record. The time between hospital registration and admission is calculated, and if the readmission is registered more than 24 hours prior to admission, the readmission is considered planned.¹⁰ Hereafter, this measure will be referred to as the time flag measure.

3. Pediatric all-condition readmission (PACR): A measure intended to identify unplanned readmission through the exclusion of certain procedures and diagnoses.¹¹

4. Potentially preventable readmission (PPR): A method to identify preventable readmissions based on a proprietary algorithm developed by 3M Health Information Systems.^12,13

While all four of these measures are used to assess quality, there is little known about these measures’ ability to exclude planned readmissions and identify only preventable pediatric readmission, which conceptually is most relevant to the quality of care. However, many of these measures were not intended to capture preventability, but instead capture the related issue of whether the readmission was planned. Therefore, we sought to evaluate the four readmission measures as they relate to both preventability and unplanned status as determined through medical record review with multidisciplinary care provider input.

METHODS

As part of a hospital-wide readmission reduction quality improvement collaborative at a free-standing tertiary care children’s hospital, clinicians from hospital medicine, cardiology, neonatology, and neurology teams reviewed 30-day readmissions using a standardized abstraction tool. All readmission events (observation or inpatient encounter) after any discharge (observation or inpatient encounter) from eligible units were reviewed; therefore, each hospitalization was a potential index hospitalization. We classified the preventability of each readmission with use of a previously described Likert scale with high interrater reliability.¹⁴ For these analyses, readmissions were considered preventable if the reviewing team rated them as either “more likely preventable” or “preventable in most circumstances.” Each readmission was also evaluated as planned or unplanned. Methods for readmission review and classification are in the Appendix.

We included all readmissions between July 2014 and June 2016. We compared the medical record review classifications with the assessments from each of the four measures of pediatric readmission. We calculated sensitivity and specificity for both outcomes (planned/unplanned and preventable/not preventable) for all four measures. For standardization of discussion, we categorized description of measure performance as “very poor” as less than 50%, “poor” between 50%-75%, “fair” as 75%-85%, “good” as 85%-90%, “very good” as 90%-95% and excellent as greater than 95%. We also calculated positive and negative predictive value (PPV and NPV) over plausible ranges of prevalence using the sensitivity and specificity of each comparison (Appendix).

Of note, certain exclusions are outlined by the PACR and PPR algorithms. The PACR evaluates only readmission events that occur in children younger than 18 years. The PPR algorithm does not assign preventability if either the index or readmission event is classified as an observation stay or if it is part of a larger chain of readmissions.

RESULTS

Among 30-day readmissions considered, 1,643 were eligible for medical record review; 1,125 reviews were completed by the clinical teams (68.5%). The median time to readmission was 7 days (interquartile range [IQR], 4-18). Most children were non-Hispanic White (71%) or Black (20%). The median age at hospitalization was 2.3 years (IQR 0.4-12.1). Most children had Medicaid (56%) or private (41%) insurance. Most of the reviews were performed in cardiology (43%) and hospital medicine (37%) with patients in neurology (13%) and neonatology (7%) constituting the remaining reviews. Uncontrolled advancement of chronic disease was the most common readmission category on medical record review (25.1%), followed by unrelated readmission (20.7%), scheduled readmission (20.4%), and progression of acute disease (16.6%) (Appendix Table 2).

Assessment of Preventable and Unplanned Readmissions

On multidisciplinary medical record review, most readmissions were classified as not preventable (84.5%). Specifically, 64% were not preventable and unplanned; 20% were deemed not preventable and planned. Only 15% were classified as unplanned and preventable and 1% as planned and preventable (Appendix Figure: Population A/B).

Matching Chart Review to the Four Algorithms

All 1,125 readmissions were assessed by the all-cause and time flag readmission measures (Appendix Figure: Population A/B). After applying algorithm exclusions (details in Appendix), only 804 of the 1,125 (71.5%) reviewed readmissions matched for PACR readmission comparison (Appendix Figure: Population C); 487 of the 1,125 (43.3%) of the reviewed readmissions matched for PPR comparison (Appendix Figure: Population D).

All-Cause

Because all-cause determines only if a readmission occurs, the measure is by definition 100% sensitive and 0% specific in both assessment of preventability and unplanned readmission (Table: Section A).

Time Flag

The time flag measure identified 80% (866/1,112) of the readmissions as unplanned. This measure had very good sensitivity but very poor specificity in identifying preventable readmissions, which corresponded to very poor PPV and good to excellent NPV. In terms of identifying unplanned readmissions, the time flag measure had excellent sensitivity and very good specificity, which corresponded to very good to excellent PPV and good to very good NPV (Table: Section B).

PACR

The PACR algorithm identified 75% (599/796) of readmissions as unplanned. The PACR has good sensitivity but very poor specificity in identifying preventable readmissions, which corresponded to very poor PPV and fair to very good NPV. In terms of identifying unplanned readmissions, the PACR had fair sensitivity but poor specificity, which corresponded to fair PPV and poor NPV (Table: Section C).

PPR

The PPR algorithm identified 53% (257/487) of admissions as potentially preventable. The PPR algorithm had poor sensitivity and specificity in identifying preventable readmissions, which corresponded to very poor PPV and fair to very good NPV. In terms of identifying unplanned readmissions, the PPR algorithm had poor sensitivity and fair specificity in identifying unplanned readmissions, which corresponded to fair to good PPV and very poor to poor NPV (Table: Section D).

Evaluation of Excluded Readmission Events

Because both the PACR and PPR had large numbers of algorithm exclusions, we describe the preventability and unplanned assessment of the excluded readmission events. Both algorithms excluded preventable events. Of the 321 readmissions excluded by the PACR algorithm, 13.4% were classified as preventable by chart review. Likewise, 14.9% of 638 readmissions excluded by PPR were classified as preventable by chart review.

DISCUSSION

The ability to accurately capture preventable pediatric readmission is a goal for hospital quality experts and health policymakers alike. Of the four commonly used readmission measures to assess readmission, only PPR is designed to focus on preventability. Unfortunately, none of these four measures is adequately sensitive or specific to identify preventable readmissions; all measures had very poor PPV for preventability. Of the four measures, the time flag measure had the best sensitivity, specificity, PPV, and NPV for identifying unplanned readmissions.

The overall percentage of unplanned readmissions identified by both the time flag and by PACR measures match the overall percentage of unplanned readmissions identified in chart review: The time flag measure identified 80% of admissions as unplanned versus 79% identified by chart review (Appendix Figure: Population A/B); PACR classified 75% as unplanned versus 81% identified by chart review for PACR-eligible readmissions (Appendix Figure: Population C). In contrast, the PPR algorithm classified many more readmissions as potentially preventable (53%) than were identified by chart review at only 16% (Appendix Figure: Population D). The PACR and PPR algorithms also exclude a significant number of readmissions that are unplanned and a smaller, but not trivial, number of readmissions that are preventable; these exclusions limit their accuracy.

The ability to apply these four measures in real time during a hospitalization varies by metric. Two of the measures, the all-cause and time flag, can be applied during a readmission event, which is appealing for quality improvement initiatives. These measures allow for notification of providers that a current hospitalization is a readmission event, which allows providers the opportunity to learn from these events as they occur (Appendix Table 1). While “unplanned” is not the same as “potentially preventable,” almost all potentially preventable readmissions are unplanned; therefore, accurately identifying unplanned readmissions is more beneficial than all-cause. Additionally, a low all-cause readmission rate can be indicative of poor access to scheduled procedures. Nevertheless, all-cause readmission is sometimes used to measure quality.^1,8 While the time flag measure may be more useful for quality improvement initiatives and hospital providers, it relies on hospital registration time, which is not widely available in administrative data sources and, therefore, has limited usefulness to policymakers.

Both PACR and PPR require administrative claims analysis, which is appealing from a policy standpoint. However, the reliance on claims data means the inclusion/exclusion of events can occur only retrospectively, which limits the usefulness of these measures in learning and intervening in real time. When the two measures are compared, PACR offers better sensitivity and PPR offers better specificity with regard to identifying unplanned readmission. The PPR software overcalls preventable readmissions, identifying more readmissions as preventable than there actually are. Nevertheless, Medicaid in several states uses PPR for payment incentive.^1,15-17 Given the poor performance of PPR in assessing both preventable and unplanned pediatric readmission, the use of this measure as a quality metric should be limited.

This study should be considered in the context of several limitations. Because the assessment of preventability was determined as part of a learning quality improvement collaborative and not as a planned research endeavor, not all readmission reviews were completed nor were other existent tools¹⁸ that allow for preventability assessment via more structured medical record review used. Second, we reviewed cases only from certain clinical services, which would limit generalizability of these findings to all pediatric admissions. However, given the low sensitivity and specificity of some of the metrics, we would not anticipate that the addition of other types of admissions would improve the sensitivity and specificity enough to ensure reliability. Third, while we relied on an established method to determine preventability, prior work has demonstrated that additional information gathered from families may change preventability.¹⁹ Finally, due to the exclusions required by the PPR and PACR algorithms, not all readmission events were reviewed. However, these exclusions reflect the actual specifications of use for both measures.

CONCLUSION

The PPR software has poor fidelity in identifying preventable and unplanned pediatric readmission; this finding has broad policy implications given how widely it is used by state Medicaid offices to assess financial penalties. Among the four pediatric readmission measures used, the time flag metric best identifies unplanned readmissions.

Disclosures

The authors have no conflicts of interest or financial relationships relevant to this article to disclose.

Funding

Dr Auger’s research is supported by a grant from the Agency for Healthcare Research and Quality (1K08HS204735-01A1). The project described was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health, under Award Number 5UL1TR001425-04. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

Readmission rates are frequently used as a hospital quality metric, with use including payment incentive at the hospital level,¹ specific condition quality measurement,² balancing measures for quality improvement projects,^3-5 transition success,^6,7 and use in public hospital rankings.⁸ Currently, four methods are commonly used to evaluate pediatric readmissions, each with strengths and limitations, including the following (Appendix Table 1):

1. All-cause readmissions: A measure of any readmission within a given time period regardless of the reason for readmission.⁹

2. Unplanned readmission/time flag: A measure intended to identify unplanned readmissions. This measure relies on time designations within the electronic health record. The time between hospital registration and admission is calculated, and if the readmission is registered more than 24 hours prior to admission, the readmission is considered planned.¹⁰ Hereafter, this measure will be referred to as the time flag measure.

3. Pediatric all-condition readmission (PACR): A measure intended to identify unplanned readmission through the exclusion of certain procedures and diagnoses.¹¹

4. Potentially preventable readmission (PPR): A method to identify preventable readmissions based on a proprietary algorithm developed by 3M Health Information Systems.^12,13

While all four of these measures are used to assess quality, there is little known about these measures’ ability to exclude planned readmissions and identify only preventable pediatric readmission, which conceptually is most relevant to the quality of care. However, many of these measures were not intended to capture preventability, but instead capture the related issue of whether the readmission was planned. Therefore, we sought to evaluate the four readmission measures as they relate to both preventability and unplanned status as determined through medical record review with multidisciplinary care provider input.

METHODS

As part of a hospital-wide readmission reduction quality improvement collaborative at a free-standing tertiary care children’s hospital, clinicians from hospital medicine, cardiology, neonatology, and neurology teams reviewed 30-day readmissions using a standardized abstraction tool. All readmission events (observation or inpatient encounter) after any discharge (observation or inpatient encounter) from eligible units were reviewed; therefore, each hospitalization was a potential index hospitalization. We classified the preventability of each readmission with use of a previously described Likert scale with high interrater reliability.¹⁴ For these analyses, readmissions were considered preventable if the reviewing team rated them as either “more likely preventable” or “preventable in most circumstances.” Each readmission was also evaluated as planned or unplanned. Methods for readmission review and classification are in the Appendix.

We included all readmissions between July 2014 and June 2016. We compared the medical record review classifications with the assessments from each of the four measures of pediatric readmission. We calculated sensitivity and specificity for both outcomes (planned/unplanned and preventable/not preventable) for all four measures. For standardization of discussion, we categorized description of measure performance as “very poor” as less than 50%, “poor” between 50%-75%, “fair” as 75%-85%, “good” as 85%-90%, “very good” as 90%-95% and excellent as greater than 95%. We also calculated positive and negative predictive value (PPV and NPV) over plausible ranges of prevalence using the sensitivity and specificity of each comparison (Appendix).

Of note, certain exclusions are outlined by the PACR and PPR algorithms. The PACR evaluates only readmission events that occur in children younger than 18 years. The PPR algorithm does not assign preventability if either the index or readmission event is classified as an observation stay or if it is part of a larger chain of readmissions.

RESULTS

Among 30-day readmissions considered, 1,643 were eligible for medical record review; 1,125 reviews were completed by the clinical teams (68.5%). The median time to readmission was 7 days (interquartile range [IQR], 4-18). Most children were non-Hispanic White (71%) or Black (20%). The median age at hospitalization was 2.3 years (IQR 0.4-12.1). Most children had Medicaid (56%) or private (41%) insurance. Most of the reviews were performed in cardiology (43%) and hospital medicine (37%) with patients in neurology (13%) and neonatology (7%) constituting the remaining reviews. Uncontrolled advancement of chronic disease was the most common readmission category on medical record review (25.1%), followed by unrelated readmission (20.7%), scheduled readmission (20.4%), and progression of acute disease (16.6%) (Appendix Table 2).

Assessment of Preventable and Unplanned Readmissions

On multidisciplinary medical record review, most readmissions were classified as not preventable (84.5%). Specifically, 64% were not preventable and unplanned; 20% were deemed not preventable and planned. Only 15% were classified as unplanned and preventable and 1% as planned and preventable (Appendix Figure: Population A/B).

Matching Chart Review to the Four Algorithms

All 1,125 readmissions were assessed by the all-cause and time flag readmission measures (Appendix Figure: Population A/B). After applying algorithm exclusions (details in Appendix), only 804 of the 1,125 (71.5%) reviewed readmissions matched for PACR readmission comparison (Appendix Figure: Population C); 487 of the 1,125 (43.3%) of the reviewed readmissions matched for PPR comparison (Appendix Figure: Population D).

All-Cause

Because all-cause determines only if a readmission occurs, the measure is by definition 100% sensitive and 0% specific in both assessment of preventability and unplanned readmission (Table: Section A).

Time Flag

The time flag measure identified 80% (866/1,112) of the readmissions as unplanned. This measure had very good sensitivity but very poor specificity in identifying preventable readmissions, which corresponded to very poor PPV and good to excellent NPV. In terms of identifying unplanned readmissions, the time flag measure had excellent sensitivity and very good specificity, which corresponded to very good to excellent PPV and good to very good NPV (Table: Section B).

PACR

The PACR algorithm identified 75% (599/796) of readmissions as unplanned. The PACR has good sensitivity but very poor specificity in identifying preventable readmissions, which corresponded to very poor PPV and fair to very good NPV. In terms of identifying unplanned readmissions, the PACR had fair sensitivity but poor specificity, which corresponded to fair PPV and poor NPV (Table: Section C).

PPR

The PPR algorithm identified 53% (257/487) of admissions as potentially preventable. The PPR algorithm had poor sensitivity and specificity in identifying preventable readmissions, which corresponded to very poor PPV and fair to very good NPV. In terms of identifying unplanned readmissions, the PPR algorithm had poor sensitivity and fair specificity in identifying unplanned readmissions, which corresponded to fair to good PPV and very poor to poor NPV (Table: Section D).

Evaluation of Excluded Readmission Events

Because both the PACR and PPR had large numbers of algorithm exclusions, we describe the preventability and unplanned assessment of the excluded readmission events. Both algorithms excluded preventable events. Of the 321 readmissions excluded by the PACR algorithm, 13.4% were classified as preventable by chart review. Likewise, 14.9% of 638 readmissions excluded by PPR were classified as preventable by chart review.

DISCUSSION

The ability to accurately capture preventable pediatric readmission is a goal for hospital quality experts and health policymakers alike. Of the four commonly used readmission measures to assess readmission, only PPR is designed to focus on preventability. Unfortunately, none of these four measures is adequately sensitive or specific to identify preventable readmissions; all measures had very poor PPV for preventability. Of the four measures, the time flag measure had the best sensitivity, specificity, PPV, and NPV for identifying unplanned readmissions.

The overall percentage of unplanned readmissions identified by both the time flag and by PACR measures match the overall percentage of unplanned readmissions identified in chart review: The time flag measure identified 80% of admissions as unplanned versus 79% identified by chart review (Appendix Figure: Population A/B); PACR classified 75% as unplanned versus 81% identified by chart review for PACR-eligible readmissions (Appendix Figure: Population C). In contrast, the PPR algorithm classified many more readmissions as potentially preventable (53%) than were identified by chart review at only 16% (Appendix Figure: Population D). The PACR and PPR algorithms also exclude a significant number of readmissions that are unplanned and a smaller, but not trivial, number of readmissions that are preventable; these exclusions limit their accuracy.

The ability to apply these four measures in real time during a hospitalization varies by metric. Two of the measures, the all-cause and time flag, can be applied during a readmission event, which is appealing for quality improvement initiatives. These measures allow for notification of providers that a current hospitalization is a readmission event, which allows providers the opportunity to learn from these events as they occur (Appendix Table 1). While “unplanned” is not the same as “potentially preventable,” almost all potentially preventable readmissions are unplanned; therefore, accurately identifying unplanned readmissions is more beneficial than all-cause. Additionally, a low all-cause readmission rate can be indicative of poor access to scheduled procedures. Nevertheless, all-cause readmission is sometimes used to measure quality.^1,8 While the time flag measure may be more useful for quality improvement initiatives and hospital providers, it relies on hospital registration time, which is not widely available in administrative data sources and, therefore, has limited usefulness to policymakers.

Both PACR and PPR require administrative claims analysis, which is appealing from a policy standpoint. However, the reliance on claims data means the inclusion/exclusion of events can occur only retrospectively, which limits the usefulness of these measures in learning and intervening in real time. When the two measures are compared, PACR offers better sensitivity and PPR offers better specificity with regard to identifying unplanned readmission. The PPR software overcalls preventable readmissions, identifying more readmissions as preventable than there actually are. Nevertheless, Medicaid in several states uses PPR for payment incentive.^1,15-17 Given the poor performance of PPR in assessing both preventable and unplanned pediatric readmission, the use of this measure as a quality metric should be limited.

This study should be considered in the context of several limitations. Because the assessment of preventability was determined as part of a learning quality improvement collaborative and not as a planned research endeavor, not all readmission reviews were completed nor were other existent tools¹⁸ that allow for preventability assessment via more structured medical record review used. Second, we reviewed cases only from certain clinical services, which would limit generalizability of these findings to all pediatric admissions. However, given the low sensitivity and specificity of some of the metrics, we would not anticipate that the addition of other types of admissions would improve the sensitivity and specificity enough to ensure reliability. Third, while we relied on an established method to determine preventability, prior work has demonstrated that additional information gathered from families may change preventability.¹⁹ Finally, due to the exclusions required by the PPR and PACR algorithms, not all readmission events were reviewed. However, these exclusions reflect the actual specifications of use for both measures.

CONCLUSION

The PPR software has poor fidelity in identifying preventable and unplanned pediatric readmission; this finding has broad policy implications given how widely it is used by state Medicaid offices to assess financial penalties. Among the four pediatric readmission measures used, the time flag metric best identifies unplanned readmissions.

Disclosures

The authors have no conflicts of interest or financial relationships relevant to this article to disclose.

Funding

Dr Auger’s research is supported by a grant from the Agency for Healthcare Research and Quality (1K08HS204735-01A1). The project described was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health, under Award Number 5UL1TR001425-04. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

Discharges against medical advice (DAMAs), in which a patient leaves the hospital prior to a physician-recommended endpoint, represent approximately 1% to 2% of inpatient discharges in the United States.¹ When compared with routine discharges, a DAMA is associated with adverse clinical consequences, including an increased risk of all-cause mortality.^2,3 Additionally, due to incomplete care, a DAMA may result in increased healthcare resource utilization (HcRU), including the use of inpatient, emergency department (ED), and outpatient services in the postdischarge period. Quantifying these relationships can provide important information regarding an individual’s healthcare-seeking behavior following a DAMA.

Prior literature has focused on the association between a DAMA and the risk of inpatient readmission. Relative to routine discharges, a DAMA is associated with a 1.5 to 2 times increased risk of a 30-day readmission.^3-9 However, these estimates are based on mixed-payer populations primarily composed (65%-80%) of individuals with public (Medicaid, Medicare) or no insurance. Further, they do not differentiate this association by payer type. It is unclear if prior results apply to commercially insured adults. These individuals represent a small but nonnegligible proportion (19%) of all DAMAs in the United States.¹⁰ Quantifying relationships among commercially insured adults can help advance our understanding of readmission patterns in the DAMA population.

There is limited evidence regarding the relationship between a DAMA and outpatient HcRU in the postdischarge period. Use of ED services after a DAMA has been explored only in specific disease populations such as asthma.⁴ Additionally, prior studies have reported a reduced frequency in the receipt of medication prescriptions and outpatient follow-up plans among individuals with a DAMA at the time of discharge.^11,12 Whether these practices translate to altered patterns of postdischarge prescription drug fills or use of outpatient services is not known.

To address these substantive gaps in the literature, the present study evaluates the association between a DAMA and all-cause HcRU in the postdischarge period among commercially insured adults. We examined HcRU across all points of service including inpatient readmissions, ED visits, physician office visits, nonphysician outpatient encounters, and prescription drug fills. These results can serve as a benchmark for comparison to future studies on DAMAs among publicly insured or uninsured individuals. Furthermore, such knowledge can help providers, payers, and policy planners make evidence-based decisions regarding postdischarge healthcare delivery.

METHODS

Data Source

This retrospective study used a 10% random sample of enrollees in the IQVIA PharMetrics^® Plus database (purchased by University of Maryland, Baltimore, under license from IQVIA). The database is composed of fully adjudicated claims and enrollment information from over 70 contributing US health plans and self-insured employer groups for over 140 million unique enrollees from 2006 onward. The enrollee population is generally representative of the commercially insured population that is younger than 65 years of age (with a subset of commercial Medicare and Medicaid) with respect to age and gender.

The database allows longitudinal follow-up for individuals using three files: medical claims, pharmacy claims, and insurance eligibility. The average length of enrollment is 39 months. The claims data represent payments to providers for services rendered to individuals covered by health plans. The medical claims file contains information on diagnostic and therapeutic services rendered in the inpatient and outpatient settings. The pharmacy claims file captures data on prescription drugs dispensed in retail and mail-order settings. The eligibility file contains demographic and insurance eligibility information for individuals.

Study Population

We identified all individuals aged 18 to 64 years with an inpatient admission record between January 1, 2007, and December 31, 2015. All individuals with continuous medical and prescription drug coverage from 6 months prior to the hospital admission date (baseline period) through 30 days following the discharge date (follow-up period) were included. Inpatient admissions with a missing discharge disposition or those that resulted in in-hospital death, discharge to a short-term hospital, skilled nursing facility, intermediate care facility, or any other type of facility were not considered for analysis. Only the first eligible inpatient admission was considered for analysis.

Main Predictor Variable

Individuals with a DAMA were analyzed as the case group. A DAMA was identified using the “Patient Status Code” variable, which represents the discharge disposition of each individual. Individuals who were discharged to home/self-care or discharged to a home health organization formed the control group (hereafter referred to as routine discharge).

Demographic, Clinical, and Hospitalization Characteristics

An individual’s age, sex, and region of residence were determined at the date of hospital admission. The Elixhauser algorithm was used to categorize comorbid conditions (as scores of 0, 1-2, ≥3 depending on number of comorbidities) based on International Classification of Diseases, Ninth Revision, Clinical Modification, diagnosis codes during the baseline period.^13,14 The following characteristics of each individual’s eligible inpatient admission were captured: year, timing (weekday or weekend), length of stay (LOS, measured in days), and receipt of a surgical procedure.

Outcomes

All-cause HcRU was identified during the 30-day postdischarge period. Specifically, we identified inpatient readmissions, ED visits, physician office visits, nonphysician outpatient encounters (for example, pathology, radiology, outpatient surgical services), and prescription drug fills. Binary variables (yes or no) were created for inpatient readmissions and ED visits while the remaining HcRU categories (ie, physician office visits, nonphysician outpatient encounters, and prescription drug fills) were analyzed as count variables. In the sensitivity analyses, we provide results for HcRU outcomes among a subgroup of individuals who had at least 90 days of continuous medical and prescription drug benefits following the hospital discharge.

Statistical Analysis

Descriptive Analysis

Measures of interest were reported using summary statistics depending on the nature of the variable. Continuous variables were described using t tests, and categorical variables were described using chi-square tests.

Propensity Score Matching

Cases and controls were matched using a 1:1 greedy matching algorithm based on propensity scores.¹⁵ We developed propensity scores based on confounders that we hypothesized would be associated with a DAMA and postdischarge HcRU. The propensity score model included the following variables: age, sex, region of residence, Elixhauser comorbidity index score, year of admission, timing of admission, LOS, and presence of any surgical procedure during the inpatient admission. The best match between cases and controls was determined based on the absolute difference in their propensity scores, which allowed for a maximal caliper width of 0.2 of the standard deviation of the logit of the propensity score.¹⁶ A standardized difference value of less than 0.1 was used to assess balance in baseline patient and hospital characteristics between cases and controls consistent with prior literature.^17,18 Proportions and balance, as measured by standardized differences between baseline covariates across cases and controls in the matched sample, are displayed in tabular format (Appendix Table 1).

Healthcare Resource Utilization

We estimated the adjusted odds ratio (AOR) using a logistic regression model. The AOR quantified the association between a DAMA and the prevalence of all-cause inpatient readmissions and ED visits during the 30-day postdischarge period. We estimated incident rate ratios (IRR) for count outcomes. Given the large number of individuals with no physician office visits, nonphysician outpatient encounters, or prescription drug fills, we estimated model parameters for IRRs using a finite mixture negative binomial hurdle model.¹⁹ We considered the data to represent a mixture of a constant distribution (which always generates zero counts) and a zero-truncated distribution (which always generates nonzero counts). The finite mixture count models include two outcomes: the mixing probabilities and the count distribution. The mixing probabilities quantify the probability that an observation for the HcRU category will be drawn from either the constant distribution (with mass at zero) or the count distribution. Conditional on having positive values, a zero-truncated generalized linear model (GLM) governs the count variable. Compared with other GLM specifications (eg, Poisson, negative binomial, zero-inflated), the negative binomial hurdle model presented the best-fitting model across several information criteria statistics (Appendix Figures 1-3 and Appendix Tables 2-4).

The GLM results provided IRR for the counts of HcRU. Ratios were interpreted as evidence of increased HcRU (IRR ≥ 1.0) or decreased HcRU (IRR < 1.0) among individuals with a DAMA compared with those discharged routinely. For all HcRU analyses, we reported results for the matched sample. All analyses were conducted using SAS version 9.4 (SAS Institute), and statistical significance was determined at α= .05. The study received the University of Maryland, Baltimore, Institutional Review Board approval (HP-00081497).

RESULTS

The unmatched sample included 457,530 individuals, of whom 0.5% had a DAMA. A consort diagram illustrating cohort inclusion and exclusion criteria is presented in Appendix Figure 4. Demographic, clinical, and inpatient admission characteristics of the unmatched sample and for subgroups defined by discharge status are displayed in Table 1. In the unmatched sample, the median age at admission was higher for individuals with a DAMA than it was for those discharged routinely (43 vs 42 years, respectively), and the proportion of males was higher among those with a DAMA (58.4% vs 33.1%). There were statistically significant differences based on the geographic region of residence and the comorbidity burden across both groups. The median LOS was shorter (1 day vs 2 days), the proportion of weekend admissions was higher (22.2% vs 16.3%), and the proportion of inpatient surgical procedures was lower (12.9% vs 59.2%) among those with a DAMA compared with that among those with routine discharges. The propensity score-matched sample included 2,245 cases and 2,245 controls (Appendix Table 1). Standardized differences for all baseline factors were less than 0.1, indicating that cases and controls were matched on the included baseline factors.

Summary Statistics: Proportions and Counts

Across the DAMA and routine discharge groups, the proportion of individuals with a 30-day inpatient readmission was similar (19.5% vs 18.7%; P = .47), whereas the proportion with an ED visit was higher (18.6% vs 9.1%; P < .01). There were no differences in the median number of inpatient readmissions (median, 0) and ED visits (median, 0) across both groups. Individuals with a DAMA and those discharged routinely displayed similar median counts of 30-day physician office (median, 1) and nonphysician outpatient encounters (median, 1) (Table 2). Individuals with a DAMA displayed a lower median number of prescription drug fills (median, 2 vs 3) than that among those with a routine discharge (Table 2).

Main Analysis: Thirty-Day Healthcare Resource Utilization

The associations between a DAMA and 30-day inpatient readmissions and ED visits based on the matched sample are presented in Table 3. Individuals with a DAMA had increased odds for an ED visit (AOR, 2.28; 95% CI, 1.90-2.72) but no significant difference in the odds of a 30-day inpatient readmission (AOR, 1.06; 95% CI, 0.91-1.23) compared with those discharged routinely.

The association between a DAMA and count HcRU outcomes is presented in Table 4. Compared with those discharged routinely, individuals with a DAMA displayed no significant difference in rates for physician office visits (IRR, 1.01; 95% CI, 0.91-1.11), nonphysician outpatient encounters (IRR, 0.89; 95% CI, 0.78-1.00), and prescription drug fills (IRR, 1.03; 95% CI, 0.97-1.09) during the 30-day postdischarge period.

Sensitivity Analysis: Ninety-Day Healthcare Resource Utilization

Relative to those discharged routinely, individuals with a DAMA had statistically significant increased odds of 90-day inpatient readmissions (AOR, 1.18; 95% CI, 1.02-1.36), odds of ED visits (AOR, 2.16; 95% CI, 1.85-2.51), and rates of prescription drug fills (IRR, 1.32; 95% CI, 1.29-1.35). No statistically significant differences were observed in the rates of physician office visits and nonphysician outpatient encounters across both groups.

DISCUSSION

In this commercially insured sample of working age individuals, we identified an association between a DAMA and the likelihood and intensity of postdischarge HcRU. The direction of the association varied across categories of HcRU and the duration of follow-up. A DAMA was associated with increased odds of 30-day ED visits but not 30-day readmissions compared with routine discharges. No significant differences were observed in the rates of 30-day physician office visits, nonphysician outpatient encounters, and prescription drug fills across both groups. To our knowledge, this is the first study on DAMAs that examines postdischarge HcRU outside the inpatient setting.

The 0.5% prevalence of DAMAs in our study was lower than the approximate 1% to 2% value that is typically reported in the literature. Prior studies have typically reported results based on mixed-payer populations.^3-10 These mixed-payer populations include publicly insured (Medicare or Medicaid) or uninsured stays, which account for a disproportionate share of all DAMAs. In contrast, commercially insured stays account for the lowest proportion of all DAMAs.¹⁰ Similar to prior literature,⁵ the DAMA group in our study was younger, had a higher proportion of males, had a higher comorbidity burden, and had a shorter LOS than the routinely discharged group.

We observed a greater likelihood of ED utilization after a DAMA. Similar findings have been reported, which may indicate that patients with a DAMA receive inadequate treatment at the time of discharge and may require further acute treatment. For example, a prior study reported that, after a DAMA, individuals with asthma were four times more likely to have an ED visit within 14 days compared with those discharged routinely.⁴

Contrary to prior findings,^3-9 we found no significant difference in the odds of a 30-day inpatient readmission across the DAMA and routine discharge groups, which may be attributable to differences in the populations studied. Those previous studies used mixed payer populations and did not differentiate results by payer type. The mixed payer populations in these studies were older (mean ages, 55 years and above) and had an increased comorbidity burden compared with our commercially insured population. Furthermore, some of these studies were either limited to single sites,⁸ single state hospital systems,^3,4,9 or focused on specific medical populations.^3,4,6-9 Our national sample of commercially insured adults is considerably younger, with a mean age of 43 years. Thirty days may be too brief to observe enough inpatient readmissions for the purpose of comparative analyses. This is suggested by our results, which indicated that there is an association between DAMA and 90-day inpatient readmission. Additionally, nonsignificant findings for 30-day inpatient readmissions may also be due to the small sample size of the DAMA group in our study, which may have limited robust statistical inference. Future studies in a larger population of commercially insured individuals with a DAMA are required to confirm these findings.

Nonsignificant differences in the rates of 30-day physician office visits, nonphysician outpatient encounters, and prescription drug fills across both groups may explain the null association with 30-day inpatient readmissions. Prior literature on specific medical populations or individuals with general hospital admissions report that early outpatient follow-up can help prevent 30-day readmissions.^20-25 In our sample, we observed similar rates of outpatient follow-up across the DAMA and routinely discharged groups. Prior studies based on single hospital sites have reported that, at the time of discharge, a lower proportion of individuals with a DAMA received medication prescriptions and outpatient follow-up plans compared with those discharged routinely.^11,12 In contrast, we evaluated prescription drug fills and outpatient visits during the postdischarge period, which may explain the difference in findings.

The present study has several strengths. To the best of our knowledge, our study represents the first and largest retrospective analysis of DAMAs in a national sample of commercially insured adults. In addition to a large generalizable sample, we examine HcRU after a DAMA across major points of service over a longitudinal postdischarge period. Our results provide a comprehensive understanding of utilization outcomes in this population including those outside the inpatient setting, which has been the focus of prior literature. These findings can help guide the implementation of appropriate patient- and system-level interventions to optimize DAMA prevention and mitigate the associated utilization burden on the healthcare system in the postdischarge period.^26,27

Our findings should be interpreted with certain limitations in mind. First, this study used data based on a commercially insured sample of patients and may not be generalizable to publicly insured or uninsured samples. Second, like prior DAMA studies that used the Nationwide Readmissions Database instead,^5-7 our study was unable to account for individual-level factors such as race, marital status, family social support, income, health literacy, and activation in self-care. Further, given the limitations of our data, we were unable to control for hospital characteristics such as bed size, urban-rural designation, teaching status, and control (eg, private or government ownership). Despite the use of propensity score methods to balance both comparison groups on observable sources of confounding, we cannot rule out the possibility of residual confounding. Lastly, due to a lack of data on postdischarge mortality outcomes, we could not control for competing risk of death in our analysis. However, in a population with an average age of 43 years, we did not expect high or differential 30- or 90-day postdischarge mortality rates across both groups.

Our findings suggest several important directions for future research. First, it will be useful to examine these associations among publicly insured and uninsured samples in which a DAMA is more prevalent and in which the associations with HcRU may be more pronounced than they are in the commercially insured population. Secondly, future research should identify subgroups of DAMA patients with an increased propensity for postdischarge HcRU. This can help in the design of individualized outpatient follow-up plans that address patient-specific medical and social needs. Finally, our findings highlight the need for education, practice guidelines, and suitable interventions to help providers in the prevention and management of a DAMA.

CONCLUSION

Using data from a commercially insured population, we identified associations between a DAMA and postdischarge HcRU. The associations differed by category of HcRU. We identified a positive association with the likelihood of ED utilization but no association with the likelihood of 30-day inpatient readmission or general outpatient utilization. Our results indicate that the examination of inpatient readmissions after a DAMA should not be considered in isolation. The identification of the full range of outpatient and inpatient HcRU after a DAMA in a broad population of patients can improve our understanding of outcomes following a DAMA and support appropriate system-level interventions designed to reduce their prevalence.

Acknowledgments

The statements, findings, conclusions, views, and opinions contained and expressed in this manuscript are based in part on data obtained under license from IQVIA. Source: IQVIA PharMetrics^® Plus January 2006 – December 2015, IQVIA. All Rights Reserved. The statements, findings, conclusions, views, and opinions contained and expressed herein are not necessarily those of IQVIA or any of its affiliated or subsidiary entities.

Disclosures

Dr Onukwugha reports grants from Bayer Healthcare Pharmaceuticals, grants from Pfizer, Inc, and personal fees from Novo Nordisk outside the submitted work. The other authors have nothing to disclose. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the US Department of Veterans Affairs, the U.S. Government, or the VA National Center for Ethics in Health Care.

Funding

The authors acknowledge the support of the University of Maryland, Baltimore Institute for Clinical & Translational Research (ICTR) through the ICTR Voucher Program.

Discharges against medical advice (DAMAs), in which a patient leaves the hospital prior to a physician-recommended endpoint, represent approximately 1% to 2% of inpatient discharges in the United States.¹ When compared with routine discharges, a DAMA is associated with adverse clinical consequences, including an increased risk of all-cause mortality.^2,3 Additionally, due to incomplete care, a DAMA may result in increased healthcare resource utilization (HcRU), including the use of inpatient, emergency department (ED), and outpatient services in the postdischarge period. Quantifying these relationships can provide important information regarding an individual’s healthcare-seeking behavior following a DAMA.

Prior literature has focused on the association between a DAMA and the risk of inpatient readmission. Relative to routine discharges, a DAMA is associated with a 1.5 to 2 times increased risk of a 30-day readmission.^3-9 However, these estimates are based on mixed-payer populations primarily composed (65%-80%) of individuals with public (Medicaid, Medicare) or no insurance. Further, they do not differentiate this association by payer type. It is unclear if prior results apply to commercially insured adults. These individuals represent a small but nonnegligible proportion (19%) of all DAMAs in the United States.¹⁰ Quantifying relationships among commercially insured adults can help advance our understanding of readmission patterns in the DAMA population.

There is limited evidence regarding the relationship between a DAMA and outpatient HcRU in the postdischarge period. Use of ED services after a DAMA has been explored only in specific disease populations such as asthma.⁴ Additionally, prior studies have reported a reduced frequency in the receipt of medication prescriptions and outpatient follow-up plans among individuals with a DAMA at the time of discharge.^11,12 Whether these practices translate to altered patterns of postdischarge prescription drug fills or use of outpatient services is not known.

To address these substantive gaps in the literature, the present study evaluates the association between a DAMA and all-cause HcRU in the postdischarge period among commercially insured adults. We examined HcRU across all points of service including inpatient readmissions, ED visits, physician office visits, nonphysician outpatient encounters, and prescription drug fills. These results can serve as a benchmark for comparison to future studies on DAMAs among publicly insured or uninsured individuals. Furthermore, such knowledge can help providers, payers, and policy planners make evidence-based decisions regarding postdischarge healthcare delivery.

METHODS

Data Source

This retrospective study used a 10% random sample of enrollees in the IQVIA PharMetrics^® Plus database (purchased by University of Maryland, Baltimore, under license from IQVIA). The database is composed of fully adjudicated claims and enrollment information from over 70 contributing US health plans and self-insured employer groups for over 140 million unique enrollees from 2006 onward. The enrollee population is generally representative of the commercially insured population that is younger than 65 years of age (with a subset of commercial Medicare and Medicaid) with respect to age and gender.

The database allows longitudinal follow-up for individuals using three files: medical claims, pharmacy claims, and insurance eligibility. The average length of enrollment is 39 months. The claims data represent payments to providers for services rendered to individuals covered by health plans. The medical claims file contains information on diagnostic and therapeutic services rendered in the inpatient and outpatient settings. The pharmacy claims file captures data on prescription drugs dispensed in retail and mail-order settings. The eligibility file contains demographic and insurance eligibility information for individuals.

Study Population

We identified all individuals aged 18 to 64 years with an inpatient admission record between January 1, 2007, and December 31, 2015. All individuals with continuous medical and prescription drug coverage from 6 months prior to the hospital admission date (baseline period) through 30 days following the discharge date (follow-up period) were included. Inpatient admissions with a missing discharge disposition or those that resulted in in-hospital death, discharge to a short-term hospital, skilled nursing facility, intermediate care facility, or any other type of facility were not considered for analysis. Only the first eligible inpatient admission was considered for analysis.

Main Predictor Variable

Individuals with a DAMA were analyzed as the case group. A DAMA was identified using the “Patient Status Code” variable, which represents the discharge disposition of each individual. Individuals who were discharged to home/self-care or discharged to a home health organization formed the control group (hereafter referred to as routine discharge).

Demographic, Clinical, and Hospitalization Characteristics

An individual’s age, sex, and region of residence were determined at the date of hospital admission. The Elixhauser algorithm was used to categorize comorbid conditions (as scores of 0, 1-2, ≥3 depending on number of comorbidities) based on International Classification of Diseases, Ninth Revision, Clinical Modification, diagnosis codes during the baseline period.^13,14 The following characteristics of each individual’s eligible inpatient admission were captured: year, timing (weekday or weekend), length of stay (LOS, measured in days), and receipt of a surgical procedure.

Outcomes

All-cause HcRU was identified during the 30-day postdischarge period. Specifically, we identified inpatient readmissions, ED visits, physician office visits, nonphysician outpatient encounters (for example, pathology, radiology, outpatient surgical services), and prescription drug fills. Binary variables (yes or no) were created for inpatient readmissions and ED visits while the remaining HcRU categories (ie, physician office visits, nonphysician outpatient encounters, and prescription drug fills) were analyzed as count variables. In the sensitivity analyses, we provide results for HcRU outcomes among a subgroup of individuals who had at least 90 days of continuous medical and prescription drug benefits following the hospital discharge.

Statistical Analysis

Descriptive Analysis

Measures of interest were reported using summary statistics depending on the nature of the variable. Continuous variables were described using t tests, and categorical variables were described using chi-square tests.

Propensity Score Matching

Cases and controls were matched using a 1:1 greedy matching algorithm based on propensity scores.¹⁵ We developed propensity scores based on confounders that we hypothesized would be associated with a DAMA and postdischarge HcRU. The propensity score model included the following variables: age, sex, region of residence, Elixhauser comorbidity index score, year of admission, timing of admission, LOS, and presence of any surgical procedure during the inpatient admission. The best match between cases and controls was determined based on the absolute difference in their propensity scores, which allowed for a maximal caliper width of 0.2 of the standard deviation of the logit of the propensity score.¹⁶ A standardized difference value of less than 0.1 was used to assess balance in baseline patient and hospital characteristics between cases and controls consistent with prior literature.^17,18 Proportions and balance, as measured by standardized differences between baseline covariates across cases and controls in the matched sample, are displayed in tabular format (Appendix Table 1).

Healthcare Resource Utilization

We estimated the adjusted odds ratio (AOR) using a logistic regression model. The AOR quantified the association between a DAMA and the prevalence of all-cause inpatient readmissions and ED visits during the 30-day postdischarge period. We estimated incident rate ratios (IRR) for count outcomes. Given the large number of individuals with no physician office visits, nonphysician outpatient encounters, or prescription drug fills, we estimated model parameters for IRRs using a finite mixture negative binomial hurdle model.¹⁹ We considered the data to represent a mixture of a constant distribution (which always generates zero counts) and a zero-truncated distribution (which always generates nonzero counts). The finite mixture count models include two outcomes: the mixing probabilities and the count distribution. The mixing probabilities quantify the probability that an observation for the HcRU category will be drawn from either the constant distribution (with mass at zero) or the count distribution. Conditional on having positive values, a zero-truncated generalized linear model (GLM) governs the count variable. Compared with other GLM specifications (eg, Poisson, negative binomial, zero-inflated), the negative binomial hurdle model presented the best-fitting model across several information criteria statistics (Appendix Figures 1-3 and Appendix Tables 2-4).

The GLM results provided IRR for the counts of HcRU. Ratios were interpreted as evidence of increased HcRU (IRR ≥ 1.0) or decreased HcRU (IRR < 1.0) among individuals with a DAMA compared with those discharged routinely. For all HcRU analyses, we reported results for the matched sample. All analyses were conducted using SAS version 9.4 (SAS Institute), and statistical significance was determined at α= .05. The study received the University of Maryland, Baltimore, Institutional Review Board approval (HP-00081497).

RESULTS

The unmatched sample included 457,530 individuals, of whom 0.5% had a DAMA. A consort diagram illustrating cohort inclusion and exclusion criteria is presented in Appendix Figure 4. Demographic, clinical, and inpatient admission characteristics of the unmatched sample and for subgroups defined by discharge status are displayed in Table 1. In the unmatched sample, the median age at admission was higher for individuals with a DAMA than it was for those discharged routinely (43 vs 42 years, respectively), and the proportion of males was higher among those with a DAMA (58.4% vs 33.1%). There were statistically significant differences based on the geographic region of residence and the comorbidity burden across both groups. The median LOS was shorter (1 day vs 2 days), the proportion of weekend admissions was higher (22.2% vs 16.3%), and the proportion of inpatient surgical procedures was lower (12.9% vs 59.2%) among those with a DAMA compared with that among those with routine discharges. The propensity score-matched sample included 2,245 cases and 2,245 controls (Appendix Table 1). Standardized differences for all baseline factors were less than 0.1, indicating that cases and controls were matched on the included baseline factors.

Summary Statistics: Proportions and Counts

Across the DAMA and routine discharge groups, the proportion of individuals with a 30-day inpatient readmission was similar (19.5% vs 18.7%; P = .47), whereas the proportion with an ED visit was higher (18.6% vs 9.1%; P < .01). There were no differences in the median number of inpatient readmissions (median, 0) and ED visits (median, 0) across both groups. Individuals with a DAMA and those discharged routinely displayed similar median counts of 30-day physician office (median, 1) and nonphysician outpatient encounters (median, 1) (Table 2). Individuals with a DAMA displayed a lower median number of prescription drug fills (median, 2 vs 3) than that among those with a routine discharge (Table 2).

Main Analysis: Thirty-Day Healthcare Resource Utilization

The associations between a DAMA and 30-day inpatient readmissions and ED visits based on the matched sample are presented in Table 3. Individuals with a DAMA had increased odds for an ED visit (AOR, 2.28; 95% CI, 1.90-2.72) but no significant difference in the odds of a 30-day inpatient readmission (AOR, 1.06; 95% CI, 0.91-1.23) compared with those discharged routinely.

The association between a DAMA and count HcRU outcomes is presented in Table 4. Compared with those discharged routinely, individuals with a DAMA displayed no significant difference in rates for physician office visits (IRR, 1.01; 95% CI, 0.91-1.11), nonphysician outpatient encounters (IRR, 0.89; 95% CI, 0.78-1.00), and prescription drug fills (IRR, 1.03; 95% CI, 0.97-1.09) during the 30-day postdischarge period.

Sensitivity Analysis: Ninety-Day Healthcare Resource Utilization

Relative to those discharged routinely, individuals with a DAMA had statistically significant increased odds of 90-day inpatient readmissions (AOR, 1.18; 95% CI, 1.02-1.36), odds of ED visits (AOR, 2.16; 95% CI, 1.85-2.51), and rates of prescription drug fills (IRR, 1.32; 95% CI, 1.29-1.35). No statistically significant differences were observed in the rates of physician office visits and nonphysician outpatient encounters across both groups.

DISCUSSION

In this commercially insured sample of working age individuals, we identified an association between a DAMA and the likelihood and intensity of postdischarge HcRU. The direction of the association varied across categories of HcRU and the duration of follow-up. A DAMA was associated with increased odds of 30-day ED visits but not 30-day readmissions compared with routine discharges. No significant differences were observed in the rates of 30-day physician office visits, nonphysician outpatient encounters, and prescription drug fills across both groups. To our knowledge, this is the first study on DAMAs that examines postdischarge HcRU outside the inpatient setting.

The 0.5% prevalence of DAMAs in our study was lower than the approximate 1% to 2% value that is typically reported in the literature. Prior studies have typically reported results based on mixed-payer populations.^3-10 These mixed-payer populations include publicly insured (Medicare or Medicaid) or uninsured stays, which account for a disproportionate share of all DAMAs. In contrast, commercially insured stays account for the lowest proportion of all DAMAs.¹⁰ Similar to prior literature,⁵ the DAMA group in our study was younger, had a higher proportion of males, had a higher comorbidity burden, and had a shorter LOS than the routinely discharged group.

We observed a greater likelihood of ED utilization after a DAMA. Similar findings have been reported, which may indicate that patients with a DAMA receive inadequate treatment at the time of discharge and may require further acute treatment. For example, a prior study reported that, after a DAMA, individuals with asthma were four times more likely to have an ED visit within 14 days compared with those discharged routinely.⁴

Contrary to prior findings,^3-9 we found no significant difference in the odds of a 30-day inpatient readmission across the DAMA and routine discharge groups, which may be attributable to differences in the populations studied. Those previous studies used mixed payer populations and did not differentiate results by payer type. The mixed payer populations in these studies were older (mean ages, 55 years and above) and had an increased comorbidity burden compared with our commercially insured population. Furthermore, some of these studies were either limited to single sites,⁸ single state hospital systems,^3,4,9 or focused on specific medical populations.^3,4,6-9 Our national sample of commercially insured adults is considerably younger, with a mean age of 43 years. Thirty days may be too brief to observe enough inpatient readmissions for the purpose of comparative analyses. This is suggested by our results, which indicated that there is an association between DAMA and 90-day inpatient readmission. Additionally, nonsignificant findings for 30-day inpatient readmissions may also be due to the small sample size of the DAMA group in our study, which may have limited robust statistical inference. Future studies in a larger population of commercially insured individuals with a DAMA are required to confirm these findings.

Nonsignificant differences in the rates of 30-day physician office visits, nonphysician outpatient encounters, and prescription drug fills across both groups may explain the null association with 30-day inpatient readmissions. Prior literature on specific medical populations or individuals with general hospital admissions report that early outpatient follow-up can help prevent 30-day readmissions.^20-25 In our sample, we observed similar rates of outpatient follow-up across the DAMA and routinely discharged groups. Prior studies based on single hospital sites have reported that, at the time of discharge, a lower proportion of individuals with a DAMA received medication prescriptions and outpatient follow-up plans compared with those discharged routinely.^11,12 In contrast, we evaluated prescription drug fills and outpatient visits during the postdischarge period, which may explain the difference in findings.

The present study has several strengths. To the best of our knowledge, our study represents the first and largest retrospective analysis of DAMAs in a national sample of commercially insured adults. In addition to a large generalizable sample, we examine HcRU after a DAMA across major points of service over a longitudinal postdischarge period. Our results provide a comprehensive understanding of utilization outcomes in this population including those outside the inpatient setting, which has been the focus of prior literature. These findings can help guide the implementation of appropriate patient- and system-level interventions to optimize DAMA prevention and mitigate the associated utilization burden on the healthcare system in the postdischarge period.^26,27

Our findings should be interpreted with certain limitations in mind. First, this study used data based on a commercially insured sample of patients and may not be generalizable to publicly insured or uninsured samples. Second, like prior DAMA studies that used the Nationwide Readmissions Database instead,^5-7 our study was unable to account for individual-level factors such as race, marital status, family social support, income, health literacy, and activation in self-care. Further, given the limitations of our data, we were unable to control for hospital characteristics such as bed size, urban-rural designation, teaching status, and control (eg, private or government ownership). Despite the use of propensity score methods to balance both comparison groups on observable sources of confounding, we cannot rule out the possibility of residual confounding. Lastly, due to a lack of data on postdischarge mortality outcomes, we could not control for competing risk of death in our analysis. However, in a population with an average age of 43 years, we did not expect high or differential 30- or 90-day postdischarge mortality rates across both groups.

Our findings suggest several important directions for future research. First, it will be useful to examine these associations among publicly insured and uninsured samples in which a DAMA is more prevalent and in which the associations with HcRU may be more pronounced than they are in the commercially insured population. Secondly, future research should identify subgroups of DAMA patients with an increased propensity for postdischarge HcRU. This can help in the design of individualized outpatient follow-up plans that address patient-specific medical and social needs. Finally, our findings highlight the need for education, practice guidelines, and suitable interventions to help providers in the prevention and management of a DAMA.

CONCLUSION

Using data from a commercially insured population, we identified associations between a DAMA and postdischarge HcRU. The associations differed by category of HcRU. We identified a positive association with the likelihood of ED utilization but no association with the likelihood of 30-day inpatient readmission or general outpatient utilization. Our results indicate that the examination of inpatient readmissions after a DAMA should not be considered in isolation. The identification of the full range of outpatient and inpatient HcRU after a DAMA in a broad population of patients can improve our understanding of outcomes following a DAMA and support appropriate system-level interventions designed to reduce their prevalence.

Acknowledgments

The statements, findings, conclusions, views, and opinions contained and expressed in this manuscript are based in part on data obtained under license from IQVIA. Source: IQVIA PharMetrics^® Plus January 2006 – December 2015, IQVIA. All Rights Reserved. The statements, findings, conclusions, views, and opinions contained and expressed herein are not necessarily those of IQVIA or any of its affiliated or subsidiary entities.

Disclosures

Dr Onukwugha reports grants from Bayer Healthcare Pharmaceuticals, grants from Pfizer, Inc, and personal fees from Novo Nordisk outside the submitted work. The other authors have nothing to disclose. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the US Department of Veterans Affairs, the U.S. Government, or the VA National Center for Ethics in Health Care.

Funding

The authors acknowledge the support of the University of Maryland, Baltimore Institute for Clinical & Translational Research (ICTR) through the ICTR Voucher Program.

Discharges against medical advice (DAMAs), in which a patient leaves the hospital prior to a physician-recommended endpoint, represent approximately 1% to 2% of inpatient discharges in the United States.¹ When compared with routine discharges, a DAMA is associated with adverse clinical consequences, including an increased risk of all-cause mortality.^2,3 Additionally, due to incomplete care, a DAMA may result in increased healthcare resource utilization (HcRU), including the use of inpatient, emergency department (ED), and outpatient services in the postdischarge period. Quantifying these relationships can provide important information regarding an individual’s healthcare-seeking behavior following a DAMA.

Prior literature has focused on the association between a DAMA and the risk of inpatient readmission. Relative to routine discharges, a DAMA is associated with a 1.5 to 2 times increased risk of a 30-day readmission.^3-9 However, these estimates are based on mixed-payer populations primarily composed (65%-80%) of individuals with public (Medicaid, Medicare) or no insurance. Further, they do not differentiate this association by payer type. It is unclear if prior results apply to commercially insured adults. These individuals represent a small but nonnegligible proportion (19%) of all DAMAs in the United States.¹⁰ Quantifying relationships among commercially insured adults can help advance our understanding of readmission patterns in the DAMA population.

There is limited evidence regarding the relationship between a DAMA and outpatient HcRU in the postdischarge period. Use of ED services after a DAMA has been explored only in specific disease populations such as asthma.⁴ Additionally, prior studies have reported a reduced frequency in the receipt of medication prescriptions and outpatient follow-up plans among individuals with a DAMA at the time of discharge.^11,12 Whether these practices translate to altered patterns of postdischarge prescription drug fills or use of outpatient services is not known.

To address these substantive gaps in the literature, the present study evaluates the association between a DAMA and all-cause HcRU in the postdischarge period among commercially insured adults. We examined HcRU across all points of service including inpatient readmissions, ED visits, physician office visits, nonphysician outpatient encounters, and prescription drug fills. These results can serve as a benchmark for comparison to future studies on DAMAs among publicly insured or uninsured individuals. Furthermore, such knowledge can help providers, payers, and policy planners make evidence-based decisions regarding postdischarge healthcare delivery.

METHODS

Data Source

This retrospective study used a 10% random sample of enrollees in the IQVIA PharMetrics^® Plus database (purchased by University of Maryland, Baltimore, under license from IQVIA). The database is composed of fully adjudicated claims and enrollment information from over 70 contributing US health plans and self-insured employer groups for over 140 million unique enrollees from 2006 onward. The enrollee population is generally representative of the commercially insured population that is younger than 65 years of age (with a subset of commercial Medicare and Medicaid) with respect to age and gender.

The database allows longitudinal follow-up for individuals using three files: medical claims, pharmacy claims, and insurance eligibility. The average length of enrollment is 39 months. The claims data represent payments to providers for services rendered to individuals covered by health plans. The medical claims file contains information on diagnostic and therapeutic services rendered in the inpatient and outpatient settings. The pharmacy claims file captures data on prescription drugs dispensed in retail and mail-order settings. The eligibility file contains demographic and insurance eligibility information for individuals.

Study Population

We identified all individuals aged 18 to 64 years with an inpatient admission record between January 1, 2007, and December 31, 2015. All individuals with continuous medical and prescription drug coverage from 6 months prior to the hospital admission date (baseline period) through 30 days following the discharge date (follow-up period) were included. Inpatient admissions with a missing discharge disposition or those that resulted in in-hospital death, discharge to a short-term hospital, skilled nursing facility, intermediate care facility, or any other type of facility were not considered for analysis. Only the first eligible inpatient admission was considered for analysis.

Main Predictor Variable

Individuals with a DAMA were analyzed as the case group. A DAMA was identified using the “Patient Status Code” variable, which represents the discharge disposition of each individual. Individuals who were discharged to home/self-care or discharged to a home health organization formed the control group (hereafter referred to as routine discharge).

Demographic, Clinical, and Hospitalization Characteristics

An individual’s age, sex, and region of residence were determined at the date of hospital admission. The Elixhauser algorithm was used to categorize comorbid conditions (as scores of 0, 1-2, ≥3 depending on number of comorbidities) based on International Classification of Diseases, Ninth Revision, Clinical Modification, diagnosis codes during the baseline period.^13,14 The following characteristics of each individual’s eligible inpatient admission were captured: year, timing (weekday or weekend), length of stay (LOS, measured in days), and receipt of a surgical procedure.

Outcomes

All-cause HcRU was identified during the 30-day postdischarge period. Specifically, we identified inpatient readmissions, ED visits, physician office visits, nonphysician outpatient encounters (for example, pathology, radiology, outpatient surgical services), and prescription drug fills. Binary variables (yes or no) were created for inpatient readmissions and ED visits while the remaining HcRU categories (ie, physician office visits, nonphysician outpatient encounters, and prescription drug fills) were analyzed as count variables. In the sensitivity analyses, we provide results for HcRU outcomes among a subgroup of individuals who had at least 90 days of continuous medical and prescription drug benefits following the hospital discharge.

Statistical Analysis

Descriptive Analysis

Measures of interest were reported using summary statistics depending on the nature of the variable. Continuous variables were described using t tests, and categorical variables were described using chi-square tests.

Propensity Score Matching

Cases and controls were matched using a 1:1 greedy matching algorithm based on propensity scores.¹⁵ We developed propensity scores based on confounders that we hypothesized would be associated with a DAMA and postdischarge HcRU. The propensity score model included the following variables: age, sex, region of residence, Elixhauser comorbidity index score, year of admission, timing of admission, LOS, and presence of any surgical procedure during the inpatient admission. The best match between cases and controls was determined based on the absolute difference in their propensity scores, which allowed for a maximal caliper width of 0.2 of the standard deviation of the logit of the propensity score.¹⁶ A standardized difference value of less than 0.1 was used to assess balance in baseline patient and hospital characteristics between cases and controls consistent with prior literature.^17,18 Proportions and balance, as measured by standardized differences between baseline covariates across cases and controls in the matched sample, are displayed in tabular format (Appendix Table 1).

Healthcare Resource Utilization

We estimated the adjusted odds ratio (AOR) using a logistic regression model. The AOR quantified the association between a DAMA and the prevalence of all-cause inpatient readmissions and ED visits during the 30-day postdischarge period. We estimated incident rate ratios (IRR) for count outcomes. Given the large number of individuals with no physician office visits, nonphysician outpatient encounters, or prescription drug fills, we estimated model parameters for IRRs using a finite mixture negative binomial hurdle model.¹⁹ We considered the data to represent a mixture of a constant distribution (which always generates zero counts) and a zero-truncated distribution (which always generates nonzero counts). The finite mixture count models include two outcomes: the mixing probabilities and the count distribution. The mixing probabilities quantify the probability that an observation for the HcRU category will be drawn from either the constant distribution (with mass at zero) or the count distribution. Conditional on having positive values, a zero-truncated generalized linear model (GLM) governs the count variable. Compared with other GLM specifications (eg, Poisson, negative binomial, zero-inflated), the negative binomial hurdle model presented the best-fitting model across several information criteria statistics (Appendix Figures 1-3 and Appendix Tables 2-4).

The GLM results provided IRR for the counts of HcRU. Ratios were interpreted as evidence of increased HcRU (IRR ≥ 1.0) or decreased HcRU (IRR < 1.0) among individuals with a DAMA compared with those discharged routinely. For all HcRU analyses, we reported results for the matched sample. All analyses were conducted using SAS version 9.4 (SAS Institute), and statistical significance was determined at α= .05. The study received the University of Maryland, Baltimore, Institutional Review Board approval (HP-00081497).

RESULTS

The unmatched sample included 457,530 individuals, of whom 0.5% had a DAMA. A consort diagram illustrating cohort inclusion and exclusion criteria is presented in Appendix Figure 4. Demographic, clinical, and inpatient admission characteristics of the unmatched sample and for subgroups defined by discharge status are displayed in Table 1. In the unmatched sample, the median age at admission was higher for individuals with a DAMA than it was for those discharged routinely (43 vs 42 years, respectively), and the proportion of males was higher among those with a DAMA (58.4% vs 33.1%). There were statistically significant differences based on the geographic region of residence and the comorbidity burden across both groups. The median LOS was shorter (1 day vs 2 days), the proportion of weekend admissions was higher (22.2% vs 16.3%), and the proportion of inpatient surgical procedures was lower (12.9% vs 59.2%) among those with a DAMA compared with that among those with routine discharges. The propensity score-matched sample included 2,245 cases and 2,245 controls (Appendix Table 1). Standardized differences for all baseline factors were less than 0.1, indicating that cases and controls were matched on the included baseline factors.

Summary Statistics: Proportions and Counts

Across the DAMA and routine discharge groups, the proportion of individuals with a 30-day inpatient readmission was similar (19.5% vs 18.7%; P = .47), whereas the proportion with an ED visit was higher (18.6% vs 9.1%; P < .01). There were no differences in the median number of inpatient readmissions (median, 0) and ED visits (median, 0) across both groups. Individuals with a DAMA and those discharged routinely displayed similar median counts of 30-day physician office (median, 1) and nonphysician outpatient encounters (median, 1) (Table 2). Individuals with a DAMA displayed a lower median number of prescription drug fills (median, 2 vs 3) than that among those with a routine discharge (Table 2).

Main Analysis: Thirty-Day Healthcare Resource Utilization

The associations between a DAMA and 30-day inpatient readmissions and ED visits based on the matched sample are presented in Table 3. Individuals with a DAMA had increased odds for an ED visit (AOR, 2.28; 95% CI, 1.90-2.72) but no significant difference in the odds of a 30-day inpatient readmission (AOR, 1.06; 95% CI, 0.91-1.23) compared with those discharged routinely.

The association between a DAMA and count HcRU outcomes is presented in Table 4. Compared with those discharged routinely, individuals with a DAMA displayed no significant difference in rates for physician office visits (IRR, 1.01; 95% CI, 0.91-1.11), nonphysician outpatient encounters (IRR, 0.89; 95% CI, 0.78-1.00), and prescription drug fills (IRR, 1.03; 95% CI, 0.97-1.09) during the 30-day postdischarge period.

Sensitivity Analysis: Ninety-Day Healthcare Resource Utilization

Relative to those discharged routinely, individuals with a DAMA had statistically significant increased odds of 90-day inpatient readmissions (AOR, 1.18; 95% CI, 1.02-1.36), odds of ED visits (AOR, 2.16; 95% CI, 1.85-2.51), and rates of prescription drug fills (IRR, 1.32; 95% CI, 1.29-1.35). No statistically significant differences were observed in the rates of physician office visits and nonphysician outpatient encounters across both groups.

DISCUSSION

In this commercially insured sample of working age individuals, we identified an association between a DAMA and the likelihood and intensity of postdischarge HcRU. The direction of the association varied across categories of HcRU and the duration of follow-up. A DAMA was associated with increased odds of 30-day ED visits but not 30-day readmissions compared with routine discharges. No significant differences were observed in the rates of 30-day physician office visits, nonphysician outpatient encounters, and prescription drug fills across both groups. To our knowledge, this is the first study on DAMAs that examines postdischarge HcRU outside the inpatient setting.

The 0.5% prevalence of DAMAs in our study was lower than the approximate 1% to 2% value that is typically reported in the literature. Prior studies have typically reported results based on mixed-payer populations.^3-10 These mixed-payer populations include publicly insured (Medicare or Medicaid) or uninsured stays, which account for a disproportionate share of all DAMAs. In contrast, commercially insured stays account for the lowest proportion of all DAMAs.¹⁰ Similar to prior literature,⁵ the DAMA group in our study was younger, had a higher proportion of males, had a higher comorbidity burden, and had a shorter LOS than the routinely discharged group.

We observed a greater likelihood of ED utilization after a DAMA. Similar findings have been reported, which may indicate that patients with a DAMA receive inadequate treatment at the time of discharge and may require further acute treatment. For example, a prior study reported that, after a DAMA, individuals with asthma were four times more likely to have an ED visit within 14 days compared with those discharged routinely.⁴

Contrary to prior findings,^3-9 we found no significant difference in the odds of a 30-day inpatient readmission across the DAMA and routine discharge groups, which may be attributable to differences in the populations studied. Those previous studies used mixed payer populations and did not differentiate results by payer type. The mixed payer populations in these studies were older (mean ages, 55 years and above) and had an increased comorbidity burden compared with our commercially insured population. Furthermore, some of these studies were either limited to single sites,⁸ single state hospital systems,^3,4,9 or focused on specific medical populations.^3,4,6-9 Our national sample of commercially insured adults is considerably younger, with a mean age of 43 years. Thirty days may be too brief to observe enough inpatient readmissions for the purpose of comparative analyses. This is suggested by our results, which indicated that there is an association between DAMA and 90-day inpatient readmission. Additionally, nonsignificant findings for 30-day inpatient readmissions may also be due to the small sample size of the DAMA group in our study, which may have limited robust statistical inference. Future studies in a larger population of commercially insured individuals with a DAMA are required to confirm these findings.

Nonsignificant differences in the rates of 30-day physician office visits, nonphysician outpatient encounters, and prescription drug fills across both groups may explain the null association with 30-day inpatient readmissions. Prior literature on specific medical populations or individuals with general hospital admissions report that early outpatient follow-up can help prevent 30-day readmissions.^20-25 In our sample, we observed similar rates of outpatient follow-up across the DAMA and routinely discharged groups. Prior studies based on single hospital sites have reported that, at the time of discharge, a lower proportion of individuals with a DAMA received medication prescriptions and outpatient follow-up plans compared with those discharged routinely.^11,12 In contrast, we evaluated prescription drug fills and outpatient visits during the postdischarge period, which may explain the difference in findings.

The present study has several strengths. To the best of our knowledge, our study represents the first and largest retrospective analysis of DAMAs in a national sample of commercially insured adults. In addition to a large generalizable sample, we examine HcRU after a DAMA across major points of service over a longitudinal postdischarge period. Our results provide a comprehensive understanding of utilization outcomes in this population including those outside the inpatient setting, which has been the focus of prior literature. These findings can help guide the implementation of appropriate patient- and system-level interventions to optimize DAMA prevention and mitigate the associated utilization burden on the healthcare system in the postdischarge period.^26,27

Our findings should be interpreted with certain limitations in mind. First, this study used data based on a commercially insured sample of patients and may not be generalizable to publicly insured or uninsured samples. Second, like prior DAMA studies that used the Nationwide Readmissions Database instead,^5-7 our study was unable to account for individual-level factors such as race, marital status, family social support, income, health literacy, and activation in self-care. Further, given the limitations of our data, we were unable to control for hospital characteristics such as bed size, urban-rural designation, teaching status, and control (eg, private or government ownership). Despite the use of propensity score methods to balance both comparison groups on observable sources of confounding, we cannot rule out the possibility of residual confounding. Lastly, due to a lack of data on postdischarge mortality outcomes, we could not control for competing risk of death in our analysis. However, in a population with an average age of 43 years, we did not expect high or differential 30- or 90-day postdischarge mortality rates across both groups.

Our findings suggest several important directions for future research. First, it will be useful to examine these associations among publicly insured and uninsured samples in which a DAMA is more prevalent and in which the associations with HcRU may be more pronounced than they are in the commercially insured population. Secondly, future research should identify subgroups of DAMA patients with an increased propensity for postdischarge HcRU. This can help in the design of individualized outpatient follow-up plans that address patient-specific medical and social needs. Finally, our findings highlight the need for education, practice guidelines, and suitable interventions to help providers in the prevention and management of a DAMA.

CONCLUSION

Using data from a commercially insured population, we identified associations between a DAMA and postdischarge HcRU. The associations differed by category of HcRU. We identified a positive association with the likelihood of ED utilization but no association with the likelihood of 30-day inpatient readmission or general outpatient utilization. Our results indicate that the examination of inpatient readmissions after a DAMA should not be considered in isolation. The identification of the full range of outpatient and inpatient HcRU after a DAMA in a broad population of patients can improve our understanding of outcomes following a DAMA and support appropriate system-level interventions designed to reduce their prevalence.

Acknowledgments

The statements, findings, conclusions, views, and opinions contained and expressed in this manuscript are based in part on data obtained under license from IQVIA. Source: IQVIA PharMetrics^® Plus January 2006 – December 2015, IQVIA. All Rights Reserved. The statements, findings, conclusions, views, and opinions contained and expressed herein are not necessarily those of IQVIA or any of its affiliated or subsidiary entities.

Disclosures

Dr Onukwugha reports grants from Bayer Healthcare Pharmaceuticals, grants from Pfizer, Inc, and personal fees from Novo Nordisk outside the submitted work. The other authors have nothing to disclose. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the US Department of Veterans Affairs, the U.S. Government, or the VA National Center for Ethics in Health Care.

Funding

The authors acknowledge the support of the University of Maryland, Baltimore Institute for Clinical & Translational Research (ICTR) through the ICTR Voucher Program.

Standardized template design is a useful tool to improve clinical documentation and reliable reporting of health care outcomes when constructed with clear objectives and with collaboration of key stakeholders. A standardized template should not only capture accurate diagnostic information, but also inform quality improvement (QI) measures and best practices.

Kang and colleagues showed that a correlation exists between organizational satisfaction and improved quality outcomes.¹ A new initiative should have a well-defined purpose reinforced by collaborative workgroups and engaged employees who understand their clinical care role with electronic health record (EHR) modifications.

Several studies have shown how the usefulness of templates achieve multipurpose goals, such as accurate documentation and improved care. Valluru and colleagues showed a significant increase in vaccination rates for patients with inflammatory bowel disease after implementing a standardized template.² By using a standardized template, Thaker and colleagues showed improved documentation regarding obesity and increased nutritional and physical activity counseling.³ Furthermore, Grogan and colleagues showed that templates are useful for house staff education on International Classification of Diseases (ICD) terminology and demonstrated improved documentation in the postintervention group.^4,5

This article discusses the US Department of Veterans Affairs (VA) North Florida/South Georgia Veterans Health System (NF/SGVHS) integrated informatics solutions within template design in the Veterans Health Administration (VHA) EHR system that was associated with an increase in its case severity index (CSI) through improved clinical documentation capture.

Methods

According to policy activities that constitute research at NF/SGVHS, institutional review board approval was not required as this work met the criteria for operational improvement activities exempt from ethics review.

NF/SGVHS includes 2 hospitals: Malcom Randall VA Medical Center (MRVAMC) in Gainesville, Florida, and Lake City VA Medical Center (LCVAMC) in Lake City, Florida. MRVAMC is a large, 1a, academic VA facility composed of rotating residents and fellows and includes multiple specialty care services. LCVAMC is a smaller, nonteaching facility.

Template Design Impact

CSI is a risk-adjusted formula developed by the Inpatient Evaluation Center within VHA. CSI is incorporated into the VHA quality metrics reporting system, Strategic Analytics for Improvement and Learning (SAIL). CSI risk-adjusts metrics such as length of stay and mortality before releasing SAIL reports. CSI is calculated separately for acute level of care (LOC) and for the intensive care unit (ICU). In fiscal year (FY) 2017, acute LOC preimplementation data for CSI at NF/SGVHS were 0.76 for MRVAMC and 0.81 for LCVAMC, which was significantly below the national VHA average of 0.96 (Table).

A below-average CSI conveys a less complicated case mix compared with most other VA facilities. Although smaller VA facilities may have a less complicated case mix, it is unusual for large, tertiary care 1a VA facilities to have a low CSI. This low CSI is usually due to inadequate documentation, which affects not only risk-adjusted quality metrics outcomes, but also potential reimbursement.⁶

An interdisciplinary team composed of attendings, residents, and a clinical document improvement specialist identified the below-average acute LOC CSI for MRVAMC and LCVAMC compared with that of the national VHA average. Further analysis by chart reviews showed inconsistencies with standardized documentation despite prior health care provider education on ICD terminology and specific groups of common comorbidities analyzed in administrative data reviews for risk-adjustment purposes, known as Elixhauser comorbidities.^5,7

A chart review showed lack of clarity regarding primary reason(s) for admission and chronic comorbidities within NF/SGVHS. Using Pareto chart analysis, the template team designed a standardized history and physical (H&P) medicine template based on NF/SGVHS common medicine admissions (Figure 1). A Pareto chart is a valuable QI tool that assists with identifying majority contributors to a problem(s) being analyzed when evaluating a large set of data points. Subsequently, this tool helps focus direction on QI efforts.⁸

The template had the usual H&P elements not shown (eg, chief complaint, history of present illness, etc), and highlights the assessment/plan section containing primary reason(s) for admission and chronic comorbidities (Figure 1). The complete assessment and plan section on the template can be found in the Appendix.

To simplify the template interface, only single clicks were required to expand diagnostic and chronic comorbidity checkboxes. Subcategories then appeared to select diagnosis and chronic comorbidities along with free text for additional documentation.

In addition, data objects were created within the template that permitted the ability to retrieve information from the VHA EHR and insert specific data points of interest in the template; for example, body mass index to assess degree of obesity and estimated glomerular filtration rate to determine the stage of chronic kidney disease. This allowed users to easily reference data in one template in lieu of searching for data in multiple places in the EHR.⁹

Results

The standardized H&P medicine template was implemented at MRVAMC and LCVAMC in June 2018 (the final month of the third quarter of FY 2018). As clinical providers throughout NF/SGVHS used the standardized template, acute LOC postimplementation data for CSI significantly improved. Although the national VHA average slightly decreased from 0.96 in the first quarter of FY 2017 to 0.89, in the first quarter of FY 2019, MRVAMC acute LOC CSI improved from 0.76 to 0.97, and LCVAMC acute LOC CSI improved from 0.81 to 1.07 during the same period.

In addition, compliance also was monitored within MRVAMC and LCVAMC for about 1 year after standardized H&P medicine template implementation. Compliance was determined by how often the standardized H&P medicine template was used for inpatient medicine admissions to the acute care wards vs other H&P notes used (such as personalized templates).

Methodology for compliance analysis included acquisition of completed H&P medicine notes from June 18, 2018 to June 30, 2019, within the VHA Veterans Information Systems and Technology Architecture (VistA) clinical and business information system using the search strings: “H&P admission history and physical” and “history of present illness.”¹⁰

A review identified 10,845 completed medicine H&P notes. Nine hundred eighteen notes were excluded as their search function yielded a location not corresponding to MRVAMC or LCVAMC. Of the 9,927 notes remaining, 8,025 of these were completed medicine H&P notes at MRVAMC and 1,902 were completed medicine H&P notes at LCVAMC (Figure 2).

From June 18, 2018 to June 30, 2019 at MRVAMC, compliance was reviewed monthly for the 8,025 completed H&P medicine notes. Of the completed H&P medicine notes, the standardized H&P medicine template was used 43.2% in June 2018. By June 2019, MRVAMC clinical providers demonstrated significant improvement for standardized H&P medicine template use at 89.9% (Figure 3). Total average compliance from June 18, 2018 to June 30, 2019, was 88.4%, which doubled compliance from the initial introduction of the standardized H&P medicine template.

Compliance was reviewed monthly for the 1,902 completed H&P medicine notes from June 18, 2018 to June 30, 2019, at LCVAMC. Of the completed H&P medicine notes, the standardized template was used 48.2% of the time in June 2018. By June 2019, LCVAMC clinical providers demonstrated significant improvement for standardized H&P medicine template use, which increased to 96.9%. Total average compliance from June 18, 2018 to June 30, 2019, was 93.8%, which was almost double the baseline compliance rate.

Discussion

Template design with clear objectives, strategic collaboration, and integrated informatics solutions has the potential to increase accuracy of documentation. As shown, the NF/SGVHS template design was associated with significant improvement in acute LOC CSI for both MRVAMC and LCVAMC due to more accurate documentation using the standardized H&P medicine template.

Numerous factors contributed to the success of this template design. First, a clear vision for application of the template was communicated with key stakeholders. In addition, the template design team was focused on specific goals rather than a one size fits all approach, which was crucial for sustainable execution. Although interdisciplinary teamwork has the potential to result in innovative practices, large multidisciplinary teams also may have difficulty establishing a shared vision that can result in barriers to achieving project goals.

Balancing standardization and customization was essential for user buy-in. As noted by Gardner and Pearce, inviting clinical providers to participate in template design and allowing for customization has the potential to increase acceptance and use of templates.¹¹ Although the original design for the standardized H&P medicine template started with the medicine service at NF/SGVHS, the design framework is applicable to numerous services where various clinical care elements can be customized.

Explaining the informatics tools built into the template allowed clinicians to see opportunities to improve clinical documentation and the impact it has on reporting health care outcomes. When improvement work involves integrating clinical care delivery and administrative expectations, it is essential that health care systems understand and strategically execute project initiatives at this critical juncture.

Finally, incorporation of a sustainability plan when process improvement strategies are implemented is vital. In addition to collaboration with the clinical providers during design and implementation of the standardized template, leadership buy-in was key. Compliance with standardized H&P medicine template use was monitored monthly and reviewed by the NF/SGVHS Chief of Staff.

As noted, LCVAMC postimplementation acute LOC CSI was higher than that of MRVAMC despite being a smaller facility. This might be due to the MRVAMC designation as a teaching institution. Medicine is the only inpatient service at LCVAMC staffed by hospitalists with limited specialists available for consultation, whereas MRVAMC is a tertiary care teaching facility with numerous inpatient services and subspecialties. As LCVAMC has more continuity, house staff rotating at MRVAMC require continued training/education on new templates and process changes.

Limitations

Although standardized template design was successful at NF/SGVHS, limitations should be noted. Our clinical documentation improvement (CDI) program also was expanded about the same time as the new templates were released. The expansion of the CDI program in addition to new template design likely had a synergistic effect on acute LOC CSI.

CSI is a complex, risk-adjusted model that includes numerous factors, including but not limited to diagnosis and comorbid conditions. Other factors include age, marital status, procedures, source of admission, specific laboratory values, medical or surgical diagnosis-related group, intensive care unit stays, and immunosuppressive status. CSI also includes operative and nonoperative components that average into an overall CSI. As the majority of CSI is composed of nonoperative constituents within NF/SGVHS, we do not believe this had any substantial impact on reporting of CSI improvements.

In addition, template entry into VHA EHR requires a location selection (such as a clinic name or ward name following an inpatient admission). Of the 10,845 completed H&P medicine notes identified in VistA, 918 notes were excluded from analysis as their search function yielded a location not corresponding to MRVAMC or LCVAMC. For the 918 notes excluded, we believe this was likely due to user error where locations not related to MRVAMC or LCVAMC were selected during standardized H&P medicine template entry.

Conclusions

After the NF/SGVHS implementation of a uniquely designed template embedded with informatics solutions within the VHA EHR, the CSI increased due to more accurate documentation.

Next steps include determining the impact of the NF/SGVHS template design on potential reimbursement and expanding template design into the outpatient setting where there are additional opportunities to improve clinical documentation and reliable reporting of health care outcomes.

Acknowledgments

The authors thank the following individuals for their experience and contribution: Beverley White is the Clinical Documentation Improvement Coordinator at North Florida/South Georgia Veterans Health System and provided expertise on documentation requirements. Russell Jacobitz and Susan Rozelle provided technical expertise on electronic health record system enhancements and implemented the template design. Jess Delaune, MD, and Robert Carroll, MD, provided additional physician input during template design. We also acknowledge the Inpatient Evaluation Center (IPEC) within the Veterans Health Administration (VHA). IPEC developed the case severity index, a risk-adjusted formula incorporated into the VHA quality metric reporting system, Strategic Analytics for Improvement and Learning (SAIL).

Standardized template design is a useful tool to improve clinical documentation and reliable reporting of health care outcomes when constructed with clear objectives and with collaboration of key stakeholders. A standardized template should not only capture accurate diagnostic information, but also inform quality improvement (QI) measures and best practices.

Kang and colleagues showed that a correlation exists between organizational satisfaction and improved quality outcomes.¹ A new initiative should have a well-defined purpose reinforced by collaborative workgroups and engaged employees who understand their clinical care role with electronic health record (EHR) modifications.

Several studies have shown how the usefulness of templates achieve multipurpose goals, such as accurate documentation and improved care. Valluru and colleagues showed a significant increase in vaccination rates for patients with inflammatory bowel disease after implementing a standardized template.² By using a standardized template, Thaker and colleagues showed improved documentation regarding obesity and increased nutritional and physical activity counseling.³ Furthermore, Grogan and colleagues showed that templates are useful for house staff education on International Classification of Diseases (ICD) terminology and demonstrated improved documentation in the postintervention group.^4,5

This article discusses the US Department of Veterans Affairs (VA) North Florida/South Georgia Veterans Health System (NF/SGVHS) integrated informatics solutions within template design in the Veterans Health Administration (VHA) EHR system that was associated with an increase in its case severity index (CSI) through improved clinical documentation capture.

Methods

According to policy activities that constitute research at NF/SGVHS, institutional review board approval was not required as this work met the criteria for operational improvement activities exempt from ethics review.

NF/SGVHS includes 2 hospitals: Malcom Randall VA Medical Center (MRVAMC) in Gainesville, Florida, and Lake City VA Medical Center (LCVAMC) in Lake City, Florida. MRVAMC is a large, 1a, academic VA facility composed of rotating residents and fellows and includes multiple specialty care services. LCVAMC is a smaller, nonteaching facility.

Template Design Impact

CSI is a risk-adjusted formula developed by the Inpatient Evaluation Center within VHA. CSI is incorporated into the VHA quality metrics reporting system, Strategic Analytics for Improvement and Learning (SAIL). CSI risk-adjusts metrics such as length of stay and mortality before releasing SAIL reports. CSI is calculated separately for acute level of care (LOC) and for the intensive care unit (ICU). In fiscal year (FY) 2017, acute LOC preimplementation data for CSI at NF/SGVHS were 0.76 for MRVAMC and 0.81 for LCVAMC, which was significantly below the national VHA average of 0.96 (Table).

A below-average CSI conveys a less complicated case mix compared with most other VA facilities. Although smaller VA facilities may have a less complicated case mix, it is unusual for large, tertiary care 1a VA facilities to have a low CSI. This low CSI is usually due to inadequate documentation, which affects not only risk-adjusted quality metrics outcomes, but also potential reimbursement.⁶

An interdisciplinary team composed of attendings, residents, and a clinical document improvement specialist identified the below-average acute LOC CSI for MRVAMC and LCVAMC compared with that of the national VHA average. Further analysis by chart reviews showed inconsistencies with standardized documentation despite prior health care provider education on ICD terminology and specific groups of common comorbidities analyzed in administrative data reviews for risk-adjustment purposes, known as Elixhauser comorbidities.^5,7

A chart review showed lack of clarity regarding primary reason(s) for admission and chronic comorbidities within NF/SGVHS. Using Pareto chart analysis, the template team designed a standardized history and physical (H&P) medicine template based on NF/SGVHS common medicine admissions (Figure 1). A Pareto chart is a valuable QI tool that assists with identifying majority contributors to a problem(s) being analyzed when evaluating a large set of data points. Subsequently, this tool helps focus direction on QI efforts.⁸

The template had the usual H&P elements not shown (eg, chief complaint, history of present illness, etc), and highlights the assessment/plan section containing primary reason(s) for admission and chronic comorbidities (Figure 1). The complete assessment and plan section on the template can be found in the Appendix.

To simplify the template interface, only single clicks were required to expand diagnostic and chronic comorbidity checkboxes. Subcategories then appeared to select diagnosis and chronic comorbidities along with free text for additional documentation.

In addition, data objects were created within the template that permitted the ability to retrieve information from the VHA EHR and insert specific data points of interest in the template; for example, body mass index to assess degree of obesity and estimated glomerular filtration rate to determine the stage of chronic kidney disease. This allowed users to easily reference data in one template in lieu of searching for data in multiple places in the EHR.⁹

Results

The standardized H&P medicine template was implemented at MRVAMC and LCVAMC in June 2018 (the final month of the third quarter of FY 2018). As clinical providers throughout NF/SGVHS used the standardized template, acute LOC postimplementation data for CSI significantly improved. Although the national VHA average slightly decreased from 0.96 in the first quarter of FY 2017 to 0.89, in the first quarter of FY 2019, MRVAMC acute LOC CSI improved from 0.76 to 0.97, and LCVAMC acute LOC CSI improved from 0.81 to 1.07 during the same period.

In addition, compliance also was monitored within MRVAMC and LCVAMC for about 1 year after standardized H&P medicine template implementation. Compliance was determined by how often the standardized H&P medicine template was used for inpatient medicine admissions to the acute care wards vs other H&P notes used (such as personalized templates).

Methodology for compliance analysis included acquisition of completed H&P medicine notes from June 18, 2018 to June 30, 2019, within the VHA Veterans Information Systems and Technology Architecture (VistA) clinical and business information system using the search strings: “H&P admission history and physical” and “history of present illness.”¹⁰

A review identified 10,845 completed medicine H&P notes. Nine hundred eighteen notes were excluded as their search function yielded a location not corresponding to MRVAMC or LCVAMC. Of the 9,927 notes remaining, 8,025 of these were completed medicine H&P notes at MRVAMC and 1,902 were completed medicine H&P notes at LCVAMC (Figure 2).

From June 18, 2018 to June 30, 2019 at MRVAMC, compliance was reviewed monthly for the 8,025 completed H&P medicine notes. Of the completed H&P medicine notes, the standardized H&P medicine template was used 43.2% in June 2018. By June 2019, MRVAMC clinical providers demonstrated significant improvement for standardized H&P medicine template use at 89.9% (Figure 3). Total average compliance from June 18, 2018 to June 30, 2019, was 88.4%, which doubled compliance from the initial introduction of the standardized H&P medicine template.

Compliance was reviewed monthly for the 1,902 completed H&P medicine notes from June 18, 2018 to June 30, 2019, at LCVAMC. Of the completed H&P medicine notes, the standardized template was used 48.2% of the time in June 2018. By June 2019, LCVAMC clinical providers demonstrated significant improvement for standardized H&P medicine template use, which increased to 96.9%. Total average compliance from June 18, 2018 to June 30, 2019, was 93.8%, which was almost double the baseline compliance rate.

Discussion

Template design with clear objectives, strategic collaboration, and integrated informatics solutions has the potential to increase accuracy of documentation. As shown, the NF/SGVHS template design was associated with significant improvement in acute LOC CSI for both MRVAMC and LCVAMC due to more accurate documentation using the standardized H&P medicine template.

Numerous factors contributed to the success of this template design. First, a clear vision for application of the template was communicated with key stakeholders. In addition, the template design team was focused on specific goals rather than a one size fits all approach, which was crucial for sustainable execution. Although interdisciplinary teamwork has the potential to result in innovative practices, large multidisciplinary teams also may have difficulty establishing a shared vision that can result in barriers to achieving project goals.

Balancing standardization and customization was essential for user buy-in. As noted by Gardner and Pearce, inviting clinical providers to participate in template design and allowing for customization has the potential to increase acceptance and use of templates.¹¹ Although the original design for the standardized H&P medicine template started with the medicine service at NF/SGVHS, the design framework is applicable to numerous services where various clinical care elements can be customized.

Explaining the informatics tools built into the template allowed clinicians to see opportunities to improve clinical documentation and the impact it has on reporting health care outcomes. When improvement work involves integrating clinical care delivery and administrative expectations, it is essential that health care systems understand and strategically execute project initiatives at this critical juncture.

Finally, incorporation of a sustainability plan when process improvement strategies are implemented is vital. In addition to collaboration with the clinical providers during design and implementation of the standardized template, leadership buy-in was key. Compliance with standardized H&P medicine template use was monitored monthly and reviewed by the NF/SGVHS Chief of Staff.

As noted, LCVAMC postimplementation acute LOC CSI was higher than that of MRVAMC despite being a smaller facility. This might be due to the MRVAMC designation as a teaching institution. Medicine is the only inpatient service at LCVAMC staffed by hospitalists with limited specialists available for consultation, whereas MRVAMC is a tertiary care teaching facility with numerous inpatient services and subspecialties. As LCVAMC has more continuity, house staff rotating at MRVAMC require continued training/education on new templates and process changes.

Limitations

Although standardized template design was successful at NF/SGVHS, limitations should be noted. Our clinical documentation improvement (CDI) program also was expanded about the same time as the new templates were released. The expansion of the CDI program in addition to new template design likely had a synergistic effect on acute LOC CSI.

CSI is a complex, risk-adjusted model that includes numerous factors, including but not limited to diagnosis and comorbid conditions. Other factors include age, marital status, procedures, source of admission, specific laboratory values, medical or surgical diagnosis-related group, intensive care unit stays, and immunosuppressive status. CSI also includes operative and nonoperative components that average into an overall CSI. As the majority of CSI is composed of nonoperative constituents within NF/SGVHS, we do not believe this had any substantial impact on reporting of CSI improvements.

In addition, template entry into VHA EHR requires a location selection (such as a clinic name or ward name following an inpatient admission). Of the 10,845 completed H&P medicine notes identified in VistA, 918 notes were excluded from analysis as their search function yielded a location not corresponding to MRVAMC or LCVAMC. For the 918 notes excluded, we believe this was likely due to user error where locations not related to MRVAMC or LCVAMC were selected during standardized H&P medicine template entry.

Conclusions

After the NF/SGVHS implementation of a uniquely designed template embedded with informatics solutions within the VHA EHR, the CSI increased due to more accurate documentation.

Next steps include determining the impact of the NF/SGVHS template design on potential reimbursement and expanding template design into the outpatient setting where there are additional opportunities to improve clinical documentation and reliable reporting of health care outcomes.

Acknowledgments

The authors thank the following individuals for their experience and contribution: Beverley White is the Clinical Documentation Improvement Coordinator at North Florida/South Georgia Veterans Health System and provided expertise on documentation requirements. Russell Jacobitz and Susan Rozelle provided technical expertise on electronic health record system enhancements and implemented the template design. Jess Delaune, MD, and Robert Carroll, MD, provided additional physician input during template design. We also acknowledge the Inpatient Evaluation Center (IPEC) within the Veterans Health Administration (VHA). IPEC developed the case severity index, a risk-adjusted formula incorporated into the VHA quality metric reporting system, Strategic Analytics for Improvement and Learning (SAIL).

Standardized template design is a useful tool to improve clinical documentation and reliable reporting of health care outcomes when constructed with clear objectives and with collaboration of key stakeholders. A standardized template should not only capture accurate diagnostic information, but also inform quality improvement (QI) measures and best practices.

Kang and colleagues showed that a correlation exists between organizational satisfaction and improved quality outcomes.¹ A new initiative should have a well-defined purpose reinforced by collaborative workgroups and engaged employees who understand their clinical care role with electronic health record (EHR) modifications.

Several studies have shown how the usefulness of templates achieve multipurpose goals, such as accurate documentation and improved care. Valluru and colleagues showed a significant increase in vaccination rates for patients with inflammatory bowel disease after implementing a standardized template.² By using a standardized template, Thaker and colleagues showed improved documentation regarding obesity and increased nutritional and physical activity counseling.³ Furthermore, Grogan and colleagues showed that templates are useful for house staff education on International Classification of Diseases (ICD) terminology and demonstrated improved documentation in the postintervention group.^4,5

This article discusses the US Department of Veterans Affairs (VA) North Florida/South Georgia Veterans Health System (NF/SGVHS) integrated informatics solutions within template design in the Veterans Health Administration (VHA) EHR system that was associated with an increase in its case severity index (CSI) through improved clinical documentation capture.

Methods

According to policy activities that constitute research at NF/SGVHS, institutional review board approval was not required as this work met the criteria for operational improvement activities exempt from ethics review.

NF/SGVHS includes 2 hospitals: Malcom Randall VA Medical Center (MRVAMC) in Gainesville, Florida, and Lake City VA Medical Center (LCVAMC) in Lake City, Florida. MRVAMC is a large, 1a, academic VA facility composed of rotating residents and fellows and includes multiple specialty care services. LCVAMC is a smaller, nonteaching facility.

Template Design Impact

CSI is a risk-adjusted formula developed by the Inpatient Evaluation Center within VHA. CSI is incorporated into the VHA quality metrics reporting system, Strategic Analytics for Improvement and Learning (SAIL). CSI risk-adjusts metrics such as length of stay and mortality before releasing SAIL reports. CSI is calculated separately for acute level of care (LOC) and for the intensive care unit (ICU). In fiscal year (FY) 2017, acute LOC preimplementation data for CSI at NF/SGVHS were 0.76 for MRVAMC and 0.81 for LCVAMC, which was significantly below the national VHA average of 0.96 (Table).

A below-average CSI conveys a less complicated case mix compared with most other VA facilities. Although smaller VA facilities may have a less complicated case mix, it is unusual for large, tertiary care 1a VA facilities to have a low CSI. This low CSI is usually due to inadequate documentation, which affects not only risk-adjusted quality metrics outcomes, but also potential reimbursement.⁶

An interdisciplinary team composed of attendings, residents, and a clinical document improvement specialist identified the below-average acute LOC CSI for MRVAMC and LCVAMC compared with that of the national VHA average. Further analysis by chart reviews showed inconsistencies with standardized documentation despite prior health care provider education on ICD terminology and specific groups of common comorbidities analyzed in administrative data reviews for risk-adjustment purposes, known as Elixhauser comorbidities.^5,7

A chart review showed lack of clarity regarding primary reason(s) for admission and chronic comorbidities within NF/SGVHS. Using Pareto chart analysis, the template team designed a standardized history and physical (H&P) medicine template based on NF/SGVHS common medicine admissions (Figure 1). A Pareto chart is a valuable QI tool that assists with identifying majority contributors to a problem(s) being analyzed when evaluating a large set of data points. Subsequently, this tool helps focus direction on QI efforts.⁸

The template had the usual H&P elements not shown (eg, chief complaint, history of present illness, etc), and highlights the assessment/plan section containing primary reason(s) for admission and chronic comorbidities (Figure 1). The complete assessment and plan section on the template can be found in the Appendix.

To simplify the template interface, only single clicks were required to expand diagnostic and chronic comorbidity checkboxes. Subcategories then appeared to select diagnosis and chronic comorbidities along with free text for additional documentation.

In addition, data objects were created within the template that permitted the ability to retrieve information from the VHA EHR and insert specific data points of interest in the template; for example, body mass index to assess degree of obesity and estimated glomerular filtration rate to determine the stage of chronic kidney disease. This allowed users to easily reference data in one template in lieu of searching for data in multiple places in the EHR.⁹

Results

The standardized H&P medicine template was implemented at MRVAMC and LCVAMC in June 2018 (the final month of the third quarter of FY 2018). As clinical providers throughout NF/SGVHS used the standardized template, acute LOC postimplementation data for CSI significantly improved. Although the national VHA average slightly decreased from 0.96 in the first quarter of FY 2017 to 0.89, in the first quarter of FY 2019, MRVAMC acute LOC CSI improved from 0.76 to 0.97, and LCVAMC acute LOC CSI improved from 0.81 to 1.07 during the same period.

In addition, compliance also was monitored within MRVAMC and LCVAMC for about 1 year after standardized H&P medicine template implementation. Compliance was determined by how often the standardized H&P medicine template was used for inpatient medicine admissions to the acute care wards vs other H&P notes used (such as personalized templates).

Methodology for compliance analysis included acquisition of completed H&P medicine notes from June 18, 2018 to June 30, 2019, within the VHA Veterans Information Systems and Technology Architecture (VistA) clinical and business information system using the search strings: “H&P admission history and physical” and “history of present illness.”¹⁰

A review identified 10,845 completed medicine H&P notes. Nine hundred eighteen notes were excluded as their search function yielded a location not corresponding to MRVAMC or LCVAMC. Of the 9,927 notes remaining, 8,025 of these were completed medicine H&P notes at MRVAMC and 1,902 were completed medicine H&P notes at LCVAMC (Figure 2).

From June 18, 2018 to June 30, 2019 at MRVAMC, compliance was reviewed monthly for the 8,025 completed H&P medicine notes. Of the completed H&P medicine notes, the standardized H&P medicine template was used 43.2% in June 2018. By June 2019, MRVAMC clinical providers demonstrated significant improvement for standardized H&P medicine template use at 89.9% (Figure 3). Total average compliance from June 18, 2018 to June 30, 2019, was 88.4%, which doubled compliance from the initial introduction of the standardized H&P medicine template.

Compliance was reviewed monthly for the 1,902 completed H&P medicine notes from June 18, 2018 to June 30, 2019, at LCVAMC. Of the completed H&P medicine notes, the standardized template was used 48.2% of the time in June 2018. By June 2019, LCVAMC clinical providers demonstrated significant improvement for standardized H&P medicine template use, which increased to 96.9%. Total average compliance from June 18, 2018 to June 30, 2019, was 93.8%, which was almost double the baseline compliance rate.

Discussion

Template design with clear objectives, strategic collaboration, and integrated informatics solutions has the potential to increase accuracy of documentation. As shown, the NF/SGVHS template design was associated with significant improvement in acute LOC CSI for both MRVAMC and LCVAMC due to more accurate documentation using the standardized H&P medicine template.

Numerous factors contributed to the success of this template design. First, a clear vision for application of the template was communicated with key stakeholders. In addition, the template design team was focused on specific goals rather than a one size fits all approach, which was crucial for sustainable execution. Although interdisciplinary teamwork has the potential to result in innovative practices, large multidisciplinary teams also may have difficulty establishing a shared vision that can result in barriers to achieving project goals.

Balancing standardization and customization was essential for user buy-in. As noted by Gardner and Pearce, inviting clinical providers to participate in template design and allowing for customization has the potential to increase acceptance and use of templates.¹¹ Although the original design for the standardized H&P medicine template started with the medicine service at NF/SGVHS, the design framework is applicable to numerous services where various clinical care elements can be customized.

Explaining the informatics tools built into the template allowed clinicians to see opportunities to improve clinical documentation and the impact it has on reporting health care outcomes. When improvement work involves integrating clinical care delivery and administrative expectations, it is essential that health care systems understand and strategically execute project initiatives at this critical juncture.

Finally, incorporation of a sustainability plan when process improvement strategies are implemented is vital. In addition to collaboration with the clinical providers during design and implementation of the standardized template, leadership buy-in was key. Compliance with standardized H&P medicine template use was monitored monthly and reviewed by the NF/SGVHS Chief of Staff.

As noted, LCVAMC postimplementation acute LOC CSI was higher than that of MRVAMC despite being a smaller facility. This might be due to the MRVAMC designation as a teaching institution. Medicine is the only inpatient service at LCVAMC staffed by hospitalists with limited specialists available for consultation, whereas MRVAMC is a tertiary care teaching facility with numerous inpatient services and subspecialties. As LCVAMC has more continuity, house staff rotating at MRVAMC require continued training/education on new templates and process changes.

Limitations

Although standardized template design was successful at NF/SGVHS, limitations should be noted. Our clinical documentation improvement (CDI) program also was expanded about the same time as the new templates were released. The expansion of the CDI program in addition to new template design likely had a synergistic effect on acute LOC CSI.

CSI is a complex, risk-adjusted model that includes numerous factors, including but not limited to diagnosis and comorbid conditions. Other factors include age, marital status, procedures, source of admission, specific laboratory values, medical or surgical diagnosis-related group, intensive care unit stays, and immunosuppressive status. CSI also includes operative and nonoperative components that average into an overall CSI. As the majority of CSI is composed of nonoperative constituents within NF/SGVHS, we do not believe this had any substantial impact on reporting of CSI improvements.

In addition, template entry into VHA EHR requires a location selection (such as a clinic name or ward name following an inpatient admission). Of the 10,845 completed H&P medicine notes identified in VistA, 918 notes were excluded from analysis as their search function yielded a location not corresponding to MRVAMC or LCVAMC. For the 918 notes excluded, we believe this was likely due to user error where locations not related to MRVAMC or LCVAMC were selected during standardized H&P medicine template entry.

Conclusions

After the NF/SGVHS implementation of a uniquely designed template embedded with informatics solutions within the VHA EHR, the CSI increased due to more accurate documentation.

Next steps include determining the impact of the NF/SGVHS template design on potential reimbursement and expanding template design into the outpatient setting where there are additional opportunities to improve clinical documentation and reliable reporting of health care outcomes.

Acknowledgments

The authors thank the following individuals for their experience and contribution: Beverley White is the Clinical Documentation Improvement Coordinator at North Florida/South Georgia Veterans Health System and provided expertise on documentation requirements. Russell Jacobitz and Susan Rozelle provided technical expertise on electronic health record system enhancements and implemented the template design. Jess Delaune, MD, and Robert Carroll, MD, provided additional physician input during template design. We also acknowledge the Inpatient Evaluation Center (IPEC) within the Veterans Health Administration (VHA). IPEC developed the case severity index, a risk-adjusted formula incorporated into the VHA quality metric reporting system, Strategic Analytics for Improvement and Learning (SAIL).