Brief Reports

Patient satisfaction surveys are widely used to empower patients to voice their concerns and point out areas of deficiency or excellence in the patient‐physician partnership and in the delivery of healthcare services.[1] In 2002, the Centers for Medicare and Medicaid Service (CMS) led an initiative to develop the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey questionnaire.[2] This survey is sent to a randomly selected subset of patients after hospital discharge. The HCAHPS instrument assesses patient ratings of physician communication, nursing communication, pain control, responsiveness, room cleanliness and quietness, discharge process, and overall satisfaction. Over 4500 acute‐care facilities routinely use this survey.[3] HCAHPS scores are publicly reported, and patients can utilize these scores to compare hospitals and make informed choices about where to get care. At an institutional level, scores are used as a tool to identify and improve deficiencies in care delivery. Additionally, HCAHPS survey data results have been analyzed in numerous research studies.[4, 5, 6]

Specialty hospitals are a subset of acute‐care hospitals that provide a narrower set of services than general medical hospitals (GMHs), predominantly in a few specialty areas such as cardiac disease and surgical fields. Many specialty hospitals advertise high rates of patient satisfaction.[7, 8, 9, 10, 11] However, specialty hospitals differ from GMHs in significant ways. Patients at specialty hospitals may be less severely ill[10, 12] and may have more generous insurance coverage.[13] Many specialty hospitals do not have an emergency department (ED), and their outcomes may reflect care of relatively stable patients.[14] A significant number of the specialty hospitals are physician‐owned, which may provide an opportunity for physicians to deliver more patient‐focused healthcare.[14] It is also thought that specialty hospitals can provide high‐quality care by designing their facilities and service structure entirely to meet the needs of a narrow set of medical conditions.

HCAHPS survey results provide an opportunity to compare satisfaction scores among various types of hospitals. We analyzed national HCAHPS data to compare satisfaction scores of specialty hospitals and GMHs and identify factors that may be responsible for this difference.

METHODS

This was a cross‐sectional analysis of national HCAHPS survey data. The methods for administration and reporting of the HCAHPS survey have been described.[15] HCAHPS patient satisfaction data and hospital characteristics, such as location, presence of an ED, and for‐profit status, were obtained from Hospital Compare database. Teaching hospital status was identified using the CMS 2013 Open Payment teaching hospital listing.[16]

For this study, we defined specialty hospitals as acute‐care hospitals that predominantly provide care in a medical or surgical specialty and do not provide care to general medical patients. Based on this definition, specialty hospitals include cardiac hospitals, orthopedic and spine hospitals, oncology hospitals, and hospitals providing multispecialty surgical and procedure‐based services. Children's hospitals, long‐term acute‐care hospitals, and psychiatry hospitals were excluded.

Specialty hospitals were identified using hospital name searches in the HCAHPS database, the American Hospital Association 2013 Annual Survey, the Physician Hospital Association hospitals directory, and through contact with experts. The specialty hospital status of hospitals was further confirmed by checking hospital websites or by directly contacting the hospital.

We analyzed 3‐year HCAHPS patient satisfaction data that included the reporting period from July 2007 to June 2010. HCAHPS data are reported for 12‐month periods at a time. Hospital information, such as address, presence of an ED, and for‐profit status were obtained from the CMS Hospital Compare 2010 dataset. Teaching hospital status was identified using the CMS 2013 Open Payment teaching hospital listing.[16] For the purpose of this study, scores on the HCAHPS survey item definitely recommend the hospital was considered to represent overall satisfaction for the hospital. This is consistent with use of this measure in other sectors in the service industry.[17, 18] Other survey items were considered subdomains of satisfaction. For each hospital, the simple mean of satisfaction scores for overall satisfaction and each of the subdomains for the three 12‐month periods was calculated. Data were summarized using frequencies and meanstandard deviation. The primary dependent variable was overall satisfaction. The main independent variables were specialty hospital status (yes or no), teaching hospital status (yes or no), for‐profit status (yes or no), and the presence of an ED (yes or no). Multiple linear regression analysis was used to adjust for the above‐noted independent variables. A P value0.05 was considered significant. All analyses were performed on Stata 10.1 IC (StataCorp, College Station, TX).

RESULTS

We identified 188 specialty hospitals and 4638 GMHs within the HCAHPS dataset. Fewer specialty hospitals had emergency care services when compared with GMHs (53.2% for specialty hospitals vs 93.6% for GMHs, P0.0001), and 47.9% of all specialty hospitals were in states that do not require a Certificate of Need, whereas only 25% of all GMHs were present in these states. For example, Texas, which has 7.2% of all GMHs across the nation, has 24.7% of all specialty hospitals. As compared to GMHs, a majority of specialty hospitals were for profit (14.5% vs 66.9%).

In unadjusted analyses, specialty hospitals had significantly higher patient satisfaction scores compared with GMHs. Overall satisfaction, as measured by the proportion of patients that will definitely recommend that hospital, was 18.8% higher for specialty hospitals than GMHs (86.6% vs 67.8%, P0.0001). This was also true for subdomains of satisfaction including physician communication, nursing communication, and cleanliness (Table 1).

We next examined the effect of survey response rate. The survey response rate for specialty hospitals was on average 17.4 percentage points higher than that of GMHs (49.6% vs 32.2%, P0.0001). When adjusted for survey response rate, the difference in overall satisfaction for specialty hospitals was reduced to 8.6% (6.7%10.5%, P0.0001). Similarly, the differences in score for subdomains of satisfaction were more modest when adjusted for higher survey response rate. In the multiple regression models, specialty hospital status, survey response rate, for‐profit status, and the presence of an ED were independently associated with higher overall satisfaction, whereas teaching hospital status was not associated with overall satisfaction. Addition of for‐profit status and presence of an ED in the regression model did not change our results. Further, the satisfaction subdomain scores for specialty hospitals remained significantly higher than for GMHs in the regression models (Table 1).

DISCUSSION

In this national study, we found that specialty hospitals had significantly higher overall satisfaction scores on the HCAHPS satisfaction survey. Similarly, significantly higher satisfaction was noted across all the satisfaction subdomains. We found that a large proportion of the difference between specialty hospitals and GMHs in overall satisfaction and subdomains of satisfaction could be explained by a higher survey response rate in specialty hospitals. After adjusting for survey response rate, the differences were comparatively modest, although remained statistically significant. Adjustment for additional confounding variables did not change our results.

Studies have shown that specialty hospitals, when compared to GMHs, may treat more patients in their area of specialization, care for fewer sick and Medicaid patients, have greater physician ownership, and are less likely to have ED services.[11, 12, 13, 14] Two small studies comparing specialty hospitals to GMHs suggest that higher satisfaction with specialty hospitals was attributable to the presence of private rooms, quiet environment, accommodation for family members, and accessible, attentive, and well‐trained nursing staff.[10, 11] Although our analysis did not account for various other hospital and patient characteristics, we expect that these factors likely play a significant role in the observed differences in patient satisfaction.

Survey response rate can be an important determinant of the validity of survey results, and a response rate >70% is often considered desirable.[19, 20] However, the mean survey response rate for the HCAHPS survey was only 32.8% for all hospitals during the survey period. In the outpatient setting, a higher survey response rate has been shown to be associated with higher satisfaction rates.[21] In the hospital setting, a randomized study of a HCAHPS survey for 45 hospitals found that patient mix explained the nonresponse bias. However, this study did not examine the roles of severity of illness or insurance status, which may account for the differences in satisfaction seen between specialty hospitals and GMHs.[22] In contrast, we found that in the hospital setting, higher survey response rate was associated with higher patient satisfaction scores.

Our study has some limitations. First, it was not possible to determine from the dataset whether higher response rate is a result of differences in the patient population characteristics between specialty hospitals and GMHs or it represents the association between higher satisfaction and higher response rate noted by other investigators. Although we used various resources to identify all specialty hospitals, we may have missed some or misclassified others due to lack of a standardized definition.[10, 12, 13] However, the total number of specialty hospitals and their distribution across various states in the current study are consistent with previous studies, supporting our belief that few, if any, hospitals were misclassified.[13]

In summary, we found significant difference in satisfaction rates reported on HCAHPS in a national study of patients attending specialty hospitals versus GMHs. However, the observed differences in satisfaction scores were sensitive to differences in survey response rates among hospitals. Teaching hospital status, for‐profit status, and the presence of an ED did not appear to further explain the differences. Additional studies incorporating other hospital and patient characteristics are needed to fully understand factors associated with differences in the observed patient satisfaction between specialty hospitals and GMHs. Additionally, strategies to increase survey HCAHPS response rates should be a priority.

Patient satisfaction surveys are widely used to empower patients to voice their concerns and point out areas of deficiency or excellence in the patient‐physician partnership and in the delivery of healthcare services.[1] In 2002, the Centers for Medicare and Medicaid Service (CMS) led an initiative to develop the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey questionnaire.[2] This survey is sent to a randomly selected subset of patients after hospital discharge. The HCAHPS instrument assesses patient ratings of physician communication, nursing communication, pain control, responsiveness, room cleanliness and quietness, discharge process, and overall satisfaction. Over 4500 acute‐care facilities routinely use this survey.[3] HCAHPS scores are publicly reported, and patients can utilize these scores to compare hospitals and make informed choices about where to get care. At an institutional level, scores are used as a tool to identify and improve deficiencies in care delivery. Additionally, HCAHPS survey data results have been analyzed in numerous research studies.[4, 5, 6]

Specialty hospitals are a subset of acute‐care hospitals that provide a narrower set of services than general medical hospitals (GMHs), predominantly in a few specialty areas such as cardiac disease and surgical fields. Many specialty hospitals advertise high rates of patient satisfaction.[7, 8, 9, 10, 11] However, specialty hospitals differ from GMHs in significant ways. Patients at specialty hospitals may be less severely ill[10, 12] and may have more generous insurance coverage.[13] Many specialty hospitals do not have an emergency department (ED), and their outcomes may reflect care of relatively stable patients.[14] A significant number of the specialty hospitals are physician‐owned, which may provide an opportunity for physicians to deliver more patient‐focused healthcare.[14] It is also thought that specialty hospitals can provide high‐quality care by designing their facilities and service structure entirely to meet the needs of a narrow set of medical conditions.

HCAHPS survey results provide an opportunity to compare satisfaction scores among various types of hospitals. We analyzed national HCAHPS data to compare satisfaction scores of specialty hospitals and GMHs and identify factors that may be responsible for this difference.

METHODS

This was a cross‐sectional analysis of national HCAHPS survey data. The methods for administration and reporting of the HCAHPS survey have been described.[15] HCAHPS patient satisfaction data and hospital characteristics, such as location, presence of an ED, and for‐profit status, were obtained from Hospital Compare database. Teaching hospital status was identified using the CMS 2013 Open Payment teaching hospital listing.[16]

For this study, we defined specialty hospitals as acute‐care hospitals that predominantly provide care in a medical or surgical specialty and do not provide care to general medical patients. Based on this definition, specialty hospitals include cardiac hospitals, orthopedic and spine hospitals, oncology hospitals, and hospitals providing multispecialty surgical and procedure‐based services. Children's hospitals, long‐term acute‐care hospitals, and psychiatry hospitals were excluded.

Specialty hospitals were identified using hospital name searches in the HCAHPS database, the American Hospital Association 2013 Annual Survey, the Physician Hospital Association hospitals directory, and through contact with experts. The specialty hospital status of hospitals was further confirmed by checking hospital websites or by directly contacting the hospital.

We analyzed 3‐year HCAHPS patient satisfaction data that included the reporting period from July 2007 to June 2010. HCAHPS data are reported for 12‐month periods at a time. Hospital information, such as address, presence of an ED, and for‐profit status were obtained from the CMS Hospital Compare 2010 dataset. Teaching hospital status was identified using the CMS 2013 Open Payment teaching hospital listing.[16] For the purpose of this study, scores on the HCAHPS survey item definitely recommend the hospital was considered to represent overall satisfaction for the hospital. This is consistent with use of this measure in other sectors in the service industry.[17, 18] Other survey items were considered subdomains of satisfaction. For each hospital, the simple mean of satisfaction scores for overall satisfaction and each of the subdomains for the three 12‐month periods was calculated. Data were summarized using frequencies and meanstandard deviation. The primary dependent variable was overall satisfaction. The main independent variables were specialty hospital status (yes or no), teaching hospital status (yes or no), for‐profit status (yes or no), and the presence of an ED (yes or no). Multiple linear regression analysis was used to adjust for the above‐noted independent variables. A P value0.05 was considered significant. All analyses were performed on Stata 10.1 IC (StataCorp, College Station, TX).

RESULTS

We identified 188 specialty hospitals and 4638 GMHs within the HCAHPS dataset. Fewer specialty hospitals had emergency care services when compared with GMHs (53.2% for specialty hospitals vs 93.6% for GMHs, P0.0001), and 47.9% of all specialty hospitals were in states that do not require a Certificate of Need, whereas only 25% of all GMHs were present in these states. For example, Texas, which has 7.2% of all GMHs across the nation, has 24.7% of all specialty hospitals. As compared to GMHs, a majority of specialty hospitals were for profit (14.5% vs 66.9%).

In unadjusted analyses, specialty hospitals had significantly higher patient satisfaction scores compared with GMHs. Overall satisfaction, as measured by the proportion of patients that will definitely recommend that hospital, was 18.8% higher for specialty hospitals than GMHs (86.6% vs 67.8%, P0.0001). This was also true for subdomains of satisfaction including physician communication, nursing communication, and cleanliness (Table 1).

We next examined the effect of survey response rate. The survey response rate for specialty hospitals was on average 17.4 percentage points higher than that of GMHs (49.6% vs 32.2%, P0.0001). When adjusted for survey response rate, the difference in overall satisfaction for specialty hospitals was reduced to 8.6% (6.7%10.5%, P0.0001). Similarly, the differences in score for subdomains of satisfaction were more modest when adjusted for higher survey response rate. In the multiple regression models, specialty hospital status, survey response rate, for‐profit status, and the presence of an ED were independently associated with higher overall satisfaction, whereas teaching hospital status was not associated with overall satisfaction. Addition of for‐profit status and presence of an ED in the regression model did not change our results. Further, the satisfaction subdomain scores for specialty hospitals remained significantly higher than for GMHs in the regression models (Table 1).

DISCUSSION

In this national study, we found that specialty hospitals had significantly higher overall satisfaction scores on the HCAHPS satisfaction survey. Similarly, significantly higher satisfaction was noted across all the satisfaction subdomains. We found that a large proportion of the difference between specialty hospitals and GMHs in overall satisfaction and subdomains of satisfaction could be explained by a higher survey response rate in specialty hospitals. After adjusting for survey response rate, the differences were comparatively modest, although remained statistically significant. Adjustment for additional confounding variables did not change our results.

Studies have shown that specialty hospitals, when compared to GMHs, may treat more patients in their area of specialization, care for fewer sick and Medicaid patients, have greater physician ownership, and are less likely to have ED services.[11, 12, 13, 14] Two small studies comparing specialty hospitals to GMHs suggest that higher satisfaction with specialty hospitals was attributable to the presence of private rooms, quiet environment, accommodation for family members, and accessible, attentive, and well‐trained nursing staff.[10, 11] Although our analysis did not account for various other hospital and patient characteristics, we expect that these factors likely play a significant role in the observed differences in patient satisfaction.

Survey response rate can be an important determinant of the validity of survey results, and a response rate >70% is often considered desirable.[19, 20] However, the mean survey response rate for the HCAHPS survey was only 32.8% for all hospitals during the survey period. In the outpatient setting, a higher survey response rate has been shown to be associated with higher satisfaction rates.[21] In the hospital setting, a randomized study of a HCAHPS survey for 45 hospitals found that patient mix explained the nonresponse bias. However, this study did not examine the roles of severity of illness or insurance status, which may account for the differences in satisfaction seen between specialty hospitals and GMHs.[22] In contrast, we found that in the hospital setting, higher survey response rate was associated with higher patient satisfaction scores.

Our study has some limitations. First, it was not possible to determine from the dataset whether higher response rate is a result of differences in the patient population characteristics between specialty hospitals and GMHs or it represents the association between higher satisfaction and higher response rate noted by other investigators. Although we used various resources to identify all specialty hospitals, we may have missed some or misclassified others due to lack of a standardized definition.[10, 12, 13] However, the total number of specialty hospitals and their distribution across various states in the current study are consistent with previous studies, supporting our belief that few, if any, hospitals were misclassified.[13]

In summary, we found significant difference in satisfaction rates reported on HCAHPS in a national study of patients attending specialty hospitals versus GMHs. However, the observed differences in satisfaction scores were sensitive to differences in survey response rates among hospitals. Teaching hospital status, for‐profit status, and the presence of an ED did not appear to further explain the differences. Additional studies incorporating other hospital and patient characteristics are needed to fully understand factors associated with differences in the observed patient satisfaction between specialty hospitals and GMHs. Additionally, strategies to increase survey HCAHPS response rates should be a priority.

Patient satisfaction surveys are widely used to empower patients to voice their concerns and point out areas of deficiency or excellence in the patient‐physician partnership and in the delivery of healthcare services.[1] In 2002, the Centers for Medicare and Medicaid Service (CMS) led an initiative to develop the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey questionnaire.[2] This survey is sent to a randomly selected subset of patients after hospital discharge. The HCAHPS instrument assesses patient ratings of physician communication, nursing communication, pain control, responsiveness, room cleanliness and quietness, discharge process, and overall satisfaction. Over 4500 acute‐care facilities routinely use this survey.[3] HCAHPS scores are publicly reported, and patients can utilize these scores to compare hospitals and make informed choices about where to get care. At an institutional level, scores are used as a tool to identify and improve deficiencies in care delivery. Additionally, HCAHPS survey data results have been analyzed in numerous research studies.[4, 5, 6]

Specialty hospitals are a subset of acute‐care hospitals that provide a narrower set of services than general medical hospitals (GMHs), predominantly in a few specialty areas such as cardiac disease and surgical fields. Many specialty hospitals advertise high rates of patient satisfaction.[7, 8, 9, 10, 11] However, specialty hospitals differ from GMHs in significant ways. Patients at specialty hospitals may be less severely ill[10, 12] and may have more generous insurance coverage.[13] Many specialty hospitals do not have an emergency department (ED), and their outcomes may reflect care of relatively stable patients.[14] A significant number of the specialty hospitals are physician‐owned, which may provide an opportunity for physicians to deliver more patient‐focused healthcare.[14] It is also thought that specialty hospitals can provide high‐quality care by designing their facilities and service structure entirely to meet the needs of a narrow set of medical conditions.

HCAHPS survey results provide an opportunity to compare satisfaction scores among various types of hospitals. We analyzed national HCAHPS data to compare satisfaction scores of specialty hospitals and GMHs and identify factors that may be responsible for this difference.

METHODS

This was a cross‐sectional analysis of national HCAHPS survey data. The methods for administration and reporting of the HCAHPS survey have been described.[15] HCAHPS patient satisfaction data and hospital characteristics, such as location, presence of an ED, and for‐profit status, were obtained from Hospital Compare database. Teaching hospital status was identified using the CMS 2013 Open Payment teaching hospital listing.[16]

For this study, we defined specialty hospitals as acute‐care hospitals that predominantly provide care in a medical or surgical specialty and do not provide care to general medical patients. Based on this definition, specialty hospitals include cardiac hospitals, orthopedic and spine hospitals, oncology hospitals, and hospitals providing multispecialty surgical and procedure‐based services. Children's hospitals, long‐term acute‐care hospitals, and psychiatry hospitals were excluded.

Specialty hospitals were identified using hospital name searches in the HCAHPS database, the American Hospital Association 2013 Annual Survey, the Physician Hospital Association hospitals directory, and through contact with experts. The specialty hospital status of hospitals was further confirmed by checking hospital websites or by directly contacting the hospital.

We analyzed 3‐year HCAHPS patient satisfaction data that included the reporting period from July 2007 to June 2010. HCAHPS data are reported for 12‐month periods at a time. Hospital information, such as address, presence of an ED, and for‐profit status were obtained from the CMS Hospital Compare 2010 dataset. Teaching hospital status was identified using the CMS 2013 Open Payment teaching hospital listing.[16] For the purpose of this study, scores on the HCAHPS survey item definitely recommend the hospital was considered to represent overall satisfaction for the hospital. This is consistent with use of this measure in other sectors in the service industry.[17, 18] Other survey items were considered subdomains of satisfaction. For each hospital, the simple mean of satisfaction scores for overall satisfaction and each of the subdomains for the three 12‐month periods was calculated. Data were summarized using frequencies and meanstandard deviation. The primary dependent variable was overall satisfaction. The main independent variables were specialty hospital status (yes or no), teaching hospital status (yes or no), for‐profit status (yes or no), and the presence of an ED (yes or no). Multiple linear regression analysis was used to adjust for the above‐noted independent variables. A P value0.05 was considered significant. All analyses were performed on Stata 10.1 IC (StataCorp, College Station, TX).

RESULTS

We identified 188 specialty hospitals and 4638 GMHs within the HCAHPS dataset. Fewer specialty hospitals had emergency care services when compared with GMHs (53.2% for specialty hospitals vs 93.6% for GMHs, P0.0001), and 47.9% of all specialty hospitals were in states that do not require a Certificate of Need, whereas only 25% of all GMHs were present in these states. For example, Texas, which has 7.2% of all GMHs across the nation, has 24.7% of all specialty hospitals. As compared to GMHs, a majority of specialty hospitals were for profit (14.5% vs 66.9%).

In unadjusted analyses, specialty hospitals had significantly higher patient satisfaction scores compared with GMHs. Overall satisfaction, as measured by the proportion of patients that will definitely recommend that hospital, was 18.8% higher for specialty hospitals than GMHs (86.6% vs 67.8%, P0.0001). This was also true for subdomains of satisfaction including physician communication, nursing communication, and cleanliness (Table 1).

We next examined the effect of survey response rate. The survey response rate for specialty hospitals was on average 17.4 percentage points higher than that of GMHs (49.6% vs 32.2%, P0.0001). When adjusted for survey response rate, the difference in overall satisfaction for specialty hospitals was reduced to 8.6% (6.7%10.5%, P0.0001). Similarly, the differences in score for subdomains of satisfaction were more modest when adjusted for higher survey response rate. In the multiple regression models, specialty hospital status, survey response rate, for‐profit status, and the presence of an ED were independently associated with higher overall satisfaction, whereas teaching hospital status was not associated with overall satisfaction. Addition of for‐profit status and presence of an ED in the regression model did not change our results. Further, the satisfaction subdomain scores for specialty hospitals remained significantly higher than for GMHs in the regression models (Table 1).

DISCUSSION

In this national study, we found that specialty hospitals had significantly higher overall satisfaction scores on the HCAHPS satisfaction survey. Similarly, significantly higher satisfaction was noted across all the satisfaction subdomains. We found that a large proportion of the difference between specialty hospitals and GMHs in overall satisfaction and subdomains of satisfaction could be explained by a higher survey response rate in specialty hospitals. After adjusting for survey response rate, the differences were comparatively modest, although remained statistically significant. Adjustment for additional confounding variables did not change our results.

Studies have shown that specialty hospitals, when compared to GMHs, may treat more patients in their area of specialization, care for fewer sick and Medicaid patients, have greater physician ownership, and are less likely to have ED services.[11, 12, 13, 14] Two small studies comparing specialty hospitals to GMHs suggest that higher satisfaction with specialty hospitals was attributable to the presence of private rooms, quiet environment, accommodation for family members, and accessible, attentive, and well‐trained nursing staff.[10, 11] Although our analysis did not account for various other hospital and patient characteristics, we expect that these factors likely play a significant role in the observed differences in patient satisfaction.

Survey response rate can be an important determinant of the validity of survey results, and a response rate >70% is often considered desirable.[19, 20] However, the mean survey response rate for the HCAHPS survey was only 32.8% for all hospitals during the survey period. In the outpatient setting, a higher survey response rate has been shown to be associated with higher satisfaction rates.[21] In the hospital setting, a randomized study of a HCAHPS survey for 45 hospitals found that patient mix explained the nonresponse bias. However, this study did not examine the roles of severity of illness or insurance status, which may account for the differences in satisfaction seen between specialty hospitals and GMHs.[22] In contrast, we found that in the hospital setting, higher survey response rate was associated with higher patient satisfaction scores.

Our study has some limitations. First, it was not possible to determine from the dataset whether higher response rate is a result of differences in the patient population characteristics between specialty hospitals and GMHs or it represents the association between higher satisfaction and higher response rate noted by other investigators. Although we used various resources to identify all specialty hospitals, we may have missed some or misclassified others due to lack of a standardized definition.[10, 12, 13] However, the total number of specialty hospitals and their distribution across various states in the current study are consistent with previous studies, supporting our belief that few, if any, hospitals were misclassified.[13]

In summary, we found significant difference in satisfaction rates reported on HCAHPS in a national study of patients attending specialty hospitals versus GMHs. However, the observed differences in satisfaction scores were sensitive to differences in survey response rates among hospitals. Teaching hospital status, for‐profit status, and the presence of an ED did not appear to further explain the differences. Additional studies incorporating other hospital and patient characteristics are needed to fully understand factors associated with differences in the observed patient satisfaction between specialty hospitals and GMHs. Additionally, strategies to increase survey HCAHPS response rates should be a priority.

Applications of point‐of‐care ultrasonography (POC‐US) have grown rapidly over the past 20 years. POC‐US training is required by the Accreditation Council for Graduate Medical Education for several graduate medical education training programs, including emergency medicine residency and pulmonary/critical care fellowships.[1] Recent efforts have examined the utility of ultrasound in the education of medical students[2] and the diagnostic and procedural applications performed by residents.[3] One powerful application of POC‐US is the use of lung ultrasound to diagnose causes of respiratory failure at the bedside.[4] Although lung ultrasound has been shown to have superior diagnostic accuracy to chest x‐rays,[5] limited availability of expert physicians and ultrasound equipment have presented barriers to wider application. The advent of lower cost pocket ultrasounds may present a solution given the early reports of similar efficacy to traditional devices in the assessment of left ventricular dysfunction, acute decompensated heart failure,[6] and focused assessment with sonography for trauma.[7] We assessed the feasibility and diagnostic accuracy of residents trained in lung ultrasound with a pocket device for evaluating patients with dyspnea.

MATERIALS AND METHODS

Diagnostic Protocol

Upon admission, residents recorded a clinical diagnosis of dyspnea based on a standard diagnostic evaluation including complete history, physical exam, and all relevant laboratory and imaging studies, including chest x‐ray and computed tomography (CT) scans. A diagnosis of dyspnea after lung ultrasound was then recorded based on the lung ultrasound findings and integrated with all other clinical information available. Standard lung ultrasound patterns and diagnostic correlates are shown in Figure 1. Diagnoses of dyspnea were recorded as one of 7 possibilities; 1) exacerbation of chronic obstructive pulmonary disease or asthma (COPD/asthma), 2) acute pulmonary edema (APE), 3) pneumonia (PNA), 4) pulmonary embolus (PE), 5) pneumothorax (PTX), 6) pleural effusion (PLEFF), and 7) other (OTH), namely anemia, ascites, and dehydration.

Figure 1

RESULTS

Five residents performed lung ultrasound on a convenience sample of 69 newly admitted patients. Patient baseline characteristics are shown in Table 1. Three residents made up the focused training group and examined 21 patients, resulting in 27 clinical diagnoses, 27 ultrasound diagnoses, and 31 final attending diagnoses. Two residents made up the extended training group and examined 48 patients, resulting in 61 clinical diagnoses, 60 ultrasound diagnoses, and 60 final attending diagnoses. Improvements in sensitivity and specificity using lung ultrasound were more pronounced for the extended training group and are shown for each diagnosis in Table 2.

Overall diagnostic accuracy using lung ultrasound improved only for the extended training group (clinical 92% vs ultrasound 97%), whereas the focused training group's accuracy was unchanged (clinical 87% vs ultrasound 88%).

ROC analysis demonstrated a superior diagnostic performance of ultrasound when compared to clinical diagnosis (Table 3).

DISCUSSION

In this prospective, observational study of residents performing lung ultrasound of patients with dyspnea, the diagnostic accuracy incorporating ultrasound increased compared to a standard diagnostic approach relying on history, physical exam, blood tests, and radiography. To our knowledge, this is the first study of residents independently performing lung ultrasound with a pocket ultrasound to diagnose dyspnea. Receiver operating curve analysis shows improvements in diagnostic accuracy for causes such as PNA, pleural effusion and COPD/asthma and demonstrates the feasibility and clinical utility of residents using pocket ultrasounds. The finding that improvements in sensitivity and specificity were larger in the extended training group highlights the need for sufficient training to demonstrate increased utility. Although a 2‐week critical care ultrasound elective may not be possible for all residents, perhaps training of intensity somewhere in between these 2 levels would be most feasible.

Challenges in diagnosing dyspnea have been well described, attributed to a lack of accurate history combined with often insensitive and nonspecific physical exam findings, blood tests, and radiographs.[8, 9] Further, patients often present with multiple contributing causes as was evidenced in this study.[10] Lack of initial, accurate diagnoses often leads to the provision of multiple, incorrect treatment regimens that may increase mortality.[11] The high accuracy of lung ultrasound in defining causes of respiratory failure suggests potential as a low‐cost solution.[12]

This study design differed from prior work in several respects. First, it included patients presenting with dyspnea to a hospital ward rather than acute respiratory failure to an intensive care unit (ICU), suggesting its diagnostic potential in a broader population of patients and settings. Second, the lung ultrasound was integrated with traditional clinical information rather than relied upon alone, a situation mimicking real‐world application of POC‐US. Third, operators were residents with limited amounts of training rather than highly trained experts. Finally, the lung ultrasound exams were performed using a pocket ultrasound with inferior imaging capability than larger, more established ultrasound devices. Despite these constraints, the utility of lung ultrasound was still evident, particularly in the diagnosis or exclusion of pneumonia and PLEFF.

Limitations include reliance on a small cohort of highly motivated residents with an interest in pulmonary and critical care, 2 who are authors of this article, making reproducibility a concern. Although convenience sampling may more closely mimic real world practices of POC‐US, a bias toward less challenging patients is possible and may limit conclusions regarding utility. Over‐reading and feedback were not provided to residents to improve their performance of lung ultrasound exams. Also, because chest CT is considered the gold standard in most studies examining the diagnostic accuracy of lung ultrasound, all residents aware of these data may underestimate the potential impact of integrating lung ultrasound with all clinical findings. Finally, the high cost of pocket ultrasounds is a barrier to general use. Recent studies on the significant cost savings associated with POC‐US make a further analysis of cost‐benefit ratios mandatory before broad use can be recommended.[13]

CONCLUSIONS

Residents participating in lung ultrasound training with a pocket ultrasound device showed improved diagnostic accuracy in their evaluation of patients with dyspnea. Those who received extended training had greater improvements across all causes of dyspnea. Training residents to apply lung ultrasound in non‐ICU settings appears to be feasible. Further study with a larger cohort of internal medicine residents and perhaps training duration that lies in between the focused and extended training groups is warranted.

Applications of point‐of‐care ultrasonography (POC‐US) have grown rapidly over the past 20 years. POC‐US training is required by the Accreditation Council for Graduate Medical Education for several graduate medical education training programs, including emergency medicine residency and pulmonary/critical care fellowships.[1] Recent efforts have examined the utility of ultrasound in the education of medical students[2] and the diagnostic and procedural applications performed by residents.[3] One powerful application of POC‐US is the use of lung ultrasound to diagnose causes of respiratory failure at the bedside.[4] Although lung ultrasound has been shown to have superior diagnostic accuracy to chest x‐rays,[5] limited availability of expert physicians and ultrasound equipment have presented barriers to wider application. The advent of lower cost pocket ultrasounds may present a solution given the early reports of similar efficacy to traditional devices in the assessment of left ventricular dysfunction, acute decompensated heart failure,[6] and focused assessment with sonography for trauma.[7] We assessed the feasibility and diagnostic accuracy of residents trained in lung ultrasound with a pocket device for evaluating patients with dyspnea.

MATERIALS AND METHODS

Diagnostic Protocol

Upon admission, residents recorded a clinical diagnosis of dyspnea based on a standard diagnostic evaluation including complete history, physical exam, and all relevant laboratory and imaging studies, including chest x‐ray and computed tomography (CT) scans. A diagnosis of dyspnea after lung ultrasound was then recorded based on the lung ultrasound findings and integrated with all other clinical information available. Standard lung ultrasound patterns and diagnostic correlates are shown in Figure 1. Diagnoses of dyspnea were recorded as one of 7 possibilities; 1) exacerbation of chronic obstructive pulmonary disease or asthma (COPD/asthma), 2) acute pulmonary edema (APE), 3) pneumonia (PNA), 4) pulmonary embolus (PE), 5) pneumothorax (PTX), 6) pleural effusion (PLEFF), and 7) other (OTH), namely anemia, ascites, and dehydration.

Figure 1

RESULTS

Five residents performed lung ultrasound on a convenience sample of 69 newly admitted patients. Patient baseline characteristics are shown in Table 1. Three residents made up the focused training group and examined 21 patients, resulting in 27 clinical diagnoses, 27 ultrasound diagnoses, and 31 final attending diagnoses. Two residents made up the extended training group and examined 48 patients, resulting in 61 clinical diagnoses, 60 ultrasound diagnoses, and 60 final attending diagnoses. Improvements in sensitivity and specificity using lung ultrasound were more pronounced for the extended training group and are shown for each diagnosis in Table 2.

Overall diagnostic accuracy using lung ultrasound improved only for the extended training group (clinical 92% vs ultrasound 97%), whereas the focused training group's accuracy was unchanged (clinical 87% vs ultrasound 88%).

ROC analysis demonstrated a superior diagnostic performance of ultrasound when compared to clinical diagnosis (Table 3).

DISCUSSION

In this prospective, observational study of residents performing lung ultrasound of patients with dyspnea, the diagnostic accuracy incorporating ultrasound increased compared to a standard diagnostic approach relying on history, physical exam, blood tests, and radiography. To our knowledge, this is the first study of residents independently performing lung ultrasound with a pocket ultrasound to diagnose dyspnea. Receiver operating curve analysis shows improvements in diagnostic accuracy for causes such as PNA, pleural effusion and COPD/asthma and demonstrates the feasibility and clinical utility of residents using pocket ultrasounds. The finding that improvements in sensitivity and specificity were larger in the extended training group highlights the need for sufficient training to demonstrate increased utility. Although a 2‐week critical care ultrasound elective may not be possible for all residents, perhaps training of intensity somewhere in between these 2 levels would be most feasible.

Challenges in diagnosing dyspnea have been well described, attributed to a lack of accurate history combined with often insensitive and nonspecific physical exam findings, blood tests, and radiographs.[8, 9] Further, patients often present with multiple contributing causes as was evidenced in this study.[10] Lack of initial, accurate diagnoses often leads to the provision of multiple, incorrect treatment regimens that may increase mortality.[11] The high accuracy of lung ultrasound in defining causes of respiratory failure suggests potential as a low‐cost solution.[12]

This study design differed from prior work in several respects. First, it included patients presenting with dyspnea to a hospital ward rather than acute respiratory failure to an intensive care unit (ICU), suggesting its diagnostic potential in a broader population of patients and settings. Second, the lung ultrasound was integrated with traditional clinical information rather than relied upon alone, a situation mimicking real‐world application of POC‐US. Third, operators were residents with limited amounts of training rather than highly trained experts. Finally, the lung ultrasound exams were performed using a pocket ultrasound with inferior imaging capability than larger, more established ultrasound devices. Despite these constraints, the utility of lung ultrasound was still evident, particularly in the diagnosis or exclusion of pneumonia and PLEFF.

Limitations include reliance on a small cohort of highly motivated residents with an interest in pulmonary and critical care, 2 who are authors of this article, making reproducibility a concern. Although convenience sampling may more closely mimic real world practices of POC‐US, a bias toward less challenging patients is possible and may limit conclusions regarding utility. Over‐reading and feedback were not provided to residents to improve their performance of lung ultrasound exams. Also, because chest CT is considered the gold standard in most studies examining the diagnostic accuracy of lung ultrasound, all residents aware of these data may underestimate the potential impact of integrating lung ultrasound with all clinical findings. Finally, the high cost of pocket ultrasounds is a barrier to general use. Recent studies on the significant cost savings associated with POC‐US make a further analysis of cost‐benefit ratios mandatory before broad use can be recommended.[13]

CONCLUSIONS

Residents participating in lung ultrasound training with a pocket ultrasound device showed improved diagnostic accuracy in their evaluation of patients with dyspnea. Those who received extended training had greater improvements across all causes of dyspnea. Training residents to apply lung ultrasound in non‐ICU settings appears to be feasible. Further study with a larger cohort of internal medicine residents and perhaps training duration that lies in between the focused and extended training groups is warranted.

Applications of point‐of‐care ultrasonography (POC‐US) have grown rapidly over the past 20 years. POC‐US training is required by the Accreditation Council for Graduate Medical Education for several graduate medical education training programs, including emergency medicine residency and pulmonary/critical care fellowships.[1] Recent efforts have examined the utility of ultrasound in the education of medical students[2] and the diagnostic and procedural applications performed by residents.[3] One powerful application of POC‐US is the use of lung ultrasound to diagnose causes of respiratory failure at the bedside.[4] Although lung ultrasound has been shown to have superior diagnostic accuracy to chest x‐rays,[5] limited availability of expert physicians and ultrasound equipment have presented barriers to wider application. The advent of lower cost pocket ultrasounds may present a solution given the early reports of similar efficacy to traditional devices in the assessment of left ventricular dysfunction, acute decompensated heart failure,[6] and focused assessment with sonography for trauma.[7] We assessed the feasibility and diagnostic accuracy of residents trained in lung ultrasound with a pocket device for evaluating patients with dyspnea.

MATERIALS AND METHODS

Diagnostic Protocol

Upon admission, residents recorded a clinical diagnosis of dyspnea based on a standard diagnostic evaluation including complete history, physical exam, and all relevant laboratory and imaging studies, including chest x‐ray and computed tomography (CT) scans. A diagnosis of dyspnea after lung ultrasound was then recorded based on the lung ultrasound findings and integrated with all other clinical information available. Standard lung ultrasound patterns and diagnostic correlates are shown in Figure 1. Diagnoses of dyspnea were recorded as one of 7 possibilities; 1) exacerbation of chronic obstructive pulmonary disease or asthma (COPD/asthma), 2) acute pulmonary edema (APE), 3) pneumonia (PNA), 4) pulmonary embolus (PE), 5) pneumothorax (PTX), 6) pleural effusion (PLEFF), and 7) other (OTH), namely anemia, ascites, and dehydration.

Figure 1

RESULTS

Five residents performed lung ultrasound on a convenience sample of 69 newly admitted patients. Patient baseline characteristics are shown in Table 1. Three residents made up the focused training group and examined 21 patients, resulting in 27 clinical diagnoses, 27 ultrasound diagnoses, and 31 final attending diagnoses. Two residents made up the extended training group and examined 48 patients, resulting in 61 clinical diagnoses, 60 ultrasound diagnoses, and 60 final attending diagnoses. Improvements in sensitivity and specificity using lung ultrasound were more pronounced for the extended training group and are shown for each diagnosis in Table 2.

Overall diagnostic accuracy using lung ultrasound improved only for the extended training group (clinical 92% vs ultrasound 97%), whereas the focused training group's accuracy was unchanged (clinical 87% vs ultrasound 88%).

ROC analysis demonstrated a superior diagnostic performance of ultrasound when compared to clinical diagnosis (Table 3).

DISCUSSION

In this prospective, observational study of residents performing lung ultrasound of patients with dyspnea, the diagnostic accuracy incorporating ultrasound increased compared to a standard diagnostic approach relying on history, physical exam, blood tests, and radiography. To our knowledge, this is the first study of residents independently performing lung ultrasound with a pocket ultrasound to diagnose dyspnea. Receiver operating curve analysis shows improvements in diagnostic accuracy for causes such as PNA, pleural effusion and COPD/asthma and demonstrates the feasibility and clinical utility of residents using pocket ultrasounds. The finding that improvements in sensitivity and specificity were larger in the extended training group highlights the need for sufficient training to demonstrate increased utility. Although a 2‐week critical care ultrasound elective may not be possible for all residents, perhaps training of intensity somewhere in between these 2 levels would be most feasible.

Challenges in diagnosing dyspnea have been well described, attributed to a lack of accurate history combined with often insensitive and nonspecific physical exam findings, blood tests, and radiographs.[8, 9] Further, patients often present with multiple contributing causes as was evidenced in this study.[10] Lack of initial, accurate diagnoses often leads to the provision of multiple, incorrect treatment regimens that may increase mortality.[11] The high accuracy of lung ultrasound in defining causes of respiratory failure suggests potential as a low‐cost solution.[12]

This study design differed from prior work in several respects. First, it included patients presenting with dyspnea to a hospital ward rather than acute respiratory failure to an intensive care unit (ICU), suggesting its diagnostic potential in a broader population of patients and settings. Second, the lung ultrasound was integrated with traditional clinical information rather than relied upon alone, a situation mimicking real‐world application of POC‐US. Third, operators were residents with limited amounts of training rather than highly trained experts. Finally, the lung ultrasound exams were performed using a pocket ultrasound with inferior imaging capability than larger, more established ultrasound devices. Despite these constraints, the utility of lung ultrasound was still evident, particularly in the diagnosis or exclusion of pneumonia and PLEFF.

Limitations include reliance on a small cohort of highly motivated residents with an interest in pulmonary and critical care, 2 who are authors of this article, making reproducibility a concern. Although convenience sampling may more closely mimic real world practices of POC‐US, a bias toward less challenging patients is possible and may limit conclusions regarding utility. Over‐reading and feedback were not provided to residents to improve their performance of lung ultrasound exams. Also, because chest CT is considered the gold standard in most studies examining the diagnostic accuracy of lung ultrasound, all residents aware of these data may underestimate the potential impact of integrating lung ultrasound with all clinical findings. Finally, the high cost of pocket ultrasounds is a barrier to general use. Recent studies on the significant cost savings associated with POC‐US make a further analysis of cost‐benefit ratios mandatory before broad use can be recommended.[13]

CONCLUSIONS

Residents participating in lung ultrasound training with a pocket ultrasound device showed improved diagnostic accuracy in their evaluation of patients with dyspnea. Those who received extended training had greater improvements across all causes of dyspnea. Training residents to apply lung ultrasound in non‐ICU settings appears to be feasible. Further study with a larger cohort of internal medicine residents and perhaps training duration that lies in between the focused and extended training groups is warranted.

The fraction of the US population identifying themselves as ethnic minorities was 36% in 2010 and will exceed 50% by 2050.[1, 2] This has resulted in an increasing gap in healthcare, as minorities have well‐documented disparities in access to healthcare and a disproportionately high morbidity and mortality.[3] In 2008, only 12.3% of US physicians were from under‐represented minority (URM) groups (see Figure in Castillo‐Page 4) (ie, those racial and ethnic populations that are underrepresented in the medical profession relative to their numbers in the general population as defined by the American Association of Medical Colleges[4, 5]). Diversifying the healthcare workforce may be an effective approach to reducing healthcare disparities, as URM physicians are more likely to choose primary care specialties,[6] work in underserved communities with socioeconomic or racial mixes similar to their own, thereby increasing access to care,[6, 7, 8] increasing minority patient satisfaction, and improving the quality of care received by minorities.[9, 10, 11]

The number of URM students attending medical school is slowly increasing, but in 2011, only 15% of the matriculating medical school students were URMs (see Figure 12 and Table 10 in Castillo‐Page[12]), and medical schools actively compete for this limited number of applicants. To increase the pool of qualified candidates, more URM students need to graduate college and pursue postgraduate healthcare training.[12]

URM undergraduate freshmen with intentions to enter medical school are 50% less likely to apply to medical school by the time they are seniors than their non‐Latino, white, and Asian counterparts.[13] Higher attrition rates have been linked to students having negative experiences in the basic science courses and with a lack of role models and exposure to careers in healthcare.[13, 14, 15, 16] We developed a hospitalist‐led mentoring program that was focused on overcoming these perceived limitations. This report describes the program and follow‐up data from our first year cohort documenting its success.

METHODS

The Healthcare Interest Program (HIP) was developed by 2 hospitalists (L. C., E. C.) and a physician's assistant (C. N.) who worked at Denver Health (DH), a university‐affiliated public hospital. We worked in conjunction with the chief diversity officer of the University of Colorado, Denver (UCD), primarily a commuter university in metropolitan Denver, where URMs composed 51% of the 2011 freshmen class. We reviewed articles describing mentoring programs for undergraduate students, and by consensus, designed a 7‐component program, each of which was intended to address a specific barrier identified in the literature as possibly contributing to reduced interest of minority students in pursuing medical careers (Table 1).[13, 14, 15, 16]

During the 2009 to 2010 academic year, information about the program, together with an application, was e‐mailed to all students at UCD who self‐identified as having interest in healthcare careers. This information was also distributed at all prehealth clubs and gatherings (ie, to students expressing interest in graduate and professional programs in healthcare‐related fields). All sophomore and junior students who submitted an application and had grade point averages (GPA) 2.8 were interviewed by the program director. Twenty‐three students were selected on the basis of their GPAs (attempting to include those with a range of GPAs), interviews, and the essays prepared as part of their applications.

An e‐mail soliciting mentors was sent to all hospitalists physicians and midlevels working at DH; 25/30 volunteered, and 20 were selected on the basis of their gender (as mentors were matched to students based on gender). The HIP director met with the mentors in person to introduce the program and its goals. All mentors had been practicing hospital medicine for 10 years after their training, and all but 3 were non‐Latino white. Each student accepted into the program was paired with a hospitalist who served as their mentor for the year.

The mentors were instructed in life coaching in both e‐mails and individual discussions. Every 2 or 3 months each hospitalist was contacted by e‐mail to see if questions or problems had arisen and to emphasize the need to meet with their mentees monthly.

Students filled out a written survey after each Books‐to‐Bedside (described in Table 1) discussion. The HIP director met with each student for at least 1 hour per semester and gathered feedback regarding mentor‐mentee success, shadowing experience, and the quality of the book club. At the end of the academic year, students completed a written, anonymous survey assessing their impressions of the program and their intentions of pursuing additional training in healthcare careers (Table 2). We used descriptive statistics to analyze the data including frequencies and mean tests.

Two years after completing the program, each student was contacted via e‐mail and/or phone to determine whether they were still pursuing healthcare careers.

RESULTS

Twenty‐three students were accepted into the program (14 female, 9 male, mean age 19 [standard deviation1]). Their GPAs ranged from 2.8 to 4.0. Eleven (48%) were the first in their family to attend college, 6 (26%) indicated that English was not their primary language, and 16 (70%) were working while attending school. All 23 students stayed in the HIP program for the full academic year.

Nineteen of the 23 students (83%) completed the survey at the end of the year. Of these, 19 (100%) strongly agreed that the HIP expanded their perceptions of what they might accomplish and increased their confidence in being able to succeed in a healthcare profession. All 19 (100%) stated that they hoped to care for underserved minority patients in the future. Sixteen (84%) strongly agreed that their role model in life was their HIP mentor. These findings suggest that many of the HIP components successfully accomplished their goals (Table 1).

Two‐year follow‐up was available for 21 of the 23 students (91%). Twenty (95%) remained committed to a career in healthcare, 18 (86%) had graduated college, 6 (29%) were enrolled in graduate training in the healthcare professions (2 in medical school, 1 in nursing school, and 3 in a master's programs in public health, counseling, and medical science, respectively), and 9 (43%) were in the process of applying to postgraduate healthcare training programs (7 to medical school, 1 to dental school, and 1 to nursing school, respectively). Five students were preparing to take the Medical College Admissions Test, and 7 were working at various jobs in the healthcare field (eg, phlebotomists, certified nurse assistants, research assistants). Of the 16 students who expressed an interest in attending medical school at the beginning of the program, 15 (94%) maintained that interest.

DISCUSSION

HIP was extremely well‐received by the participating students, the majority graduated college and remained committed to a career in healthcare, and 29% were enrolled in postgraduate training in healthcare professions 2 years after graduation.

The 86% graduation rate that we observed compares highly favorably to the UCD campus‐wide graduation rates for minority students of 12.5% at 4 years and 30.8% at 5 years. Although there may be selection bias in the students participating in HIP, the extremely high graduation rate is consistent with HIP meeting 1 or more of its stated objectives.

Many universities have prehealthcare pipeline programs that are designed to provide short‐term summer medical experiences, research opportunities, and assistance with the Medical College Admissions Test.[17, 18, 19] We believe, however, that several aspects of our program are unique. First, we designed HIP to be year‐long, rather than a summertime program. Continuing the mentoring and life coaching throughout the year may allow stronger relationships to develop between the mentor and the student. In addition, ongoing student‐mentor interactions during the time when a student may be encountering problems with their undergraduate basic science courses may be beneficial. Second, the Books‐to‐Bedside lectures series, which was designed to link the students' basic science training with clinical medicine, has not previously been described and may contribute to a higher rate of completion of their basic science training. Third, those aspects of the program resulting in increased peer interactions (eg, book club discussions, diversity lectures, and social gatherings) provided an important venue for students with similar interests to interact, an opportunity that is limited at UCD as it is primarily a commuter university.

A number of lessons were learned during the first year of the program. First, a program such as ours must include rigorous evaluation from the start to make a case for support to the university and key stakeholders. With this in mind, it is possible to obtain funding and ensure long‐term sustainability. Second, by involving UCD's chief diversity officer in the development, the program fostered a strong partnership between DH and UCD and facilitated growing the program. Third, the hospitalists who attended the diversity‐training aspects of the program stated through informal feedback that they felt better equipped to care for the underserved and felt that providing mentorship increased their personal job satisfaction. Fourth, the students requested more opportunities for them to participate in health disparities research and in shadowing in subspecialties in addition to internal medicine. In response to this feedback, we now offer research opportunities, lectures on health disparities research, and interactions with community leaders working in improving healthcare for the underserved.

Although influencing the graduation rate from graduate level schooling is beyond the scope of HIP, we can conclude that the large majority of students participating in HIP maintained their interest in the healthcare professions, graduated college, and that many went on to postgraduate healthcare training. The data we present pertain to the cohort of students in the first year of the HIP. As the program matures, we will continue to evaluate the long‐term outcomes of our students and hospitalist mentors. This may provide opportunities for other academic hospitalists to replicate our program in their own communities.

The fraction of the US population identifying themselves as ethnic minorities was 36% in 2010 and will exceed 50% by 2050.[1, 2] This has resulted in an increasing gap in healthcare, as minorities have well‐documented disparities in access to healthcare and a disproportionately high morbidity and mortality.[3] In 2008, only 12.3% of US physicians were from under‐represented minority (URM) groups (see Figure in Castillo‐Page 4) (ie, those racial and ethnic populations that are underrepresented in the medical profession relative to their numbers in the general population as defined by the American Association of Medical Colleges[4, 5]). Diversifying the healthcare workforce may be an effective approach to reducing healthcare disparities, as URM physicians are more likely to choose primary care specialties,[6] work in underserved communities with socioeconomic or racial mixes similar to their own, thereby increasing access to care,[6, 7, 8] increasing minority patient satisfaction, and improving the quality of care received by minorities.[9, 10, 11]

The number of URM students attending medical school is slowly increasing, but in 2011, only 15% of the matriculating medical school students were URMs (see Figure 12 and Table 10 in Castillo‐Page[12]), and medical schools actively compete for this limited number of applicants. To increase the pool of qualified candidates, more URM students need to graduate college and pursue postgraduate healthcare training.[12]

URM undergraduate freshmen with intentions to enter medical school are 50% less likely to apply to medical school by the time they are seniors than their non‐Latino, white, and Asian counterparts.[13] Higher attrition rates have been linked to students having negative experiences in the basic science courses and with a lack of role models and exposure to careers in healthcare.[13, 14, 15, 16] We developed a hospitalist‐led mentoring program that was focused on overcoming these perceived limitations. This report describes the program and follow‐up data from our first year cohort documenting its success.

METHODS

The Healthcare Interest Program (HIP) was developed by 2 hospitalists (L. C., E. C.) and a physician's assistant (C. N.) who worked at Denver Health (DH), a university‐affiliated public hospital. We worked in conjunction with the chief diversity officer of the University of Colorado, Denver (UCD), primarily a commuter university in metropolitan Denver, where URMs composed 51% of the 2011 freshmen class. We reviewed articles describing mentoring programs for undergraduate students, and by consensus, designed a 7‐component program, each of which was intended to address a specific barrier identified in the literature as possibly contributing to reduced interest of minority students in pursuing medical careers (Table 1).[13, 14, 15, 16]

During the 2009 to 2010 academic year, information about the program, together with an application, was e‐mailed to all students at UCD who self‐identified as having interest in healthcare careers. This information was also distributed at all prehealth clubs and gatherings (ie, to students expressing interest in graduate and professional programs in healthcare‐related fields). All sophomore and junior students who submitted an application and had grade point averages (GPA) 2.8 were interviewed by the program director. Twenty‐three students were selected on the basis of their GPAs (attempting to include those with a range of GPAs), interviews, and the essays prepared as part of their applications.

An e‐mail soliciting mentors was sent to all hospitalists physicians and midlevels working at DH; 25/30 volunteered, and 20 were selected on the basis of their gender (as mentors were matched to students based on gender). The HIP director met with the mentors in person to introduce the program and its goals. All mentors had been practicing hospital medicine for 10 years after their training, and all but 3 were non‐Latino white. Each student accepted into the program was paired with a hospitalist who served as their mentor for the year.

The mentors were instructed in life coaching in both e‐mails and individual discussions. Every 2 or 3 months each hospitalist was contacted by e‐mail to see if questions or problems had arisen and to emphasize the need to meet with their mentees monthly.

Students filled out a written survey after each Books‐to‐Bedside (described in Table 1) discussion. The HIP director met with each student for at least 1 hour per semester and gathered feedback regarding mentor‐mentee success, shadowing experience, and the quality of the book club. At the end of the academic year, students completed a written, anonymous survey assessing their impressions of the program and their intentions of pursuing additional training in healthcare careers (Table 2). We used descriptive statistics to analyze the data including frequencies and mean tests.

Two years after completing the program, each student was contacted via e‐mail and/or phone to determine whether they were still pursuing healthcare careers.

RESULTS

Twenty‐three students were accepted into the program (14 female, 9 male, mean age 19 [standard deviation1]). Their GPAs ranged from 2.8 to 4.0. Eleven (48%) were the first in their family to attend college, 6 (26%) indicated that English was not their primary language, and 16 (70%) were working while attending school. All 23 students stayed in the HIP program for the full academic year.

Nineteen of the 23 students (83%) completed the survey at the end of the year. Of these, 19 (100%) strongly agreed that the HIP expanded their perceptions of what they might accomplish and increased their confidence in being able to succeed in a healthcare profession. All 19 (100%) stated that they hoped to care for underserved minority patients in the future. Sixteen (84%) strongly agreed that their role model in life was their HIP mentor. These findings suggest that many of the HIP components successfully accomplished their goals (Table 1).

Two‐year follow‐up was available for 21 of the 23 students (91%). Twenty (95%) remained committed to a career in healthcare, 18 (86%) had graduated college, 6 (29%) were enrolled in graduate training in the healthcare professions (2 in medical school, 1 in nursing school, and 3 in a master's programs in public health, counseling, and medical science, respectively), and 9 (43%) were in the process of applying to postgraduate healthcare training programs (7 to medical school, 1 to dental school, and 1 to nursing school, respectively). Five students were preparing to take the Medical College Admissions Test, and 7 were working at various jobs in the healthcare field (eg, phlebotomists, certified nurse assistants, research assistants). Of the 16 students who expressed an interest in attending medical school at the beginning of the program, 15 (94%) maintained that interest.

DISCUSSION

HIP was extremely well‐received by the participating students, the majority graduated college and remained committed to a career in healthcare, and 29% were enrolled in postgraduate training in healthcare professions 2 years after graduation.

The 86% graduation rate that we observed compares highly favorably to the UCD campus‐wide graduation rates for minority students of 12.5% at 4 years and 30.8% at 5 years. Although there may be selection bias in the students participating in HIP, the extremely high graduation rate is consistent with HIP meeting 1 or more of its stated objectives.

Many universities have prehealthcare pipeline programs that are designed to provide short‐term summer medical experiences, research opportunities, and assistance with the Medical College Admissions Test.[17, 18, 19] We believe, however, that several aspects of our program are unique. First, we designed HIP to be year‐long, rather than a summertime program. Continuing the mentoring and life coaching throughout the year may allow stronger relationships to develop between the mentor and the student. In addition, ongoing student‐mentor interactions during the time when a student may be encountering problems with their undergraduate basic science courses may be beneficial. Second, the Books‐to‐Bedside lectures series, which was designed to link the students' basic science training with clinical medicine, has not previously been described and may contribute to a higher rate of completion of their basic science training. Third, those aspects of the program resulting in increased peer interactions (eg, book club discussions, diversity lectures, and social gatherings) provided an important venue for students with similar interests to interact, an opportunity that is limited at UCD as it is primarily a commuter university.

A number of lessons were learned during the first year of the program. First, a program such as ours must include rigorous evaluation from the start to make a case for support to the university and key stakeholders. With this in mind, it is possible to obtain funding and ensure long‐term sustainability. Second, by involving UCD's chief diversity officer in the development, the program fostered a strong partnership between DH and UCD and facilitated growing the program. Third, the hospitalists who attended the diversity‐training aspects of the program stated through informal feedback that they felt better equipped to care for the underserved and felt that providing mentorship increased their personal job satisfaction. Fourth, the students requested more opportunities for them to participate in health disparities research and in shadowing in subspecialties in addition to internal medicine. In response to this feedback, we now offer research opportunities, lectures on health disparities research, and interactions with community leaders working in improving healthcare for the underserved.

Although influencing the graduation rate from graduate level schooling is beyond the scope of HIP, we can conclude that the large majority of students participating in HIP maintained their interest in the healthcare professions, graduated college, and that many went on to postgraduate healthcare training. The data we present pertain to the cohort of students in the first year of the HIP. As the program matures, we will continue to evaluate the long‐term outcomes of our students and hospitalist mentors. This may provide opportunities for other academic hospitalists to replicate our program in their own communities.

The fraction of the US population identifying themselves as ethnic minorities was 36% in 2010 and will exceed 50% by 2050.[1, 2] This has resulted in an increasing gap in healthcare, as minorities have well‐documented disparities in access to healthcare and a disproportionately high morbidity and mortality.[3] In 2008, only 12.3% of US physicians were from under‐represented minority (URM) groups (see Figure in Castillo‐Page 4) (ie, those racial and ethnic populations that are underrepresented in the medical profession relative to their numbers in the general population as defined by the American Association of Medical Colleges[4, 5]). Diversifying the healthcare workforce may be an effective approach to reducing healthcare disparities, as URM physicians are more likely to choose primary care specialties,[6] work in underserved communities with socioeconomic or racial mixes similar to their own, thereby increasing access to care,[6, 7, 8] increasing minority patient satisfaction, and improving the quality of care received by minorities.[9, 10, 11]

The number of URM students attending medical school is slowly increasing, but in 2011, only 15% of the matriculating medical school students were URMs (see Figure 12 and Table 10 in Castillo‐Page[12]), and medical schools actively compete for this limited number of applicants. To increase the pool of qualified candidates, more URM students need to graduate college and pursue postgraduate healthcare training.[12]

URM undergraduate freshmen with intentions to enter medical school are 50% less likely to apply to medical school by the time they are seniors than their non‐Latino, white, and Asian counterparts.[13] Higher attrition rates have been linked to students having negative experiences in the basic science courses and with a lack of role models and exposure to careers in healthcare.[13, 14, 15, 16] We developed a hospitalist‐led mentoring program that was focused on overcoming these perceived limitations. This report describes the program and follow‐up data from our first year cohort documenting its success.

METHODS

The Healthcare Interest Program (HIP) was developed by 2 hospitalists (L. C., E. C.) and a physician's assistant (C. N.) who worked at Denver Health (DH), a university‐affiliated public hospital. We worked in conjunction with the chief diversity officer of the University of Colorado, Denver (UCD), primarily a commuter university in metropolitan Denver, where URMs composed 51% of the 2011 freshmen class. We reviewed articles describing mentoring programs for undergraduate students, and by consensus, designed a 7‐component program, each of which was intended to address a specific barrier identified in the literature as possibly contributing to reduced interest of minority students in pursuing medical careers (Table 1).[13, 14, 15, 16]

During the 2009 to 2010 academic year, information about the program, together with an application, was e‐mailed to all students at UCD who self‐identified as having interest in healthcare careers. This information was also distributed at all prehealth clubs and gatherings (ie, to students expressing interest in graduate and professional programs in healthcare‐related fields). All sophomore and junior students who submitted an application and had grade point averages (GPA) 2.8 were interviewed by the program director. Twenty‐three students were selected on the basis of their GPAs (attempting to include those with a range of GPAs), interviews, and the essays prepared as part of their applications.

An e‐mail soliciting mentors was sent to all hospitalists physicians and midlevels working at DH; 25/30 volunteered, and 20 were selected on the basis of their gender (as mentors were matched to students based on gender). The HIP director met with the mentors in person to introduce the program and its goals. All mentors had been practicing hospital medicine for 10 years after their training, and all but 3 were non‐Latino white. Each student accepted into the program was paired with a hospitalist who served as their mentor for the year.

The mentors were instructed in life coaching in both e‐mails and individual discussions. Every 2 or 3 months each hospitalist was contacted by e‐mail to see if questions or problems had arisen and to emphasize the need to meet with their mentees monthly.

Students filled out a written survey after each Books‐to‐Bedside (described in Table 1) discussion. The HIP director met with each student for at least 1 hour per semester and gathered feedback regarding mentor‐mentee success, shadowing experience, and the quality of the book club. At the end of the academic year, students completed a written, anonymous survey assessing their impressions of the program and their intentions of pursuing additional training in healthcare careers (Table 2). We used descriptive statistics to analyze the data including frequencies and mean tests.

Two years after completing the program, each student was contacted via e‐mail and/or phone to determine whether they were still pursuing healthcare careers.

RESULTS

Twenty‐three students were accepted into the program (14 female, 9 male, mean age 19 [standard deviation1]). Their GPAs ranged from 2.8 to 4.0. Eleven (48%) were the first in their family to attend college, 6 (26%) indicated that English was not their primary language, and 16 (70%) were working while attending school. All 23 students stayed in the HIP program for the full academic year.

Nineteen of the 23 students (83%) completed the survey at the end of the year. Of these, 19 (100%) strongly agreed that the HIP expanded their perceptions of what they might accomplish and increased their confidence in being able to succeed in a healthcare profession. All 19 (100%) stated that they hoped to care for underserved minority patients in the future. Sixteen (84%) strongly agreed that their role model in life was their HIP mentor. These findings suggest that many of the HIP components successfully accomplished their goals (Table 1).

Two‐year follow‐up was available for 21 of the 23 students (91%). Twenty (95%) remained committed to a career in healthcare, 18 (86%) had graduated college, 6 (29%) were enrolled in graduate training in the healthcare professions (2 in medical school, 1 in nursing school, and 3 in a master's programs in public health, counseling, and medical science, respectively), and 9 (43%) were in the process of applying to postgraduate healthcare training programs (7 to medical school, 1 to dental school, and 1 to nursing school, respectively). Five students were preparing to take the Medical College Admissions Test, and 7 were working at various jobs in the healthcare field (eg, phlebotomists, certified nurse assistants, research assistants). Of the 16 students who expressed an interest in attending medical school at the beginning of the program, 15 (94%) maintained that interest.

DISCUSSION

HIP was extremely well‐received by the participating students, the majority graduated college and remained committed to a career in healthcare, and 29% were enrolled in postgraduate training in healthcare professions 2 years after graduation.

The 86% graduation rate that we observed compares highly favorably to the UCD campus‐wide graduation rates for minority students of 12.5% at 4 years and 30.8% at 5 years. Although there may be selection bias in the students participating in HIP, the extremely high graduation rate is consistent with HIP meeting 1 or more of its stated objectives.

Many universities have prehealthcare pipeline programs that are designed to provide short‐term summer medical experiences, research opportunities, and assistance with the Medical College Admissions Test.[17, 18, 19] We believe, however, that several aspects of our program are unique. First, we designed HIP to be year‐long, rather than a summertime program. Continuing the mentoring and life coaching throughout the year may allow stronger relationships to develop between the mentor and the student. In addition, ongoing student‐mentor interactions during the time when a student may be encountering problems with their undergraduate basic science courses may be beneficial. Second, the Books‐to‐Bedside lectures series, which was designed to link the students' basic science training with clinical medicine, has not previously been described and may contribute to a higher rate of completion of their basic science training. Third, those aspects of the program resulting in increased peer interactions (eg, book club discussions, diversity lectures, and social gatherings) provided an important venue for students with similar interests to interact, an opportunity that is limited at UCD as it is primarily a commuter university.

A number of lessons were learned during the first year of the program. First, a program such as ours must include rigorous evaluation from the start to make a case for support to the university and key stakeholders. With this in mind, it is possible to obtain funding and ensure long‐term sustainability. Second, by involving UCD's chief diversity officer in the development, the program fostered a strong partnership between DH and UCD and facilitated growing the program. Third, the hospitalists who attended the diversity‐training aspects of the program stated through informal feedback that they felt better equipped to care for the underserved and felt that providing mentorship increased their personal job satisfaction. Fourth, the students requested more opportunities for them to participate in health disparities research and in shadowing in subspecialties in addition to internal medicine. In response to this feedback, we now offer research opportunities, lectures on health disparities research, and interactions with community leaders working in improving healthcare for the underserved.

Although influencing the graduation rate from graduate level schooling is beyond the scope of HIP, we can conclude that the large majority of students participating in HIP maintained their interest in the healthcare professions, graduated college, and that many went on to postgraduate healthcare training. The data we present pertain to the cohort of students in the first year of the HIP. As the program matures, we will continue to evaluate the long‐term outcomes of our students and hospitalist mentors. This may provide opportunities for other academic hospitalists to replicate our program in their own communities.

In 2002, based on consensus practice guidelines,[1] the Centers for Medicare and Medicaid Services (CMS) and the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) announced a core measure mandating the collection of routine blood cultures in the emergency department (ED) for all patients hospitalized with community‐acquired pneumonia (CAP) to benchmark the quality of hospital care. However, due to the limited utility and false‐positive results of routine blood cultures,[2, 3, 4, 5, 6] performance measures and practice guidelines were modified in 2005 and 2007, respectively, to recommend routine collection in only the sickest patients with CAP.[2, 7] Despite recommendations for a more narrow set of indications, the collection of blood cultures in patients hospitalized with CAP continued to increase.[8]

Distinguishing CAP from other respiratory illnesses may be challenging. Among patients presenting to the ED with an acute respiratory illness, only a minority of patients (10%30%) are diagnosed with pneumonia.[9] Therefore, the harms and costs of inappropriate diagnostic tests for CAP may be further magnified if applied to a larger population of patients who present to the ED with similar clinical signs and symptoms as pneumonia. Using a national sample of ED visits, we examined whether there was a similar increase in the frequency of blood culture collection among patients who were hospitalized with respiratory symptoms due to an illness other than pneumonia.

METHOD

RESULTS

This study included 4854 ED visits, representing approximately 17 million visits by adult patients hospitalized with respiratory symptoms due to a nonpneumonia illness. The most common primary ED provider's diagnoses for these visits included heart failure (15.9%), chronic obstructive pulmonary disease (12.6%), chest pain (11.9%), respiratory insufficiency or failure (8.8%), and asthma (5.5%). The characteristics of these visits are shown in Table 1.

The proportion of blood cultures collected in the ED for patients hospitalized with respiratory symptoms due to a nonpneumonia illness increased from 9.9% (95% confidence interval [CI]: 7.1%‐13.5%) in 2002 to 20.4% (95% CI: 16.1%‐25.6%) in 2010 (P0.001 for the trend). This observed increase paralleled the increase in the frequency of culture collection in patients hospitalized with CAP (P=0.12 for the difference in temporal trends). The estimated absolute number of visits for respiratory symptoms due a nonpneumonia illness with a blood culture collected increased from 211,000 (95% CI: 126,000296,000) in 2002 to 526,000 (95% CI: 361,000692,000) in 2010, which was similar in magnitude to the estimated number of visits for CAP with a culture collected (Table 2).

DISCUSSION

In this national study of ED visits, we found that the collection of blood cultures in patients hospitalized with respiratory symptoms due to an illness other than pneumonia continued to increase from 2002 to 2010 in a parallel fashion to the trend observed for patients hospitalized with CAP. Our findings suggest that the heightened attention of collecting blood cultures for suspected pneumonia had unintended consequences, which led to an increase in the collection of blood cultures in patients hospitalized with conditions that mimic pneumonia in the ED.

There can be a great deal of diagnostic uncertainty when treating patients in the ED who present with acute respiratory symptoms. Unfortunately, the initial history and physical exam are often insufficient to effectively rule in CAP.[13] Furthermore, the challenge of diagnosing pneumonia is amplified in the subset of patients who present with evolving, atypical, or occult disease. Faced with this diagnostic uncertainty, ED providers may feel pressured to comply with performance measures for CAP, promoting the overuse of inappropriate diagnostic tests and treatments. For instance, efforts to comply with early antibiotic administration in patients with CAP have led to an increase in unnecessary antibiotic use among patients with a diagnosis other than CAP.[14] Due to concerns for these unintended consequences, the core measure for early antibiotic administration was effectively retired in 2012.

Although a smaller percentage of ED visits for respiratory symptoms had a blood culture collected compared to CAP visits, there was a similar absolute number of visits with a blood culture collected during the study period. While a fraction of these patients may present with an infectious etiology aside from pneumonia, the majority of these cases likely represent situations where blood cultures add little diagnostic value at the expense of potentially longer hospital stays and broad spectrum antimicrobial use due to false‐positive results,[5, 15] as well as higher costs incurred by the test itself.[15, 16]

Although recommendations for routine culture collection for all patients hospitalized with CAP have been revised, the JCAHO/CMS core measure (PN‐3b) announced in 2002 mandates that if a culture is collected in the ED, it should be collected prior to antibiotic administration. Due to inherent uncertainty and challenges in making a timely diagnosis of pneumonia, this measure may encourage providers to reflexively order cultures in all patients presenting with respiratory symptoms in whom antibiotic administration is anticipated. The observed increasing trend in culture collection in patients hospitalized with respiratory symptoms due to a nonpneumonia illness should prompt JCAHO and CMS to reevaluate the risks and benefits of this core measure, with consideration of eliminating it altogether to discourage overuse in this population.

Our study had certain limitations. First, the omission of 2005 and 2006 data prohibited an evaluation of whether culture rates slowed down among patients hospitalized with respiratory symptoms due to a nonpneumonia illness after revisions in recommendations for obtaining cultures in patients with CAP. Second, there may have been misclassification of culture collection due to errors in chart abstraction. However, abstraction errors in the NHAMCS typically result in undercoding.[17] Therefore, our findings likely underestimate the magnitude and frequency of culture collection in this population.

In conclusion, collecting blood cultures in the ED for patients hospitalized with respiratory symptoms due to a nonpneumonia illness has increased in a parallel fashion compared to the trend in culture collection in patients hospitalized with CAP from 2002 to 2010. This suggests an important potential unintended consequence of blood culture recommendations for CAP on patients who present with conditions that resemble pneumonia. More attention to the judicious use of blood cultures in these patients to reduce harm and costs is needed.

In 2002, based on consensus practice guidelines,[1] the Centers for Medicare and Medicaid Services (CMS) and the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) announced a core measure mandating the collection of routine blood cultures in the emergency department (ED) for all patients hospitalized with community‐acquired pneumonia (CAP) to benchmark the quality of hospital care. However, due to the limited utility and false‐positive results of routine blood cultures,[2, 3, 4, 5, 6] performance measures and practice guidelines were modified in 2005 and 2007, respectively, to recommend routine collection in only the sickest patients with CAP.[2, 7] Despite recommendations for a more narrow set of indications, the collection of blood cultures in patients hospitalized with CAP continued to increase.[8]

Distinguishing CAP from other respiratory illnesses may be challenging. Among patients presenting to the ED with an acute respiratory illness, only a minority of patients (10%30%) are diagnosed with pneumonia.[9] Therefore, the harms and costs of inappropriate diagnostic tests for CAP may be further magnified if applied to a larger population of patients who present to the ED with similar clinical signs and symptoms as pneumonia. Using a national sample of ED visits, we examined whether there was a similar increase in the frequency of blood culture collection among patients who were hospitalized with respiratory symptoms due to an illness other than pneumonia.

METHOD

RESULTS

This study included 4854 ED visits, representing approximately 17 million visits by adult patients hospitalized with respiratory symptoms due to a nonpneumonia illness. The most common primary ED provider's diagnoses for these visits included heart failure (15.9%), chronic obstructive pulmonary disease (12.6%), chest pain (11.9%), respiratory insufficiency or failure (8.8%), and asthma (5.5%). The characteristics of these visits are shown in Table 1.

The proportion of blood cultures collected in the ED for patients hospitalized with respiratory symptoms due to a nonpneumonia illness increased from 9.9% (95% confidence interval [CI]: 7.1%‐13.5%) in 2002 to 20.4% (95% CI: 16.1%‐25.6%) in 2010 (P0.001 for the trend). This observed increase paralleled the increase in the frequency of culture collection in patients hospitalized with CAP (P=0.12 for the difference in temporal trends). The estimated absolute number of visits for respiratory symptoms due a nonpneumonia illness with a blood culture collected increased from 211,000 (95% CI: 126,000296,000) in 2002 to 526,000 (95% CI: 361,000692,000) in 2010, which was similar in magnitude to the estimated number of visits for CAP with a culture collected (Table 2).

DISCUSSION

In this national study of ED visits, we found that the collection of blood cultures in patients hospitalized with respiratory symptoms due to an illness other than pneumonia continued to increase from 2002 to 2010 in a parallel fashion to the trend observed for patients hospitalized with CAP. Our findings suggest that the heightened attention of collecting blood cultures for suspected pneumonia had unintended consequences, which led to an increase in the collection of blood cultures in patients hospitalized with conditions that mimic pneumonia in the ED.

There can be a great deal of diagnostic uncertainty when treating patients in the ED who present with acute respiratory symptoms. Unfortunately, the initial history and physical exam are often insufficient to effectively rule in CAP.[13] Furthermore, the challenge of diagnosing pneumonia is amplified in the subset of patients who present with evolving, atypical, or occult disease. Faced with this diagnostic uncertainty, ED providers may feel pressured to comply with performance measures for CAP, promoting the overuse of inappropriate diagnostic tests and treatments. For instance, efforts to comply with early antibiotic administration in patients with CAP have led to an increase in unnecessary antibiotic use among patients with a diagnosis other than CAP.[14] Due to concerns for these unintended consequences, the core measure for early antibiotic administration was effectively retired in 2012.

Although a smaller percentage of ED visits for respiratory symptoms had a blood culture collected compared to CAP visits, there was a similar absolute number of visits with a blood culture collected during the study period. While a fraction of these patients may present with an infectious etiology aside from pneumonia, the majority of these cases likely represent situations where blood cultures add little diagnostic value at the expense of potentially longer hospital stays and broad spectrum antimicrobial use due to false‐positive results,[5, 15] as well as higher costs incurred by the test itself.[15, 16]

Although recommendations for routine culture collection for all patients hospitalized with CAP have been revised, the JCAHO/CMS core measure (PN‐3b) announced in 2002 mandates that if a culture is collected in the ED, it should be collected prior to antibiotic administration. Due to inherent uncertainty and challenges in making a timely diagnosis of pneumonia, this measure may encourage providers to reflexively order cultures in all patients presenting with respiratory symptoms in whom antibiotic administration is anticipated. The observed increasing trend in culture collection in patients hospitalized with respiratory symptoms due to a nonpneumonia illness should prompt JCAHO and CMS to reevaluate the risks and benefits of this core measure, with consideration of eliminating it altogether to discourage overuse in this population.

Our study had certain limitations. First, the omission of 2005 and 2006 data prohibited an evaluation of whether culture rates slowed down among patients hospitalized with respiratory symptoms due to a nonpneumonia illness after revisions in recommendations for obtaining cultures in patients with CAP. Second, there may have been misclassification of culture collection due to errors in chart abstraction. However, abstraction errors in the NHAMCS typically result in undercoding.[17] Therefore, our findings likely underestimate the magnitude and frequency of culture collection in this population.

In conclusion, collecting blood cultures in the ED for patients hospitalized with respiratory symptoms due to a nonpneumonia illness has increased in a parallel fashion compared to the trend in culture collection in patients hospitalized with CAP from 2002 to 2010. This suggests an important potential unintended consequence of blood culture recommendations for CAP on patients who present with conditions that resemble pneumonia. More attention to the judicious use of blood cultures in these patients to reduce harm and costs is needed.

In 2002, based on consensus practice guidelines,[1] the Centers for Medicare and Medicaid Services (CMS) and the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) announced a core measure mandating the collection of routine blood cultures in the emergency department (ED) for all patients hospitalized with community‐acquired pneumonia (CAP) to benchmark the quality of hospital care. However, due to the limited utility and false‐positive results of routine blood cultures,[2, 3, 4, 5, 6] performance measures and practice guidelines were modified in 2005 and 2007, respectively, to recommend routine collection in only the sickest patients with CAP.[2, 7] Despite recommendations for a more narrow set of indications, the collection of blood cultures in patients hospitalized with CAP continued to increase.[8]

Distinguishing CAP from other respiratory illnesses may be challenging. Among patients presenting to the ED with an acute respiratory illness, only a minority of patients (10%30%) are diagnosed with pneumonia.[9] Therefore, the harms and costs of inappropriate diagnostic tests for CAP may be further magnified if applied to a larger population of patients who present to the ED with similar clinical signs and symptoms as pneumonia. Using a national sample of ED visits, we examined whether there was a similar increase in the frequency of blood culture collection among patients who were hospitalized with respiratory symptoms due to an illness other than pneumonia.

METHOD

RESULTS

This study included 4854 ED visits, representing approximately 17 million visits by adult patients hospitalized with respiratory symptoms due to a nonpneumonia illness. The most common primary ED provider's diagnoses for these visits included heart failure (15.9%), chronic obstructive pulmonary disease (12.6%), chest pain (11.9%), respiratory insufficiency or failure (8.8%), and asthma (5.5%). The characteristics of these visits are shown in Table 1.

The proportion of blood cultures collected in the ED for patients hospitalized with respiratory symptoms due to a nonpneumonia illness increased from 9.9% (95% confidence interval [CI]: 7.1%‐13.5%) in 2002 to 20.4% (95% CI: 16.1%‐25.6%) in 2010 (P0.001 for the trend). This observed increase paralleled the increase in the frequency of culture collection in patients hospitalized with CAP (P=0.12 for the difference in temporal trends). The estimated absolute number of visits for respiratory symptoms due a nonpneumonia illness with a blood culture collected increased from 211,000 (95% CI: 126,000296,000) in 2002 to 526,000 (95% CI: 361,000692,000) in 2010, which was similar in magnitude to the estimated number of visits for CAP with a culture collected (Table 2).

DISCUSSION

In this national study of ED visits, we found that the collection of blood cultures in patients hospitalized with respiratory symptoms due to an illness other than pneumonia continued to increase from 2002 to 2010 in a parallel fashion to the trend observed for patients hospitalized with CAP. Our findings suggest that the heightened attention of collecting blood cultures for suspected pneumonia had unintended consequences, which led to an increase in the collection of blood cultures in patients hospitalized with conditions that mimic pneumonia in the ED.

There can be a great deal of diagnostic uncertainty when treating patients in the ED who present with acute respiratory symptoms. Unfortunately, the initial history and physical exam are often insufficient to effectively rule in CAP.[13] Furthermore, the challenge of diagnosing pneumonia is amplified in the subset of patients who present with evolving, atypical, or occult disease. Faced with this diagnostic uncertainty, ED providers may feel pressured to comply with performance measures for CAP, promoting the overuse of inappropriate diagnostic tests and treatments. For instance, efforts to comply with early antibiotic administration in patients with CAP have led to an increase in unnecessary antibiotic use among patients with a diagnosis other than CAP.[14] Due to concerns for these unintended consequences, the core measure for early antibiotic administration was effectively retired in 2012.

Although a smaller percentage of ED visits for respiratory symptoms had a blood culture collected compared to CAP visits, there was a similar absolute number of visits with a blood culture collected during the study period. While a fraction of these patients may present with an infectious etiology aside from pneumonia, the majority of these cases likely represent situations where blood cultures add little diagnostic value at the expense of potentially longer hospital stays and broad spectrum antimicrobial use due to false‐positive results,[5, 15] as well as higher costs incurred by the test itself.[15, 16]

Although recommendations for routine culture collection for all patients hospitalized with CAP have been revised, the JCAHO/CMS core measure (PN‐3b) announced in 2002 mandates that if a culture is collected in the ED, it should be collected prior to antibiotic administration. Due to inherent uncertainty and challenges in making a timely diagnosis of pneumonia, this measure may encourage providers to reflexively order cultures in all patients presenting with respiratory symptoms in whom antibiotic administration is anticipated. The observed increasing trend in culture collection in patients hospitalized with respiratory symptoms due to a nonpneumonia illness should prompt JCAHO and CMS to reevaluate the risks and benefits of this core measure, with consideration of eliminating it altogether to discourage overuse in this population.

Our study had certain limitations. First, the omission of 2005 and 2006 data prohibited an evaluation of whether culture rates slowed down among patients hospitalized with respiratory symptoms due to a nonpneumonia illness after revisions in recommendations for obtaining cultures in patients with CAP. Second, there may have been misclassification of culture collection due to errors in chart abstraction. However, abstraction errors in the NHAMCS typically result in undercoding.[17] Therefore, our findings likely underestimate the magnitude and frequency of culture collection in this population.

In conclusion, collecting blood cultures in the ED for patients hospitalized with respiratory symptoms due to a nonpneumonia illness has increased in a parallel fashion compared to the trend in culture collection in patients hospitalized with CAP from 2002 to 2010. This suggests an important potential unintended consequence of blood culture recommendations for CAP on patients who present with conditions that resemble pneumonia. More attention to the judicious use of blood cultures in these patients to reduce harm and costs is needed.

The volume of existing knowledge and the pace of discovery in medical science challenge a clinician's ability to access relevant information at the point of care. Knowledge gaps that arise in practice usually involve matters related to diagnosis, drug therapy, or treatment.[1] In the clinical setting, healthcare providers (HCPs) answer questions using a variety of online and print resources. Ironically, HCPs often lack the training required to find details regarding uncommon disorders or complex medical decisions that are not easily found or well represented in the published literature.[2] Instead, HCPs turn to trusted colleagues who possess the necessary expertise.[3]

Closing the knowledge‐to‐practice gap involves a range of factual information and data derived from published evidence, anecdotal experience, as well as organization‐ and region‐specific practices.[4] The inability to codify both explicit and tacit information has been linked to variability in prescription practices, excessive use of surgical services, and delayed decisions involving the appropriate provision of end‐of‐life care.[5] Although electronic medical record systems are not configured to support peer collaboration,[6] alternative strategies including crowdsourcing has been used successfully in other domains to tap collective intelligence of skilled workers.[7] Crowdsourcing allows organizations to explore problems at low cost, gain access a wide range of complementary expertise, and capture large amounts of data for analysis.[8, 9] Although an increasing number of physicians use either smartphones or tablets on the job,[10] peer‐to‐peer medical crowdsourcing has not been investigated, despite the fact that processes involving team‐based clinical decision making are associated with better outcomes.[11] Here we field tested the mobile crowdsourcing application DocCHIRP (Crowdsourcing Health Information Retrieval Protocol for Doctors) and assessed user opinion regarding its utility in the clinical setting.

MATERIALS AND METHODS

DocCHIRP Program Design

The authors (M.W.H., J.B., H.K.) conceptualized and designed DocCHIRP for mobile (iOS [Apple Inc., Cupertino, CA] and Android [Google Inc., Mountain View, CA]) and desktop use. Email prompts and push notifications, which were modeled after the application VizWiz (Rochester Human Computer Interaction Group, University of Rochester, Rochester, NY), supported near real‐time communication between HCPs. According to recent US Food and Drug Administration guidelines, DocCHIRP is considered a medical reference,[12] intended to share domain‐specific knowledge on diagnosis, therapy, and other medically relevant topics. Devices were password protected and encrypted according to university standards. A typical workflow involves an index provider faced with a clinical question that sends a consult question to 1 or more trusted providers. The crowd receiving the notification responds when available using either free‐text responses or agree/disagree prompts (Figure 1A,B). Providers use preference settings to manage crowd membership, notification settings, and demographics describing their expertise.

Figure 1

RESULTS

Attending and resident physicians represented the majority of DocCHIRP account holders (91%), with nurse practitioners accounting for the remaining sample (9%). There were 50 male and 35 female participants, with an age range of 28 to 78 years (median age, 43 years). Departmental affiliations included Pediatrics (n = 28, 33%), Neurology (n = 27, 32%), Internal Medicine (n = 10, 12%), Psychiatry (n = 4, 5%), the Division of Pediatric Neurology (n = 11, 13%), and others (n = 5, 6%). Of the 1544 total visits to the DocCHIRP site, providers favored using smart phones (56.8%) and tablets (9.5%) over the desktop interface (33.6%; Figure 1C). iPhone use (81.7%) surpassed the other platforms combined. Desktop users visited twice as many pages (16.8 pages/visit) compared to those using smart phones (5.5 pages/visit) or tablets (8.6 pages/visit). Desktop users remained engaged longer than mobile users (13 vs 5 minutes). In the post‐trial user survey, we received 72 valid surveys from 85 potential participants (85% response rate).

We used a tiered enrollment design, sending invitations to potential participants in 3 phases to study the relationship between the size of the HCP crowd and sustained use as reported in other social networks.[13] Using a cutoff of >3 visits per week to demarcate active periods of use, we saw during the initial phase of enrollment that 20 providers generated a total of 170 visits over 22 days (Figure 2A). The addition of 28 members (phase II, n = 48 total) extended active use by 28 days, with a total of 476 page visits. The addition of 32 members (phase III, n = 85 total) resulted in 56 days of active participation with 612 visits to the site. When plotted (Figure 2B), the relationship between crowd size (total number of registered users) and cumulative visits (R² = 0.951), as well as crowd size and days of high activity (R² = 0.953) were linear and direct. We also investigated the timing of user engagement by pooling the data and breaking down use by time of day and day of the week (Figure 3A,B). In addition to observing peak engagement during the midmorning and afternoon, times of anticipated physician‐patient contact, we observed a third use peak in the evening. With the exception of sporadic weekend use, DocCHIRP use clustered during midweek.

Figure 2

Figure 3

DocCHIRP users generated 45 questions. The fastest first response was returned in less than 4 minutes, with a median first response time of 19 minutes (Figure 3C). Analysis of the consult requests received revealed a clustering of 7 principal question‐response groups: (1) the effective use of medications, (2) complex medical decision making, (3) use of the application itself, (4) guidance regarding the standard of care, (5) selection and interpretation of diagnostic tests, (6) differential diagnosis, and (7) patient referral (Figure 3D). Consults regarding medication use and complex decision making were dominant themes (63%). Several consults generated multiple responses, broadening the scope of the original query or requesting additional information (Table 1).

To better understand factors influencing use of the mobile crowdsourcing application, we surveyed users, receiving 68 comments related to the overall approach and barriers to adoption among other aspects (Table 2). The 40 comments regarding the use of medical crowdsourcing were divided evenly between supporters and critics. Enthusiasm for cross‐discipline collaboration, having tools to codify expert knowledge, and discovering consensus opinion from the expert crowd was offset by concerns that push notifications would distract providers, compromise efficiency, and potentially lead providers to act on inaccurate information.

DISCUSSION

In the current study, we developed and field‐tested the application DocCHIRP, which helps HCPs crowdsource information from each other in near real time. The average response latency in this pilot trial was 20 minutes, which was unexpectedly fast given the relatively small size of the participating crowd. Additionally, nearly one‐third of users accessed the server in the evening using the web interface rather than their mobile phone. This suggests that although HCPs liked having direct access to colleagues in near real time, the also valued the opportunity to connect asynchronously after hours.

Relative to the total number of page views, the number of HCPs using the technology for peer‐to‐peer consultation was low. Feedback provided in the post‐trial survey suggested several reasons for this effect. Some providers viewed the application without posting because they were reluctant to disclose knowledge gaps to their peers. Several users suggested implementing a system that supports anonymous posting, but others thought this would undermine the value of the information provided. Additionally, users recognized the potential for crowdsourcing to adversely effect HCP's productivity and daily workflow. This is relevant given growing concerns about distracted doctoring and association with reduced safety and quality of medical care.[14] This concept is further echoed in a paper by Wu et al. demonstrating that frequent interruptions offset the perceived benefit of increased mobility afforded by the use of mobile technology.[15] However, it is worth considering that if implemented properly, study participants believe crowdsourcing could have a net neutral impact on clinical workflow by improving the efficiency of provider communication and saving time otherwise spent problem solving. Participants also felt the approach could infringe on an already threatened work‐life boundary, and could also lead to unprofessional and antisocial behaviors.[16] Collectively, these problems are not unique to medical crowdsourcing, and prior experience in this area may offer several viable solutions. First, because crowd burnout is inversely proportional to crowd size, successful adoption in practice will require growing a provider base of sufficient depth and expertise to handle the consult demand. With the expansion of accountable care organizations across the United States, this will not likely be a limiting factor. And although not implemented here, flexible notification settings, user‐defined identity rules, and other higher‐level software design elements should alleviate the issues related to provider reputation and workflow interruptions.

Overall, HCPs are optimistic that mobile handheld technologies will benefit their practice.[17] Yet, software‐based approaches including expert decision support systems must overcome particular hurdles including lack of provider trust in the algorithms used in these approaches.[18] In the end, trust is ultimately a human phenomena; users will only trust the system if they know the information came from a trusted and highly reputable individual or institution. By tapping the expertise of a network of institutional colleagues, crowdsourcing addresses this issue of trust. Appropriately, providers were also concerned about the legality and personal risk of using crowdsourcing to discuss matters related to patient care. The technology was not intended to share protected health information, and as with other forms of digital communication, providers were cautioned during the consent process to monitor their behavior in this regard. Although soliciting advice from the medical crowd has an inherently higher level of risk compared to the use of crowdsourcing in education, research, or business, the index provider is ultimately responsible for considering all available information before making any treatment decision.

Though our pilot trial was not designed to assess effects on HCP efficiency or on the quality of care delivered, our work provides a unique window on the information‐seeking behaviors HCPs and highlights potential modifications that could enhance the utility of future crowdsourcing programs. Because the trial was performed within the context of an academic health center, it remains to be seen how medical crowdsourcing will translate in private practice, rural clinics, and other clinical environments where peer‐to‐peer consultation is sought. Given the potential for high‐stakes information exchanges, further study regarding the use of medical crowdsourcing in a controlled environment will be required before the technology can be disseminated to a broader audience. If future iterations of the mobile crowdsourcing application can address the aforementioned adoption barriers and support the organic growth of the crowd of HCPs, we believe the approach could have a positive and transformative effect on how providers acquire relevant knowledge and care for patients.

The volume of existing knowledge and the pace of discovery in medical science challenge a clinician's ability to access relevant information at the point of care. Knowledge gaps that arise in practice usually involve matters related to diagnosis, drug therapy, or treatment.[1] In the clinical setting, healthcare providers (HCPs) answer questions using a variety of online and print resources. Ironically, HCPs often lack the training required to find details regarding uncommon disorders or complex medical decisions that are not easily found or well represented in the published literature.[2] Instead, HCPs turn to trusted colleagues who possess the necessary expertise.[3]

Closing the knowledge‐to‐practice gap involves a range of factual information and data derived from published evidence, anecdotal experience, as well as organization‐ and region‐specific practices.[4] The inability to codify both explicit and tacit information has been linked to variability in prescription practices, excessive use of surgical services, and delayed decisions involving the appropriate provision of end‐of‐life care.[5] Although electronic medical record systems are not configured to support peer collaboration,[6] alternative strategies including crowdsourcing has been used successfully in other domains to tap collective intelligence of skilled workers.[7] Crowdsourcing allows organizations to explore problems at low cost, gain access a wide range of complementary expertise, and capture large amounts of data for analysis.[8, 9] Although an increasing number of physicians use either smartphones or tablets on the job,[10] peer‐to‐peer medical crowdsourcing has not been investigated, despite the fact that processes involving team‐based clinical decision making are associated with better outcomes.[11] Here we field tested the mobile crowdsourcing application DocCHIRP (Crowdsourcing Health Information Retrieval Protocol for Doctors) and assessed user opinion regarding its utility in the clinical setting.

MATERIALS AND METHODS

DocCHIRP Program Design

The authors (M.W.H., J.B., H.K.) conceptualized and designed DocCHIRP for mobile (iOS [Apple Inc., Cupertino, CA] and Android [Google Inc., Mountain View, CA]) and desktop use. Email prompts and push notifications, which were modeled after the application VizWiz (Rochester Human Computer Interaction Group, University of Rochester, Rochester, NY), supported near real‐time communication between HCPs. According to recent US Food and Drug Administration guidelines, DocCHIRP is considered a medical reference,[12] intended to share domain‐specific knowledge on diagnosis, therapy, and other medically relevant topics. Devices were password protected and encrypted according to university standards. A typical workflow involves an index provider faced with a clinical question that sends a consult question to 1 or more trusted providers. The crowd receiving the notification responds when available using either free‐text responses or agree/disagree prompts (Figure 1A,B). Providers use preference settings to manage crowd membership, notification settings, and demographics describing their expertise.

Figure 1

RESULTS

Attending and resident physicians represented the majority of DocCHIRP account holders (91%), with nurse practitioners accounting for the remaining sample (9%). There were 50 male and 35 female participants, with an age range of 28 to 78 years (median age, 43 years). Departmental affiliations included Pediatrics (n = 28, 33%), Neurology (n = 27, 32%), Internal Medicine (n = 10, 12%), Psychiatry (n = 4, 5%), the Division of Pediatric Neurology (n = 11, 13%), and others (n = 5, 6%). Of the 1544 total visits to the DocCHIRP site, providers favored using smart phones (56.8%) and tablets (9.5%) over the desktop interface (33.6%; Figure 1C). iPhone use (81.7%) surpassed the other platforms combined. Desktop users visited twice as many pages (16.8 pages/visit) compared to those using smart phones (5.5 pages/visit) or tablets (8.6 pages/visit). Desktop users remained engaged longer than mobile users (13 vs 5 minutes). In the post‐trial user survey, we received 72 valid surveys from 85 potential participants (85% response rate).

We used a tiered enrollment design, sending invitations to potential participants in 3 phases to study the relationship between the size of the HCP crowd and sustained use as reported in other social networks.[13] Using a cutoff of >3 visits per week to demarcate active periods of use, we saw during the initial phase of enrollment that 20 providers generated a total of 170 visits over 22 days (Figure 2A). The addition of 28 members (phase II, n = 48 total) extended active use by 28 days, with a total of 476 page visits. The addition of 32 members (phase III, n = 85 total) resulted in 56 days of active participation with 612 visits to the site. When plotted (Figure 2B), the relationship between crowd size (total number of registered users) and cumulative visits (R² = 0.951), as well as crowd size and days of high activity (R² = 0.953) were linear and direct. We also investigated the timing of user engagement by pooling the data and breaking down use by time of day and day of the week (Figure 3A,B). In addition to observing peak engagement during the midmorning and afternoon, times of anticipated physician‐patient contact, we observed a third use peak in the evening. With the exception of sporadic weekend use, DocCHIRP use clustered during midweek.

Figure 2

Figure 3

DocCHIRP users generated 45 questions. The fastest first response was returned in less than 4 minutes, with a median first response time of 19 minutes (Figure 3C). Analysis of the consult requests received revealed a clustering of 7 principal question‐response groups: (1) the effective use of medications, (2) complex medical decision making, (3) use of the application itself, (4) guidance regarding the standard of care, (5) selection and interpretation of diagnostic tests, (6) differential diagnosis, and (7) patient referral (Figure 3D). Consults regarding medication use and complex decision making were dominant themes (63%). Several consults generated multiple responses, broadening the scope of the original query or requesting additional information (Table 1).

To better understand factors influencing use of the mobile crowdsourcing application, we surveyed users, receiving 68 comments related to the overall approach and barriers to adoption among other aspects (Table 2). The 40 comments regarding the use of medical crowdsourcing were divided evenly between supporters and critics. Enthusiasm for cross‐discipline collaboration, having tools to codify expert knowledge, and discovering consensus opinion from the expert crowd was offset by concerns that push notifications would distract providers, compromise efficiency, and potentially lead providers to act on inaccurate information.

DISCUSSION

In the current study, we developed and field‐tested the application DocCHIRP, which helps HCPs crowdsource information from each other in near real time. The average response latency in this pilot trial was 20 minutes, which was unexpectedly fast given the relatively small size of the participating crowd. Additionally, nearly one‐third of users accessed the server in the evening using the web interface rather than their mobile phone. This suggests that although HCPs liked having direct access to colleagues in near real time, the also valued the opportunity to connect asynchronously after hours.

Relative to the total number of page views, the number of HCPs using the technology for peer‐to‐peer consultation was low. Feedback provided in the post‐trial survey suggested several reasons for this effect. Some providers viewed the application without posting because they were reluctant to disclose knowledge gaps to their peers. Several users suggested implementing a system that supports anonymous posting, but others thought this would undermine the value of the information provided. Additionally, users recognized the potential for crowdsourcing to adversely effect HCP's productivity and daily workflow. This is relevant given growing concerns about distracted doctoring and association with reduced safety and quality of medical care.[14] This concept is further echoed in a paper by Wu et al. demonstrating that frequent interruptions offset the perceived benefit of increased mobility afforded by the use of mobile technology.[15] However, it is worth considering that if implemented properly, study participants believe crowdsourcing could have a net neutral impact on clinical workflow by improving the efficiency of provider communication and saving time otherwise spent problem solving. Participants also felt the approach could infringe on an already threatened work‐life boundary, and could also lead to unprofessional and antisocial behaviors.[16] Collectively, these problems are not unique to medical crowdsourcing, and prior experience in this area may offer several viable solutions. First, because crowd burnout is inversely proportional to crowd size, successful adoption in practice will require growing a provider base of sufficient depth and expertise to handle the consult demand. With the expansion of accountable care organizations across the United States, this will not likely be a limiting factor. And although not implemented here, flexible notification settings, user‐defined identity rules, and other higher‐level software design elements should alleviate the issues related to provider reputation and workflow interruptions.

Overall, HCPs are optimistic that mobile handheld technologies will benefit their practice.[17] Yet, software‐based approaches including expert decision support systems must overcome particular hurdles including lack of provider trust in the algorithms used in these approaches.[18] In the end, trust is ultimately a human phenomena; users will only trust the system if they know the information came from a trusted and highly reputable individual or institution. By tapping the expertise of a network of institutional colleagues, crowdsourcing addresses this issue of trust. Appropriately, providers were also concerned about the legality and personal risk of using crowdsourcing to discuss matters related to patient care. The technology was not intended to share protected health information, and as with other forms of digital communication, providers were cautioned during the consent process to monitor their behavior in this regard. Although soliciting advice from the medical crowd has an inherently higher level of risk compared to the use of crowdsourcing in education, research, or business, the index provider is ultimately responsible for considering all available information before making any treatment decision.

Though our pilot trial was not designed to assess effects on HCP efficiency or on the quality of care delivered, our work provides a unique window on the information‐seeking behaviors HCPs and highlights potential modifications that could enhance the utility of future crowdsourcing programs. Because the trial was performed within the context of an academic health center, it remains to be seen how medical crowdsourcing will translate in private practice, rural clinics, and other clinical environments where peer‐to‐peer consultation is sought. Given the potential for high‐stakes information exchanges, further study regarding the use of medical crowdsourcing in a controlled environment will be required before the technology can be disseminated to a broader audience. If future iterations of the mobile crowdsourcing application can address the aforementioned adoption barriers and support the organic growth of the crowd of HCPs, we believe the approach could have a positive and transformative effect on how providers acquire relevant knowledge and care for patients.

The volume of existing knowledge and the pace of discovery in medical science challenge a clinician's ability to access relevant information at the point of care. Knowledge gaps that arise in practice usually involve matters related to diagnosis, drug therapy, or treatment.[1] In the clinical setting, healthcare providers (HCPs) answer questions using a variety of online and print resources. Ironically, HCPs often lack the training required to find details regarding uncommon disorders or complex medical decisions that are not easily found or well represented in the published literature.[2] Instead, HCPs turn to trusted colleagues who possess the necessary expertise.[3]

Closing the knowledge‐to‐practice gap involves a range of factual information and data derived from published evidence, anecdotal experience, as well as organization‐ and region‐specific practices.[4] The inability to codify both explicit and tacit information has been linked to variability in prescription practices, excessive use of surgical services, and delayed decisions involving the appropriate provision of end‐of‐life care.[5] Although electronic medical record systems are not configured to support peer collaboration,[6] alternative strategies including crowdsourcing has been used successfully in other domains to tap collective intelligence of skilled workers.[7] Crowdsourcing allows organizations to explore problems at low cost, gain access a wide range of complementary expertise, and capture large amounts of data for analysis.[8, 9] Although an increasing number of physicians use either smartphones or tablets on the job,[10] peer‐to‐peer medical crowdsourcing has not been investigated, despite the fact that processes involving team‐based clinical decision making are associated with better outcomes.[11] Here we field tested the mobile crowdsourcing application DocCHIRP (Crowdsourcing Health Information Retrieval Protocol for Doctors) and assessed user opinion regarding its utility in the clinical setting.

MATERIALS AND METHODS

DocCHIRP Program Design

The authors (M.W.H., J.B., H.K.) conceptualized and designed DocCHIRP for mobile (iOS [Apple Inc., Cupertino, CA] and Android [Google Inc., Mountain View, CA]) and desktop use. Email prompts and push notifications, which were modeled after the application VizWiz (Rochester Human Computer Interaction Group, University of Rochester, Rochester, NY), supported near real‐time communication between HCPs. According to recent US Food and Drug Administration guidelines, DocCHIRP is considered a medical reference,[12] intended to share domain‐specific knowledge on diagnosis, therapy, and other medically relevant topics. Devices were password protected and encrypted according to university standards. A typical workflow involves an index provider faced with a clinical question that sends a consult question to 1 or more trusted providers. The crowd receiving the notification responds when available using either free‐text responses or agree/disagree prompts (Figure 1A,B). Providers use preference settings to manage crowd membership, notification settings, and demographics describing their expertise.

Figure 1

RESULTS

Attending and resident physicians represented the majority of DocCHIRP account holders (91%), with nurse practitioners accounting for the remaining sample (9%). There were 50 male and 35 female participants, with an age range of 28 to 78 years (median age, 43 years). Departmental affiliations included Pediatrics (n = 28, 33%), Neurology (n = 27, 32%), Internal Medicine (n = 10, 12%), Psychiatry (n = 4, 5%), the Division of Pediatric Neurology (n = 11, 13%), and others (n = 5, 6%). Of the 1544 total visits to the DocCHIRP site, providers favored using smart phones (56.8%) and tablets (9.5%) over the desktop interface (33.6%; Figure 1C). iPhone use (81.7%) surpassed the other platforms combined. Desktop users visited twice as many pages (16.8 pages/visit) compared to those using smart phones (5.5 pages/visit) or tablets (8.6 pages/visit). Desktop users remained engaged longer than mobile users (13 vs 5 minutes). In the post‐trial user survey, we received 72 valid surveys from 85 potential participants (85% response rate).

We used a tiered enrollment design, sending invitations to potential participants in 3 phases to study the relationship between the size of the HCP crowd and sustained use as reported in other social networks.[13] Using a cutoff of >3 visits per week to demarcate active periods of use, we saw during the initial phase of enrollment that 20 providers generated a total of 170 visits over 22 days (Figure 2A). The addition of 28 members (phase II, n = 48 total) extended active use by 28 days, with a total of 476 page visits. The addition of 32 members (phase III, n = 85 total) resulted in 56 days of active participation with 612 visits to the site. When plotted (Figure 2B), the relationship between crowd size (total number of registered users) and cumulative visits (R² = 0.951), as well as crowd size and days of high activity (R² = 0.953) were linear and direct. We also investigated the timing of user engagement by pooling the data and breaking down use by time of day and day of the week (Figure 3A,B). In addition to observing peak engagement during the midmorning and afternoon, times of anticipated physician‐patient contact, we observed a third use peak in the evening. With the exception of sporadic weekend use, DocCHIRP use clustered during midweek.

Figure 2

Figure 3

DocCHIRP users generated 45 questions. The fastest first response was returned in less than 4 minutes, with a median first response time of 19 minutes (Figure 3C). Analysis of the consult requests received revealed a clustering of 7 principal question‐response groups: (1) the effective use of medications, (2) complex medical decision making, (3) use of the application itself, (4) guidance regarding the standard of care, (5) selection and interpretation of diagnostic tests, (6) differential diagnosis, and (7) patient referral (Figure 3D). Consults regarding medication use and complex decision making were dominant themes (63%). Several consults generated multiple responses, broadening the scope of the original query or requesting additional information (Table 1).

To better understand factors influencing use of the mobile crowdsourcing application, we surveyed users, receiving 68 comments related to the overall approach and barriers to adoption among other aspects (Table 2). The 40 comments regarding the use of medical crowdsourcing were divided evenly between supporters and critics. Enthusiasm for cross‐discipline collaboration, having tools to codify expert knowledge, and discovering consensus opinion from the expert crowd was offset by concerns that push notifications would distract providers, compromise efficiency, and potentially lead providers to act on inaccurate information.

DISCUSSION

In the current study, we developed and field‐tested the application DocCHIRP, which helps HCPs crowdsource information from each other in near real time. The average response latency in this pilot trial was 20 minutes, which was unexpectedly fast given the relatively small size of the participating crowd. Additionally, nearly one‐third of users accessed the server in the evening using the web interface rather than their mobile phone. This suggests that although HCPs liked having direct access to colleagues in near real time, the also valued the opportunity to connect asynchronously after hours.

Relative to the total number of page views, the number of HCPs using the technology for peer‐to‐peer consultation was low. Feedback provided in the post‐trial survey suggested several reasons for this effect. Some providers viewed the application without posting because they were reluctant to disclose knowledge gaps to their peers. Several users suggested implementing a system that supports anonymous posting, but others thought this would undermine the value of the information provided. Additionally, users recognized the potential for crowdsourcing to adversely effect HCP's productivity and daily workflow. This is relevant given growing concerns about distracted doctoring and association with reduced safety and quality of medical care.[14] This concept is further echoed in a paper by Wu et al. demonstrating that frequent interruptions offset the perceived benefit of increased mobility afforded by the use of mobile technology.[15] However, it is worth considering that if implemented properly, study participants believe crowdsourcing could have a net neutral impact on clinical workflow by improving the efficiency of provider communication and saving time otherwise spent problem solving. Participants also felt the approach could infringe on an already threatened work‐life boundary, and could also lead to unprofessional and antisocial behaviors.[16] Collectively, these problems are not unique to medical crowdsourcing, and prior experience in this area may offer several viable solutions. First, because crowd burnout is inversely proportional to crowd size, successful adoption in practice will require growing a provider base of sufficient depth and expertise to handle the consult demand. With the expansion of accountable care organizations across the United States, this will not likely be a limiting factor. And although not implemented here, flexible notification settings, user‐defined identity rules, and other higher‐level software design elements should alleviate the issues related to provider reputation and workflow interruptions.

Overall, HCPs are optimistic that mobile handheld technologies will benefit their practice.[17] Yet, software‐based approaches including expert decision support systems must overcome particular hurdles including lack of provider trust in the algorithms used in these approaches.[18] In the end, trust is ultimately a human phenomena; users will only trust the system if they know the information came from a trusted and highly reputable individual or institution. By tapping the expertise of a network of institutional colleagues, crowdsourcing addresses this issue of trust. Appropriately, providers were also concerned about the legality and personal risk of using crowdsourcing to discuss matters related to patient care. The technology was not intended to share protected health information, and as with other forms of digital communication, providers were cautioned during the consent process to monitor their behavior in this regard. Although soliciting advice from the medical crowd has an inherently higher level of risk compared to the use of crowdsourcing in education, research, or business, the index provider is ultimately responsible for considering all available information before making any treatment decision.

Though our pilot trial was not designed to assess effects on HCP efficiency or on the quality of care delivered, our work provides a unique window on the information‐seeking behaviors HCPs and highlights potential modifications that could enhance the utility of future crowdsourcing programs. Because the trial was performed within the context of an academic health center, it remains to be seen how medical crowdsourcing will translate in private practice, rural clinics, and other clinical environments where peer‐to‐peer consultation is sought. Given the potential for high‐stakes information exchanges, further study regarding the use of medical crowdsourcing in a controlled environment will be required before the technology can be disseminated to a broader audience. If future iterations of the mobile crowdsourcing application can address the aforementioned adoption barriers and support the organic growth of the crowd of HCPs, we believe the approach could have a positive and transformative effect on how providers acquire relevant knowledge and care for patients.

Pneumonia remains one of the most common indications for hospital admissions. In the United States in 2010, more than 1 million patients were discharged with a diagnosis of pneumonia.[1] A diagnosis of pneumonia is based on typical clinical findings with recommendations to identify a demonstrable infiltrate on appropriate imaging modalities.[2] Although computed tomography (CT) imaging of the chest is much more sensitive than plain radiography at detecting infiltrates, the greater cost and higher radiation exposure limits its use as a screening modality.[3, 4] Additional imaging studies are recommended for patients who fail to respond to therapy.[2] There are, however, no published studies to determine the exact impact of chest CT scans on the management of pneumonia.

We conducted a retrospective assessment of CT scan use in patients admitted with a diagnosis of pneumonia. The study was designed to assess (1) the overall utilization rate of chest CT scans at our institution and (2) the impact of CT findings on patient management.

METHODS

This retrospective study was conducted at St. John Hospital and Medical Center, an 808‐bed tertiary care community teaching hospital in Detroit. The study was approved by the St. John Hospital and Medical Center's institutional review board.

Patients admitted to our institution between January 1, 2008 and November 1, 2011 were evaluated for study inclusion by searching the hospital's computer database using the discharge International Classification of Diseases, 9th Revision, Clinical Modification (ICD‐9‐CM) codes for pneumonia, pleural effusion, and empyema. Patients were included for initial review if the appropriate ICD‐9‐CM codes were included within the list of discharge diagnoses and were not restricted based on hierarchy within that list. Patients were included in further analysis if they were 18 years of age, a diagnosis of pneumonia was made within 48 hours of admission, and records were available for review. Patients were excluded if they did not meet the above criteria or a diagnosis of pneumonia could not be confirmed by chart review. The electronic medical record was reviewed and patient demographics, hospital admission source, microbiology results, radiographic findings, and outcomes were recorded. Additional procedures such as thoracentesis, open lung biopsy and/or chest tube placement were recorded for patients if performed. The Charlson Weighted Index of Comorbidity and Confusion, Urea, Respiratory rate, Blood pressure, Age > 65 (CURB 65) scores were calculated as described elsewhere.[5, 6] CT scans were assessed for time and date of study after admission along with all relevant findings.

RESULTS

A total of 264 patients were identified by discharge diagnosis, and 195 (73.9%) patients met the inclusion criteria. Among the 69 patients who were excluded, 37 patients were diagnosed more than 48 hours after admission, 19 patients did not have a radiographically demonstrable abnormality, 5 patients had an incomplete medical record, and 8 patients received no antibiotics. The overall mean age of the cases was 63.4 19.1 years, with an average length of stay of 7.4 5.7 days. Sixty‐nine (35.3%) of the case patients had a chest CT scan performed. A CT scan was performed more often in younger patients (58.1 19.0 vs 66.8 18.6, P = 0.002) and in patients with lower CURB 65 scores (1.7 1.4 vs 2.2 1.4, P = 0.037). A CT scan was also performed more often in patients with no infiltrates or consolidation on plain radiographs (26.9% vs 7.1%, P 0.0001). Patients were also more likely to have a procedure performed if they had a CT performed (21.7% vs 3.1%, P 0.0001) and were admitted from home versus a long‐term care facility or other healthcare institution (92.8% vs 78.6%, P = 0.011). Comparisons are shown in Table 1. After controlling for age, CURB 65 score on admission, admission source, and the presence of consolidation or infiltrates on initial chest radiograph (CXR), individuals were 4.76 times less likely to have a CT scan performed if the CXR showed consolidation and/or infiltrates (odds ratio: 0.21, P = 0.001; 95% confidence interval: 0.08‐0.53) (Table 2).