Anthony F Jerant, MD

The handwritten record has been the standard way of documenting medical information since the 19th century.¹ However, the rapid evolution of computer technology has led to the development and use of electronic medical records (EMRs) during the past several decades.^2,3 EMRs have many theoretical advantages over paper charts, including: the ease of transfer of information between medical providers and facilities;¹ the ease of querying databases and tracking information for research and quality improvement;¹ the ability to monitor for adverse events, such as drug interactions, at the point of care;⁴ and the ability to expand the range of documentation in medical records by incorporating multimedia elements, such as digitized photographs, heart and lung sounds, and even patient interview video clips.⁵

However, while the potential for EMRs to transform medical care has been recognized since at least the 1970s,⁶ implementation of EMRs in the outpatient primary care setting has been disappointingly slow.² There are many reasons for the slow adoption of EMRs; cost and the complexity of health care delivery systems are major factors.⁷ Even so, it is likely that more primary care physicians would make the leap to electronic records, despite high initial costs and complexity, if evidence suggested their use could improve patient care. Therefore, we conducted a systematic review of the literature to answer the question: In the outpatientprimary care setting, does the use of EMRs lead to improved surrogate patient outcomes?

METHODS

Throughout 1998 and 1999, each author independently searched the MEDLINE database (1966 through 1999) via the University of California Digital Library interface8 to find relevant articles for inclusion in the systematic review. Each author used Medical Subject Headings (MeSH), key words, and publication type restrictions, in all possible combinations, to conduct the literature search.* The lists of citations retrieved were compared, and citations appearing on either list (or both) were reviewed in detail as described below.

Initial Research Question. Our initial goal was to review the original reports of all prospective studies about the impact of EMRs on patient morbidity and mortality in the outpatient primary care setting. We limited our systematic review to papers involving hybrid or complete EMR systems used by primary care physicians in the outpatient setting. A hybrid EMR was defined as a system that includes integrated access to all of the following resources: clinical laboratory and radiology data; master problem lists; inpatient and outpatient encounter diagnoses and dates; prescriptions; and billing information. Physician notes are not included in such systems; they are kept in traditional paper format. A complete EMR was defined as a system that includes all of these resources, plus full outpatient encounter progress notes, histories and physicals, and consultation notes. Some complete EMRs function as truly paperless records, while others involve the use of paper encounter forms from which information is later entered into the EMR by data entry personnel. Readers interested in the results of studies involving more limited-practice computer systems are referred to 2 recent systematic reviews.^9,10

Revised Research Question. After conducting the literature search and reviewing the retrieved citations, it was apparent that there were no published studies examining the impact of EMRs on patient morbidity and mortality. We felt that pursuing a systematic review of papers evaluating the impact of EMR systems on surrogate outcomes that have been clearly linked with changes in morbidity and mortality¹¹ would still be of value to primary care physicians. An example of a paper involving such an outcome would be one examining the impact of an EMR system on the rate of screening mammography in women age 50-69.¹²

All retrieved citations, including abstracts when available, were reviewed independently by each author. For papers in which a difference of opinion or uncertainty on the part of either reviewer existed, final consensus was reached by joint review and discussion. Relevant citations from the reference lists of reviewed papers that were not retrieved in the MEDLINE searches were also requested and reviewed.

Quality Assessment. We assessed the quality of included reports using the scale outlined in Table 1. However, given the preliminary nature of research regarding the clinical impact of EMRs, we used only the absence of a control group as a criterion for article exclusion. Of the 3 elements of our scale that were based on the instrument developed by Jadad et al¹³ (randomization, blinding, and treatment of withdrawals), 2 were modified so as to be more applicable to EMR intervention studies. Double blinding, which would require blinding of study physicians to EMR interventions, would not be feasible. However, single blinding of outcome assessment would be desirable and was included as a scoring element. In regard to withdrawals and dropouts, both physicians and their patients were considered subjects in several EMR studies (eg, both provider compliance rates with screening recommendations and patient follow-up rates following provider recommendations were outcomes). In these cases, the number and reasons for withdrawal of subjects (providers and, when applicable, patients) had to be stated or no points could be assigned. Finally, in addition to adding a control group element to Jadad’s original scale, we also added an element concerning measures taken to minimize simultaneous and potentially confounding interventions. We felt this was important, since initiatives such as clinic educational seminars could confound the results of an EMR intervention trial even in the presence of randomized, controlled clinic group assignment.

Results

Our search strategy retrieved a total of 406 citations. Three-hundred eight citations were not relevant to our study subject and were rejected without further review, leaving 98 articles requiring retrieval and detailed review (a full listing of these articles is available upon request). After independent review of each, followed by joint discussion for several articles that were initially in dispute, a total of 16 articles containing data that was not duplicated in any other reports met our inclusion criteria. Following are the EMR systems that studied in those 16 papers included in our final systematic review. (Table 2 presents the full list and details of the complete EMRs, and Table 3 presents the hybrid EMRs included in our study.)

Studies of Complete EMRs

Computer-Stored Ambulatory Record. The Computer-Stored Ambulatory Record (COSTAR) was developed in the early 1970s at Massachusetts General Hospital.⁶ In COSTAR, all clinical data collected by any provider is recorded on a paper encounter form and later entered into the computer system by support personnel. For each study, the clinical information of interest was retrieved from the EMR using a system-specific query language. Subsequently, COSTAR was used to generate printed reminder sheets for providers, prompting them to either complete problem-specific tasks noted as incomplete in the EMR query or to indicate why they had not been accomplished (eg, patient refusal). In earlier studies,^14-17 the printed reminder was not linked to a patient encounter, while in the most recent study¹⁸ the printed reminder was provided at the time of the next encounter with the patient.

Several studies employed a sequential design, with subjects serving as their own historical controls.^14-16 All general internists’ compliance with tasks was measured at baseline, then during a period when printed reminders were provided, and then again after the intervention was discontinued. Using this approach, they achieved an improved rate of antibiotic treatment in response to Group A b-hemolytic Streptococcus-positive throat culture.¹⁶ While the authors did not report exact rates, approximately 10% of patients with positive throat cultures had no documentation of antibiotic treatment in their computerized records at baseline. This rate declined to approximately 5% during the reminder intervention, and then rapidly returned to the baseline rate when the intervention was discontinued. In 2 similarly designed studies, general internists’ rates of compliance with institutionally determined lithium carbonate prescribing standards¹⁵ and syphilis quality-of-care standards¹⁴ showed significant improvement during the reminder period. In the syphilis study, providers’ mean global syphilis care scores (maximum score = 100) improved from 72.7 at baseline to 90.5 (P < 0.05 by Mann-Whitney test) during the reminder period but had dropped to 86.2 3 months after the end of the reminder period.

Two other COSTAR studies involved the use of a concurrent control group of providers that did not receive computer-generated reminders.^17-18 In the first study, 12-month rates of follow-up blood pressure measurement in response to an index diastolic blood pressure measurement of higher than 100 mm Hg in previously non-hypertensive patients were determined.¹⁷ Forty-nine percent of patients with providers in the reminder group had a follow-up blood pressure measurement within 12 months, while only 31% of patients with providers in the control group had a follow-up measurement.

Finally, in the most recent COSTAR study, rates of completion for 8 preventive health care maneuvers were determined for 2 groups.¹⁸ One group of internal medicine house staff in the University of Nebraska program received patient-specific printed reminders at the time of patient encounters, and a second group did not. Assignment was not random; residents who had clinic on alternating weeks were arbitrarily assigned to study or control groups. Computerized reminders were found to improve overall rates of tetanus, influenza, and pneumococcal vaccination and flexible sigmoidoscopy completion but had no significant impact on rates of fecal occult blood testing, mammography, Papanicolaou tests, or serum thyroxine screening in the elderly. Using factorial analysis, strong interaction was noted among group assignment, the supervising attending, and resident level of training, such that compliance scores doubled among first-year residents supervised by 2 particular attending physicians but did not substantially improve in other subgroups. Overall rates of compliance with health care maintenance maneuvers were quite low even for the intervention group (eg, 7.1% for flexible sigmoidoscopy).

Other Complete Systems. A controlled trial using the Medical University of South Carolina’s EMR evaluated 1-year rates of family medicine house staff and faculty compliance with 5 preventive health maneuvers.¹⁹ Providers and their patients were randomly assigned to 1 of 4 groups: physician and patient reminders from the EMR; patient reminders only; physician reminders only; or no reminders (control group). Patient letters were generated by the computer system, and printed patient-specific physician reminders were available at the time of encounters. Adherence to 4 of the 5 preventive services (cholesterol measurement, fecal occult blood testing, mammography, and tetanus immunization) increased significantly for all intervention groups compared with the control group, but the largest gains were seen for the group that received both physician and patient reminders. Consistent with the findings for COSTAR,¹⁸ no clinically or statistically significant impact of any reminder approach could be demonstrated for Papanicolaou test rates.

A final study involving the use of the Beth Israel Hospital’s Center for Clinical Computing system was reflective of current trends, because it involved direct provider entry of clinical encounter data into the EMR and on-screen (rather than printed) reminder prompts.²⁰ The primary outcome in this controlled trial was general internists’ median response time to EMR-generated on-screen alerts and reminders regarding care for patients with human immunodeficiency virus (HIV) infection; secondary outcomes included primary care, specialty clinic, and emergency department visit rates and hospitalizations. Providers at 5 practice sites were nonrandomly divided into a study group (on-screen prompts) and control group (no prompts of any kind). Alerts included items such as consideration of Pneumocystis carinii prophylaxis when the CD4 cell count dropped below 200 cells per cubic millimeter, and reminders included items such as the need for purified protein derivative skin testing. The median response times to 303 alerts in the intervention group and 388 in the control group were 11 and 52 days, respectively, and the median response times to 432 reminders in the intervention group and 360 reminders in the control group were 114 days and more than 500 days, respectively (both P <0001 by log-rank test). There was no effect of the intervention on health system use outcomes except for a significant increase in the rate of ophthalmologic screening examinations for the intervention group.

Studies Involving Hybrid EMRs

The Regenstrief System. The largest number of reports involving a hybrid EMR have come from the Regenstrief Institute for Health Care at Indiana University.^21-24 Most studies were randomized controlled trials monitoring rates of house staff or faculty compliance with preventive health care, prophylactic treatment (eg, b-blockers for patients who have had a myocardial infarction), and active problem treatment reminders. The reminders were generated by the EMR and supplied in printed form, either at the time of encounters or via delayed feedback messages.^21-23 Studies have built upon one another sequentially, beginning with a 1984 trial that examined the impact of encounter-based printed reminders. For 61 internal medicine residents who received computer-generated printed reminders, a 49% response rate was reported, and 54 residents who received no reminder reported a 29% response rate (P <0001 by analysis of variance). Subsequently, a 1986 trial comparing the impact of encounter-based reminders with delayed feedback messages found that the impact of the encounter reminder was approximately double that of delayed feedback, and that combining the 2 reminders had no additive effect.²¹ Finally, in 1989 a third study used encounter-based reminders for the providers in the control group and compared action rates for this group with those in which an accompanying form required providers to circle which (if any) action was taken on each reminder.²³ Compliance for faculty in the response form group was not further improved, while compliance for house staff in the response form group was significantly improved for fecal occult blood testing (63% vs 46%, P <0001) and mammography (55% vs 45%, P = .013), but not Papanicolaou tests.

One additional Regenstrief study outside this line of inquiry evaluated the impact of general internists’ remote access to EMRs on emergency department visits and hospitalization rates.²⁴ Patients receiving care in a general internal medicine clinic were randomly allocated to 1 of 3 groups: after-hours access to a study internist by phone; after-hours access to a study internist by phone who also had remote access to the EMR; or no after-hours access except via the emergency department (control). This is the only study meeting our criteria that reported no benefit of EMR use on the primary outcomes. Provider use of the remote EMR was low, accessed for only 55.8% of all calls in this intervention group. Common reasons for not using the resource were that the access device was not with the provider at the time of the call and that it did not function properly.

Ottawa System. Another series of reports came from the Family Medicine Center at Ottawa Civic Hospital.^24-28 All data appear to have been collected concurrently as part of a single randomized controlled trial and published over a 6-year period. Patients in 4 practices were randomly assigned to the control group or an intervention group. Those allocated to an intervention were then sub-randomized to 1 of 3 approaches to improving preventive health maintenance, all based on EMR-generated reminder prompts: (1) physician-provided reminder during an encounter in response to an EMR prompt; (2) computer-generated letter to the patient; or (3) telephone call from a nurse in response to an EMR prompt. Using these methods, rates of influenza,²⁵ tetanus vaccination,²⁸ Papanicolaou tests, and blood pressure screening²⁷ were substantially higher in all intervention groups than in the control group. For tetanus vaccination, the highest rate was achieved in the patient letter group (27.4%), followed by the telephone call group (20.8%), physician reminder group (19.6%), and control group (3.2%). The patient letter group also achieved the highest rate of Papanicolaou test and blood pressure screening. The telephone call group, however, had the highest rate of influenza vaccination, possibly because of the more time-sensitive, seasonal nature of this vaccination.

Other Hybrid Systems. A more recent trial involved the use of The Medical Record (TMR), developed at Duke University.²⁹ In a randomized controlled trial, rates of house staff and faculty adherence to diabetes mellitus care standards were compared in 2 groups: an intervention group, which received patient-specific, computer-generated, printed reminders at the time of patient encounters; and a control group, which received no reminders. The reminder system resulted in a 32% median compliance rate for the study group compared with a 15.6% rate in the control group (P = .01). While mean patient encounter lengths for the study group were not significantly increased, the subgroup of encounters that addressed at least some aspect of diabetes care were 10 minutes longer than those that did not involve diabetes care. Study group physicians cited encounter time constraints as the primary reason for ignoring recommendations.

Discussion

It is not possible to draw firm conclusions from the results of these trials because they were of varying quality, conducted in dissimilar centers, and employed a variety of EMRs. However, it is apparent that EMR systems offer great potential for improving rates of patient completion of health maintenance and screening maneuvers. One could argue that an EMR is not required to generate reminders, since other studies have shown similar results with non-EMR–based reminder initiatives.^9,10 However, for most medium to large practice settings, EMRs are likely to be a far superior use of time and resources than any manual or stand-alone computer system in facilitating this task.

While all the EMR-based reminder methods in these studies were superior to no method at all, EMR-generated patient reminder letters and EMR-prompted nurse reminder phone calls have been associated with screening rates superior to those resulting from EMR-prompted physician reminders to patients during clinical encounters. When physicians are relied on to make reminders to patients, success appears more likely if they are supplied with patient-specific, printed or on-screen point-of-encounter prompts rather than delayed feedback letters that are not linked to an encounter. It also appears that the ability of EMR-based reminder systems to increase the rates of screening maneuvers is greater for those interventions that can be quickly completed (eg, serum cholesterol level) than for those that require a second appointment and more inconvenience (eg, Papanicolaou test).

Limitations

A number of factors limit the strength of our conclusions. While most EMR research has been conducted in a few centers, studies of TMR30 and COSTAR¹⁸ conducted outside their originating centers reveal the difficulty in moving these systems into the community setting. In addition, the reviewed studies generally lacked rigor, using sequential designs or nonrandom assignment to study groups and implementing potentially confounding interventions, such as provider health maintenance education seminars, that make it difficult to determine the incremental benefit of the EMR system. Furthermore, studies have examined surrogate end points, such as the completion of screening interventions, rather than changes in morbidity and mortality, which are of much greater importance to primary care clinicians.¹¹ Finally, many of the studies were poorly reported, with incomplete definitions of the EMR and practice setting and only partial coverage of outcomes data.Future studies must set a more rigorous standard for EMR research.^9,31

Implications for further research

Because of the rapid pace of technology, some of the EMRs discussed in our paper are legacy systems and do not accurately reflect those that are currently on the market. To prevent this data lag phenomenon in the future, efforts must be made to report findings generated from EMR implementation projects as quickly as possible. This goal is likely to be realized if academic health centers make EMR research and implementation a high priority. Another crucial research issue concerns the relative merits of EMR components. For example, is the presence of electronic progress notes in complete EMRs associated with incremental clinical value beyond that provided by hybrid systems? Current EMR manufacturers appear to be focusing on complete systems,³² many of which involve direct physician entry of encounter data and orders, which might decrease physician time efficiency. Yet the positive findings of the Regenstrief and Ottawa trials imply that a great deal of the benefit of EMRs may derive from the coordination of laboratory, radiographic, diagnosis, medication, and administrative data (all entered by nonphysician support personnel) onto a single platform. If physicians are to embrace the financial and encounter time costs likely to be entailed by currently marketed complete EMR systems, comparative trials demonstrating tangible advantages compared with hybrid systems will be required. Such data are particularly important because physicians in several studies cited lack of time as a major reason for ignoring the EMR prompts.^22-29

The design of such trials will need to be carefully considered if the potential benefits of complete EMRs are to be discovered. EMRs are promising tools for raising the possibility of relatively uncommon diagnoses on the basis of computer-algorithmic detection of clinical data constellations from a database. A future study might examine the ability of a complete EMR system to recognize not only that a patient has poorly controlled hypertension but also that the patient is older than 65 years; has concurrent diagnoses of tobacco abuse and peripheral vascular disease; has a creatinine level of 1.7; is on an optimal triple-drug antihypertensive regimen including a diuretic; had stated compliance with that regimen at his last outpatient encounter; and had an abdominal bruit on recent physical examination. The EMR, recognizing this constellation via a pre-programmed detection algorithm, would generate a provider prompt detailing the relatively high probability of the diagnosis of renal artery stenosis as a cause for the patient’s uncontrolled hypertension. If the physician could then compare the diagnostic performance of this complete EMR with that of a hybrid EMR running a similar algorithm, the potential value of electronic progress notes could be clarified.

The handwritten record has been the standard way of documenting medical information since the 19th century.¹ However, the rapid evolution of computer technology has led to the development and use of electronic medical records (EMRs) during the past several decades.^2,3 EMRs have many theoretical advantages over paper charts, including: the ease of transfer of information between medical providers and facilities;¹ the ease of querying databases and tracking information for research and quality improvement;¹ the ability to monitor for adverse events, such as drug interactions, at the point of care;⁴ and the ability to expand the range of documentation in medical records by incorporating multimedia elements, such as digitized photographs, heart and lung sounds, and even patient interview video clips.⁵

However, while the potential for EMRs to transform medical care has been recognized since at least the 1970s,⁶ implementation of EMRs in the outpatient primary care setting has been disappointingly slow.² There are many reasons for the slow adoption of EMRs; cost and the complexity of health care delivery systems are major factors.⁷ Even so, it is likely that more primary care physicians would make the leap to electronic records, despite high initial costs and complexity, if evidence suggested their use could improve patient care. Therefore, we conducted a systematic review of the literature to answer the question: In the outpatientprimary care setting, does the use of EMRs lead to improved surrogate patient outcomes?

METHODS

Throughout 1998 and 1999, each author independently searched the MEDLINE database (1966 through 1999) via the University of California Digital Library interface8 to find relevant articles for inclusion in the systematic review. Each author used Medical Subject Headings (MeSH), key words, and publication type restrictions, in all possible combinations, to conduct the literature search.* The lists of citations retrieved were compared, and citations appearing on either list (or both) were reviewed in detail as described below.

Initial Research Question. Our initial goal was to review the original reports of all prospective studies about the impact of EMRs on patient morbidity and mortality in the outpatient primary care setting. We limited our systematic review to papers involving hybrid or complete EMR systems used by primary care physicians in the outpatient setting. A hybrid EMR was defined as a system that includes integrated access to all of the following resources: clinical laboratory and radiology data; master problem lists; inpatient and outpatient encounter diagnoses and dates; prescriptions; and billing information. Physician notes are not included in such systems; they are kept in traditional paper format. A complete EMR was defined as a system that includes all of these resources, plus full outpatient encounter progress notes, histories and physicals, and consultation notes. Some complete EMRs function as truly paperless records, while others involve the use of paper encounter forms from which information is later entered into the EMR by data entry personnel. Readers interested in the results of studies involving more limited-practice computer systems are referred to 2 recent systematic reviews.^9,10

Revised Research Question. After conducting the literature search and reviewing the retrieved citations, it was apparent that there were no published studies examining the impact of EMRs on patient morbidity and mortality. We felt that pursuing a systematic review of papers evaluating the impact of EMR systems on surrogate outcomes that have been clearly linked with changes in morbidity and mortality¹¹ would still be of value to primary care physicians. An example of a paper involving such an outcome would be one examining the impact of an EMR system on the rate of screening mammography in women age 50-69.¹²

All retrieved citations, including abstracts when available, were reviewed independently by each author. For papers in which a difference of opinion or uncertainty on the part of either reviewer existed, final consensus was reached by joint review and discussion. Relevant citations from the reference lists of reviewed papers that were not retrieved in the MEDLINE searches were also requested and reviewed.

Quality Assessment. We assessed the quality of included reports using the scale outlined in Table 1. However, given the preliminary nature of research regarding the clinical impact of EMRs, we used only the absence of a control group as a criterion for article exclusion. Of the 3 elements of our scale that were based on the instrument developed by Jadad et al¹³ (randomization, blinding, and treatment of withdrawals), 2 were modified so as to be more applicable to EMR intervention studies. Double blinding, which would require blinding of study physicians to EMR interventions, would not be feasible. However, single blinding of outcome assessment would be desirable and was included as a scoring element. In regard to withdrawals and dropouts, both physicians and their patients were considered subjects in several EMR studies (eg, both provider compliance rates with screening recommendations and patient follow-up rates following provider recommendations were outcomes). In these cases, the number and reasons for withdrawal of subjects (providers and, when applicable, patients) had to be stated or no points could be assigned. Finally, in addition to adding a control group element to Jadad’s original scale, we also added an element concerning measures taken to minimize simultaneous and potentially confounding interventions. We felt this was important, since initiatives such as clinic educational seminars could confound the results of an EMR intervention trial even in the presence of randomized, controlled clinic group assignment.

Results

Our search strategy retrieved a total of 406 citations. Three-hundred eight citations were not relevant to our study subject and were rejected without further review, leaving 98 articles requiring retrieval and detailed review (a full listing of these articles is available upon request). After independent review of each, followed by joint discussion for several articles that were initially in dispute, a total of 16 articles containing data that was not duplicated in any other reports met our inclusion criteria. Following are the EMR systems that studied in those 16 papers included in our final systematic review. (Table 2 presents the full list and details of the complete EMRs, and Table 3 presents the hybrid EMRs included in our study.)

Studies of Complete EMRs

Computer-Stored Ambulatory Record. The Computer-Stored Ambulatory Record (COSTAR) was developed in the early 1970s at Massachusetts General Hospital.⁶ In COSTAR, all clinical data collected by any provider is recorded on a paper encounter form and later entered into the computer system by support personnel. For each study, the clinical information of interest was retrieved from the EMR using a system-specific query language. Subsequently, COSTAR was used to generate printed reminder sheets for providers, prompting them to either complete problem-specific tasks noted as incomplete in the EMR query or to indicate why they had not been accomplished (eg, patient refusal). In earlier studies,^14-17 the printed reminder was not linked to a patient encounter, while in the most recent study¹⁸ the printed reminder was provided at the time of the next encounter with the patient.

Several studies employed a sequential design, with subjects serving as their own historical controls.^14-16 All general internists’ compliance with tasks was measured at baseline, then during a period when printed reminders were provided, and then again after the intervention was discontinued. Using this approach, they achieved an improved rate of antibiotic treatment in response to Group A b-hemolytic Streptococcus-positive throat culture.¹⁶ While the authors did not report exact rates, approximately 10% of patients with positive throat cultures had no documentation of antibiotic treatment in their computerized records at baseline. This rate declined to approximately 5% during the reminder intervention, and then rapidly returned to the baseline rate when the intervention was discontinued. In 2 similarly designed studies, general internists’ rates of compliance with institutionally determined lithium carbonate prescribing standards¹⁵ and syphilis quality-of-care standards¹⁴ showed significant improvement during the reminder period. In the syphilis study, providers’ mean global syphilis care scores (maximum score = 100) improved from 72.7 at baseline to 90.5 (P < 0.05 by Mann-Whitney test) during the reminder period but had dropped to 86.2 3 months after the end of the reminder period.

Two other COSTAR studies involved the use of a concurrent control group of providers that did not receive computer-generated reminders.^17-18 In the first study, 12-month rates of follow-up blood pressure measurement in response to an index diastolic blood pressure measurement of higher than 100 mm Hg in previously non-hypertensive patients were determined.¹⁷ Forty-nine percent of patients with providers in the reminder group had a follow-up blood pressure measurement within 12 months, while only 31% of patients with providers in the control group had a follow-up measurement.

Finally, in the most recent COSTAR study, rates of completion for 8 preventive health care maneuvers were determined for 2 groups.¹⁸ One group of internal medicine house staff in the University of Nebraska program received patient-specific printed reminders at the time of patient encounters, and a second group did not. Assignment was not random; residents who had clinic on alternating weeks were arbitrarily assigned to study or control groups. Computerized reminders were found to improve overall rates of tetanus, influenza, and pneumococcal vaccination and flexible sigmoidoscopy completion but had no significant impact on rates of fecal occult blood testing, mammography, Papanicolaou tests, or serum thyroxine screening in the elderly. Using factorial analysis, strong interaction was noted among group assignment, the supervising attending, and resident level of training, such that compliance scores doubled among first-year residents supervised by 2 particular attending physicians but did not substantially improve in other subgroups. Overall rates of compliance with health care maintenance maneuvers were quite low even for the intervention group (eg, 7.1% for flexible sigmoidoscopy).

Other Complete Systems. A controlled trial using the Medical University of South Carolina’s EMR evaluated 1-year rates of family medicine house staff and faculty compliance with 5 preventive health maneuvers.¹⁹ Providers and their patients were randomly assigned to 1 of 4 groups: physician and patient reminders from the EMR; patient reminders only; physician reminders only; or no reminders (control group). Patient letters were generated by the computer system, and printed patient-specific physician reminders were available at the time of encounters. Adherence to 4 of the 5 preventive services (cholesterol measurement, fecal occult blood testing, mammography, and tetanus immunization) increased significantly for all intervention groups compared with the control group, but the largest gains were seen for the group that received both physician and patient reminders. Consistent with the findings for COSTAR,¹⁸ no clinically or statistically significant impact of any reminder approach could be demonstrated for Papanicolaou test rates.

A final study involving the use of the Beth Israel Hospital’s Center for Clinical Computing system was reflective of current trends, because it involved direct provider entry of clinical encounter data into the EMR and on-screen (rather than printed) reminder prompts.²⁰ The primary outcome in this controlled trial was general internists’ median response time to EMR-generated on-screen alerts and reminders regarding care for patients with human immunodeficiency virus (HIV) infection; secondary outcomes included primary care, specialty clinic, and emergency department visit rates and hospitalizations. Providers at 5 practice sites were nonrandomly divided into a study group (on-screen prompts) and control group (no prompts of any kind). Alerts included items such as consideration of Pneumocystis carinii prophylaxis when the CD4 cell count dropped below 200 cells per cubic millimeter, and reminders included items such as the need for purified protein derivative skin testing. The median response times to 303 alerts in the intervention group and 388 in the control group were 11 and 52 days, respectively, and the median response times to 432 reminders in the intervention group and 360 reminders in the control group were 114 days and more than 500 days, respectively (both P <0001 by log-rank test). There was no effect of the intervention on health system use outcomes except for a significant increase in the rate of ophthalmologic screening examinations for the intervention group.

Studies Involving Hybrid EMRs

The Regenstrief System. The largest number of reports involving a hybrid EMR have come from the Regenstrief Institute for Health Care at Indiana University.^21-24 Most studies were randomized controlled trials monitoring rates of house staff or faculty compliance with preventive health care, prophylactic treatment (eg, b-blockers for patients who have had a myocardial infarction), and active problem treatment reminders. The reminders were generated by the EMR and supplied in printed form, either at the time of encounters or via delayed feedback messages.^21-23 Studies have built upon one another sequentially, beginning with a 1984 trial that examined the impact of encounter-based printed reminders. For 61 internal medicine residents who received computer-generated printed reminders, a 49% response rate was reported, and 54 residents who received no reminder reported a 29% response rate (P <0001 by analysis of variance). Subsequently, a 1986 trial comparing the impact of encounter-based reminders with delayed feedback messages found that the impact of the encounter reminder was approximately double that of delayed feedback, and that combining the 2 reminders had no additive effect.²¹ Finally, in 1989 a third study used encounter-based reminders for the providers in the control group and compared action rates for this group with those in which an accompanying form required providers to circle which (if any) action was taken on each reminder.²³ Compliance for faculty in the response form group was not further improved, while compliance for house staff in the response form group was significantly improved for fecal occult blood testing (63% vs 46%, P <0001) and mammography (55% vs 45%, P = .013), but not Papanicolaou tests.

One additional Regenstrief study outside this line of inquiry evaluated the impact of general internists’ remote access to EMRs on emergency department visits and hospitalization rates.²⁴ Patients receiving care in a general internal medicine clinic were randomly allocated to 1 of 3 groups: after-hours access to a study internist by phone; after-hours access to a study internist by phone who also had remote access to the EMR; or no after-hours access except via the emergency department (control). This is the only study meeting our criteria that reported no benefit of EMR use on the primary outcomes. Provider use of the remote EMR was low, accessed for only 55.8% of all calls in this intervention group. Common reasons for not using the resource were that the access device was not with the provider at the time of the call and that it did not function properly.

Ottawa System. Another series of reports came from the Family Medicine Center at Ottawa Civic Hospital.^24-28 All data appear to have been collected concurrently as part of a single randomized controlled trial and published over a 6-year period. Patients in 4 practices were randomly assigned to the control group or an intervention group. Those allocated to an intervention were then sub-randomized to 1 of 3 approaches to improving preventive health maintenance, all based on EMR-generated reminder prompts: (1) physician-provided reminder during an encounter in response to an EMR prompt; (2) computer-generated letter to the patient; or (3) telephone call from a nurse in response to an EMR prompt. Using these methods, rates of influenza,²⁵ tetanus vaccination,²⁸ Papanicolaou tests, and blood pressure screening²⁷ were substantially higher in all intervention groups than in the control group. For tetanus vaccination, the highest rate was achieved in the patient letter group (27.4%), followed by the telephone call group (20.8%), physician reminder group (19.6%), and control group (3.2%). The patient letter group also achieved the highest rate of Papanicolaou test and blood pressure screening. The telephone call group, however, had the highest rate of influenza vaccination, possibly because of the more time-sensitive, seasonal nature of this vaccination.

Other Hybrid Systems. A more recent trial involved the use of The Medical Record (TMR), developed at Duke University.²⁹ In a randomized controlled trial, rates of house staff and faculty adherence to diabetes mellitus care standards were compared in 2 groups: an intervention group, which received patient-specific, computer-generated, printed reminders at the time of patient encounters; and a control group, which received no reminders. The reminder system resulted in a 32% median compliance rate for the study group compared with a 15.6% rate in the control group (P = .01). While mean patient encounter lengths for the study group were not significantly increased, the subgroup of encounters that addressed at least some aspect of diabetes care were 10 minutes longer than those that did not involve diabetes care. Study group physicians cited encounter time constraints as the primary reason for ignoring recommendations.

Discussion

It is not possible to draw firm conclusions from the results of these trials because they were of varying quality, conducted in dissimilar centers, and employed a variety of EMRs. However, it is apparent that EMR systems offer great potential for improving rates of patient completion of health maintenance and screening maneuvers. One could argue that an EMR is not required to generate reminders, since other studies have shown similar results with non-EMR–based reminder initiatives.^9,10 However, for most medium to large practice settings, EMRs are likely to be a far superior use of time and resources than any manual or stand-alone computer system in facilitating this task.

While all the EMR-based reminder methods in these studies were superior to no method at all, EMR-generated patient reminder letters and EMR-prompted nurse reminder phone calls have been associated with screening rates superior to those resulting from EMR-prompted physician reminders to patients during clinical encounters. When physicians are relied on to make reminders to patients, success appears more likely if they are supplied with patient-specific, printed or on-screen point-of-encounter prompts rather than delayed feedback letters that are not linked to an encounter. It also appears that the ability of EMR-based reminder systems to increase the rates of screening maneuvers is greater for those interventions that can be quickly completed (eg, serum cholesterol level) than for those that require a second appointment and more inconvenience (eg, Papanicolaou test).

Limitations

A number of factors limit the strength of our conclusions. While most EMR research has been conducted in a few centers, studies of TMR30 and COSTAR¹⁸ conducted outside their originating centers reveal the difficulty in moving these systems into the community setting. In addition, the reviewed studies generally lacked rigor, using sequential designs or nonrandom assignment to study groups and implementing potentially confounding interventions, such as provider health maintenance education seminars, that make it difficult to determine the incremental benefit of the EMR system. Furthermore, studies have examined surrogate end points, such as the completion of screening interventions, rather than changes in morbidity and mortality, which are of much greater importance to primary care clinicians.¹¹ Finally, many of the studies were poorly reported, with incomplete definitions of the EMR and practice setting and only partial coverage of outcomes data.Future studies must set a more rigorous standard for EMR research.^9,31

Implications for further research

Because of the rapid pace of technology, some of the EMRs discussed in our paper are legacy systems and do not accurately reflect those that are currently on the market. To prevent this data lag phenomenon in the future, efforts must be made to report findings generated from EMR implementation projects as quickly as possible. This goal is likely to be realized if academic health centers make EMR research and implementation a high priority. Another crucial research issue concerns the relative merits of EMR components. For example, is the presence of electronic progress notes in complete EMRs associated with incremental clinical value beyond that provided by hybrid systems? Current EMR manufacturers appear to be focusing on complete systems,³² many of which involve direct physician entry of encounter data and orders, which might decrease physician time efficiency. Yet the positive findings of the Regenstrief and Ottawa trials imply that a great deal of the benefit of EMRs may derive from the coordination of laboratory, radiographic, diagnosis, medication, and administrative data (all entered by nonphysician support personnel) onto a single platform. If physicians are to embrace the financial and encounter time costs likely to be entailed by currently marketed complete EMR systems, comparative trials demonstrating tangible advantages compared with hybrid systems will be required. Such data are particularly important because physicians in several studies cited lack of time as a major reason for ignoring the EMR prompts.^22-29

The design of such trials will need to be carefully considered if the potential benefits of complete EMRs are to be discovered. EMRs are promising tools for raising the possibility of relatively uncommon diagnoses on the basis of computer-algorithmic detection of clinical data constellations from a database. A future study might examine the ability of a complete EMR system to recognize not only that a patient has poorly controlled hypertension but also that the patient is older than 65 years; has concurrent diagnoses of tobacco abuse and peripheral vascular disease; has a creatinine level of 1.7; is on an optimal triple-drug antihypertensive regimen including a diuretic; had stated compliance with that regimen at his last outpatient encounter; and had an abdominal bruit on recent physical examination. The EMR, recognizing this constellation via a pre-programmed detection algorithm, would generate a provider prompt detailing the relatively high probability of the diagnosis of renal artery stenosis as a cause for the patient’s uncontrolled hypertension. If the physician could then compare the diagnostic performance of this complete EMR with that of a hybrid EMR running a similar algorithm, the potential value of electronic progress notes could be clarified.

The handwritten record has been the standard way of documenting medical information since the 19th century.¹ However, the rapid evolution of computer technology has led to the development and use of electronic medical records (EMRs) during the past several decades.^2,3 EMRs have many theoretical advantages over paper charts, including: the ease of transfer of information between medical providers and facilities;¹ the ease of querying databases and tracking information for research and quality improvement;¹ the ability to monitor for adverse events, such as drug interactions, at the point of care;⁴ and the ability to expand the range of documentation in medical records by incorporating multimedia elements, such as digitized photographs, heart and lung sounds, and even patient interview video clips.⁵

However, while the potential for EMRs to transform medical care has been recognized since at least the 1970s,⁶ implementation of EMRs in the outpatient primary care setting has been disappointingly slow.² There are many reasons for the slow adoption of EMRs; cost and the complexity of health care delivery systems are major factors.⁷ Even so, it is likely that more primary care physicians would make the leap to electronic records, despite high initial costs and complexity, if evidence suggested their use could improve patient care. Therefore, we conducted a systematic review of the literature to answer the question: In the outpatientprimary care setting, does the use of EMRs lead to improved surrogate patient outcomes?

METHODS

Throughout 1998 and 1999, each author independently searched the MEDLINE database (1966 through 1999) via the University of California Digital Library interface8 to find relevant articles for inclusion in the systematic review. Each author used Medical Subject Headings (MeSH), key words, and publication type restrictions, in all possible combinations, to conduct the literature search.* The lists of citations retrieved were compared, and citations appearing on either list (or both) were reviewed in detail as described below.

Initial Research Question. Our initial goal was to review the original reports of all prospective studies about the impact of EMRs on patient morbidity and mortality in the outpatient primary care setting. We limited our systematic review to papers involving hybrid or complete EMR systems used by primary care physicians in the outpatient setting. A hybrid EMR was defined as a system that includes integrated access to all of the following resources: clinical laboratory and radiology data; master problem lists; inpatient and outpatient encounter diagnoses and dates; prescriptions; and billing information. Physician notes are not included in such systems; they are kept in traditional paper format. A complete EMR was defined as a system that includes all of these resources, plus full outpatient encounter progress notes, histories and physicals, and consultation notes. Some complete EMRs function as truly paperless records, while others involve the use of paper encounter forms from which information is later entered into the EMR by data entry personnel. Readers interested in the results of studies involving more limited-practice computer systems are referred to 2 recent systematic reviews.^9,10

Revised Research Question. After conducting the literature search and reviewing the retrieved citations, it was apparent that there were no published studies examining the impact of EMRs on patient morbidity and mortality. We felt that pursuing a systematic review of papers evaluating the impact of EMR systems on surrogate outcomes that have been clearly linked with changes in morbidity and mortality¹¹ would still be of value to primary care physicians. An example of a paper involving such an outcome would be one examining the impact of an EMR system on the rate of screening mammography in women age 50-69.¹²

All retrieved citations, including abstracts when available, were reviewed independently by each author. For papers in which a difference of opinion or uncertainty on the part of either reviewer existed, final consensus was reached by joint review and discussion. Relevant citations from the reference lists of reviewed papers that were not retrieved in the MEDLINE searches were also requested and reviewed.

Quality Assessment. We assessed the quality of included reports using the scale outlined in Table 1. However, given the preliminary nature of research regarding the clinical impact of EMRs, we used only the absence of a control group as a criterion for article exclusion. Of the 3 elements of our scale that were based on the instrument developed by Jadad et al¹³ (randomization, blinding, and treatment of withdrawals), 2 were modified so as to be more applicable to EMR intervention studies. Double blinding, which would require blinding of study physicians to EMR interventions, would not be feasible. However, single blinding of outcome assessment would be desirable and was included as a scoring element. In regard to withdrawals and dropouts, both physicians and their patients were considered subjects in several EMR studies (eg, both provider compliance rates with screening recommendations and patient follow-up rates following provider recommendations were outcomes). In these cases, the number and reasons for withdrawal of subjects (providers and, when applicable, patients) had to be stated or no points could be assigned. Finally, in addition to adding a control group element to Jadad’s original scale, we also added an element concerning measures taken to minimize simultaneous and potentially confounding interventions. We felt this was important, since initiatives such as clinic educational seminars could confound the results of an EMR intervention trial even in the presence of randomized, controlled clinic group assignment.

Results

Our search strategy retrieved a total of 406 citations. Three-hundred eight citations were not relevant to our study subject and were rejected without further review, leaving 98 articles requiring retrieval and detailed review (a full listing of these articles is available upon request). After independent review of each, followed by joint discussion for several articles that were initially in dispute, a total of 16 articles containing data that was not duplicated in any other reports met our inclusion criteria. Following are the EMR systems that studied in those 16 papers included in our final systematic review. (Table 2 presents the full list and details of the complete EMRs, and Table 3 presents the hybrid EMRs included in our study.)

Studies of Complete EMRs

Computer-Stored Ambulatory Record. The Computer-Stored Ambulatory Record (COSTAR) was developed in the early 1970s at Massachusetts General Hospital.⁶ In COSTAR, all clinical data collected by any provider is recorded on a paper encounter form and later entered into the computer system by support personnel. For each study, the clinical information of interest was retrieved from the EMR using a system-specific query language. Subsequently, COSTAR was used to generate printed reminder sheets for providers, prompting them to either complete problem-specific tasks noted as incomplete in the EMR query or to indicate why they had not been accomplished (eg, patient refusal). In earlier studies,^14-17 the printed reminder was not linked to a patient encounter, while in the most recent study¹⁸ the printed reminder was provided at the time of the next encounter with the patient.

Several studies employed a sequential design, with subjects serving as their own historical controls.^14-16 All general internists’ compliance with tasks was measured at baseline, then during a period when printed reminders were provided, and then again after the intervention was discontinued. Using this approach, they achieved an improved rate of antibiotic treatment in response to Group A b-hemolytic Streptococcus-positive throat culture.¹⁶ While the authors did not report exact rates, approximately 10% of patients with positive throat cultures had no documentation of antibiotic treatment in their computerized records at baseline. This rate declined to approximately 5% during the reminder intervention, and then rapidly returned to the baseline rate when the intervention was discontinued. In 2 similarly designed studies, general internists’ rates of compliance with institutionally determined lithium carbonate prescribing standards¹⁵ and syphilis quality-of-care standards¹⁴ showed significant improvement during the reminder period. In the syphilis study, providers’ mean global syphilis care scores (maximum score = 100) improved from 72.7 at baseline to 90.5 (P < 0.05 by Mann-Whitney test) during the reminder period but had dropped to 86.2 3 months after the end of the reminder period.

Two other COSTAR studies involved the use of a concurrent control group of providers that did not receive computer-generated reminders.^17-18 In the first study, 12-month rates of follow-up blood pressure measurement in response to an index diastolic blood pressure measurement of higher than 100 mm Hg in previously non-hypertensive patients were determined.¹⁷ Forty-nine percent of patients with providers in the reminder group had a follow-up blood pressure measurement within 12 months, while only 31% of patients with providers in the control group had a follow-up measurement.

Finally, in the most recent COSTAR study, rates of completion for 8 preventive health care maneuvers were determined for 2 groups.¹⁸ One group of internal medicine house staff in the University of Nebraska program received patient-specific printed reminders at the time of patient encounters, and a second group did not. Assignment was not random; residents who had clinic on alternating weeks were arbitrarily assigned to study or control groups. Computerized reminders were found to improve overall rates of tetanus, influenza, and pneumococcal vaccination and flexible sigmoidoscopy completion but had no significant impact on rates of fecal occult blood testing, mammography, Papanicolaou tests, or serum thyroxine screening in the elderly. Using factorial analysis, strong interaction was noted among group assignment, the supervising attending, and resident level of training, such that compliance scores doubled among first-year residents supervised by 2 particular attending physicians but did not substantially improve in other subgroups. Overall rates of compliance with health care maintenance maneuvers were quite low even for the intervention group (eg, 7.1% for flexible sigmoidoscopy).

Other Complete Systems. A controlled trial using the Medical University of South Carolina’s EMR evaluated 1-year rates of family medicine house staff and faculty compliance with 5 preventive health maneuvers.¹⁹ Providers and their patients were randomly assigned to 1 of 4 groups: physician and patient reminders from the EMR; patient reminders only; physician reminders only; or no reminders (control group). Patient letters were generated by the computer system, and printed patient-specific physician reminders were available at the time of encounters. Adherence to 4 of the 5 preventive services (cholesterol measurement, fecal occult blood testing, mammography, and tetanus immunization) increased significantly for all intervention groups compared with the control group, but the largest gains were seen for the group that received both physician and patient reminders. Consistent with the findings for COSTAR,¹⁸ no clinically or statistically significant impact of any reminder approach could be demonstrated for Papanicolaou test rates.

A final study involving the use of the Beth Israel Hospital’s Center for Clinical Computing system was reflective of current trends, because it involved direct provider entry of clinical encounter data into the EMR and on-screen (rather than printed) reminder prompts.²⁰ The primary outcome in this controlled trial was general internists’ median response time to EMR-generated on-screen alerts and reminders regarding care for patients with human immunodeficiency virus (HIV) infection; secondary outcomes included primary care, specialty clinic, and emergency department visit rates and hospitalizations. Providers at 5 practice sites were nonrandomly divided into a study group (on-screen prompts) and control group (no prompts of any kind). Alerts included items such as consideration of Pneumocystis carinii prophylaxis when the CD4 cell count dropped below 200 cells per cubic millimeter, and reminders included items such as the need for purified protein derivative skin testing. The median response times to 303 alerts in the intervention group and 388 in the control group were 11 and 52 days, respectively, and the median response times to 432 reminders in the intervention group and 360 reminders in the control group were 114 days and more than 500 days, respectively (both P <0001 by log-rank test). There was no effect of the intervention on health system use outcomes except for a significant increase in the rate of ophthalmologic screening examinations for the intervention group.

Studies Involving Hybrid EMRs

The Regenstrief System. The largest number of reports involving a hybrid EMR have come from the Regenstrief Institute for Health Care at Indiana University.^21-24 Most studies were randomized controlled trials monitoring rates of house staff or faculty compliance with preventive health care, prophylactic treatment (eg, b-blockers for patients who have had a myocardial infarction), and active problem treatment reminders. The reminders were generated by the EMR and supplied in printed form, either at the time of encounters or via delayed feedback messages.^21-23 Studies have built upon one another sequentially, beginning with a 1984 trial that examined the impact of encounter-based printed reminders. For 61 internal medicine residents who received computer-generated printed reminders, a 49% response rate was reported, and 54 residents who received no reminder reported a 29% response rate (P <0001 by analysis of variance). Subsequently, a 1986 trial comparing the impact of encounter-based reminders with delayed feedback messages found that the impact of the encounter reminder was approximately double that of delayed feedback, and that combining the 2 reminders had no additive effect.²¹ Finally, in 1989 a third study used encounter-based reminders for the providers in the control group and compared action rates for this group with those in which an accompanying form required providers to circle which (if any) action was taken on each reminder.²³ Compliance for faculty in the response form group was not further improved, while compliance for house staff in the response form group was significantly improved for fecal occult blood testing (63% vs 46%, P <0001) and mammography (55% vs 45%, P = .013), but not Papanicolaou tests.

One additional Regenstrief study outside this line of inquiry evaluated the impact of general internists’ remote access to EMRs on emergency department visits and hospitalization rates.²⁴ Patients receiving care in a general internal medicine clinic were randomly allocated to 1 of 3 groups: after-hours access to a study internist by phone; after-hours access to a study internist by phone who also had remote access to the EMR; or no after-hours access except via the emergency department (control). This is the only study meeting our criteria that reported no benefit of EMR use on the primary outcomes. Provider use of the remote EMR was low, accessed for only 55.8% of all calls in this intervention group. Common reasons for not using the resource were that the access device was not with the provider at the time of the call and that it did not function properly.

Ottawa System. Another series of reports came from the Family Medicine Center at Ottawa Civic Hospital.^24-28 All data appear to have been collected concurrently as part of a single randomized controlled trial and published over a 6-year period. Patients in 4 practices were randomly assigned to the control group or an intervention group. Those allocated to an intervention were then sub-randomized to 1 of 3 approaches to improving preventive health maintenance, all based on EMR-generated reminder prompts: (1) physician-provided reminder during an encounter in response to an EMR prompt; (2) computer-generated letter to the patient; or (3) telephone call from a nurse in response to an EMR prompt. Using these methods, rates of influenza,²⁵ tetanus vaccination,²⁸ Papanicolaou tests, and blood pressure screening²⁷ were substantially higher in all intervention groups than in the control group. For tetanus vaccination, the highest rate was achieved in the patient letter group (27.4%), followed by the telephone call group (20.8%), physician reminder group (19.6%), and control group (3.2%). The patient letter group also achieved the highest rate of Papanicolaou test and blood pressure screening. The telephone call group, however, had the highest rate of influenza vaccination, possibly because of the more time-sensitive, seasonal nature of this vaccination.

Other Hybrid Systems. A more recent trial involved the use of The Medical Record (TMR), developed at Duke University.²⁹ In a randomized controlled trial, rates of house staff and faculty adherence to diabetes mellitus care standards were compared in 2 groups: an intervention group, which received patient-specific, computer-generated, printed reminders at the time of patient encounters; and a control group, which received no reminders. The reminder system resulted in a 32% median compliance rate for the study group compared with a 15.6% rate in the control group (P = .01). While mean patient encounter lengths for the study group were not significantly increased, the subgroup of encounters that addressed at least some aspect of diabetes care were 10 minutes longer than those that did not involve diabetes care. Study group physicians cited encounter time constraints as the primary reason for ignoring recommendations.

Discussion

It is not possible to draw firm conclusions from the results of these trials because they were of varying quality, conducted in dissimilar centers, and employed a variety of EMRs. However, it is apparent that EMR systems offer great potential for improving rates of patient completion of health maintenance and screening maneuvers. One could argue that an EMR is not required to generate reminders, since other studies have shown similar results with non-EMR–based reminder initiatives.^9,10 However, for most medium to large practice settings, EMRs are likely to be a far superior use of time and resources than any manual or stand-alone computer system in facilitating this task.

While all the EMR-based reminder methods in these studies were superior to no method at all, EMR-generated patient reminder letters and EMR-prompted nurse reminder phone calls have been associated with screening rates superior to those resulting from EMR-prompted physician reminders to patients during clinical encounters. When physicians are relied on to make reminders to patients, success appears more likely if they are supplied with patient-specific, printed or on-screen point-of-encounter prompts rather than delayed feedback letters that are not linked to an encounter. It also appears that the ability of EMR-based reminder systems to increase the rates of screening maneuvers is greater for those interventions that can be quickly completed (eg, serum cholesterol level) than for those that require a second appointment and more inconvenience (eg, Papanicolaou test).

Limitations

A number of factors limit the strength of our conclusions. While most EMR research has been conducted in a few centers, studies of TMR30 and COSTAR¹⁸ conducted outside their originating centers reveal the difficulty in moving these systems into the community setting. In addition, the reviewed studies generally lacked rigor, using sequential designs or nonrandom assignment to study groups and implementing potentially confounding interventions, such as provider health maintenance education seminars, that make it difficult to determine the incremental benefit of the EMR system. Furthermore, studies have examined surrogate end points, such as the completion of screening interventions, rather than changes in morbidity and mortality, which are of much greater importance to primary care clinicians.¹¹ Finally, many of the studies were poorly reported, with incomplete definitions of the EMR and practice setting and only partial coverage of outcomes data.Future studies must set a more rigorous standard for EMR research.^9,31

Implications for further research

Because of the rapid pace of technology, some of the EMRs discussed in our paper are legacy systems and do not accurately reflect those that are currently on the market. To prevent this data lag phenomenon in the future, efforts must be made to report findings generated from EMR implementation projects as quickly as possible. This goal is likely to be realized if academic health centers make EMR research and implementation a high priority. Another crucial research issue concerns the relative merits of EMR components. For example, is the presence of electronic progress notes in complete EMRs associated with incremental clinical value beyond that provided by hybrid systems? Current EMR manufacturers appear to be focusing on complete systems,³² many of which involve direct physician entry of encounter data and orders, which might decrease physician time efficiency. Yet the positive findings of the Regenstrief and Ottawa trials imply that a great deal of the benefit of EMRs may derive from the coordination of laboratory, radiographic, diagnosis, medication, and administrative data (all entered by nonphysician support personnel) onto a single platform. If physicians are to embrace the financial and encounter time costs likely to be entailed by currently marketed complete EMR systems, comparative trials demonstrating tangible advantages compared with hybrid systems will be required. Such data are particularly important because physicians in several studies cited lack of time as a major reason for ignoring the EMR prompts.^22-29

The design of such trials will need to be carefully considered if the potential benefits of complete EMRs are to be discovered. EMRs are promising tools for raising the possibility of relatively uncommon diagnoses on the basis of computer-algorithmic detection of clinical data constellations from a database. A future study might examine the ability of a complete EMR system to recognize not only that a patient has poorly controlled hypertension but also that the patient is older than 65 years; has concurrent diagnoses of tobacco abuse and peripheral vascular disease; has a creatinine level of 1.7; is on an optimal triple-drug antihypertensive regimen including a diuretic; had stated compliance with that regimen at his last outpatient encounter; and had an abdominal bruit on recent physical examination. The EMR, recognizing this constellation via a pre-programmed detection algorithm, would generate a provider prompt detailing the relatively high probability of the diagnosis of renal artery stenosis as a cause for the patient’s uncontrolled hypertension. If the physician could then compare the diagnostic performance of this complete EMR with that of a hybrid EMR running a similar algorithm, the potential value of electronic progress notes could be clarified.