User login
METHODS: Eighteen purposefully selected family medicine practices were directly observed for 4 to 12 weeks each. A total of 57 providers were shadowed by a research nurse, and detailed field notes on 1600 patient encounters were recorded. A 3-member analysis team reviewed the qualitative data and identified emergent patterns.
RESULTS: Clinics’ use of patient education materials fell mostly into 2 distinct patterns. “Stockpilers” were providers who relied on the clinic staff to develop and organize a common library of patient education handouts. Providers with a “personal stash” collected much smaller numbers of materials that they personally maintained. Providers in the latter group had a known repertoire of a limited amount of educational material and used it more often than providers with access to a greater variety and number of handouts. In all practices, providers distributed most handouts; staff and self-selection by patients played a minor role.
CONCLUSIONS: It appears that provider involvement and familiarity with patient education materials are key to their use in clinical practice. Clinicians use written patient education materials most efficiently by personally selecting and maintaining a small number of handouts that address topics most relevant to their practice.
Patient education has been defined as a “learning experience using a combination of methods such as teaching, counseling and behavior modification techniques which influence patient’s knowledge and health behavior.”1 Health information provided by physicians is a type of patient education valued by patients2,3 and that plays a role in the complex issue of improving overall health status and psychosocial functioning.4,5 Various educational formats are used by physicians including verbal instruction, counseling, referral to education specialists, audio or videotapes, and written education material. Printed handouts, however, have several theoretical advantages over other methods of patient education:6 Patients may have difficulty remembering instructions that are presented only verbally,3,7,8 and written handouts can supplement or reinforce information presented during the office visit. Handouts are also proposed to be a cost-effective means for the busy clinician to educate patients without having to convey all the information personally.
Several studies have confirmed the theoretical utility of written patient education materials. Patients both read and retain written materials given to them by health professionals.9-11 This written information has been shown to increase compliance with medication use,12-14 physicians’ instructions,15,16 smoking cessation,4,8 and the use of preventive health services.8,17 Information handouts have also been shown to increase patient knowledge,18-20 to increase satisfaction with patient care,9,10,21 and to reduce patient anxiety.5,8 Written information has even been shown to reduce unnecessary medical visits22-24 and inappropriate telephone calls.25 For these reasons, both providers and patients report they value written materials.26-28
Large amounts of patient education materials are available, covering topics ranging from prevention and safety issues to strategies for coping with chronic diseases. The American Cancer Society, for example, spent $56 million on producing patient education materials in 1989.29 In spite of the amount of material available, little information has been published on how handouts are used in actual clinical practice.
Few studies in the medical literature describe strategies clinicians can use to organize and disseminate written materials in the office setting. Most authors recommend compiling a comprehensive set of written handouts covering a broad array of clinical topics and indexing these materials to facilitate retrieval by medical personnel.3,30-32 Some of these authors have also described the use of computers to both index and generate written materials.3,30,33-35 The various strategies described for disseminating written materials to patients have included using the clinician, nursing staff or receptionists,36 patient libraries,32,37 and racks in waiting rooms.3,8 None of these strategies have been adequately evaluated in terms of their effectiveness in enhancing the use of written patient education materials, and no studies have described what office systems exist in actual practices to support their use.
This paper describes how patient education materials are organized and disseminated in family practices, and the office system factors that contribute to their use in real world settings.
Methods
The “Prevention and Competing Demands in Primary Care Practice” study was designed to examine the organizational contexts that support or inhibit the delivery of preventive services in family medicine practices. From November 1996 to February 1999, extensive descriptive field notes were recorded throughout this large comparative case study of family practice organizations. Eighteen practices were studied using a multimethod ethnographic design that involved observation of clinical encounters and the office system by a research nurse who spent 4 weeks or more in each practice. A total of 44 physicians and 13 other primary care providers were shadowed and interviewed, and approximately 1600 patient encounters were directly observed. Data collection generated approximately 20,000 pages of text materials. Descriptions of how patient education materials were organized and used in practices were drawn from this larger data set that looked more broadly at preventive services delivery.
Sampling
A total of 18 family practices were purposefully selected38 from across Nebraska. Based on results of a previous study,39 practices known to deliver both high and low levels of tobacco prevention services were included. The sites were chosen to include a wide variety of practice types, with maximum variation with respect to rural/urban, small/large, and privately owned/part of a larger health system. After preliminary analyses of the initial 10 practices, 8 additional practices were selected to search for confirming or challenging cases using replication logic.40Table 1 summarizes the characteristics of the practices and physicians studied.
Study participation was solicited by sending an invitation letter, followed by a phone call to one of the physicians within the practice. Later, the consent of all clinicians to conduct research in the practice was obtained. Only 5 practices declined to participate. Three individual physicians subsequently declined participation after their practice was enrolled in the study (2 family physicians who were on the verge of retirement and one gynecologist who saw patients part-time in one family practice). Data collection proceeded for the other clinicians enrolled in the study for those sites.
Data Collection
A research nurse trained in qualitative methods was sent to each practice where she used a variety of data collection methods to produce a comprehensive picture of the practice as a functioning organization. It took 4 to 12 weeks for the nurse to complete the data collection in each practice depending on its size.
Field Notes. The research nurse observed the physical environment and functioning of the practice and dictated extensive field notes at the end of each day.41 These notes contained detailed descriptions of the clinic location and environment, patient characteristics, nursing station, examination rooms, the waiting area, bulletin boards, posters, and patient education materials. Photographs were taken of each room. The nurse specifically noted the location and organization of patient education materials and their accessibility to both providers and patients. She inventoried the available patient education materials, noting the number of brochures available, the topics covered, and who produced them. Samples of each patient education handout were obtained whenever possible. She also noted who was responsible for maintaining and organizing patient education supplies.
Checklist of Office Environment. Structured checklists of the office environment facilitated quantification of specific areas of interest and served as a template for standardized field note descriptions of the practice.42 Items on this 5-page instrument included the number of patients scheduled and seen per day, the number of personnel in the office, and the percentage of patients covered by managed care plans. The accessibility, quality, and patient use of education materials were also specifically recorded.
Patient Encounters. Approximately 30 patient encounters for each provider were observed. After obtaining written informed consent from the patient, the research nurse shadowed the provider and took notes for later dictation. The patient encounter field notes contained descriptions of any verbal patient teaching and the context of that education, including the reason for the visit, how the visit unfolded, and how the provider and patient interacted. The patient encounter structured checklist captured the number of times patient education materials were used.43 Thus it was possible to quantify how frequently patient education handouts were used during the observed patient visits, as well as to describe the context of their use.
Patient Pathways. The research nurse also followed 2 of each clinician’s patients from the time they entered the practice until they left. These patient pathways provided a minute by minute recording of events from the time of entry into the health center, encounter with the receptionist, nurse, physician, checkout, and until the patient left.44 To collect this data, the research nurse followed patients during their clinic visit, noting the places they visited, how long they waited and what happened to them during each stage. Opportunities for patients to select patient education materials intended for self-service were noted.
Interviews. Depth interviews with each provider in the practice explored themes related to the delivery of preventive services.45,46 Patient education materials were sometimes discussed during the interviews, but specific questions about their use were not included on the interview guide.
Data Analysis and Interpretation
All quantitative and qualitative data were checked for accuracy and entered into Folioviews, an infobase software package.47,48 This software program facilitates the organization of text documents and allows computerized searches and coding of the qualitative database.
The first phase of data analysis was an immersion/crystallization process49,50 that lead to the development of a code book—an organizational scheme for understanding the qualitative data—that could be applied to the entire data set. Initially, one of the authors (MV) immersed herself in all the data from 5 purposefully selected practices to understand the functioning, organization, and dynamics of the practice. She read the field notes on the computer and made written notes on each practice, then reviewed patient encounter checklists to see how often handouts were used by each provider. Using this approach, she worked with the other authors to crystallize hypotheses and form an initial organizational scheme. Group discussions among all of the authors led to the development of our code book.51
A strategy for sampling the data from other practices was used to identify relevant portions of the larger data set for secondary data analysis. A sample of at least 10 patient encounters with each provider from the 18 practices was read, noting the content and type of patient education. We found that only 10 encounters were needed to reach saturation of our understanding of a provider’s educational style. In addition, all encounters in which a patient education handout was given were reviewed, for a total of 500 patient encounters. Computer searches using key words helped find places where patient education was discussed in the field notes and patient encounters. Photographs of all the practices were reviewed to provide a clear mental picture of the location of the patient education materials in different areas of the clinic.
During the second phase of analysis, the code book was applied to the relevant portions of the database using a template organizing style.52 Segments of the field notes and patient encounter notes that discussed aspects of patient education were identified and organized, and eventually used to construct matrices or tables.51 These matrices allowed visualization of emergent patterns and facilitated comparisons across cases.
Results
Use of patient education materials and the strategies to organize and distribute the handouts varied among providers. Some practices had acquired large numbers of patient education materials while others focused on a small number of handouts. Similarly, staff involvement in acquiring and organizing patient education materials varied from practice to practice. Two distinct patterns of organizational style emerged from the data related to these themes that had implications for the use of the materials Table 2 summarizes this data from each of the practices.
Stockpile Organization
Most of the practices in the study had accumulated large amounts of patient education materials, and many had more than 150 different handouts available. These were eclectic collections of materials that consisted of the pooled contributions of providers, nurses, office staff, and pharmaceutical representatives. Once assembled, these collections were intended for communal use by all providers in the clinic. In some instances, larger health systems had provided substantial collections of printed handouts to each of their clinics. Some practices had the ability to access and reproduce computer-generated patient education materials, further expanding the range of topics available. In these large collections there was often a great deal of redundancy. One 3-physician practice, for example, had 5 different brochures on childhood lead screening. Responsibility for maintaining and organizing these large collections of patient education materials was a time-consuming task that was usually delegated to a staff member. As a result, most physicians were unfamiliar with the handouts available in their practice. In those practices with extensive shared collections of patient education materials, very few handouts were actually used by the providers. The first case provides an example of this type of practice.
The Stockpile. This was a large physician group that recognized patient education as a priority for the practice. They had tried a variety of strategies to enhance their use of written materials, including the hiring of a health education coordinator. She was assigned the task of organizing and maintaining all of their patient handouts.
In this role, the health education coordinator had accumulated a huge number of handouts on a broad range of topics (45 different handouts on well-child care alone). The materials came from pharmaceutical companies, the clinic physicians, professional organizations, and the coordinator’s personal files, and more continued to filter in as the physicians brought in new materials. These education materials were stored at the nurses’ station in neatly color-coded file drawers. Other nursing staff distributed materials in response to physician requests.
Despite the tremendous amount of handouts available, their organization and quality, the staff involvement, and the physicians’ own commitment to using them, written patient education materials were rarely distributed (14/272 or 5% of observed visits).
This was a model clinic in the sense that it had the most staff resources dedicated to maintaining the stockpile of handouts and one of the most organized collections. Many other clinics used the same overall strategy, with similar infrequency of use.
Providers stated one of the reasons they did not use the materials organized in the stockpile was that they were unsure of the quality and accuracy of the information presented. One physician said, “I don’t always necessarily agree 100% with what’s in there, and I feel like if I’m going to hand it out, it’s something I should have read myself.” Another problem with the stockpile approach was that the providers did not know what was available. “There is so much there that it’s almost overwhelming,” one physician stated. Having too many handouts may have made it difficult for providers to familiarize themselves with the materials or to locate the ones they wanted to use.
Personal Stash Organization
Another set of practices adopted a different strategy of using patient education materials. In these practices, the physicians themselves took responsibility for the patient education handouts. The individual physicians each selected, organized, and maintained a small private collection of materials. Even within group practices, physicians using this organizational scheme would maintain separate collections of patient handouts that fit their unique instructional needs. In these practices, patient education materials were used more frequently.
The Personal Stash. One physician and a recently hired nurse practitioner worked in a clinic with a diverse patient population ranging from trendy young professionals to homeless families. The physician consistently incorporated verbal patient education into almost every clinical encounter. In counseling patients, he demonstrated remarkable versatility in acknowledging patients’ unique social and familial circumstances, as well as their readiness to adopt healthier behaviors.
Very few (28) patient handouts were available in the practice. The physician had developed many of these handouts himself by photocopying articles from popular magazines and books. He was responsible for organizing and maintaining his supply of handouts, and he personally retrieved them from his office filing cabinet to distribute to patients. This physician used patient education handouts quite frequently (in 13 of 52 observed visits—25%).
Providers using the personal stash approach were observed using handouts more frequently than those using shared stockpiles. Where providers drew from a small but known repertoire of patient education materials they used them more often than those in practices with a large number of handouts.
Hybrid Approach
Exceptions to this overall pattern were 4 cliniciansin 2 separate group practices that used the stockpile approach. These clinics had large numbers of patient education handouts maintained by staff members for communal use. In contrast to their partners, these 4 clinicians used handouts extensively (30/122 observed visits). These providers had a personal interest and involvement in the practice’s library of patient education materials. They each maintained a portfolio of selected handouts kept in their own examination rooms or offices, and they had each personally developed some materials. These 4 clinicians had forged a hybrid approach of maintaining hands-on involvement within a system that delegated responsibility to staff. In this way, they were able to maintain familiarity with the materials and use them frequently.
Role of Staff and Patient Self-Selection
In all of the practices, clinicians distributed the majority of patient education materials, and the role of other clinic staff and patient self-selection was minor. In a few clinics, staff gave out written information in response to established protocols (eg, to all new patients or before certain procedures). The clinic staff did not appear to have access to sufficient clinical information to tailor the patient education materials to meet specific patient needs. Frequently they were privy only to the patient’s stated chief complaint and not to the final diagnosis. In addition, many nurses were observed functioning in a very mechanical, highly structured way that did not allow them to respond to the educational needs of patients. Some nursing assistants spent all day escorting patients from the waiting room to the examination room, obtaining a set of vital signs en route. Their professional role was highly circumscribed and did not involve more complex tasks or in-depth interactions with patients.
Even though all of the clinics had education materials available for patients to help themselves, they were only rarely taken. This may have been because sometimes the racks of patient education materials were placed in areas that were difficult for patients to access.
The Information Hallway. In one practice, patient educational material was kept in 2 racks in the hallway. One of these racks was positioned at an exit that was accessible to patients, but the congestion caused by traffic made it difficult for patients to stop and browse the selection of handouts. Another wall rack full of patient education handouts was at the end of a hallway next to the restroom, which was out of the way for most patients. In this practice, patients did not seem to have the opportunity to review the education topics available and select any materials to take with them, so these racks were almost never used.
Other inconvenient places for patients to access materials were in nurse check-in areas where they rarely spent enough time to pick up a handout. However, even when materials were easily accessible, patients were seldom seen taking any handouts. Patients observed in this study also rarely accessed racks of patient education materials placed in waiting rooms or in special patient libraries.
Further Observations. Surprisingly, those physicians who used large numbers of handouts were not necessarily those who had the most organized or accessible patient education materials. Many of the practices with large but unused collections of patient education materials had them neatly arranged and indexed for easy retrieval. Several practices had computer programs that could generate patient education materials from an easily searchable list of many patient education topics. These computer programs were only rarely observed being used. In addition, although a few providers who used large amounts of patient education materials had them available in the examination room, others used handouts frequently though they had to retrieve them from a less convenient place (eg, an office down the hall). Similarly, the physicians’ interest in educating patients did not seem to be related to their use of informational handouts. Some physicians spent a great deal of time counseling and educating their patients but relied solely on verbal instruction, without the use of printed materials.
Discussion
Previously published studies have suggested that clinics should accumulate and organize large amounts of material on a great variety of topics as a means of increasing the use of patient education materials.3,30-32,37 The results of our study suggest that this strategy is associated with lower usage in actual practice. Providers that concentrated their attention on maintaining a small repertoire of patient education handouts used those materials more frequently. It appears that provider involvement and familiarity with patient education materials are key to their use in clinical practice. Clinicians were less likely to use handouts when they were selected and maintained by other clinical staff or by the larger health system. This suggests that when physicians assume a passive role in the collection of educational materials they have less awareness of the topics available, less certainty about the quality of the information, and therefore use the handouts less often. A more efficient strategy might be for each provider in a practice to choose and maintain a small number of patient education materials particularly suited to their educational style, practice profile, and the perceived informational needs of their patients.
The engagement of physicians to maintain a mportant because of the dominant role they play in distributing such materials. Physicians distributed most of the patient education materials in the study practices, with staff members playing only a minor role. Nurses and other staff members gave out materials only in response to set protocols developed for a narrow range of topics. They did not offer patient education materials targeted to individual informational needs, because they did not have access to the clinical information necessary, they lacked the clinical skills to do so, or because patient education fell outside of their defined professional role in the clinic. Handouts tailored to specific patient concerns have been shown to be more effective than generic ones,8 so this strategy of distributing materials using fixed clinical protocols may have less impact on patient behavior.
The use of libraries or displays of education materials for self-selection by patients was a strategy used to some extent by all of the clinics, but these resources were underused. Patient education materials were often not located in convenient areas. Other factors such as privacy concerns may have inhibited patients from picking up brochures; this is an area that should be explored in further research.
Because this was a qualitative study, these findings may not be generalizable to all practices; other practices outside of our study may have developed different strategies for organizing patient education materials. Another limitation of this study was that it was observational. Future intervention studies are needed to determine if physicians who adopt the personal stash approach are able to more efficiently use patient education materials. It is also necessary to explore how willing physicians would be to accept some greater responsibility for maintaining personal collections of patient education to enhance their distribution. Government agencies, private health organizations, health systems, and individual practices expend considerable resources on patient education materials. Future research should focus not simply on the development of new handouts but also on exploring ways they can be incorporated into actual clinical practice.
Acknowledgements
The study was funded by the Agency for Health Care Policy and Research Grant #5 RO1 HSO8776092 and the State of Nebraska Department of Health and Human Services LB 506–Cancer and Smoking Funds. The authors would like to thank Jason Lebsack and Diane Dodendorf for data management, Constance Gibb and Jenine Rouse for data collection, Linda Swoboda for manuscript preparation, and Helen McIlvain and Jeff Susman for manuscript review.
1. Bartlett E. At last, a definition. Patient Educ Counsel 1985;7:323-4.
2. Ackroyd E. The patient’s complaint. Br J Hosp Med 1986;36:454.-
3. Wise PH, Pietroni RG, Bhatt VB, Bond CS, Hirst S, Hooker RJ. Development and evaluation of a novel patient information system. J R Soc Med 1996;89:557-60.
4. Mullen PD, Laville EA, Biddle AK, Lorig K. Efficacy of psychoeducational interventions on pain, depression, and disability in people with arthritis: a meta analysis. J Rheumatol 1987;14:33-9.
5. Bondy LR, Sims N, Schroeder DR, Offord KP, Narr BJ. The effect of anesthestic patient education on preoperative patient anxiety. Reg Anesth Pain Med 1999;24:158-64.
6. Weinman J. Providing written information for patients: psychological considerations. J Royal Soc Med 1990;83:303-5.
7. Kitching JB. Patient information leaflets—the state of the art. J Royal Soc Med 1990;3:298-300.
8. Glascoe FP, Oberklaid F, Dworkin PH, Trimm F. Brief approaches to educating patients and parents in primary care. Pediatr 1998;101:1068.-
9. Gibbs S, Water WE, George CF. The benefits of prescription information leaflets (1). Brit J Clin Pharmacol 1989a;27:723-39.
10. Gibbs S, Water WE, George CF. The benefits of prescription information leaflets (2). Brit J Clin Pharmacol 1989b;28:345-51.
11. Gibbs S, Waters WE, George CF. Prescription information leaflets: a national survey. J R Soc Med 1990;83:292-97.
12. Fisher RC. Patient education and compliance: a pharmacist’s perspective. Patient Educ Counsel 1992;19:261-71
13. Hawe P, Higgins G. Can medication education improve the drug compliance of the elderly? Evaluation of an in hospital programme. Patient Educ Counsel 1990;16:151-60.
14. Myers ED, Calvert EJ. Information, compliance, and side effects: a study of patients on anti-depressant medication. Br J Clin Pharmacol 1984;17:221-5.
15. Ellis DA, Hopkin JM, Leitch AG, J CS. Doctor’s orders. BMJ 1979;1:456.-
16. Young FK, Brooks BR. Patient teaching manuals improve retention of treatment information: a controlled clinical trial in multiple sclerosis. Neurosci Nurs 1986;18:26-8.
17. Grady K, Lemkau J, McVay J, Reisine S. The importance of physician encouragement in breast cancer screening of older women. Prevent Med 1992;21:766-80.
18. Vignos PJ, Parker WT, Thompson HM. Evaluation of a clinic education programme for patients with RA. J Rheumatol 1976;3:155-65.
19. Cook B, Noteloviz M. An osteoporosis patient education and screening programme: Results and implications. Patient Educ Counsel 1991;17:135-45.
20. Vetto JT, Dubois PM, Vetto IP. The impact of distribution of patient-education pamphlet in a multidisciplinary breast clinic. J Cancer Educ 1996;11:148-52.
21. Sandler DA, Heaton C, Garner TG, Mitchell JR. Patients’ and general practitioners’ satisfaction with information given on discharge from hospital: audit of a new information card. BMJ 1989;299:1511-13.
22. Anderson JE, Morrell DC, Avery AJ, Watkins CJ. Evaluation of a patient education manual. BMJ 1980;281:924-5.
23. Casey R, McMahon F, McCormick MC. Fever therapy: educational intervention for parents. Pediatr 1984;73:600-03.
24. Roberts CR, Imrey PB, Turner JD. Reducing physician visits for colds through consumer education. JAMA 1983;250:1986-9.
25. Bhopal RS, Gilmour WH, Fallon CW, Bhopal JS, Hamilton I. Evaluation of a practice information leaflet. Fam Pract 1990;7:132-7.
26. Mottram DR, Reed C. Comparative evaluation of patient information leaflets by pharmacists, doctors, and the general public. J Clin Pharm Therapeutics 1997;22:127-34.
27. Arthur VAM. Written patient information: a review of the literature. J Adv Nurs 1995;21:1081-6.
28. Busson M, Dunn APM. Patient knowledge about prescribed medicines. Pharm J 1986;236:624-6.
29. Becker SH, Mcphee SJ. Health-care professionals’ use of cancer-related patient education materials: a pilot study. J Cancer Ed 1993;8:43-6.
30. Thoma GB. Evolution of a patient education program in a rural hospital. Nurs Manage 1994;25:46-8.
31. Smith JL, Levitt C, Franco ED. Innovative system to improve use of patient education materials. Can Fam physician 1997;43:58-62.
32. Collings LH, Pike LC, Binder AI, McClymont ME, Knight ST. Value of written health information in the general practice setting. Br J Gen Pract 1991;41:466-7.
33. Tronni C, Welebob E. End-user satisfaction of a patient education tool manual versus computer-generated tool. Computers Nurs 1996;14:235-8.
34. Korn R, Wieczorek RR. Computerization of standards and patient education material. J Nurs Staff Develop 1995;11:307-12.
35. Weaver J. Patient education: an innovative computer approach. Nurs Manage 1995;26:78-9, 81,83.
36. Williams RB, Boles M, Johnson RE. Patient use of a computer for prevention in primary care practice. Patient Educ Counsel 1995;25:283-92.
37. Childhood Asthma Management Program Research Group. Design and implementation of a patient education center for the Childhood Asthma Management Program. Ann Allergy Asthma Immunol 1998;81:571-81.
38. Patton MQ. Qualitative evaluation and research methods. 2nd ed. Newbury Park, Calif: Sage Publications; 1990.
39. McIlvain HE, Crabtree BF, Gilbert C, Havranek R, Backer EL. Current trends in tobacco prevention and cessation in Nebraska physicians’ offices. J Fam Pract 1997;44:193-202.
40. Yin RK. Case study research: design and methods. Applied social research methods series. 2nd ed. Newbury Park, Calif: Sage Publications; 1989;166.-
41. Bogdewic SP. Participant observation. In: Crabtree BF, Miller WL, eds. Doing qualitative research. 2nd ed. Newbury Park. Calif: Sage Publications; 1999;37-70.
42. Stange KC, Zyzanski SJ, Jaen CR, et al. Illuminating the ‘black box’ a description of 4454 patient visits to 138 family physicians. J Fam Pract 1998;46:377-89.
43. Crabtree B, Miller W. Researching practice settings: a case study approach. In: Crabtree B, Miller W, eds. Doing qualitative research. 2nd ed. Thousand Oaks, Calif: Sage Publications; 1999.
44. Pommerenke FA, Dietrich AJ. Improving and maintaining preventive services, part I: applying the patient model. J Fam Pract 1992;34:86-91.
45. Miller W, Crabtree B. Depth interviewing. In: Crabtree B, Miller W, eds. Doing qualitative research. 2nd ed. Thousand Oaks, Calif: Sage Publications; 1999;89-108.
46. McCracken G. The long interview. Newbury Park, Calif: Sage Publications; 1988.
47. Meadows L, Dodendorf D. Data management and interpretation: using computer programs to assist. In: Crabtree B, Miller W, eds. Doing qualitative research. 2nd ed. Newbury Park, Calif: Sage Publications; 1999.
48. Ray LD. Qualitative data management using Folio VIEWS. Qual Health Research 1997;7:301-8.
49. Miller W, Crabtree B. The dance of interpretation. In: Crabtree B, Miller W, eds. Doing qualitative research. 2nd ed. Thousand Oaks, Calif: Sage Publications; 1999.
50. Borkan J. Immersion/crystallization. In: Crabtree BF, Miller WL, eds. Doing qualitative research. 2nd ed. Thousand Oaks, Calif: Sage Publications; 1999;179-94.
51. Miles MB, Huberman AM. Qualitative data analysis: an expanded sourcebook. 2nd ed. Thousand Oaks, Calif: Sage Publications; 1994;337.-
52. Crabtree B, Miller W. Using codes and code manuals: a template organizing style of interpretation. In: Crabtree B, Miller W, eds. Doing qualitative research. 2nd ed. Thousand Oaks, Calif: Sage Publications; 1999;163-78.
METHODS: Eighteen purposefully selected family medicine practices were directly observed for 4 to 12 weeks each. A total of 57 providers were shadowed by a research nurse, and detailed field notes on 1600 patient encounters were recorded. A 3-member analysis team reviewed the qualitative data and identified emergent patterns.
RESULTS: Clinics’ use of patient education materials fell mostly into 2 distinct patterns. “Stockpilers” were providers who relied on the clinic staff to develop and organize a common library of patient education handouts. Providers with a “personal stash” collected much smaller numbers of materials that they personally maintained. Providers in the latter group had a known repertoire of a limited amount of educational material and used it more often than providers with access to a greater variety and number of handouts. In all practices, providers distributed most handouts; staff and self-selection by patients played a minor role.
CONCLUSIONS: It appears that provider involvement and familiarity with patient education materials are key to their use in clinical practice. Clinicians use written patient education materials most efficiently by personally selecting and maintaining a small number of handouts that address topics most relevant to their practice.
Patient education has been defined as a “learning experience using a combination of methods such as teaching, counseling and behavior modification techniques which influence patient’s knowledge and health behavior.”1 Health information provided by physicians is a type of patient education valued by patients2,3 and that plays a role in the complex issue of improving overall health status and psychosocial functioning.4,5 Various educational formats are used by physicians including verbal instruction, counseling, referral to education specialists, audio or videotapes, and written education material. Printed handouts, however, have several theoretical advantages over other methods of patient education:6 Patients may have difficulty remembering instructions that are presented only verbally,3,7,8 and written handouts can supplement or reinforce information presented during the office visit. Handouts are also proposed to be a cost-effective means for the busy clinician to educate patients without having to convey all the information personally.
Several studies have confirmed the theoretical utility of written patient education materials. Patients both read and retain written materials given to them by health professionals.9-11 This written information has been shown to increase compliance with medication use,12-14 physicians’ instructions,15,16 smoking cessation,4,8 and the use of preventive health services.8,17 Information handouts have also been shown to increase patient knowledge,18-20 to increase satisfaction with patient care,9,10,21 and to reduce patient anxiety.5,8 Written information has even been shown to reduce unnecessary medical visits22-24 and inappropriate telephone calls.25 For these reasons, both providers and patients report they value written materials.26-28
Large amounts of patient education materials are available, covering topics ranging from prevention and safety issues to strategies for coping with chronic diseases. The American Cancer Society, for example, spent $56 million on producing patient education materials in 1989.29 In spite of the amount of material available, little information has been published on how handouts are used in actual clinical practice.
Few studies in the medical literature describe strategies clinicians can use to organize and disseminate written materials in the office setting. Most authors recommend compiling a comprehensive set of written handouts covering a broad array of clinical topics and indexing these materials to facilitate retrieval by medical personnel.3,30-32 Some of these authors have also described the use of computers to both index and generate written materials.3,30,33-35 The various strategies described for disseminating written materials to patients have included using the clinician, nursing staff or receptionists,36 patient libraries,32,37 and racks in waiting rooms.3,8 None of these strategies have been adequately evaluated in terms of their effectiveness in enhancing the use of written patient education materials, and no studies have described what office systems exist in actual practices to support their use.
This paper describes how patient education materials are organized and disseminated in family practices, and the office system factors that contribute to their use in real world settings.
Methods
The “Prevention and Competing Demands in Primary Care Practice” study was designed to examine the organizational contexts that support or inhibit the delivery of preventive services in family medicine practices. From November 1996 to February 1999, extensive descriptive field notes were recorded throughout this large comparative case study of family practice organizations. Eighteen practices were studied using a multimethod ethnographic design that involved observation of clinical encounters and the office system by a research nurse who spent 4 weeks or more in each practice. A total of 44 physicians and 13 other primary care providers were shadowed and interviewed, and approximately 1600 patient encounters were directly observed. Data collection generated approximately 20,000 pages of text materials. Descriptions of how patient education materials were organized and used in practices were drawn from this larger data set that looked more broadly at preventive services delivery.
Sampling
A total of 18 family practices were purposefully selected38 from across Nebraska. Based on results of a previous study,39 practices known to deliver both high and low levels of tobacco prevention services were included. The sites were chosen to include a wide variety of practice types, with maximum variation with respect to rural/urban, small/large, and privately owned/part of a larger health system. After preliminary analyses of the initial 10 practices, 8 additional practices were selected to search for confirming or challenging cases using replication logic.40Table 1 summarizes the characteristics of the practices and physicians studied.
Study participation was solicited by sending an invitation letter, followed by a phone call to one of the physicians within the practice. Later, the consent of all clinicians to conduct research in the practice was obtained. Only 5 practices declined to participate. Three individual physicians subsequently declined participation after their practice was enrolled in the study (2 family physicians who were on the verge of retirement and one gynecologist who saw patients part-time in one family practice). Data collection proceeded for the other clinicians enrolled in the study for those sites.
Data Collection
A research nurse trained in qualitative methods was sent to each practice where she used a variety of data collection methods to produce a comprehensive picture of the practice as a functioning organization. It took 4 to 12 weeks for the nurse to complete the data collection in each practice depending on its size.
Field Notes. The research nurse observed the physical environment and functioning of the practice and dictated extensive field notes at the end of each day.41 These notes contained detailed descriptions of the clinic location and environment, patient characteristics, nursing station, examination rooms, the waiting area, bulletin boards, posters, and patient education materials. Photographs were taken of each room. The nurse specifically noted the location and organization of patient education materials and their accessibility to both providers and patients. She inventoried the available patient education materials, noting the number of brochures available, the topics covered, and who produced them. Samples of each patient education handout were obtained whenever possible. She also noted who was responsible for maintaining and organizing patient education supplies.
Checklist of Office Environment. Structured checklists of the office environment facilitated quantification of specific areas of interest and served as a template for standardized field note descriptions of the practice.42 Items on this 5-page instrument included the number of patients scheduled and seen per day, the number of personnel in the office, and the percentage of patients covered by managed care plans. The accessibility, quality, and patient use of education materials were also specifically recorded.
Patient Encounters. Approximately 30 patient encounters for each provider were observed. After obtaining written informed consent from the patient, the research nurse shadowed the provider and took notes for later dictation. The patient encounter field notes contained descriptions of any verbal patient teaching and the context of that education, including the reason for the visit, how the visit unfolded, and how the provider and patient interacted. The patient encounter structured checklist captured the number of times patient education materials were used.43 Thus it was possible to quantify how frequently patient education handouts were used during the observed patient visits, as well as to describe the context of their use.
Patient Pathways. The research nurse also followed 2 of each clinician’s patients from the time they entered the practice until they left. These patient pathways provided a minute by minute recording of events from the time of entry into the health center, encounter with the receptionist, nurse, physician, checkout, and until the patient left.44 To collect this data, the research nurse followed patients during their clinic visit, noting the places they visited, how long they waited and what happened to them during each stage. Opportunities for patients to select patient education materials intended for self-service were noted.
Interviews. Depth interviews with each provider in the practice explored themes related to the delivery of preventive services.45,46 Patient education materials were sometimes discussed during the interviews, but specific questions about their use were not included on the interview guide.
Data Analysis and Interpretation
All quantitative and qualitative data were checked for accuracy and entered into Folioviews, an infobase software package.47,48 This software program facilitates the organization of text documents and allows computerized searches and coding of the qualitative database.
The first phase of data analysis was an immersion/crystallization process49,50 that lead to the development of a code book—an organizational scheme for understanding the qualitative data—that could be applied to the entire data set. Initially, one of the authors (MV) immersed herself in all the data from 5 purposefully selected practices to understand the functioning, organization, and dynamics of the practice. She read the field notes on the computer and made written notes on each practice, then reviewed patient encounter checklists to see how often handouts were used by each provider. Using this approach, she worked with the other authors to crystallize hypotheses and form an initial organizational scheme. Group discussions among all of the authors led to the development of our code book.51
A strategy for sampling the data from other practices was used to identify relevant portions of the larger data set for secondary data analysis. A sample of at least 10 patient encounters with each provider from the 18 practices was read, noting the content and type of patient education. We found that only 10 encounters were needed to reach saturation of our understanding of a provider’s educational style. In addition, all encounters in which a patient education handout was given were reviewed, for a total of 500 patient encounters. Computer searches using key words helped find places where patient education was discussed in the field notes and patient encounters. Photographs of all the practices were reviewed to provide a clear mental picture of the location of the patient education materials in different areas of the clinic.
During the second phase of analysis, the code book was applied to the relevant portions of the database using a template organizing style.52 Segments of the field notes and patient encounter notes that discussed aspects of patient education were identified and organized, and eventually used to construct matrices or tables.51 These matrices allowed visualization of emergent patterns and facilitated comparisons across cases.
Results
Use of patient education materials and the strategies to organize and distribute the handouts varied among providers. Some practices had acquired large numbers of patient education materials while others focused on a small number of handouts. Similarly, staff involvement in acquiring and organizing patient education materials varied from practice to practice. Two distinct patterns of organizational style emerged from the data related to these themes that had implications for the use of the materials Table 2 summarizes this data from each of the practices.
Stockpile Organization
Most of the practices in the study had accumulated large amounts of patient education materials, and many had more than 150 different handouts available. These were eclectic collections of materials that consisted of the pooled contributions of providers, nurses, office staff, and pharmaceutical representatives. Once assembled, these collections were intended for communal use by all providers in the clinic. In some instances, larger health systems had provided substantial collections of printed handouts to each of their clinics. Some practices had the ability to access and reproduce computer-generated patient education materials, further expanding the range of topics available. In these large collections there was often a great deal of redundancy. One 3-physician practice, for example, had 5 different brochures on childhood lead screening. Responsibility for maintaining and organizing these large collections of patient education materials was a time-consuming task that was usually delegated to a staff member. As a result, most physicians were unfamiliar with the handouts available in their practice. In those practices with extensive shared collections of patient education materials, very few handouts were actually used by the providers. The first case provides an example of this type of practice.
The Stockpile. This was a large physician group that recognized patient education as a priority for the practice. They had tried a variety of strategies to enhance their use of written materials, including the hiring of a health education coordinator. She was assigned the task of organizing and maintaining all of their patient handouts.
In this role, the health education coordinator had accumulated a huge number of handouts on a broad range of topics (45 different handouts on well-child care alone). The materials came from pharmaceutical companies, the clinic physicians, professional organizations, and the coordinator’s personal files, and more continued to filter in as the physicians brought in new materials. These education materials were stored at the nurses’ station in neatly color-coded file drawers. Other nursing staff distributed materials in response to physician requests.
Despite the tremendous amount of handouts available, their organization and quality, the staff involvement, and the physicians’ own commitment to using them, written patient education materials were rarely distributed (14/272 or 5% of observed visits).
This was a model clinic in the sense that it had the most staff resources dedicated to maintaining the stockpile of handouts and one of the most organized collections. Many other clinics used the same overall strategy, with similar infrequency of use.
Providers stated one of the reasons they did not use the materials organized in the stockpile was that they were unsure of the quality and accuracy of the information presented. One physician said, “I don’t always necessarily agree 100% with what’s in there, and I feel like if I’m going to hand it out, it’s something I should have read myself.” Another problem with the stockpile approach was that the providers did not know what was available. “There is so much there that it’s almost overwhelming,” one physician stated. Having too many handouts may have made it difficult for providers to familiarize themselves with the materials or to locate the ones they wanted to use.
Personal Stash Organization
Another set of practices adopted a different strategy of using patient education materials. In these practices, the physicians themselves took responsibility for the patient education handouts. The individual physicians each selected, organized, and maintained a small private collection of materials. Even within group practices, physicians using this organizational scheme would maintain separate collections of patient handouts that fit their unique instructional needs. In these practices, patient education materials were used more frequently.
The Personal Stash. One physician and a recently hired nurse practitioner worked in a clinic with a diverse patient population ranging from trendy young professionals to homeless families. The physician consistently incorporated verbal patient education into almost every clinical encounter. In counseling patients, he demonstrated remarkable versatility in acknowledging patients’ unique social and familial circumstances, as well as their readiness to adopt healthier behaviors.
Very few (28) patient handouts were available in the practice. The physician had developed many of these handouts himself by photocopying articles from popular magazines and books. He was responsible for organizing and maintaining his supply of handouts, and he personally retrieved them from his office filing cabinet to distribute to patients. This physician used patient education handouts quite frequently (in 13 of 52 observed visits—25%).
Providers using the personal stash approach were observed using handouts more frequently than those using shared stockpiles. Where providers drew from a small but known repertoire of patient education materials they used them more often than those in practices with a large number of handouts.
Hybrid Approach
Exceptions to this overall pattern were 4 cliniciansin 2 separate group practices that used the stockpile approach. These clinics had large numbers of patient education handouts maintained by staff members for communal use. In contrast to their partners, these 4 clinicians used handouts extensively (30/122 observed visits). These providers had a personal interest and involvement in the practice’s library of patient education materials. They each maintained a portfolio of selected handouts kept in their own examination rooms or offices, and they had each personally developed some materials. These 4 clinicians had forged a hybrid approach of maintaining hands-on involvement within a system that delegated responsibility to staff. In this way, they were able to maintain familiarity with the materials and use them frequently.
Role of Staff and Patient Self-Selection
In all of the practices, clinicians distributed the majority of patient education materials, and the role of other clinic staff and patient self-selection was minor. In a few clinics, staff gave out written information in response to established protocols (eg, to all new patients or before certain procedures). The clinic staff did not appear to have access to sufficient clinical information to tailor the patient education materials to meet specific patient needs. Frequently they were privy only to the patient’s stated chief complaint and not to the final diagnosis. In addition, many nurses were observed functioning in a very mechanical, highly structured way that did not allow them to respond to the educational needs of patients. Some nursing assistants spent all day escorting patients from the waiting room to the examination room, obtaining a set of vital signs en route. Their professional role was highly circumscribed and did not involve more complex tasks or in-depth interactions with patients.
Even though all of the clinics had education materials available for patients to help themselves, they were only rarely taken. This may have been because sometimes the racks of patient education materials were placed in areas that were difficult for patients to access.
The Information Hallway. In one practice, patient educational material was kept in 2 racks in the hallway. One of these racks was positioned at an exit that was accessible to patients, but the congestion caused by traffic made it difficult for patients to stop and browse the selection of handouts. Another wall rack full of patient education handouts was at the end of a hallway next to the restroom, which was out of the way for most patients. In this practice, patients did not seem to have the opportunity to review the education topics available and select any materials to take with them, so these racks were almost never used.
Other inconvenient places for patients to access materials were in nurse check-in areas where they rarely spent enough time to pick up a handout. However, even when materials were easily accessible, patients were seldom seen taking any handouts. Patients observed in this study also rarely accessed racks of patient education materials placed in waiting rooms or in special patient libraries.
Further Observations. Surprisingly, those physicians who used large numbers of handouts were not necessarily those who had the most organized or accessible patient education materials. Many of the practices with large but unused collections of patient education materials had them neatly arranged and indexed for easy retrieval. Several practices had computer programs that could generate patient education materials from an easily searchable list of many patient education topics. These computer programs were only rarely observed being used. In addition, although a few providers who used large amounts of patient education materials had them available in the examination room, others used handouts frequently though they had to retrieve them from a less convenient place (eg, an office down the hall). Similarly, the physicians’ interest in educating patients did not seem to be related to their use of informational handouts. Some physicians spent a great deal of time counseling and educating their patients but relied solely on verbal instruction, without the use of printed materials.
Discussion
Previously published studies have suggested that clinics should accumulate and organize large amounts of material on a great variety of topics as a means of increasing the use of patient education materials.3,30-32,37 The results of our study suggest that this strategy is associated with lower usage in actual practice. Providers that concentrated their attention on maintaining a small repertoire of patient education handouts used those materials more frequently. It appears that provider involvement and familiarity with patient education materials are key to their use in clinical practice. Clinicians were less likely to use handouts when they were selected and maintained by other clinical staff or by the larger health system. This suggests that when physicians assume a passive role in the collection of educational materials they have less awareness of the topics available, less certainty about the quality of the information, and therefore use the handouts less often. A more efficient strategy might be for each provider in a practice to choose and maintain a small number of patient education materials particularly suited to their educational style, practice profile, and the perceived informational needs of their patients.
The engagement of physicians to maintain a mportant because of the dominant role they play in distributing such materials. Physicians distributed most of the patient education materials in the study practices, with staff members playing only a minor role. Nurses and other staff members gave out materials only in response to set protocols developed for a narrow range of topics. They did not offer patient education materials targeted to individual informational needs, because they did not have access to the clinical information necessary, they lacked the clinical skills to do so, or because patient education fell outside of their defined professional role in the clinic. Handouts tailored to specific patient concerns have been shown to be more effective than generic ones,8 so this strategy of distributing materials using fixed clinical protocols may have less impact on patient behavior.
The use of libraries or displays of education materials for self-selection by patients was a strategy used to some extent by all of the clinics, but these resources were underused. Patient education materials were often not located in convenient areas. Other factors such as privacy concerns may have inhibited patients from picking up brochures; this is an area that should be explored in further research.
Because this was a qualitative study, these findings may not be generalizable to all practices; other practices outside of our study may have developed different strategies for organizing patient education materials. Another limitation of this study was that it was observational. Future intervention studies are needed to determine if physicians who adopt the personal stash approach are able to more efficiently use patient education materials. It is also necessary to explore how willing physicians would be to accept some greater responsibility for maintaining personal collections of patient education to enhance their distribution. Government agencies, private health organizations, health systems, and individual practices expend considerable resources on patient education materials. Future research should focus not simply on the development of new handouts but also on exploring ways they can be incorporated into actual clinical practice.
Acknowledgements
The study was funded by the Agency for Health Care Policy and Research Grant #5 RO1 HSO8776092 and the State of Nebraska Department of Health and Human Services LB 506–Cancer and Smoking Funds. The authors would like to thank Jason Lebsack and Diane Dodendorf for data management, Constance Gibb and Jenine Rouse for data collection, Linda Swoboda for manuscript preparation, and Helen McIlvain and Jeff Susman for manuscript review.
METHODS: Eighteen purposefully selected family medicine practices were directly observed for 4 to 12 weeks each. A total of 57 providers were shadowed by a research nurse, and detailed field notes on 1600 patient encounters were recorded. A 3-member analysis team reviewed the qualitative data and identified emergent patterns.
RESULTS: Clinics’ use of patient education materials fell mostly into 2 distinct patterns. “Stockpilers” were providers who relied on the clinic staff to develop and organize a common library of patient education handouts. Providers with a “personal stash” collected much smaller numbers of materials that they personally maintained. Providers in the latter group had a known repertoire of a limited amount of educational material and used it more often than providers with access to a greater variety and number of handouts. In all practices, providers distributed most handouts; staff and self-selection by patients played a minor role.
CONCLUSIONS: It appears that provider involvement and familiarity with patient education materials are key to their use in clinical practice. Clinicians use written patient education materials most efficiently by personally selecting and maintaining a small number of handouts that address topics most relevant to their practice.
Patient education has been defined as a “learning experience using a combination of methods such as teaching, counseling and behavior modification techniques which influence patient’s knowledge and health behavior.”1 Health information provided by physicians is a type of patient education valued by patients2,3 and that plays a role in the complex issue of improving overall health status and psychosocial functioning.4,5 Various educational formats are used by physicians including verbal instruction, counseling, referral to education specialists, audio or videotapes, and written education material. Printed handouts, however, have several theoretical advantages over other methods of patient education:6 Patients may have difficulty remembering instructions that are presented only verbally,3,7,8 and written handouts can supplement or reinforce information presented during the office visit. Handouts are also proposed to be a cost-effective means for the busy clinician to educate patients without having to convey all the information personally.
Several studies have confirmed the theoretical utility of written patient education materials. Patients both read and retain written materials given to them by health professionals.9-11 This written information has been shown to increase compliance with medication use,12-14 physicians’ instructions,15,16 smoking cessation,4,8 and the use of preventive health services.8,17 Information handouts have also been shown to increase patient knowledge,18-20 to increase satisfaction with patient care,9,10,21 and to reduce patient anxiety.5,8 Written information has even been shown to reduce unnecessary medical visits22-24 and inappropriate telephone calls.25 For these reasons, both providers and patients report they value written materials.26-28
Large amounts of patient education materials are available, covering topics ranging from prevention and safety issues to strategies for coping with chronic diseases. The American Cancer Society, for example, spent $56 million on producing patient education materials in 1989.29 In spite of the amount of material available, little information has been published on how handouts are used in actual clinical practice.
Few studies in the medical literature describe strategies clinicians can use to organize and disseminate written materials in the office setting. Most authors recommend compiling a comprehensive set of written handouts covering a broad array of clinical topics and indexing these materials to facilitate retrieval by medical personnel.3,30-32 Some of these authors have also described the use of computers to both index and generate written materials.3,30,33-35 The various strategies described for disseminating written materials to patients have included using the clinician, nursing staff or receptionists,36 patient libraries,32,37 and racks in waiting rooms.3,8 None of these strategies have been adequately evaluated in terms of their effectiveness in enhancing the use of written patient education materials, and no studies have described what office systems exist in actual practices to support their use.
This paper describes how patient education materials are organized and disseminated in family practices, and the office system factors that contribute to their use in real world settings.
Methods
The “Prevention and Competing Demands in Primary Care Practice” study was designed to examine the organizational contexts that support or inhibit the delivery of preventive services in family medicine practices. From November 1996 to February 1999, extensive descriptive field notes were recorded throughout this large comparative case study of family practice organizations. Eighteen practices were studied using a multimethod ethnographic design that involved observation of clinical encounters and the office system by a research nurse who spent 4 weeks or more in each practice. A total of 44 physicians and 13 other primary care providers were shadowed and interviewed, and approximately 1600 patient encounters were directly observed. Data collection generated approximately 20,000 pages of text materials. Descriptions of how patient education materials were organized and used in practices were drawn from this larger data set that looked more broadly at preventive services delivery.
Sampling
A total of 18 family practices were purposefully selected38 from across Nebraska. Based on results of a previous study,39 practices known to deliver both high and low levels of tobacco prevention services were included. The sites were chosen to include a wide variety of practice types, with maximum variation with respect to rural/urban, small/large, and privately owned/part of a larger health system. After preliminary analyses of the initial 10 practices, 8 additional practices were selected to search for confirming or challenging cases using replication logic.40Table 1 summarizes the characteristics of the practices and physicians studied.
Study participation was solicited by sending an invitation letter, followed by a phone call to one of the physicians within the practice. Later, the consent of all clinicians to conduct research in the practice was obtained. Only 5 practices declined to participate. Three individual physicians subsequently declined participation after their practice was enrolled in the study (2 family physicians who were on the verge of retirement and one gynecologist who saw patients part-time in one family practice). Data collection proceeded for the other clinicians enrolled in the study for those sites.
Data Collection
A research nurse trained in qualitative methods was sent to each practice where she used a variety of data collection methods to produce a comprehensive picture of the practice as a functioning organization. It took 4 to 12 weeks for the nurse to complete the data collection in each practice depending on its size.
Field Notes. The research nurse observed the physical environment and functioning of the practice and dictated extensive field notes at the end of each day.41 These notes contained detailed descriptions of the clinic location and environment, patient characteristics, nursing station, examination rooms, the waiting area, bulletin boards, posters, and patient education materials. Photographs were taken of each room. The nurse specifically noted the location and organization of patient education materials and their accessibility to both providers and patients. She inventoried the available patient education materials, noting the number of brochures available, the topics covered, and who produced them. Samples of each patient education handout were obtained whenever possible. She also noted who was responsible for maintaining and organizing patient education supplies.
Checklist of Office Environment. Structured checklists of the office environment facilitated quantification of specific areas of interest and served as a template for standardized field note descriptions of the practice.42 Items on this 5-page instrument included the number of patients scheduled and seen per day, the number of personnel in the office, and the percentage of patients covered by managed care plans. The accessibility, quality, and patient use of education materials were also specifically recorded.
Patient Encounters. Approximately 30 patient encounters for each provider were observed. After obtaining written informed consent from the patient, the research nurse shadowed the provider and took notes for later dictation. The patient encounter field notes contained descriptions of any verbal patient teaching and the context of that education, including the reason for the visit, how the visit unfolded, and how the provider and patient interacted. The patient encounter structured checklist captured the number of times patient education materials were used.43 Thus it was possible to quantify how frequently patient education handouts were used during the observed patient visits, as well as to describe the context of their use.
Patient Pathways. The research nurse also followed 2 of each clinician’s patients from the time they entered the practice until they left. These patient pathways provided a minute by minute recording of events from the time of entry into the health center, encounter with the receptionist, nurse, physician, checkout, and until the patient left.44 To collect this data, the research nurse followed patients during their clinic visit, noting the places they visited, how long they waited and what happened to them during each stage. Opportunities for patients to select patient education materials intended for self-service were noted.
Interviews. Depth interviews with each provider in the practice explored themes related to the delivery of preventive services.45,46 Patient education materials were sometimes discussed during the interviews, but specific questions about their use were not included on the interview guide.
Data Analysis and Interpretation
All quantitative and qualitative data were checked for accuracy and entered into Folioviews, an infobase software package.47,48 This software program facilitates the organization of text documents and allows computerized searches and coding of the qualitative database.
The first phase of data analysis was an immersion/crystallization process49,50 that lead to the development of a code book—an organizational scheme for understanding the qualitative data—that could be applied to the entire data set. Initially, one of the authors (MV) immersed herself in all the data from 5 purposefully selected practices to understand the functioning, organization, and dynamics of the practice. She read the field notes on the computer and made written notes on each practice, then reviewed patient encounter checklists to see how often handouts were used by each provider. Using this approach, she worked with the other authors to crystallize hypotheses and form an initial organizational scheme. Group discussions among all of the authors led to the development of our code book.51
A strategy for sampling the data from other practices was used to identify relevant portions of the larger data set for secondary data analysis. A sample of at least 10 patient encounters with each provider from the 18 practices was read, noting the content and type of patient education. We found that only 10 encounters were needed to reach saturation of our understanding of a provider’s educational style. In addition, all encounters in which a patient education handout was given were reviewed, for a total of 500 patient encounters. Computer searches using key words helped find places where patient education was discussed in the field notes and patient encounters. Photographs of all the practices were reviewed to provide a clear mental picture of the location of the patient education materials in different areas of the clinic.
During the second phase of analysis, the code book was applied to the relevant portions of the database using a template organizing style.52 Segments of the field notes and patient encounter notes that discussed aspects of patient education were identified and organized, and eventually used to construct matrices or tables.51 These matrices allowed visualization of emergent patterns and facilitated comparisons across cases.
Results
Use of patient education materials and the strategies to organize and distribute the handouts varied among providers. Some practices had acquired large numbers of patient education materials while others focused on a small number of handouts. Similarly, staff involvement in acquiring and organizing patient education materials varied from practice to practice. Two distinct patterns of organizational style emerged from the data related to these themes that had implications for the use of the materials Table 2 summarizes this data from each of the practices.
Stockpile Organization
Most of the practices in the study had accumulated large amounts of patient education materials, and many had more than 150 different handouts available. These were eclectic collections of materials that consisted of the pooled contributions of providers, nurses, office staff, and pharmaceutical representatives. Once assembled, these collections were intended for communal use by all providers in the clinic. In some instances, larger health systems had provided substantial collections of printed handouts to each of their clinics. Some practices had the ability to access and reproduce computer-generated patient education materials, further expanding the range of topics available. In these large collections there was often a great deal of redundancy. One 3-physician practice, for example, had 5 different brochures on childhood lead screening. Responsibility for maintaining and organizing these large collections of patient education materials was a time-consuming task that was usually delegated to a staff member. As a result, most physicians were unfamiliar with the handouts available in their practice. In those practices with extensive shared collections of patient education materials, very few handouts were actually used by the providers. The first case provides an example of this type of practice.
The Stockpile. This was a large physician group that recognized patient education as a priority for the practice. They had tried a variety of strategies to enhance their use of written materials, including the hiring of a health education coordinator. She was assigned the task of organizing and maintaining all of their patient handouts.
In this role, the health education coordinator had accumulated a huge number of handouts on a broad range of topics (45 different handouts on well-child care alone). The materials came from pharmaceutical companies, the clinic physicians, professional organizations, and the coordinator’s personal files, and more continued to filter in as the physicians brought in new materials. These education materials were stored at the nurses’ station in neatly color-coded file drawers. Other nursing staff distributed materials in response to physician requests.
Despite the tremendous amount of handouts available, their organization and quality, the staff involvement, and the physicians’ own commitment to using them, written patient education materials were rarely distributed (14/272 or 5% of observed visits).
This was a model clinic in the sense that it had the most staff resources dedicated to maintaining the stockpile of handouts and one of the most organized collections. Many other clinics used the same overall strategy, with similar infrequency of use.
Providers stated one of the reasons they did not use the materials organized in the stockpile was that they were unsure of the quality and accuracy of the information presented. One physician said, “I don’t always necessarily agree 100% with what’s in there, and I feel like if I’m going to hand it out, it’s something I should have read myself.” Another problem with the stockpile approach was that the providers did not know what was available. “There is so much there that it’s almost overwhelming,” one physician stated. Having too many handouts may have made it difficult for providers to familiarize themselves with the materials or to locate the ones they wanted to use.
Personal Stash Organization
Another set of practices adopted a different strategy of using patient education materials. In these practices, the physicians themselves took responsibility for the patient education handouts. The individual physicians each selected, organized, and maintained a small private collection of materials. Even within group practices, physicians using this organizational scheme would maintain separate collections of patient handouts that fit their unique instructional needs. In these practices, patient education materials were used more frequently.
The Personal Stash. One physician and a recently hired nurse practitioner worked in a clinic with a diverse patient population ranging from trendy young professionals to homeless families. The physician consistently incorporated verbal patient education into almost every clinical encounter. In counseling patients, he demonstrated remarkable versatility in acknowledging patients’ unique social and familial circumstances, as well as their readiness to adopt healthier behaviors.
Very few (28) patient handouts were available in the practice. The physician had developed many of these handouts himself by photocopying articles from popular magazines and books. He was responsible for organizing and maintaining his supply of handouts, and he personally retrieved them from his office filing cabinet to distribute to patients. This physician used patient education handouts quite frequently (in 13 of 52 observed visits—25%).
Providers using the personal stash approach were observed using handouts more frequently than those using shared stockpiles. Where providers drew from a small but known repertoire of patient education materials they used them more often than those in practices with a large number of handouts.
Hybrid Approach
Exceptions to this overall pattern were 4 cliniciansin 2 separate group practices that used the stockpile approach. These clinics had large numbers of patient education handouts maintained by staff members for communal use. In contrast to their partners, these 4 clinicians used handouts extensively (30/122 observed visits). These providers had a personal interest and involvement in the practice’s library of patient education materials. They each maintained a portfolio of selected handouts kept in their own examination rooms or offices, and they had each personally developed some materials. These 4 clinicians had forged a hybrid approach of maintaining hands-on involvement within a system that delegated responsibility to staff. In this way, they were able to maintain familiarity with the materials and use them frequently.
Role of Staff and Patient Self-Selection
In all of the practices, clinicians distributed the majority of patient education materials, and the role of other clinic staff and patient self-selection was minor. In a few clinics, staff gave out written information in response to established protocols (eg, to all new patients or before certain procedures). The clinic staff did not appear to have access to sufficient clinical information to tailor the patient education materials to meet specific patient needs. Frequently they were privy only to the patient’s stated chief complaint and not to the final diagnosis. In addition, many nurses were observed functioning in a very mechanical, highly structured way that did not allow them to respond to the educational needs of patients. Some nursing assistants spent all day escorting patients from the waiting room to the examination room, obtaining a set of vital signs en route. Their professional role was highly circumscribed and did not involve more complex tasks or in-depth interactions with patients.
Even though all of the clinics had education materials available for patients to help themselves, they were only rarely taken. This may have been because sometimes the racks of patient education materials were placed in areas that were difficult for patients to access.
The Information Hallway. In one practice, patient educational material was kept in 2 racks in the hallway. One of these racks was positioned at an exit that was accessible to patients, but the congestion caused by traffic made it difficult for patients to stop and browse the selection of handouts. Another wall rack full of patient education handouts was at the end of a hallway next to the restroom, which was out of the way for most patients. In this practice, patients did not seem to have the opportunity to review the education topics available and select any materials to take with them, so these racks were almost never used.
Other inconvenient places for patients to access materials were in nurse check-in areas where they rarely spent enough time to pick up a handout. However, even when materials were easily accessible, patients were seldom seen taking any handouts. Patients observed in this study also rarely accessed racks of patient education materials placed in waiting rooms or in special patient libraries.
Further Observations. Surprisingly, those physicians who used large numbers of handouts were not necessarily those who had the most organized or accessible patient education materials. Many of the practices with large but unused collections of patient education materials had them neatly arranged and indexed for easy retrieval. Several practices had computer programs that could generate patient education materials from an easily searchable list of many patient education topics. These computer programs were only rarely observed being used. In addition, although a few providers who used large amounts of patient education materials had them available in the examination room, others used handouts frequently though they had to retrieve them from a less convenient place (eg, an office down the hall). Similarly, the physicians’ interest in educating patients did not seem to be related to their use of informational handouts. Some physicians spent a great deal of time counseling and educating their patients but relied solely on verbal instruction, without the use of printed materials.
Discussion
Previously published studies have suggested that clinics should accumulate and organize large amounts of material on a great variety of topics as a means of increasing the use of patient education materials.3,30-32,37 The results of our study suggest that this strategy is associated with lower usage in actual practice. Providers that concentrated their attention on maintaining a small repertoire of patient education handouts used those materials more frequently. It appears that provider involvement and familiarity with patient education materials are key to their use in clinical practice. Clinicians were less likely to use handouts when they were selected and maintained by other clinical staff or by the larger health system. This suggests that when physicians assume a passive role in the collection of educational materials they have less awareness of the topics available, less certainty about the quality of the information, and therefore use the handouts less often. A more efficient strategy might be for each provider in a practice to choose and maintain a small number of patient education materials particularly suited to their educational style, practice profile, and the perceived informational needs of their patients.
The engagement of physicians to maintain a mportant because of the dominant role they play in distributing such materials. Physicians distributed most of the patient education materials in the study practices, with staff members playing only a minor role. Nurses and other staff members gave out materials only in response to set protocols developed for a narrow range of topics. They did not offer patient education materials targeted to individual informational needs, because they did not have access to the clinical information necessary, they lacked the clinical skills to do so, or because patient education fell outside of their defined professional role in the clinic. Handouts tailored to specific patient concerns have been shown to be more effective than generic ones,8 so this strategy of distributing materials using fixed clinical protocols may have less impact on patient behavior.
The use of libraries or displays of education materials for self-selection by patients was a strategy used to some extent by all of the clinics, but these resources were underused. Patient education materials were often not located in convenient areas. Other factors such as privacy concerns may have inhibited patients from picking up brochures; this is an area that should be explored in further research.
Because this was a qualitative study, these findings may not be generalizable to all practices; other practices outside of our study may have developed different strategies for organizing patient education materials. Another limitation of this study was that it was observational. Future intervention studies are needed to determine if physicians who adopt the personal stash approach are able to more efficiently use patient education materials. It is also necessary to explore how willing physicians would be to accept some greater responsibility for maintaining personal collections of patient education to enhance their distribution. Government agencies, private health organizations, health systems, and individual practices expend considerable resources on patient education materials. Future research should focus not simply on the development of new handouts but also on exploring ways they can be incorporated into actual clinical practice.
Acknowledgements
The study was funded by the Agency for Health Care Policy and Research Grant #5 RO1 HSO8776092 and the State of Nebraska Department of Health and Human Services LB 506–Cancer and Smoking Funds. The authors would like to thank Jason Lebsack and Diane Dodendorf for data management, Constance Gibb and Jenine Rouse for data collection, Linda Swoboda for manuscript preparation, and Helen McIlvain and Jeff Susman for manuscript review.
1. Bartlett E. At last, a definition. Patient Educ Counsel 1985;7:323-4.
2. Ackroyd E. The patient’s complaint. Br J Hosp Med 1986;36:454.-
3. Wise PH, Pietroni RG, Bhatt VB, Bond CS, Hirst S, Hooker RJ. Development and evaluation of a novel patient information system. J R Soc Med 1996;89:557-60.
4. Mullen PD, Laville EA, Biddle AK, Lorig K. Efficacy of psychoeducational interventions on pain, depression, and disability in people with arthritis: a meta analysis. J Rheumatol 1987;14:33-9.
5. Bondy LR, Sims N, Schroeder DR, Offord KP, Narr BJ. The effect of anesthestic patient education on preoperative patient anxiety. Reg Anesth Pain Med 1999;24:158-64.
6. Weinman J. Providing written information for patients: psychological considerations. J Royal Soc Med 1990;83:303-5.
7. Kitching JB. Patient information leaflets—the state of the art. J Royal Soc Med 1990;3:298-300.
8. Glascoe FP, Oberklaid F, Dworkin PH, Trimm F. Brief approaches to educating patients and parents in primary care. Pediatr 1998;101:1068.-
9. Gibbs S, Water WE, George CF. The benefits of prescription information leaflets (1). Brit J Clin Pharmacol 1989a;27:723-39.
10. Gibbs S, Water WE, George CF. The benefits of prescription information leaflets (2). Brit J Clin Pharmacol 1989b;28:345-51.
11. Gibbs S, Waters WE, George CF. Prescription information leaflets: a national survey. J R Soc Med 1990;83:292-97.
12. Fisher RC. Patient education and compliance: a pharmacist’s perspective. Patient Educ Counsel 1992;19:261-71
13. Hawe P, Higgins G. Can medication education improve the drug compliance of the elderly? Evaluation of an in hospital programme. Patient Educ Counsel 1990;16:151-60.
14. Myers ED, Calvert EJ. Information, compliance, and side effects: a study of patients on anti-depressant medication. Br J Clin Pharmacol 1984;17:221-5.
15. Ellis DA, Hopkin JM, Leitch AG, J CS. Doctor’s orders. BMJ 1979;1:456.-
16. Young FK, Brooks BR. Patient teaching manuals improve retention of treatment information: a controlled clinical trial in multiple sclerosis. Neurosci Nurs 1986;18:26-8.
17. Grady K, Lemkau J, McVay J, Reisine S. The importance of physician encouragement in breast cancer screening of older women. Prevent Med 1992;21:766-80.
18. Vignos PJ, Parker WT, Thompson HM. Evaluation of a clinic education programme for patients with RA. J Rheumatol 1976;3:155-65.
19. Cook B, Noteloviz M. An osteoporosis patient education and screening programme: Results and implications. Patient Educ Counsel 1991;17:135-45.
20. Vetto JT, Dubois PM, Vetto IP. The impact of distribution of patient-education pamphlet in a multidisciplinary breast clinic. J Cancer Educ 1996;11:148-52.
21. Sandler DA, Heaton C, Garner TG, Mitchell JR. Patients’ and general practitioners’ satisfaction with information given on discharge from hospital: audit of a new information card. BMJ 1989;299:1511-13.
22. Anderson JE, Morrell DC, Avery AJ, Watkins CJ. Evaluation of a patient education manual. BMJ 1980;281:924-5.
23. Casey R, McMahon F, McCormick MC. Fever therapy: educational intervention for parents. Pediatr 1984;73:600-03.
24. Roberts CR, Imrey PB, Turner JD. Reducing physician visits for colds through consumer education. JAMA 1983;250:1986-9.
25. Bhopal RS, Gilmour WH, Fallon CW, Bhopal JS, Hamilton I. Evaluation of a practice information leaflet. Fam Pract 1990;7:132-7.
26. Mottram DR, Reed C. Comparative evaluation of patient information leaflets by pharmacists, doctors, and the general public. J Clin Pharm Therapeutics 1997;22:127-34.
27. Arthur VAM. Written patient information: a review of the literature. J Adv Nurs 1995;21:1081-6.
28. Busson M, Dunn APM. Patient knowledge about prescribed medicines. Pharm J 1986;236:624-6.
29. Becker SH, Mcphee SJ. Health-care professionals’ use of cancer-related patient education materials: a pilot study. J Cancer Ed 1993;8:43-6.
30. Thoma GB. Evolution of a patient education program in a rural hospital. Nurs Manage 1994;25:46-8.
31. Smith JL, Levitt C, Franco ED. Innovative system to improve use of patient education materials. Can Fam physician 1997;43:58-62.
32. Collings LH, Pike LC, Binder AI, McClymont ME, Knight ST. Value of written health information in the general practice setting. Br J Gen Pract 1991;41:466-7.
33. Tronni C, Welebob E. End-user satisfaction of a patient education tool manual versus computer-generated tool. Computers Nurs 1996;14:235-8.
34. Korn R, Wieczorek RR. Computerization of standards and patient education material. J Nurs Staff Develop 1995;11:307-12.
35. Weaver J. Patient education: an innovative computer approach. Nurs Manage 1995;26:78-9, 81,83.
36. Williams RB, Boles M, Johnson RE. Patient use of a computer for prevention in primary care practice. Patient Educ Counsel 1995;25:283-92.
37. Childhood Asthma Management Program Research Group. Design and implementation of a patient education center for the Childhood Asthma Management Program. Ann Allergy Asthma Immunol 1998;81:571-81.
38. Patton MQ. Qualitative evaluation and research methods. 2nd ed. Newbury Park, Calif: Sage Publications; 1990.
39. McIlvain HE, Crabtree BF, Gilbert C, Havranek R, Backer EL. Current trends in tobacco prevention and cessation in Nebraska physicians’ offices. J Fam Pract 1997;44:193-202.
40. Yin RK. Case study research: design and methods. Applied social research methods series. 2nd ed. Newbury Park, Calif: Sage Publications; 1989;166.-
41. Bogdewic SP. Participant observation. In: Crabtree BF, Miller WL, eds. Doing qualitative research. 2nd ed. Newbury Park. Calif: Sage Publications; 1999;37-70.
42. Stange KC, Zyzanski SJ, Jaen CR, et al. Illuminating the ‘black box’ a description of 4454 patient visits to 138 family physicians. J Fam Pract 1998;46:377-89.
43. Crabtree B, Miller W. Researching practice settings: a case study approach. In: Crabtree B, Miller W, eds. Doing qualitative research. 2nd ed. Thousand Oaks, Calif: Sage Publications; 1999.
44. Pommerenke FA, Dietrich AJ. Improving and maintaining preventive services, part I: applying the patient model. J Fam Pract 1992;34:86-91.
45. Miller W, Crabtree B. Depth interviewing. In: Crabtree B, Miller W, eds. Doing qualitative research. 2nd ed. Thousand Oaks, Calif: Sage Publications; 1999;89-108.
46. McCracken G. The long interview. Newbury Park, Calif: Sage Publications; 1988.
47. Meadows L, Dodendorf D. Data management and interpretation: using computer programs to assist. In: Crabtree B, Miller W, eds. Doing qualitative research. 2nd ed. Newbury Park, Calif: Sage Publications; 1999.
48. Ray LD. Qualitative data management using Folio VIEWS. Qual Health Research 1997;7:301-8.
49. Miller W, Crabtree B. The dance of interpretation. In: Crabtree B, Miller W, eds. Doing qualitative research. 2nd ed. Thousand Oaks, Calif: Sage Publications; 1999.
50. Borkan J. Immersion/crystallization. In: Crabtree BF, Miller WL, eds. Doing qualitative research. 2nd ed. Thousand Oaks, Calif: Sage Publications; 1999;179-94.
51. Miles MB, Huberman AM. Qualitative data analysis: an expanded sourcebook. 2nd ed. Thousand Oaks, Calif: Sage Publications; 1994;337.-
52. Crabtree B, Miller W. Using codes and code manuals: a template organizing style of interpretation. In: Crabtree B, Miller W, eds. Doing qualitative research. 2nd ed. Thousand Oaks, Calif: Sage Publications; 1999;163-78.
1. Bartlett E. At last, a definition. Patient Educ Counsel 1985;7:323-4.
2. Ackroyd E. The patient’s complaint. Br J Hosp Med 1986;36:454.-
3. Wise PH, Pietroni RG, Bhatt VB, Bond CS, Hirst S, Hooker RJ. Development and evaluation of a novel patient information system. J R Soc Med 1996;89:557-60.
4. Mullen PD, Laville EA, Biddle AK, Lorig K. Efficacy of psychoeducational interventions on pain, depression, and disability in people with arthritis: a meta analysis. J Rheumatol 1987;14:33-9.
5. Bondy LR, Sims N, Schroeder DR, Offord KP, Narr BJ. The effect of anesthestic patient education on preoperative patient anxiety. Reg Anesth Pain Med 1999;24:158-64.
6. Weinman J. Providing written information for patients: psychological considerations. J Royal Soc Med 1990;83:303-5.
7. Kitching JB. Patient information leaflets—the state of the art. J Royal Soc Med 1990;3:298-300.
8. Glascoe FP, Oberklaid F, Dworkin PH, Trimm F. Brief approaches to educating patients and parents in primary care. Pediatr 1998;101:1068.-
9. Gibbs S, Water WE, George CF. The benefits of prescription information leaflets (1). Brit J Clin Pharmacol 1989a;27:723-39.
10. Gibbs S, Water WE, George CF. The benefits of prescription information leaflets (2). Brit J Clin Pharmacol 1989b;28:345-51.
11. Gibbs S, Waters WE, George CF. Prescription information leaflets: a national survey. J R Soc Med 1990;83:292-97.
12. Fisher RC. Patient education and compliance: a pharmacist’s perspective. Patient Educ Counsel 1992;19:261-71
13. Hawe P, Higgins G. Can medication education improve the drug compliance of the elderly? Evaluation of an in hospital programme. Patient Educ Counsel 1990;16:151-60.
14. Myers ED, Calvert EJ. Information, compliance, and side effects: a study of patients on anti-depressant medication. Br J Clin Pharmacol 1984;17:221-5.
15. Ellis DA, Hopkin JM, Leitch AG, J CS. Doctor’s orders. BMJ 1979;1:456.-
16. Young FK, Brooks BR. Patient teaching manuals improve retention of treatment information: a controlled clinical trial in multiple sclerosis. Neurosci Nurs 1986;18:26-8.
17. Grady K, Lemkau J, McVay J, Reisine S. The importance of physician encouragement in breast cancer screening of older women. Prevent Med 1992;21:766-80.
18. Vignos PJ, Parker WT, Thompson HM. Evaluation of a clinic education programme for patients with RA. J Rheumatol 1976;3:155-65.
19. Cook B, Noteloviz M. An osteoporosis patient education and screening programme: Results and implications. Patient Educ Counsel 1991;17:135-45.
20. Vetto JT, Dubois PM, Vetto IP. The impact of distribution of patient-education pamphlet in a multidisciplinary breast clinic. J Cancer Educ 1996;11:148-52.
21. Sandler DA, Heaton C, Garner TG, Mitchell JR. Patients’ and general practitioners’ satisfaction with information given on discharge from hospital: audit of a new information card. BMJ 1989;299:1511-13.
22. Anderson JE, Morrell DC, Avery AJ, Watkins CJ. Evaluation of a patient education manual. BMJ 1980;281:924-5.
23. Casey R, McMahon F, McCormick MC. Fever therapy: educational intervention for parents. Pediatr 1984;73:600-03.
24. Roberts CR, Imrey PB, Turner JD. Reducing physician visits for colds through consumer education. JAMA 1983;250:1986-9.
25. Bhopal RS, Gilmour WH, Fallon CW, Bhopal JS, Hamilton I. Evaluation of a practice information leaflet. Fam Pract 1990;7:132-7.
26. Mottram DR, Reed C. Comparative evaluation of patient information leaflets by pharmacists, doctors, and the general public. J Clin Pharm Therapeutics 1997;22:127-34.
27. Arthur VAM. Written patient information: a review of the literature. J Adv Nurs 1995;21:1081-6.
28. Busson M, Dunn APM. Patient knowledge about prescribed medicines. Pharm J 1986;236:624-6.
29. Becker SH, Mcphee SJ. Health-care professionals’ use of cancer-related patient education materials: a pilot study. J Cancer Ed 1993;8:43-6.
30. Thoma GB. Evolution of a patient education program in a rural hospital. Nurs Manage 1994;25:46-8.
31. Smith JL, Levitt C, Franco ED. Innovative system to improve use of patient education materials. Can Fam physician 1997;43:58-62.
32. Collings LH, Pike LC, Binder AI, McClymont ME, Knight ST. Value of written health information in the general practice setting. Br J Gen Pract 1991;41:466-7.
33. Tronni C, Welebob E. End-user satisfaction of a patient education tool manual versus computer-generated tool. Computers Nurs 1996;14:235-8.
34. Korn R, Wieczorek RR. Computerization of standards and patient education material. J Nurs Staff Develop 1995;11:307-12.
35. Weaver J. Patient education: an innovative computer approach. Nurs Manage 1995;26:78-9, 81,83.
36. Williams RB, Boles M, Johnson RE. Patient use of a computer for prevention in primary care practice. Patient Educ Counsel 1995;25:283-92.
37. Childhood Asthma Management Program Research Group. Design and implementation of a patient education center for the Childhood Asthma Management Program. Ann Allergy Asthma Immunol 1998;81:571-81.
38. Patton MQ. Qualitative evaluation and research methods. 2nd ed. Newbury Park, Calif: Sage Publications; 1990.
39. McIlvain HE, Crabtree BF, Gilbert C, Havranek R, Backer EL. Current trends in tobacco prevention and cessation in Nebraska physicians’ offices. J Fam Pract 1997;44:193-202.
40. Yin RK. Case study research: design and methods. Applied social research methods series. 2nd ed. Newbury Park, Calif: Sage Publications; 1989;166.-
41. Bogdewic SP. Participant observation. In: Crabtree BF, Miller WL, eds. Doing qualitative research. 2nd ed. Newbury Park. Calif: Sage Publications; 1999;37-70.
42. Stange KC, Zyzanski SJ, Jaen CR, et al. Illuminating the ‘black box’ a description of 4454 patient visits to 138 family physicians. J Fam Pract 1998;46:377-89.
43. Crabtree B, Miller W. Researching practice settings: a case study approach. In: Crabtree B, Miller W, eds. Doing qualitative research. 2nd ed. Thousand Oaks, Calif: Sage Publications; 1999.
44. Pommerenke FA, Dietrich AJ. Improving and maintaining preventive services, part I: applying the patient model. J Fam Pract 1992;34:86-91.
45. Miller W, Crabtree B. Depth interviewing. In: Crabtree B, Miller W, eds. Doing qualitative research. 2nd ed. Thousand Oaks, Calif: Sage Publications; 1999;89-108.
46. McCracken G. The long interview. Newbury Park, Calif: Sage Publications; 1988.
47. Meadows L, Dodendorf D. Data management and interpretation: using computer programs to assist. In: Crabtree B, Miller W, eds. Doing qualitative research. 2nd ed. Newbury Park, Calif: Sage Publications; 1999.
48. Ray LD. Qualitative data management using Folio VIEWS. Qual Health Research 1997;7:301-8.
49. Miller W, Crabtree B. The dance of interpretation. In: Crabtree B, Miller W, eds. Doing qualitative research. 2nd ed. Thousand Oaks, Calif: Sage Publications; 1999.
50. Borkan J. Immersion/crystallization. In: Crabtree BF, Miller WL, eds. Doing qualitative research. 2nd ed. Thousand Oaks, Calif: Sage Publications; 1999;179-94.
51. Miles MB, Huberman AM. Qualitative data analysis: an expanded sourcebook. 2nd ed. Thousand Oaks, Calif: Sage Publications; 1994;337.-
52. Crabtree B, Miller W. Using codes and code manuals: a template organizing style of interpretation. In: Crabtree B, Miller W, eds. Doing qualitative research. 2nd ed. Thousand Oaks, Calif: Sage Publications; 1999;163-78.