User login
Researchers from the Pediatric Research in Inpatient Settings (PRIS) network, an open pediatric hospitalist research network,[1] have identified inpatient pediatric medical and surgical conditions considered high priority for quality improvement (QI) initiatives and/or comparative effectiveness research based on prevalence, cost, and interhospital variation in resource utilization.[2] One approach for improving the quality of care within hospitals is to operationalize evidence‐based guidelines into practice.[3] Although guidelines may be used by individual clinicians, systematic adoption by hospitals into clinical workflow has the potential to influence providers to adhere to evidence‐based care, reduce unwarranted variation, and ultimately improve patient outcomes.[3, 4, 5, 6]
There are critical appraisal tools to measure the methodological quality, as defined by the Institute of Medicine (IOM) and others in their guidelines.[7, 8, 9, 10, 11, 12] One such validated tool is the AGREE II instrument, created by the AGREE (Appraisal of Guidelines for REsearch and Evaluation) collaboration.[13, 14] It defines methodological quality as the confidence that the biases linked to the rigor of development, presentation, and applicability of a clinical practice guideline have been minimized and that each step of the development process is clearly reported.[13]
The objective of our study was to rate the methodological quality of national guidelines for 20 of the PRIS priority pediatric inpatient conditions.[2] Our intent in pursuing this project was 2‐fold: first, to inform pediatric inpatient QI initiatives, and second, to call out priority pediatric inpatient conditions for which high methodological‐quality guidelines are currently lacking.
METHODS
The study methods involved (1) prioritizing pediatric inpatient conditions, (2) identifying national guidelines for the priority conditions, and (3) rating the methodological quality of available guidelines. This study was considered nonhuman‐subject research (A. Johnson, personal e‐mail communication, November 14, 2012), and the original prioritization study was deemed exempt from review by the institutional review board of the Children's Hospital of Philadelphia under 45 CFR 46.102(f).[2]
Prioritizing Pediatric Inpatient Conditions
Methods for developing the prioritization list are published elsewhere in detail and briefly described here.[1] An International Classification of Diseases, 9th Revision, Clinical Modification‐based clinical condition grouper was created for primary discharge diagnosis codes for inpatient, ambulatory surgery, and observation unit encounters accounting for either 80% of all encounters or 80% of all charges for over 3.4 million discharges from 2004 to 2009 for 38 children's hospitals in the Pediatric Health Information Systems (PHIS) database, which includes administrative and billing data.[15] A standardized cost master index was created to assign the same unit cost for each billable item (calculated as the median of median hospital unit costs) to allow for comparisons of resource utilization across hospitals (eg, the cost of a chest x‐ray was set to be the same across all hospitals in 2009 dollars). Total hospital costs were then recalculated for every admission by multiplying the standardized cost master index by the number of units for each item in the hospital bill, and then summing the standardized costs of each line item in every bill. Conditions were ranked based on prevalence and total cost across all hospitals in the study period. The variation in standardized costs across hospitals for each condition was determined.
For the current study, conditions were considered if they had a top 20 prevalence rank, a top 20 cost rank, high variation (intraclass correlation coefficient >0.1) in standardized costs across hospitals, a minimum number of PHIS hospitals with annualized overexpenditures (using the standardized cost master) of at least $50,000 when compared to the mean, or a minimum median of 200 cases per hospital over the 6‐year study period to assure sufficient hospital volume for future interventions. This resulted in 29 conditions; the selected 20 conditions matched the top 20 prevalence rank (see Supporting Information, Table 1, in the online version of this article).[2]
| Condition by PRIS Priority Rank | Guidelines Meeting Inclusion Criteriaa | Guidelines Citation | Mean Overall Reviewer Methodological Quality Rating (Rater 1, Rater 2)b | Recommended for Use in the Pediatric Inpatient Setting, Mean (Rater 1, Rater 2)c | Weighted Kappa(95% Confidence Interval) |
|---|---|---|---|---|---|
| |||||
| Otitis media, unspecified, s | 1 | American Academy of Family Physicians; American Academy of Otolaryngology‐Head and Neck Surgery; American Academy of Pediatrics Subcommittee on Otitis Media With Effusion. Clinical Practice Guidelines: Otitis media with effusion. Pediatrics. 2004 May;113(5):1412‐29. | 6 (6, 6) | 3 (3, 3) | 0.76 (0.490.93) |
| Hypertrophy of tonsils and adenoids, s | 1 | Baugh RF et al. Clinical practice guideline: tonsillectomy in children. Otolaryngol Head Neck Surg. 2011;144(1 suppl):S1S30. | 6.5 (7, 6) | 3 (3, 3) | 0.49 (0.050.81) |
| Asthma, m | 1 | National Heart, Lung, and Blood Institute; National Asthma Education and Prevention Program. Expert panel report 3 (EPR‐3): guidelines for the diagnosis and management of asthma‐full report 2007. Pages 1‐440. Available at: | 7 (7, 7) | 3 (3, 3) | 0.62 (0.210.87) |
| Bronchiolitis, m | 1 | American Academy of Pediatrics Subcommittee on Diagnosis and Management of Bronchiolitis. Diagnosis and management of bronchiolitis. Pediatrics. 2006;118:17741793. | 6.5 (6, 7) | 3 (3, 3) | 0.95 (0.871.00) |
| Pneumonia, m | 1 | Bradley JS et al.The management of community‐acquired pneumonia in infants and children older than 3 months of age: clinical practice guidelines by the Pediatric Infectious Diseases Society and the Infectious Diseases Society of America. Clin Infect Dis. 2011;53(7):e25e76. | 6 (6, 6) | 3 (3, 3) | 0.82 (0.640.96) |
| Dental caries, s | 1 | American Academy on Pediatric Dentistry Clinical Affairs CommitteePulp Therapy Subcommittee; American Academy on Pediatric Dentistry Council on Clinical Affairs. Guideline on pulp therapy for primary and young permanent teeth. Pediatr Dent. 2008;30:170174. | 3 (3, 3) | 1.5 (1, 2) | 0.51 (0.140.83) |
| Chemotherapy, m | 0 | ||||
| Cellulitis, m | 1 | Stevens DL et al. Practice guidelines for the diagnosis and management of skin and softtissue infections. Clin Infect Dis. 2005;41:13731406. | 4.5 (4, 5) | 2.5 (2, 3) | 0.52 (0.150.79) |
| Inguinal hernia, s | 0 | ||||
| Gastroesophageal reflux and esophagitis, m, s | 2 | Vandenplas Y et al. Pediatric gastroesophageal reflux clinical practice guidelines: joint recommendations of NASPGHAN and ESPGHAN. J Pediatr Gastroenterol Nutr. 2009;49(4):498547. | 5 (5, 5) | 3 (3, 3) | 0.69 (0.450.87) |
| Furuta GT et al. Eosinophilic esophagitis in children and adults: a systematic review and consensus recommendations for diagnosis and treatment. Gastroenterology. 2007;133:13421363. | 5 (5, 5) | 2.5 (2, 3) | 0.93 (0.850.98) | ||
| Dehydration, m | 0 | ||||
| Redundant prepuce and phimosis, s | 1 | American Academy of Pediatrics Task Force on Circumcision. Male circumcision. Pediatrics. 2012;130(3):e756e785. | 6 (6, 6) | 3 (3, 3) | 0.66 (0.250.89) |
| Abdominal pain, m | 0 | ||||
| Other convulsions, m | 0 | ||||
| Urinary tract infection, m | 1 | Roberts KB et al. Urinary tract infection: clinical practice guideline for the diagnosis and management of the initial UTI in febrile infants and children 2 to 24 months. Pediatrics. 2011;128:595610. | 5.5 (5, 6) | 2.5 (2, 3) | 0.62 (0.230.84) |
| Acute appendicitis without peritonitis, s | 1 | Solomkin JS et al. Diagnosis and management of complicated intra‐abdominal infection in adults and children: guidelines by the Surgical Infection Society and the Infectious Diseases Society of America. Clin Infect Dis. 2010;50:133164. | 4.5 (5, 4) | 2.5 (3, 2) | 0.37 (0.110.81) |
| Eso‐ exo‐ hetero‐, and hypertropia, s | 0 | ||||
| Fever, m | 0 | ||||
| Seizures with and without intractable epilepsy, m | 3 | Brophy GM et al; Neurocritical Care Society Status Epilepticus Guideline Writing Committee. Guidelines for the evaluation and management of status epilepticus. Neurocrit Care. 2012;17:323. | 5 (5, 5) | 3 (3, 3) | 0.95 (0.870.99) |
| Hirtz D et al. Practice parameter: treatment of the child with a first unprovoked seizure: report of the Quality Standards Subcommittee of the American Academy of Neurology and the Practice Committee of the Child Neurology Society. Neurology. 2003;60:166175. | 5 (5, 5) | 2.5 (2, 3) | 0.73 (0.410.94) | ||
| Riviello JJ Jr et al. Practice parameter: diagnostic assessment of the child with status epilepticus (an evidence‐based review): report of the Quality Standards Subcommittee of the American Academy of Neurology and the Practice Committee of the Child Neurology Society. Neurology. 2006;67:15421550. | 5 (4, 6) | 2.5 (2, 3) | 0.80 (0.630.94) | ||
| Sickle cell disease with crisis, m | 2 | Section on Hematology/Oncology Committee on Genetics; American Academy of Pediatrics. Health supervision for children with sickle cell disease. Pediatrics. 2002;109:526535. | 3.5 (3, 4) | 1.5 (1, 2) | 0.92 (0.800.98) |
| National Heart, Lung, and Blood Institute, National Institutes of Health. The management of sickle cell disease. National Heart, Lung, and Blood Institute, National Institutes of Health, Bethesda, MD. Available at: | 4 (4, 4) | 2.5 (2, 3) | 0.91 (0.800.97) | ||
Identifying National Guidelines
We developed a search protocol (see Supporting Information, Table 2, in the online version of this article) using condition‐specific keywords and the following criteria: guideline, pediatric, 2002 to 2012. A medical librarian (E.E.) used the protocol to search PubMed, National Guidelines Clearing House, and the American Academy of Pediatrics website for guidelines for the 20 selected conditions.
We limited our study to US national guidelines published or updated from 2002 to 2012 to be most relevant to the 38 US children's hospitals in the original study. Guidelines had to address either medical or surgical or both types of inpatient management for the condition, depending on how the condition was categorized on the PRIS prioritization list. For example, to target inpatient issues, otitis media was treated as a surgical condition when the prioritization list was created, therefore guidelines included in our study needed to address surgical management (ie, myringotomy or tympanostomy tubes).[2] Guidelines specific to 1 organism, test, or treatment were a priori excluded, as they would not map well to the prioritization list, and would be difficult to interpret. Guidelines focusing exclusively on condition prevention were also excluded. Guidelines with a broad subject matter (eg, abdominal infection) or unclear age were included if they contained a significant focus on the condition of interest (eg, appendicitis without peritonitis), such that the course of pediatric inpatient care was described for that condition. Retracted or outdated (superseded by a more current version) guidelines were excluded.
An investigator (G.H.) reviewed potentially relevant results from the librarian's search. For example, the search for tonsillectomy guidelines retrieved a guideline on the use of polysomnography prior to tonsillectomy in children but did not cover the inpatient management or tonsillectomy procedure.[16] This guideline was excluded from our study, as it focused on a specific test and did not discuss surgical management of the condition.
Rating Methodological Quality of Guidelines
Methodological quality of guidelines was rated with the AGREE II tool by 2 investigators (G.H. and K.N.).[13, 17] This tool has 2 overall guideline assessments and 23 subcomponents within 6 domains, reflecting many of the IOM's recommendations for methodological quality in guidelines: scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability, and editorial independence.[8, 17]
The AGREE II tool rates each of the 23 subcomponent questions using a 7‐point scale (1=strongly disagree7=strongly agree). We followed the AGREE II user's manual suggestion in rating subcomponents as 1, indicating an absence of information for that question if the question was not addressed within the guideline.[13] The AGREE II user's manual describes the option of creating standardized domain scores; however, as the objective of our study was to assess the overall methodological quality of the guideline and not to highlight particular areas of strengths/weaknesses in the domains, we elected to present raw scores only.[13]
For the overall guideline rating item 1 (Rate the overall quality of this guideline.) the AGREE II tool instructs that a score of 1 indicates lowest possible quality and 7 indicates highest possible quality.[13] As these score anchors are far apart with no guide for interpretation of intermediate results, we modified the descriptive terms on the tool to define scores <3 as low quality, scores 3 to 5 as moderate quality, and scores >5 as high quality to allow for easier interpretation of our results. We also modified the final overall recommendation score (on a 3‐point scale) from I would recommend this guideline for use to I would recommend this guideline for use in the pediatric inpatient setting.[13, 17] A score of 1 indicated to not recommend, 2 indicated to recommend with modifications, and 3 indicated to recommend without modification.
Significant discrepancies (>2‐point difference on overall rating) between the 2 raters were to be settled by consensus scoring by 3 senior investigators blinded to previous reviews, using a modified Delphi technique.[18]
Inter‐rater reliability was measured using a weighted kappa coefficient and reported using a bootstrapped method with 95% confidence intervals. Interpretation of kappa is such that 0 is the amount of agreement that would be expected by chance, and 1 is perfect agreement, with previous researchers stating scores >0.81 indicate almost perfect agreement.[19]
RESULTS
The librarian's search retrieved 2869 potential results (Figure 1). Seventeen guidelines met inclusion criteria for 13 of the 20 priority conditions. Seven conditions did not have national guidelines meeting inclusion criteria. Table 1 displays the 20 medical and surgical conditions on the modified PRIS prioritization list, including overall guideline scoring, recommendation scores, and kappa results for each guideline. The highest methodological‐quality guidelines were for asthma,[20] tonsillectomy,[21] and bronchiolitis[22] (mean overall rating 7, 6.5, and 6.5, respectively). The lowest methodological‐quality guidelines were for 2 sickle cell disease guidelines[23, 24] and 1 dental caries guideline[25] (mean overall rating 4, 3.5, and 3, respectively). Seven guidelines were rated as high overall quality, and 10 guidelines were rated as moderate overall quality. Eight of the 17 guidelines[20, 21, 22, 26, 27, 28, 29, 30] were recommended for use in the pediatric inpatient setting without modification by both reviewers. Two guidelines (for dental caries[25] and sickle cell[23]) were not recommended for use by 1 reviewer.
As an example of scoring, a national guideline for asthma had high overall scores (7 from each reviewer) and high scores across most AGREE II subcomponents. The guideline was found by both reviewers to be systematic in describing guideline development with clearly stated recommendations linked to the available evidence (including strengths and limitations) and implementation considerations.[20] Conversely, a national guideline for sickle cell disease had moderate overall scores (scores of 3 and 4) and low‐moderate scores across the majority of the subcomponent items.[23] The reviewers believe that this guideline would have been strengthened by increased transparency in guideline development, discussion of the evidence surrounding recommendations, and discussion of implementation factors. A table with detailed scoring of each guideline is available (see Supporting Information, Table 3, in the online version of this article).
Agreement between the 2 raters was almost perfect,[19] with an overall boot‐strapped weighted kappa of 0.83 (95% confidence interval 0.780.87) across 850 scores. There were no discrepancies between reviewers in overall scoring requiring consensus scoring.
DISCUSSION
Using a modified version of a published prioritization list for inpatient pediatric conditions, we found national guidelines for 13 of 20 conditions with high prevalence, cost, and interhospital variation in resource utilization. Seven conditions had no national guidelines published within the past 10 years applicable for use in the pediatric inpatient setting. Of 17 guidelines for 13 conditions, 10 had moderate and 7 had high methodological quality.
Our findings add to the literature describing methodological quality of guidelines. Many publications focus on the methodological quality of guidelines as a group and use a standardized instrument (eg, the AGREE II tool) to rate within domains (eg, domain 1: scope and purpose) across guidelines in an effort to encourage improvement in developing and reporting in guidelines.[31, 32] Our study differs in that we chose to focus on the overall quality rating of individual guidelines for specific prioritized conditions to allow hospitals to guide QI initiatives. One study that had a similar aim to ours surveyed Dutch pediatricians to select priority conditions and used the AGREE II tool to rate 17 guidelines, recommending 14 for use in the Netherlands.[33]
Identifying high methodological‐quality guidelines is only 1 in a series of steps prior to successful guideline implementation in hospitals. Other aspects of guidelines, including the strength of the evidence (eg, from randomized controlled trials) and subsequent force and clarity (eg, use of must instead of consider) of recommendations, may affect clinician or patient adherence, work processes, and ultimately patient outcomes. Strong evidence should translate into forceful and clear recommendations. Authors with the Yale Guideline Recommendation Corpus describe significant variation in reporting of guideline recommendations, and further studies have shown that the force and clarity of a recommendation is associated with adherence rates.[34, 35, 36, 37] Unfortunately, current guideline appraisal tools lack the means to score the strength of evidence, and force and clarity of recommendations.[10]
Implementation science demonstrates that there are many important factors in translating best practice into improvements in clinical care. In addition to implementation considerations such as adherence, patient preferences, and work processes, variability in methodological quality, strength of evidence, and force and clarity of recommendations may be additional reasons why evidence for the impact of guidelines on patient outcomes remains mixed in the literature.[38] One recent study found that adherence to antibiotics recommended within a national pediatric community‐acquired pneumonia guideline, which had a high methodological‐quality score in our study, did not change hospital length of stay or readmissions.[29, 39] There are several possible interpretations for this. Recommendations may not have been based upon strong evidence, research methodology assessing how adherence to recommendations impacts patient outcomes may have been limited, or the outcomes measured in current studies (such as readmission) are not the outcomes that may be improved by adherence to these recommendations (such as decreasing antimicrobial resistance). These are important considerations when hospitals are incorporating recommendations from guidelines into practice. Hospitals should assess the multiple aspects of guidelines, including methodological quality, which our study helps to identify, strength of evidence, and force and clarity of recommendations, as well as adherence, patient preferences, work processes, and key outcome measures when implementing guidelines into clinical practice. A study utilizing a robust QI methodology demonstrated that clinician adherence to several elements in an asthma guideline, which also had a high methodological‐quality score in our study, led to a significant decrease in 6‐month hospital and emergency department readmission for asthma.[6, 20]
Our study also highlights that several pediatric conditions with high prevalence, cost, and interhospital resource utilization variation lack recent national pediatric guidelines applicable to the inpatient setting. If strong evidence exists for these priority conditions, professional societies should create high methodological‐quality guidelines with strong and clear recommendations. If evidence is lacking for these priority conditions, then investigators should focus on generating research in these areas.
There are several limitations to this study. The AGREE II tool does not have a mechanism to measure the strength of evidence used in a guideline. Methodological quality of a guideline alone may not translate into improved outcomes. Conditions may have national guidelines published before 2002, institution‐specific or international guidelines, or adult guidelines that might be amenable to use in the pediatric inpatient setting but were not included in this study. Several conditions on the prioritization list are broad in nature (eg, dehydration) and may not be amenable to the creation of guidelines. Other conditions on the prioritization list (eg, chemotherapy or cellulitis) may have useful guidelines within the context of specific conditions (eg, acute lymphoblastic leukemia) or for specific organisms (eg, methicillin‐resistant Staphylococcus aureus). We elected to exclude these narrower guidelines to focus on broad and comprehensive guidelines applicable to a wider range of clinical situations. Additionally, although use of a validated tool attempts to objectively guide ratings, the rating of quality is to some degree subjective. Finally, our study used a previously published prioritization list using data from children's hospitals, and the list likely under‐represents conditions commonly managed in community hospitals (eg, hyperbilirubinemia).[2] Exclusion of these conditions was not reflective of importance or quality of available national guidelines.
CONCLUSIONS
Our study adds to recent publications on the need to prioritize conditions for QI in children's hospitals. We identified a group of moderate to high methodological‐quality national guidelines for pediatric inpatient conditions with high prevalence, cost, and variation in interhospital resource utilization. Not all prioritized conditions have national high methodological‐quality guidelines available. Hospitals should prioritize conditions with high methodological‐quality guidelines to allocate resources for QI initiatives. Professional societies should focus their efforts on producing methodologically sound guidelines for prioritized conditions currently lacking high‐quality guidelines if sufficient evidence exists.
Acknowledgements
The authors thank Christopher G. Maloney, MD, PhD, for critical review of the manuscript, and Gregory J. Stoddard, MS, for statistical support. Mr. Stoddard's work is supported by the University of Utah Study Design and Biostatistics Center, with funding in part from the National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health, through grant 8UL1TR000105 (formerly UL1RR025764).
Disclosures
Sanjay Mahant, Ron Keren, and Raj Srivastava are all Executive Council members of the Pediatric Research in Inpatient Settings (PRIS) Network. PRIS, Sanjay Mahant, Ron Keren, and Raj Srivastava are all supported by grants from the Children's Hospital Association. Sanjay Mahant is also supported by research grants from the Canadian Institute of Health Research and Physician Services Incorporated. Ron Keren and Raj Srivastava also serve as medical legal consultants. The remaining authors have no financial relationships relevant to this article to disclose.
- , . Development of the Pediatric Research in Inpatient Settings (PRIS) Network: lessons learned. J Hosp Med. 2012;7(8):661–664.
- , , , et al. Prioritization of comparative effectiveness research topics in hospital pediatrics. Arch Pediatr Adolesc Med. 2012;166(12):1–10.
- , . How Intermountain trimmed health care costs through robust quality improvement efforts. Health Aff (Millwood). 2011;30(6):1185–1191.
- . Making it easy to do it right. N Engl J Med. 2001;345(13):991–993.
- , , , , . Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ. 1999;318(7182):527–530.
- , , , et al. The Joint Commission Children's Asthma Care quality measures and asthma readmissions. Pediatrics. 2012;130(3):482–491.
- , , , et al. Toward transparent clinical policies. Pediatrics. 2008;121(3):643–646.
- Field MJ, Lohr KN, eds.; Committee to Advise the Public Health Service on Clinical Practice Guidelines, Institute of Medicine. Clinical Practice Guidelines: Directions for a New Program. Washington, DC: National Academies Press; 1990.
- , , , , . Development and application of a generic methodology to assess the quality of clinical guidelines. Int J Qual Health Care. 1999;11(1):21–28.
- , , , , . A systematic review of appraisal tools for clinical practice guidelines: multiple similarities and one common deficit. Int J Qual Health Care. 2005;17(3):235–242.
- , , . How to decide whether a clinical practice guideline is trustworthy. JAMA. 2013;309(2):139–140.
- Field MJ, Lohr KN, eds.; Committee to Advise the Public Health Service on Clinical Practice Guidelines, Institute of Medicine. Guidelines for Clinical Practice: From Development to Use. Washington, DC: National Academies Press; 1992.
- The AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care. 2003;12:18–23.
- . AGREE II‐improving the quality of clinical care. Lancet. 2010;376(9747):1128–1129.
- , , , . Corticosteroids and mortality in children with bacterial meningitis. JAMA. 2008;299(17):2048–2055.
- , , , et al. Clinical practice guideline: polysomnography for sleep‐disordered breathing prior to tonsillectomy in children. Otolaryngol Head Neck Surg. 2011;145(1 suppl):S1–S15.
- , , , et al. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ. 2010;182(18):E839–E842.
- , . The Delphi method as a research tool: an example, design considerations and applications. Inform Manag. 2004;42(1):15–29.
- , . The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174.
- National Heart, Lung, and Blood Institute; National Asthma Education and Prevention Program. Expert panel report 3 (EPR‐3): guidelines for the diagnosis and management of asthma‐full report 2007. Pages 1–440. Available at: http://www.nhlbi.nih.gov/guidelines/asthma/asthgdln.pdf. Accessed on 24 August 2012.
- , , , et al. Clinical practice guideline: tonsillectomy in children. Otolaryngol Head Neck Surg. 2011;144(1 suppl):S1–S30.
- American Academy of Pediatrics Subcommittee on Diagnosis and Management of Bronchiolitis. Diagnosis and management of bronchiolitis. Pediatrics. 2006;118:1774–1793.
- Health supervision for children with sickle cell disease. Pediatrics. 2002;109(3):526–535.
- National Heart, Lung, and Blood Institute, National Institutes of Health. The management of sickle cell disease. National Heart, Lung, and Blood Institute, National Institutes of Health, Bethesda, MD. Available at: http://www.nhlbi.nih.gov/health/prof/blood/sickle/sc_mngt.pdf. Revised June 2002. Accessed on 13 October 2012.
- American Academy on Pediatric Dentistry Clinical Affairs Committee–Pulp Therapy Subcommittee; American Academy on Pediatric Dentistry Council on Clinical Affairs. Guideline on pulp therapy for primary and young permanent teeth. Pediatr Dent. 2008;30:170–174.
- , , , et al.; Neurocritical Care Society Status Epilepticus Guideline Writing Committee. Guidelines for the evaluation and management of status epilepticus. Neurocrit Care. 2012;17:3–23.
- American Academy of Pediatrics Task Force on Circumcision. Male circumcision. Pediatrics. 2012;130(3):e756–e785.
- , , , et al. Pediatric gastroesophageal reflux clinical practice guidelines: joint recommendations of the North American Society for Pediatric Gastroenterology, Hepatology, and Nutrition (NASPGHAN) and the European Society for Pediatric Gastroenterology, Hepatology, and Nutrition (ESPGHAN). J Pediatr Gastroenterol Nutr. 2009;49(4):498–547.
- , , , et al. The management of community‐acquired pneumonia in infants and children older than 3 months of age: clinical practice guidelines by the Pediatric Infectious Diseases Society and the Infectious Diseases Society of America. Clin Infect Dis. 2011;53(7):e25–e76.
- American Academy of Family Physicians; American Academy of Otolaryngology‐Head and Neck Surgery; American Academy of Pediatrics Subcommittee on Otitis Media With Effusion. Clinical Practice Guidelines: Otitis media with effusion. Pediatrics. 2004 May;113(5):1412–29.
- , , . Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer‐reviewed medical literature. JAMA. 1999;281(20):1900–1905.
- , , , . Quality of reporting and evidence in American Academy of Pediatrics guidelines. Pediatrics. 2013;131(4):732–738.
- , , . Quality of evidence‐based pediatric guidelines. Pediatrics. 2005;115(5):1378–1391.
- , , . The Yale Guideline Recommendation Corpus: a representative sample of the knowledge content of guidelines. Int J Med Inform. 2009;78(5):354–363.
- , , . How often is strength of recommendation indicated in guidelines? Analysis of the Yale Guideline Recommendation Corpus. AMIA Annu Symp Proc. 2008:984.
- , , . Clinical practice guideline development manual, third edition: a quality‐driven approach for translating evidence into action. Otolaryngol Head Neck Surg. 2013;148(1 suppl):S1–S55.
- , , , , , . Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. BMJ. 1998;317(7162):858–861.
- , , , et al. Toward evidence‐based quality improvement. Evidence (and its limitations) of the effectiveness of guideline dissemination and implementation strategies 1966–1998. J Gen Intern Med. 2006;21(suppl 2):S14–S20.
- , , , . Effectiveness of antimicrobial guidelines for community‐acquired pneumonia in children. Pediatrics. 2012;129(5):e1326–e1333.
Researchers from the Pediatric Research in Inpatient Settings (PRIS) network, an open pediatric hospitalist research network,[1] have identified inpatient pediatric medical and surgical conditions considered high priority for quality improvement (QI) initiatives and/or comparative effectiveness research based on prevalence, cost, and interhospital variation in resource utilization.[2] One approach for improving the quality of care within hospitals is to operationalize evidence‐based guidelines into practice.[3] Although guidelines may be used by individual clinicians, systematic adoption by hospitals into clinical workflow has the potential to influence providers to adhere to evidence‐based care, reduce unwarranted variation, and ultimately improve patient outcomes.[3, 4, 5, 6]
There are critical appraisal tools to measure the methodological quality, as defined by the Institute of Medicine (IOM) and others in their guidelines.[7, 8, 9, 10, 11, 12] One such validated tool is the AGREE II instrument, created by the AGREE (Appraisal of Guidelines for REsearch and Evaluation) collaboration.[13, 14] It defines methodological quality as the confidence that the biases linked to the rigor of development, presentation, and applicability of a clinical practice guideline have been minimized and that each step of the development process is clearly reported.[13]
The objective of our study was to rate the methodological quality of national guidelines for 20 of the PRIS priority pediatric inpatient conditions.[2] Our intent in pursuing this project was 2‐fold: first, to inform pediatric inpatient QI initiatives, and second, to call out priority pediatric inpatient conditions for which high methodological‐quality guidelines are currently lacking.
METHODS
The study methods involved (1) prioritizing pediatric inpatient conditions, (2) identifying national guidelines for the priority conditions, and (3) rating the methodological quality of available guidelines. This study was considered nonhuman‐subject research (A. Johnson, personal e‐mail communication, November 14, 2012), and the original prioritization study was deemed exempt from review by the institutional review board of the Children's Hospital of Philadelphia under 45 CFR 46.102(f).[2]
Prioritizing Pediatric Inpatient Conditions
Methods for developing the prioritization list are published elsewhere in detail and briefly described here.[1] An International Classification of Diseases, 9th Revision, Clinical Modification‐based clinical condition grouper was created for primary discharge diagnosis codes for inpatient, ambulatory surgery, and observation unit encounters accounting for either 80% of all encounters or 80% of all charges for over 3.4 million discharges from 2004 to 2009 for 38 children's hospitals in the Pediatric Health Information Systems (PHIS) database, which includes administrative and billing data.[15] A standardized cost master index was created to assign the same unit cost for each billable item (calculated as the median of median hospital unit costs) to allow for comparisons of resource utilization across hospitals (eg, the cost of a chest x‐ray was set to be the same across all hospitals in 2009 dollars). Total hospital costs were then recalculated for every admission by multiplying the standardized cost master index by the number of units for each item in the hospital bill, and then summing the standardized costs of each line item in every bill. Conditions were ranked based on prevalence and total cost across all hospitals in the study period. The variation in standardized costs across hospitals for each condition was determined.
For the current study, conditions were considered if they had a top 20 prevalence rank, a top 20 cost rank, high variation (intraclass correlation coefficient >0.1) in standardized costs across hospitals, a minimum number of PHIS hospitals with annualized overexpenditures (using the standardized cost master) of at least $50,000 when compared to the mean, or a minimum median of 200 cases per hospital over the 6‐year study period to assure sufficient hospital volume for future interventions. This resulted in 29 conditions; the selected 20 conditions matched the top 20 prevalence rank (see Supporting Information, Table 1, in the online version of this article).[2]
| Condition by PRIS Priority Rank | Guidelines Meeting Inclusion Criteriaa | Guidelines Citation | Mean Overall Reviewer Methodological Quality Rating (Rater 1, Rater 2)b | Recommended for Use in the Pediatric Inpatient Setting, Mean (Rater 1, Rater 2)c | Weighted Kappa(95% Confidence Interval) |
|---|---|---|---|---|---|
| |||||
| Otitis media, unspecified, s | 1 | American Academy of Family Physicians; American Academy of Otolaryngology‐Head and Neck Surgery; American Academy of Pediatrics Subcommittee on Otitis Media With Effusion. Clinical Practice Guidelines: Otitis media with effusion. Pediatrics. 2004 May;113(5):1412‐29. | 6 (6, 6) | 3 (3, 3) | 0.76 (0.490.93) |
| Hypertrophy of tonsils and adenoids, s | 1 | Baugh RF et al. Clinical practice guideline: tonsillectomy in children. Otolaryngol Head Neck Surg. 2011;144(1 suppl):S1S30. | 6.5 (7, 6) | 3 (3, 3) | 0.49 (0.050.81) |
| Asthma, m | 1 | National Heart, Lung, and Blood Institute; National Asthma Education and Prevention Program. Expert panel report 3 (EPR‐3): guidelines for the diagnosis and management of asthma‐full report 2007. Pages 1‐440. Available at: | 7 (7, 7) | 3 (3, 3) | 0.62 (0.210.87) |
| Bronchiolitis, m | 1 | American Academy of Pediatrics Subcommittee on Diagnosis and Management of Bronchiolitis. Diagnosis and management of bronchiolitis. Pediatrics. 2006;118:17741793. | 6.5 (6, 7) | 3 (3, 3) | 0.95 (0.871.00) |
| Pneumonia, m | 1 | Bradley JS et al.The management of community‐acquired pneumonia in infants and children older than 3 months of age: clinical practice guidelines by the Pediatric Infectious Diseases Society and the Infectious Diseases Society of America. Clin Infect Dis. 2011;53(7):e25e76. | 6 (6, 6) | 3 (3, 3) | 0.82 (0.640.96) |
| Dental caries, s | 1 | American Academy on Pediatric Dentistry Clinical Affairs CommitteePulp Therapy Subcommittee; American Academy on Pediatric Dentistry Council on Clinical Affairs. Guideline on pulp therapy for primary and young permanent teeth. Pediatr Dent. 2008;30:170174. | 3 (3, 3) | 1.5 (1, 2) | 0.51 (0.140.83) |
| Chemotherapy, m | 0 | ||||
| Cellulitis, m | 1 | Stevens DL et al. Practice guidelines for the diagnosis and management of skin and softtissue infections. Clin Infect Dis. 2005;41:13731406. | 4.5 (4, 5) | 2.5 (2, 3) | 0.52 (0.150.79) |
| Inguinal hernia, s | 0 | ||||
| Gastroesophageal reflux and esophagitis, m, s | 2 | Vandenplas Y et al. Pediatric gastroesophageal reflux clinical practice guidelines: joint recommendations of NASPGHAN and ESPGHAN. J Pediatr Gastroenterol Nutr. 2009;49(4):498547. | 5 (5, 5) | 3 (3, 3) | 0.69 (0.450.87) |
| Furuta GT et al. Eosinophilic esophagitis in children and adults: a systematic review and consensus recommendations for diagnosis and treatment. Gastroenterology. 2007;133:13421363. | 5 (5, 5) | 2.5 (2, 3) | 0.93 (0.850.98) | ||
| Dehydration, m | 0 | ||||
| Redundant prepuce and phimosis, s | 1 | American Academy of Pediatrics Task Force on Circumcision. Male circumcision. Pediatrics. 2012;130(3):e756e785. | 6 (6, 6) | 3 (3, 3) | 0.66 (0.250.89) |
| Abdominal pain, m | 0 | ||||
| Other convulsions, m | 0 | ||||
| Urinary tract infection, m | 1 | Roberts KB et al. Urinary tract infection: clinical practice guideline for the diagnosis and management of the initial UTI in febrile infants and children 2 to 24 months. Pediatrics. 2011;128:595610. | 5.5 (5, 6) | 2.5 (2, 3) | 0.62 (0.230.84) |
| Acute appendicitis without peritonitis, s | 1 | Solomkin JS et al. Diagnosis and management of complicated intra‐abdominal infection in adults and children: guidelines by the Surgical Infection Society and the Infectious Diseases Society of America. Clin Infect Dis. 2010;50:133164. | 4.5 (5, 4) | 2.5 (3, 2) | 0.37 (0.110.81) |
| Eso‐ exo‐ hetero‐, and hypertropia, s | 0 | ||||
| Fever, m | 0 | ||||
| Seizures with and without intractable epilepsy, m | 3 | Brophy GM et al; Neurocritical Care Society Status Epilepticus Guideline Writing Committee. Guidelines for the evaluation and management of status epilepticus. Neurocrit Care. 2012;17:323. | 5 (5, 5) | 3 (3, 3) | 0.95 (0.870.99) |
| Hirtz D et al. Practice parameter: treatment of the child with a first unprovoked seizure: report of the Quality Standards Subcommittee of the American Academy of Neurology and the Practice Committee of the Child Neurology Society. Neurology. 2003;60:166175. | 5 (5, 5) | 2.5 (2, 3) | 0.73 (0.410.94) | ||
| Riviello JJ Jr et al. Practice parameter: diagnostic assessment of the child with status epilepticus (an evidence‐based review): report of the Quality Standards Subcommittee of the American Academy of Neurology and the Practice Committee of the Child Neurology Society. Neurology. 2006;67:15421550. | 5 (4, 6) | 2.5 (2, 3) | 0.80 (0.630.94) | ||
| Sickle cell disease with crisis, m | 2 | Section on Hematology/Oncology Committee on Genetics; American Academy of Pediatrics. Health supervision for children with sickle cell disease. Pediatrics. 2002;109:526535. | 3.5 (3, 4) | 1.5 (1, 2) | 0.92 (0.800.98) |
| National Heart, Lung, and Blood Institute, National Institutes of Health. The management of sickle cell disease. National Heart, Lung, and Blood Institute, National Institutes of Health, Bethesda, MD. Available at: | 4 (4, 4) | 2.5 (2, 3) | 0.91 (0.800.97) | ||
Identifying National Guidelines
We developed a search protocol (see Supporting Information, Table 2, in the online version of this article) using condition‐specific keywords and the following criteria: guideline, pediatric, 2002 to 2012. A medical librarian (E.E.) used the protocol to search PubMed, National Guidelines Clearing House, and the American Academy of Pediatrics website for guidelines for the 20 selected conditions.
We limited our study to US national guidelines published or updated from 2002 to 2012 to be most relevant to the 38 US children's hospitals in the original study. Guidelines had to address either medical or surgical or both types of inpatient management for the condition, depending on how the condition was categorized on the PRIS prioritization list. For example, to target inpatient issues, otitis media was treated as a surgical condition when the prioritization list was created, therefore guidelines included in our study needed to address surgical management (ie, myringotomy or tympanostomy tubes).[2] Guidelines specific to 1 organism, test, or treatment were a priori excluded, as they would not map well to the prioritization list, and would be difficult to interpret. Guidelines focusing exclusively on condition prevention were also excluded. Guidelines with a broad subject matter (eg, abdominal infection) or unclear age were included if they contained a significant focus on the condition of interest (eg, appendicitis without peritonitis), such that the course of pediatric inpatient care was described for that condition. Retracted or outdated (superseded by a more current version) guidelines were excluded.
An investigator (G.H.) reviewed potentially relevant results from the librarian's search. For example, the search for tonsillectomy guidelines retrieved a guideline on the use of polysomnography prior to tonsillectomy in children but did not cover the inpatient management or tonsillectomy procedure.[16] This guideline was excluded from our study, as it focused on a specific test and did not discuss surgical management of the condition.
Rating Methodological Quality of Guidelines
Methodological quality of guidelines was rated with the AGREE II tool by 2 investigators (G.H. and K.N.).[13, 17] This tool has 2 overall guideline assessments and 23 subcomponents within 6 domains, reflecting many of the IOM's recommendations for methodological quality in guidelines: scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability, and editorial independence.[8, 17]
The AGREE II tool rates each of the 23 subcomponent questions using a 7‐point scale (1=strongly disagree7=strongly agree). We followed the AGREE II user's manual suggestion in rating subcomponents as 1, indicating an absence of information for that question if the question was not addressed within the guideline.[13] The AGREE II user's manual describes the option of creating standardized domain scores; however, as the objective of our study was to assess the overall methodological quality of the guideline and not to highlight particular areas of strengths/weaknesses in the domains, we elected to present raw scores only.[13]
For the overall guideline rating item 1 (Rate the overall quality of this guideline.) the AGREE II tool instructs that a score of 1 indicates lowest possible quality and 7 indicates highest possible quality.[13] As these score anchors are far apart with no guide for interpretation of intermediate results, we modified the descriptive terms on the tool to define scores <3 as low quality, scores 3 to 5 as moderate quality, and scores >5 as high quality to allow for easier interpretation of our results. We also modified the final overall recommendation score (on a 3‐point scale) from I would recommend this guideline for use to I would recommend this guideline for use in the pediatric inpatient setting.[13, 17] A score of 1 indicated to not recommend, 2 indicated to recommend with modifications, and 3 indicated to recommend without modification.
Significant discrepancies (>2‐point difference on overall rating) between the 2 raters were to be settled by consensus scoring by 3 senior investigators blinded to previous reviews, using a modified Delphi technique.[18]
Inter‐rater reliability was measured using a weighted kappa coefficient and reported using a bootstrapped method with 95% confidence intervals. Interpretation of kappa is such that 0 is the amount of agreement that would be expected by chance, and 1 is perfect agreement, with previous researchers stating scores >0.81 indicate almost perfect agreement.[19]
RESULTS
The librarian's search retrieved 2869 potential results (Figure 1). Seventeen guidelines met inclusion criteria for 13 of the 20 priority conditions. Seven conditions did not have national guidelines meeting inclusion criteria. Table 1 displays the 20 medical and surgical conditions on the modified PRIS prioritization list, including overall guideline scoring, recommendation scores, and kappa results for each guideline. The highest methodological‐quality guidelines were for asthma,[20] tonsillectomy,[21] and bronchiolitis[22] (mean overall rating 7, 6.5, and 6.5, respectively). The lowest methodological‐quality guidelines were for 2 sickle cell disease guidelines[23, 24] and 1 dental caries guideline[25] (mean overall rating 4, 3.5, and 3, respectively). Seven guidelines were rated as high overall quality, and 10 guidelines were rated as moderate overall quality. Eight of the 17 guidelines[20, 21, 22, 26, 27, 28, 29, 30] were recommended for use in the pediatric inpatient setting without modification by both reviewers. Two guidelines (for dental caries[25] and sickle cell[23]) were not recommended for use by 1 reviewer.
As an example of scoring, a national guideline for asthma had high overall scores (7 from each reviewer) and high scores across most AGREE II subcomponents. The guideline was found by both reviewers to be systematic in describing guideline development with clearly stated recommendations linked to the available evidence (including strengths and limitations) and implementation considerations.[20] Conversely, a national guideline for sickle cell disease had moderate overall scores (scores of 3 and 4) and low‐moderate scores across the majority of the subcomponent items.[23] The reviewers believe that this guideline would have been strengthened by increased transparency in guideline development, discussion of the evidence surrounding recommendations, and discussion of implementation factors. A table with detailed scoring of each guideline is available (see Supporting Information, Table 3, in the online version of this article).
Agreement between the 2 raters was almost perfect,[19] with an overall boot‐strapped weighted kappa of 0.83 (95% confidence interval 0.780.87) across 850 scores. There were no discrepancies between reviewers in overall scoring requiring consensus scoring.
DISCUSSION
Using a modified version of a published prioritization list for inpatient pediatric conditions, we found national guidelines for 13 of 20 conditions with high prevalence, cost, and interhospital variation in resource utilization. Seven conditions had no national guidelines published within the past 10 years applicable for use in the pediatric inpatient setting. Of 17 guidelines for 13 conditions, 10 had moderate and 7 had high methodological quality.
Our findings add to the literature describing methodological quality of guidelines. Many publications focus on the methodological quality of guidelines as a group and use a standardized instrument (eg, the AGREE II tool) to rate within domains (eg, domain 1: scope and purpose) across guidelines in an effort to encourage improvement in developing and reporting in guidelines.[31, 32] Our study differs in that we chose to focus on the overall quality rating of individual guidelines for specific prioritized conditions to allow hospitals to guide QI initiatives. One study that had a similar aim to ours surveyed Dutch pediatricians to select priority conditions and used the AGREE II tool to rate 17 guidelines, recommending 14 for use in the Netherlands.[33]
Identifying high methodological‐quality guidelines is only 1 in a series of steps prior to successful guideline implementation in hospitals. Other aspects of guidelines, including the strength of the evidence (eg, from randomized controlled trials) and subsequent force and clarity (eg, use of must instead of consider) of recommendations, may affect clinician or patient adherence, work processes, and ultimately patient outcomes. Strong evidence should translate into forceful and clear recommendations. Authors with the Yale Guideline Recommendation Corpus describe significant variation in reporting of guideline recommendations, and further studies have shown that the force and clarity of a recommendation is associated with adherence rates.[34, 35, 36, 37] Unfortunately, current guideline appraisal tools lack the means to score the strength of evidence, and force and clarity of recommendations.[10]
Implementation science demonstrates that there are many important factors in translating best practice into improvements in clinical care. In addition to implementation considerations such as adherence, patient preferences, and work processes, variability in methodological quality, strength of evidence, and force and clarity of recommendations may be additional reasons why evidence for the impact of guidelines on patient outcomes remains mixed in the literature.[38] One recent study found that adherence to antibiotics recommended within a national pediatric community‐acquired pneumonia guideline, which had a high methodological‐quality score in our study, did not change hospital length of stay or readmissions.[29, 39] There are several possible interpretations for this. Recommendations may not have been based upon strong evidence, research methodology assessing how adherence to recommendations impacts patient outcomes may have been limited, or the outcomes measured in current studies (such as readmission) are not the outcomes that may be improved by adherence to these recommendations (such as decreasing antimicrobial resistance). These are important considerations when hospitals are incorporating recommendations from guidelines into practice. Hospitals should assess the multiple aspects of guidelines, including methodological quality, which our study helps to identify, strength of evidence, and force and clarity of recommendations, as well as adherence, patient preferences, work processes, and key outcome measures when implementing guidelines into clinical practice. A study utilizing a robust QI methodology demonstrated that clinician adherence to several elements in an asthma guideline, which also had a high methodological‐quality score in our study, led to a significant decrease in 6‐month hospital and emergency department readmission for asthma.[6, 20]
Our study also highlights that several pediatric conditions with high prevalence, cost, and interhospital resource utilization variation lack recent national pediatric guidelines applicable to the inpatient setting. If strong evidence exists for these priority conditions, professional societies should create high methodological‐quality guidelines with strong and clear recommendations. If evidence is lacking for these priority conditions, then investigators should focus on generating research in these areas.
There are several limitations to this study. The AGREE II tool does not have a mechanism to measure the strength of evidence used in a guideline. Methodological quality of a guideline alone may not translate into improved outcomes. Conditions may have national guidelines published before 2002, institution‐specific or international guidelines, or adult guidelines that might be amenable to use in the pediatric inpatient setting but were not included in this study. Several conditions on the prioritization list are broad in nature (eg, dehydration) and may not be amenable to the creation of guidelines. Other conditions on the prioritization list (eg, chemotherapy or cellulitis) may have useful guidelines within the context of specific conditions (eg, acute lymphoblastic leukemia) or for specific organisms (eg, methicillin‐resistant Staphylococcus aureus). We elected to exclude these narrower guidelines to focus on broad and comprehensive guidelines applicable to a wider range of clinical situations. Additionally, although use of a validated tool attempts to objectively guide ratings, the rating of quality is to some degree subjective. Finally, our study used a previously published prioritization list using data from children's hospitals, and the list likely under‐represents conditions commonly managed in community hospitals (eg, hyperbilirubinemia).[2] Exclusion of these conditions was not reflective of importance or quality of available national guidelines.
CONCLUSIONS
Our study adds to recent publications on the need to prioritize conditions for QI in children's hospitals. We identified a group of moderate to high methodological‐quality national guidelines for pediatric inpatient conditions with high prevalence, cost, and variation in interhospital resource utilization. Not all prioritized conditions have national high methodological‐quality guidelines available. Hospitals should prioritize conditions with high methodological‐quality guidelines to allocate resources for QI initiatives. Professional societies should focus their efforts on producing methodologically sound guidelines for prioritized conditions currently lacking high‐quality guidelines if sufficient evidence exists.
Acknowledgements
The authors thank Christopher G. Maloney, MD, PhD, for critical review of the manuscript, and Gregory J. Stoddard, MS, for statistical support. Mr. Stoddard's work is supported by the University of Utah Study Design and Biostatistics Center, with funding in part from the National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health, through grant 8UL1TR000105 (formerly UL1RR025764).
Disclosures
Sanjay Mahant, Ron Keren, and Raj Srivastava are all Executive Council members of the Pediatric Research in Inpatient Settings (PRIS) Network. PRIS, Sanjay Mahant, Ron Keren, and Raj Srivastava are all supported by grants from the Children's Hospital Association. Sanjay Mahant is also supported by research grants from the Canadian Institute of Health Research and Physician Services Incorporated. Ron Keren and Raj Srivastava also serve as medical legal consultants. The remaining authors have no financial relationships relevant to this article to disclose.
Researchers from the Pediatric Research in Inpatient Settings (PRIS) network, an open pediatric hospitalist research network,[1] have identified inpatient pediatric medical and surgical conditions considered high priority for quality improvement (QI) initiatives and/or comparative effectiveness research based on prevalence, cost, and interhospital variation in resource utilization.[2] One approach for improving the quality of care within hospitals is to operationalize evidence‐based guidelines into practice.[3] Although guidelines may be used by individual clinicians, systematic adoption by hospitals into clinical workflow has the potential to influence providers to adhere to evidence‐based care, reduce unwarranted variation, and ultimately improve patient outcomes.[3, 4, 5, 6]
There are critical appraisal tools to measure the methodological quality, as defined by the Institute of Medicine (IOM) and others in their guidelines.[7, 8, 9, 10, 11, 12] One such validated tool is the AGREE II instrument, created by the AGREE (Appraisal of Guidelines for REsearch and Evaluation) collaboration.[13, 14] It defines methodological quality as the confidence that the biases linked to the rigor of development, presentation, and applicability of a clinical practice guideline have been minimized and that each step of the development process is clearly reported.[13]
The objective of our study was to rate the methodological quality of national guidelines for 20 of the PRIS priority pediatric inpatient conditions.[2] Our intent in pursuing this project was 2‐fold: first, to inform pediatric inpatient QI initiatives, and second, to call out priority pediatric inpatient conditions for which high methodological‐quality guidelines are currently lacking.
METHODS
The study methods involved (1) prioritizing pediatric inpatient conditions, (2) identifying national guidelines for the priority conditions, and (3) rating the methodological quality of available guidelines. This study was considered nonhuman‐subject research (A. Johnson, personal e‐mail communication, November 14, 2012), and the original prioritization study was deemed exempt from review by the institutional review board of the Children's Hospital of Philadelphia under 45 CFR 46.102(f).[2]
Prioritizing Pediatric Inpatient Conditions
Methods for developing the prioritization list are published elsewhere in detail and briefly described here.[1] An International Classification of Diseases, 9th Revision, Clinical Modification‐based clinical condition grouper was created for primary discharge diagnosis codes for inpatient, ambulatory surgery, and observation unit encounters accounting for either 80% of all encounters or 80% of all charges for over 3.4 million discharges from 2004 to 2009 for 38 children's hospitals in the Pediatric Health Information Systems (PHIS) database, which includes administrative and billing data.[15] A standardized cost master index was created to assign the same unit cost for each billable item (calculated as the median of median hospital unit costs) to allow for comparisons of resource utilization across hospitals (eg, the cost of a chest x‐ray was set to be the same across all hospitals in 2009 dollars). Total hospital costs were then recalculated for every admission by multiplying the standardized cost master index by the number of units for each item in the hospital bill, and then summing the standardized costs of each line item in every bill. Conditions were ranked based on prevalence and total cost across all hospitals in the study period. The variation in standardized costs across hospitals for each condition was determined.
For the current study, conditions were considered if they had a top 20 prevalence rank, a top 20 cost rank, high variation (intraclass correlation coefficient >0.1) in standardized costs across hospitals, a minimum number of PHIS hospitals with annualized overexpenditures (using the standardized cost master) of at least $50,000 when compared to the mean, or a minimum median of 200 cases per hospital over the 6‐year study period to assure sufficient hospital volume for future interventions. This resulted in 29 conditions; the selected 20 conditions matched the top 20 prevalence rank (see Supporting Information, Table 1, in the online version of this article).[2]
| Condition by PRIS Priority Rank | Guidelines Meeting Inclusion Criteriaa | Guidelines Citation | Mean Overall Reviewer Methodological Quality Rating (Rater 1, Rater 2)b | Recommended for Use in the Pediatric Inpatient Setting, Mean (Rater 1, Rater 2)c | Weighted Kappa(95% Confidence Interval) |
|---|---|---|---|---|---|
| |||||
| Otitis media, unspecified, s | 1 | American Academy of Family Physicians; American Academy of Otolaryngology‐Head and Neck Surgery; American Academy of Pediatrics Subcommittee on Otitis Media With Effusion. Clinical Practice Guidelines: Otitis media with effusion. Pediatrics. 2004 May;113(5):1412‐29. | 6 (6, 6) | 3 (3, 3) | 0.76 (0.490.93) |
| Hypertrophy of tonsils and adenoids, s | 1 | Baugh RF et al. Clinical practice guideline: tonsillectomy in children. Otolaryngol Head Neck Surg. 2011;144(1 suppl):S1S30. | 6.5 (7, 6) | 3 (3, 3) | 0.49 (0.050.81) |
| Asthma, m | 1 | National Heart, Lung, and Blood Institute; National Asthma Education and Prevention Program. Expert panel report 3 (EPR‐3): guidelines for the diagnosis and management of asthma‐full report 2007. Pages 1‐440. Available at: | 7 (7, 7) | 3 (3, 3) | 0.62 (0.210.87) |
| Bronchiolitis, m | 1 | American Academy of Pediatrics Subcommittee on Diagnosis and Management of Bronchiolitis. Diagnosis and management of bronchiolitis. Pediatrics. 2006;118:17741793. | 6.5 (6, 7) | 3 (3, 3) | 0.95 (0.871.00) |
| Pneumonia, m | 1 | Bradley JS et al.The management of community‐acquired pneumonia in infants and children older than 3 months of age: clinical practice guidelines by the Pediatric Infectious Diseases Society and the Infectious Diseases Society of America. Clin Infect Dis. 2011;53(7):e25e76. | 6 (6, 6) | 3 (3, 3) | 0.82 (0.640.96) |
| Dental caries, s | 1 | American Academy on Pediatric Dentistry Clinical Affairs CommitteePulp Therapy Subcommittee; American Academy on Pediatric Dentistry Council on Clinical Affairs. Guideline on pulp therapy for primary and young permanent teeth. Pediatr Dent. 2008;30:170174. | 3 (3, 3) | 1.5 (1, 2) | 0.51 (0.140.83) |
| Chemotherapy, m | 0 | ||||
| Cellulitis, m | 1 | Stevens DL et al. Practice guidelines for the diagnosis and management of skin and softtissue infections. Clin Infect Dis. 2005;41:13731406. | 4.5 (4, 5) | 2.5 (2, 3) | 0.52 (0.150.79) |
| Inguinal hernia, s | 0 | ||||
| Gastroesophageal reflux and esophagitis, m, s | 2 | Vandenplas Y et al. Pediatric gastroesophageal reflux clinical practice guidelines: joint recommendations of NASPGHAN and ESPGHAN. J Pediatr Gastroenterol Nutr. 2009;49(4):498547. | 5 (5, 5) | 3 (3, 3) | 0.69 (0.450.87) |
| Furuta GT et al. Eosinophilic esophagitis in children and adults: a systematic review and consensus recommendations for diagnosis and treatment. Gastroenterology. 2007;133:13421363. | 5 (5, 5) | 2.5 (2, 3) | 0.93 (0.850.98) | ||
| Dehydration, m | 0 | ||||
| Redundant prepuce and phimosis, s | 1 | American Academy of Pediatrics Task Force on Circumcision. Male circumcision. Pediatrics. 2012;130(3):e756e785. | 6 (6, 6) | 3 (3, 3) | 0.66 (0.250.89) |
| Abdominal pain, m | 0 | ||||
| Other convulsions, m | 0 | ||||
| Urinary tract infection, m | 1 | Roberts KB et al. Urinary tract infection: clinical practice guideline for the diagnosis and management of the initial UTI in febrile infants and children 2 to 24 months. Pediatrics. 2011;128:595610. | 5.5 (5, 6) | 2.5 (2, 3) | 0.62 (0.230.84) |
| Acute appendicitis without peritonitis, s | 1 | Solomkin JS et al. Diagnosis and management of complicated intra‐abdominal infection in adults and children: guidelines by the Surgical Infection Society and the Infectious Diseases Society of America. Clin Infect Dis. 2010;50:133164. | 4.5 (5, 4) | 2.5 (3, 2) | 0.37 (0.110.81) |
| Eso‐ exo‐ hetero‐, and hypertropia, s | 0 | ||||
| Fever, m | 0 | ||||
| Seizures with and without intractable epilepsy, m | 3 | Brophy GM et al; Neurocritical Care Society Status Epilepticus Guideline Writing Committee. Guidelines for the evaluation and management of status epilepticus. Neurocrit Care. 2012;17:323. | 5 (5, 5) | 3 (3, 3) | 0.95 (0.870.99) |
| Hirtz D et al. Practice parameter: treatment of the child with a first unprovoked seizure: report of the Quality Standards Subcommittee of the American Academy of Neurology and the Practice Committee of the Child Neurology Society. Neurology. 2003;60:166175. | 5 (5, 5) | 2.5 (2, 3) | 0.73 (0.410.94) | ||
| Riviello JJ Jr et al. Practice parameter: diagnostic assessment of the child with status epilepticus (an evidence‐based review): report of the Quality Standards Subcommittee of the American Academy of Neurology and the Practice Committee of the Child Neurology Society. Neurology. 2006;67:15421550. | 5 (4, 6) | 2.5 (2, 3) | 0.80 (0.630.94) | ||
| Sickle cell disease with crisis, m | 2 | Section on Hematology/Oncology Committee on Genetics; American Academy of Pediatrics. Health supervision for children with sickle cell disease. Pediatrics. 2002;109:526535. | 3.5 (3, 4) | 1.5 (1, 2) | 0.92 (0.800.98) |
| National Heart, Lung, and Blood Institute, National Institutes of Health. The management of sickle cell disease. National Heart, Lung, and Blood Institute, National Institutes of Health, Bethesda, MD. Available at: | 4 (4, 4) | 2.5 (2, 3) | 0.91 (0.800.97) | ||
Identifying National Guidelines
We developed a search protocol (see Supporting Information, Table 2, in the online version of this article) using condition‐specific keywords and the following criteria: guideline, pediatric, 2002 to 2012. A medical librarian (E.E.) used the protocol to search PubMed, National Guidelines Clearing House, and the American Academy of Pediatrics website for guidelines for the 20 selected conditions.
We limited our study to US national guidelines published or updated from 2002 to 2012 to be most relevant to the 38 US children's hospitals in the original study. Guidelines had to address either medical or surgical or both types of inpatient management for the condition, depending on how the condition was categorized on the PRIS prioritization list. For example, to target inpatient issues, otitis media was treated as a surgical condition when the prioritization list was created, therefore guidelines included in our study needed to address surgical management (ie, myringotomy or tympanostomy tubes).[2] Guidelines specific to 1 organism, test, or treatment were a priori excluded, as they would not map well to the prioritization list, and would be difficult to interpret. Guidelines focusing exclusively on condition prevention were also excluded. Guidelines with a broad subject matter (eg, abdominal infection) or unclear age were included if they contained a significant focus on the condition of interest (eg, appendicitis without peritonitis), such that the course of pediatric inpatient care was described for that condition. Retracted or outdated (superseded by a more current version) guidelines were excluded.
An investigator (G.H.) reviewed potentially relevant results from the librarian's search. For example, the search for tonsillectomy guidelines retrieved a guideline on the use of polysomnography prior to tonsillectomy in children but did not cover the inpatient management or tonsillectomy procedure.[16] This guideline was excluded from our study, as it focused on a specific test and did not discuss surgical management of the condition.
Rating Methodological Quality of Guidelines
Methodological quality of guidelines was rated with the AGREE II tool by 2 investigators (G.H. and K.N.).[13, 17] This tool has 2 overall guideline assessments and 23 subcomponents within 6 domains, reflecting many of the IOM's recommendations for methodological quality in guidelines: scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability, and editorial independence.[8, 17]
The AGREE II tool rates each of the 23 subcomponent questions using a 7‐point scale (1=strongly disagree7=strongly agree). We followed the AGREE II user's manual suggestion in rating subcomponents as 1, indicating an absence of information for that question if the question was not addressed within the guideline.[13] The AGREE II user's manual describes the option of creating standardized domain scores; however, as the objective of our study was to assess the overall methodological quality of the guideline and not to highlight particular areas of strengths/weaknesses in the domains, we elected to present raw scores only.[13]
For the overall guideline rating item 1 (Rate the overall quality of this guideline.) the AGREE II tool instructs that a score of 1 indicates lowest possible quality and 7 indicates highest possible quality.[13] As these score anchors are far apart with no guide for interpretation of intermediate results, we modified the descriptive terms on the tool to define scores <3 as low quality, scores 3 to 5 as moderate quality, and scores >5 as high quality to allow for easier interpretation of our results. We also modified the final overall recommendation score (on a 3‐point scale) from I would recommend this guideline for use to I would recommend this guideline for use in the pediatric inpatient setting.[13, 17] A score of 1 indicated to not recommend, 2 indicated to recommend with modifications, and 3 indicated to recommend without modification.
Significant discrepancies (>2‐point difference on overall rating) between the 2 raters were to be settled by consensus scoring by 3 senior investigators blinded to previous reviews, using a modified Delphi technique.[18]
Inter‐rater reliability was measured using a weighted kappa coefficient and reported using a bootstrapped method with 95% confidence intervals. Interpretation of kappa is such that 0 is the amount of agreement that would be expected by chance, and 1 is perfect agreement, with previous researchers stating scores >0.81 indicate almost perfect agreement.[19]
RESULTS
The librarian's search retrieved 2869 potential results (Figure 1). Seventeen guidelines met inclusion criteria for 13 of the 20 priority conditions. Seven conditions did not have national guidelines meeting inclusion criteria. Table 1 displays the 20 medical and surgical conditions on the modified PRIS prioritization list, including overall guideline scoring, recommendation scores, and kappa results for each guideline. The highest methodological‐quality guidelines were for asthma,[20] tonsillectomy,[21] and bronchiolitis[22] (mean overall rating 7, 6.5, and 6.5, respectively). The lowest methodological‐quality guidelines were for 2 sickle cell disease guidelines[23, 24] and 1 dental caries guideline[25] (mean overall rating 4, 3.5, and 3, respectively). Seven guidelines were rated as high overall quality, and 10 guidelines were rated as moderate overall quality. Eight of the 17 guidelines[20, 21, 22, 26, 27, 28, 29, 30] were recommended for use in the pediatric inpatient setting without modification by both reviewers. Two guidelines (for dental caries[25] and sickle cell[23]) were not recommended for use by 1 reviewer.
As an example of scoring, a national guideline for asthma had high overall scores (7 from each reviewer) and high scores across most AGREE II subcomponents. The guideline was found by both reviewers to be systematic in describing guideline development with clearly stated recommendations linked to the available evidence (including strengths and limitations) and implementation considerations.[20] Conversely, a national guideline for sickle cell disease had moderate overall scores (scores of 3 and 4) and low‐moderate scores across the majority of the subcomponent items.[23] The reviewers believe that this guideline would have been strengthened by increased transparency in guideline development, discussion of the evidence surrounding recommendations, and discussion of implementation factors. A table with detailed scoring of each guideline is available (see Supporting Information, Table 3, in the online version of this article).
Agreement between the 2 raters was almost perfect,[19] with an overall boot‐strapped weighted kappa of 0.83 (95% confidence interval 0.780.87) across 850 scores. There were no discrepancies between reviewers in overall scoring requiring consensus scoring.
DISCUSSION
Using a modified version of a published prioritization list for inpatient pediatric conditions, we found national guidelines for 13 of 20 conditions with high prevalence, cost, and interhospital variation in resource utilization. Seven conditions had no national guidelines published within the past 10 years applicable for use in the pediatric inpatient setting. Of 17 guidelines for 13 conditions, 10 had moderate and 7 had high methodological quality.
Our findings add to the literature describing methodological quality of guidelines. Many publications focus on the methodological quality of guidelines as a group and use a standardized instrument (eg, the AGREE II tool) to rate within domains (eg, domain 1: scope and purpose) across guidelines in an effort to encourage improvement in developing and reporting in guidelines.[31, 32] Our study differs in that we chose to focus on the overall quality rating of individual guidelines for specific prioritized conditions to allow hospitals to guide QI initiatives. One study that had a similar aim to ours surveyed Dutch pediatricians to select priority conditions and used the AGREE II tool to rate 17 guidelines, recommending 14 for use in the Netherlands.[33]
Identifying high methodological‐quality guidelines is only 1 in a series of steps prior to successful guideline implementation in hospitals. Other aspects of guidelines, including the strength of the evidence (eg, from randomized controlled trials) and subsequent force and clarity (eg, use of must instead of consider) of recommendations, may affect clinician or patient adherence, work processes, and ultimately patient outcomes. Strong evidence should translate into forceful and clear recommendations. Authors with the Yale Guideline Recommendation Corpus describe significant variation in reporting of guideline recommendations, and further studies have shown that the force and clarity of a recommendation is associated with adherence rates.[34, 35, 36, 37] Unfortunately, current guideline appraisal tools lack the means to score the strength of evidence, and force and clarity of recommendations.[10]
Implementation science demonstrates that there are many important factors in translating best practice into improvements in clinical care. In addition to implementation considerations such as adherence, patient preferences, and work processes, variability in methodological quality, strength of evidence, and force and clarity of recommendations may be additional reasons why evidence for the impact of guidelines on patient outcomes remains mixed in the literature.[38] One recent study found that adherence to antibiotics recommended within a national pediatric community‐acquired pneumonia guideline, which had a high methodological‐quality score in our study, did not change hospital length of stay or readmissions.[29, 39] There are several possible interpretations for this. Recommendations may not have been based upon strong evidence, research methodology assessing how adherence to recommendations impacts patient outcomes may have been limited, or the outcomes measured in current studies (such as readmission) are not the outcomes that may be improved by adherence to these recommendations (such as decreasing antimicrobial resistance). These are important considerations when hospitals are incorporating recommendations from guidelines into practice. Hospitals should assess the multiple aspects of guidelines, including methodological quality, which our study helps to identify, strength of evidence, and force and clarity of recommendations, as well as adherence, patient preferences, work processes, and key outcome measures when implementing guidelines into clinical practice. A study utilizing a robust QI methodology demonstrated that clinician adherence to several elements in an asthma guideline, which also had a high methodological‐quality score in our study, led to a significant decrease in 6‐month hospital and emergency department readmission for asthma.[6, 20]
Our study also highlights that several pediatric conditions with high prevalence, cost, and interhospital resource utilization variation lack recent national pediatric guidelines applicable to the inpatient setting. If strong evidence exists for these priority conditions, professional societies should create high methodological‐quality guidelines with strong and clear recommendations. If evidence is lacking for these priority conditions, then investigators should focus on generating research in these areas.
There are several limitations to this study. The AGREE II tool does not have a mechanism to measure the strength of evidence used in a guideline. Methodological quality of a guideline alone may not translate into improved outcomes. Conditions may have national guidelines published before 2002, institution‐specific or international guidelines, or adult guidelines that might be amenable to use in the pediatric inpatient setting but were not included in this study. Several conditions on the prioritization list are broad in nature (eg, dehydration) and may not be amenable to the creation of guidelines. Other conditions on the prioritization list (eg, chemotherapy or cellulitis) may have useful guidelines within the context of specific conditions (eg, acute lymphoblastic leukemia) or for specific organisms (eg, methicillin‐resistant Staphylococcus aureus). We elected to exclude these narrower guidelines to focus on broad and comprehensive guidelines applicable to a wider range of clinical situations. Additionally, although use of a validated tool attempts to objectively guide ratings, the rating of quality is to some degree subjective. Finally, our study used a previously published prioritization list using data from children's hospitals, and the list likely under‐represents conditions commonly managed in community hospitals (eg, hyperbilirubinemia).[2] Exclusion of these conditions was not reflective of importance or quality of available national guidelines.
CONCLUSIONS
Our study adds to recent publications on the need to prioritize conditions for QI in children's hospitals. We identified a group of moderate to high methodological‐quality national guidelines for pediatric inpatient conditions with high prevalence, cost, and variation in interhospital resource utilization. Not all prioritized conditions have national high methodological‐quality guidelines available. Hospitals should prioritize conditions with high methodological‐quality guidelines to allocate resources for QI initiatives. Professional societies should focus their efforts on producing methodologically sound guidelines for prioritized conditions currently lacking high‐quality guidelines if sufficient evidence exists.
Acknowledgements
The authors thank Christopher G. Maloney, MD, PhD, for critical review of the manuscript, and Gregory J. Stoddard, MS, for statistical support. Mr. Stoddard's work is supported by the University of Utah Study Design and Biostatistics Center, with funding in part from the National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health, through grant 8UL1TR000105 (formerly UL1RR025764).
Disclosures
Sanjay Mahant, Ron Keren, and Raj Srivastava are all Executive Council members of the Pediatric Research in Inpatient Settings (PRIS) Network. PRIS, Sanjay Mahant, Ron Keren, and Raj Srivastava are all supported by grants from the Children's Hospital Association. Sanjay Mahant is also supported by research grants from the Canadian Institute of Health Research and Physician Services Incorporated. Ron Keren and Raj Srivastava also serve as medical legal consultants. The remaining authors have no financial relationships relevant to this article to disclose.
- , . Development of the Pediatric Research in Inpatient Settings (PRIS) Network: lessons learned. J Hosp Med. 2012;7(8):661–664.
- , , , et al. Prioritization of comparative effectiveness research topics in hospital pediatrics. Arch Pediatr Adolesc Med. 2012;166(12):1–10.
- , . How Intermountain trimmed health care costs through robust quality improvement efforts. Health Aff (Millwood). 2011;30(6):1185–1191.
- . Making it easy to do it right. N Engl J Med. 2001;345(13):991–993.
- , , , , . Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ. 1999;318(7182):527–530.
- , , , et al. The Joint Commission Children's Asthma Care quality measures and asthma readmissions. Pediatrics. 2012;130(3):482–491.
- , , , et al. Toward transparent clinical policies. Pediatrics. 2008;121(3):643–646.
- Field MJ, Lohr KN, eds.; Committee to Advise the Public Health Service on Clinical Practice Guidelines, Institute of Medicine. Clinical Practice Guidelines: Directions for a New Program. Washington, DC: National Academies Press; 1990.
- , , , , . Development and application of a generic methodology to assess the quality of clinical guidelines. Int J Qual Health Care. 1999;11(1):21–28.
- , , , , . A systematic review of appraisal tools for clinical practice guidelines: multiple similarities and one common deficit. Int J Qual Health Care. 2005;17(3):235–242.
- , , . How to decide whether a clinical practice guideline is trustworthy. JAMA. 2013;309(2):139–140.
- Field MJ, Lohr KN, eds.; Committee to Advise the Public Health Service on Clinical Practice Guidelines, Institute of Medicine. Guidelines for Clinical Practice: From Development to Use. Washington, DC: National Academies Press; 1992.
- The AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care. 2003;12:18–23.
- . AGREE II‐improving the quality of clinical care. Lancet. 2010;376(9747):1128–1129.
- , , , . Corticosteroids and mortality in children with bacterial meningitis. JAMA. 2008;299(17):2048–2055.
- , , , et al. Clinical practice guideline: polysomnography for sleep‐disordered breathing prior to tonsillectomy in children. Otolaryngol Head Neck Surg. 2011;145(1 suppl):S1–S15.
- , , , et al. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ. 2010;182(18):E839–E842.
- , . The Delphi method as a research tool: an example, design considerations and applications. Inform Manag. 2004;42(1):15–29.
- , . The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174.
- National Heart, Lung, and Blood Institute; National Asthma Education and Prevention Program. Expert panel report 3 (EPR‐3): guidelines for the diagnosis and management of asthma‐full report 2007. Pages 1–440. Available at: http://www.nhlbi.nih.gov/guidelines/asthma/asthgdln.pdf. Accessed on 24 August 2012.
- , , , et al. Clinical practice guideline: tonsillectomy in children. Otolaryngol Head Neck Surg. 2011;144(1 suppl):S1–S30.
- American Academy of Pediatrics Subcommittee on Diagnosis and Management of Bronchiolitis. Diagnosis and management of bronchiolitis. Pediatrics. 2006;118:1774–1793.
- Health supervision for children with sickle cell disease. Pediatrics. 2002;109(3):526–535.
- National Heart, Lung, and Blood Institute, National Institutes of Health. The management of sickle cell disease. National Heart, Lung, and Blood Institute, National Institutes of Health, Bethesda, MD. Available at: http://www.nhlbi.nih.gov/health/prof/blood/sickle/sc_mngt.pdf. Revised June 2002. Accessed on 13 October 2012.
- American Academy on Pediatric Dentistry Clinical Affairs Committee–Pulp Therapy Subcommittee; American Academy on Pediatric Dentistry Council on Clinical Affairs. Guideline on pulp therapy for primary and young permanent teeth. Pediatr Dent. 2008;30:170–174.
- , , , et al.; Neurocritical Care Society Status Epilepticus Guideline Writing Committee. Guidelines for the evaluation and management of status epilepticus. Neurocrit Care. 2012;17:3–23.
- American Academy of Pediatrics Task Force on Circumcision. Male circumcision. Pediatrics. 2012;130(3):e756–e785.
- , , , et al. Pediatric gastroesophageal reflux clinical practice guidelines: joint recommendations of the North American Society for Pediatric Gastroenterology, Hepatology, and Nutrition (NASPGHAN) and the European Society for Pediatric Gastroenterology, Hepatology, and Nutrition (ESPGHAN). J Pediatr Gastroenterol Nutr. 2009;49(4):498–547.
- , , , et al. The management of community‐acquired pneumonia in infants and children older than 3 months of age: clinical practice guidelines by the Pediatric Infectious Diseases Society and the Infectious Diseases Society of America. Clin Infect Dis. 2011;53(7):e25–e76.
- American Academy of Family Physicians; American Academy of Otolaryngology‐Head and Neck Surgery; American Academy of Pediatrics Subcommittee on Otitis Media With Effusion. Clinical Practice Guidelines: Otitis media with effusion. Pediatrics. 2004 May;113(5):1412–29.
- , , . Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer‐reviewed medical literature. JAMA. 1999;281(20):1900–1905.
- , , , . Quality of reporting and evidence in American Academy of Pediatrics guidelines. Pediatrics. 2013;131(4):732–738.
- , , . Quality of evidence‐based pediatric guidelines. Pediatrics. 2005;115(5):1378–1391.
- , , . The Yale Guideline Recommendation Corpus: a representative sample of the knowledge content of guidelines. Int J Med Inform. 2009;78(5):354–363.
- , , . How often is strength of recommendation indicated in guidelines? Analysis of the Yale Guideline Recommendation Corpus. AMIA Annu Symp Proc. 2008:984.
- , , . Clinical practice guideline development manual, third edition: a quality‐driven approach for translating evidence into action. Otolaryngol Head Neck Surg. 2013;148(1 suppl):S1–S55.
- , , , , , . Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. BMJ. 1998;317(7162):858–861.
- , , , et al. Toward evidence‐based quality improvement. Evidence (and its limitations) of the effectiveness of guideline dissemination and implementation strategies 1966–1998. J Gen Intern Med. 2006;21(suppl 2):S14–S20.
- , , , . Effectiveness of antimicrobial guidelines for community‐acquired pneumonia in children. Pediatrics. 2012;129(5):e1326–e1333.
- , . Development of the Pediatric Research in Inpatient Settings (PRIS) Network: lessons learned. J Hosp Med. 2012;7(8):661–664.
- , , , et al. Prioritization of comparative effectiveness research topics in hospital pediatrics. Arch Pediatr Adolesc Med. 2012;166(12):1–10.
- , . How Intermountain trimmed health care costs through robust quality improvement efforts. Health Aff (Millwood). 2011;30(6):1185–1191.
- . Making it easy to do it right. N Engl J Med. 2001;345(13):991–993.
- , , , , . Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ. 1999;318(7182):527–530.
- , , , et al. The Joint Commission Children's Asthma Care quality measures and asthma readmissions. Pediatrics. 2012;130(3):482–491.
- , , , et al. Toward transparent clinical policies. Pediatrics. 2008;121(3):643–646.
- Field MJ, Lohr KN, eds.; Committee to Advise the Public Health Service on Clinical Practice Guidelines, Institute of Medicine. Clinical Practice Guidelines: Directions for a New Program. Washington, DC: National Academies Press; 1990.
- , , , , . Development and application of a generic methodology to assess the quality of clinical guidelines. Int J Qual Health Care. 1999;11(1):21–28.
- , , , , . A systematic review of appraisal tools for clinical practice guidelines: multiple similarities and one common deficit. Int J Qual Health Care. 2005;17(3):235–242.
- , , . How to decide whether a clinical practice guideline is trustworthy. JAMA. 2013;309(2):139–140.
- Field MJ, Lohr KN, eds.; Committee to Advise the Public Health Service on Clinical Practice Guidelines, Institute of Medicine. Guidelines for Clinical Practice: From Development to Use. Washington, DC: National Academies Press; 1992.
- The AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care. 2003;12:18–23.
- . AGREE II‐improving the quality of clinical care. Lancet. 2010;376(9747):1128–1129.
- , , , . Corticosteroids and mortality in children with bacterial meningitis. JAMA. 2008;299(17):2048–2055.
- , , , et al. Clinical practice guideline: polysomnography for sleep‐disordered breathing prior to tonsillectomy in children. Otolaryngol Head Neck Surg. 2011;145(1 suppl):S1–S15.
- , , , et al. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ. 2010;182(18):E839–E842.
- , . The Delphi method as a research tool: an example, design considerations and applications. Inform Manag. 2004;42(1):15–29.
- , . The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174.
- National Heart, Lung, and Blood Institute; National Asthma Education and Prevention Program. Expert panel report 3 (EPR‐3): guidelines for the diagnosis and management of asthma‐full report 2007. Pages 1–440. Available at: http://www.nhlbi.nih.gov/guidelines/asthma/asthgdln.pdf. Accessed on 24 August 2012.
- , , , et al. Clinical practice guideline: tonsillectomy in children. Otolaryngol Head Neck Surg. 2011;144(1 suppl):S1–S30.
- American Academy of Pediatrics Subcommittee on Diagnosis and Management of Bronchiolitis. Diagnosis and management of bronchiolitis. Pediatrics. 2006;118:1774–1793.
- Health supervision for children with sickle cell disease. Pediatrics. 2002;109(3):526–535.
- National Heart, Lung, and Blood Institute, National Institutes of Health. The management of sickle cell disease. National Heart, Lung, and Blood Institute, National Institutes of Health, Bethesda, MD. Available at: http://www.nhlbi.nih.gov/health/prof/blood/sickle/sc_mngt.pdf. Revised June 2002. Accessed on 13 October 2012.
- American Academy on Pediatric Dentistry Clinical Affairs Committee–Pulp Therapy Subcommittee; American Academy on Pediatric Dentistry Council on Clinical Affairs. Guideline on pulp therapy for primary and young permanent teeth. Pediatr Dent. 2008;30:170–174.
- , , , et al.; Neurocritical Care Society Status Epilepticus Guideline Writing Committee. Guidelines for the evaluation and management of status epilepticus. Neurocrit Care. 2012;17:3–23.
- American Academy of Pediatrics Task Force on Circumcision. Male circumcision. Pediatrics. 2012;130(3):e756–e785.
- , , , et al. Pediatric gastroesophageal reflux clinical practice guidelines: joint recommendations of the North American Society for Pediatric Gastroenterology, Hepatology, and Nutrition (NASPGHAN) and the European Society for Pediatric Gastroenterology, Hepatology, and Nutrition (ESPGHAN). J Pediatr Gastroenterol Nutr. 2009;49(4):498–547.
- , , , et al. The management of community‐acquired pneumonia in infants and children older than 3 months of age: clinical practice guidelines by the Pediatric Infectious Diseases Society and the Infectious Diseases Society of America. Clin Infect Dis. 2011;53(7):e25–e76.
- American Academy of Family Physicians; American Academy of Otolaryngology‐Head and Neck Surgery; American Academy of Pediatrics Subcommittee on Otitis Media With Effusion. Clinical Practice Guidelines: Otitis media with effusion. Pediatrics. 2004 May;113(5):1412–29.
- , , . Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer‐reviewed medical literature. JAMA. 1999;281(20):1900–1905.
- , , , . Quality of reporting and evidence in American Academy of Pediatrics guidelines. Pediatrics. 2013;131(4):732–738.
- , , . Quality of evidence‐based pediatric guidelines. Pediatrics. 2005;115(5):1378–1391.
- , , . The Yale Guideline Recommendation Corpus: a representative sample of the knowledge content of guidelines. Int J Med Inform. 2009;78(5):354–363.
- , , . How often is strength of recommendation indicated in guidelines? Analysis of the Yale Guideline Recommendation Corpus. AMIA Annu Symp Proc. 2008:984.
- , , . Clinical practice guideline development manual, third edition: a quality‐driven approach for translating evidence into action. Otolaryngol Head Neck Surg. 2013;148(1 suppl):S1–S55.
- , , , , , . Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. BMJ. 1998;317(7162):858–861.
- , , , et al. Toward evidence‐based quality improvement. Evidence (and its limitations) of the effectiveness of guideline dissemination and implementation strategies 1966–1998. J Gen Intern Med. 2006;21(suppl 2):S14–S20.
- , , , . Effectiveness of antimicrobial guidelines for community‐acquired pneumonia in children. Pediatrics. 2012;129(5):e1326–e1333.
© 2014 Society of Hospital Medicine
