Studying Systematic Reviews

Two important systematic reviews of the literature appear in this issue of JFP. Smucny and colleagues¹ address the question of whether people with cough following an acute respiratory infection should be routinely offered b-agonists. Scholten and coworkers² ask which clinical examination maneuvers of the knee are accurate for diagnosing meniscus lesions. These questions are important, because the answers might change the way we practice. What do they mean, and what should we make of the increasing number of reviews of the literature? Our commentary is designed to provide a short description of systematic reviews, using these 2 papers as examples. First we will look at the requirements for a good systematic review and then see how these 2 papers matched up.

What characterizes a good systematic review?

Reviews of the literature are designed for clinicians who need to know what information published research can provide for patient care but who do not have the time (or the skills) to search it out for themselves. Systematic reviews are designed to minimize the biases of the reviewer, which are all too easily introduced unless the data coming from the literature are treated with the same objectivity and respect as the data in a primary trial. If certain questions are not addressed, the opinions of the reviewer may unduly influence his conclusions.³

Did the review look at all the evidence?

What does it matter if one study is left out? The answer is that it might introduce bias if the omission is because of the study’s results and not mere chance. Perhaps trials of b-agonists for acute bronchitis were more likely to be published if they were positive than if they were negative. Negative trials might only be found in the “grey literature”—that is, research that may not have been indexed into formal peer-reviewed journals, making it harder to locate-which would give positive papers exaggerated prominence. Omitting negative studies would give the impression that b-agonists were more effective than they really are.

The paper by Smucny and colleagues describes a very adequate electronic search of the literature (including the standard collections of intervention trials, such as The Cochrane Library, MEDLINE, and EMBASE), conference proceedings, and other similar collections. They also wrote to people who might know of unpublished research. This is a very thorough investigation of the grey literature. Scholten and coworkers limited their search to English, French, German, and Dutch, without attempting any grey literature search (although they obtained more information from the authors of one study to clear up some uncertainty). Does this create a potential bias? The answer lies in trying to decide to what extent this might have caused a bias. If Thai research written in Thai was excluded, but Thai research written in English was included,⁴ then there is such a potential. But perhaps most scientific medical papers in Thailand are written in English. A question mark remains.

Were studies properly selected on the criteria of quality?

Papers must be selected on quality rather than anything else. One sure test of objectivity is whether the methods description allows replication of the review by another group. Both these papers describe a very clear process; criteria were set before the process was started. If we go to the JFP Web site, www.jfponline.com, we can check each paper’s quality score. The way individual investigators each scored the papers independently should minimize any systematic bias. The paper by Smucny and colleagues checked each paper for randomization, blinding, and loss of patients from the analysis. These have been shown to be the most important issues to assess in randomized controlled trials (Shulz). Since the Scholten and coworkers paper is about diagnostic accuracy we should look for different items, particularly in the selection of patients, the “gold standard,” and any loss to follow-up of patients. It is always hard to know whether the positive results were really positive. So it is important to have robust gold standards against which the physical examinations for all patients were independently compared. These consisted of arthrotomy, arthroscopy, or magnetic resonance imaging. They sensibly decided to leave out studies on the dead: It would be difficult to know if such studies could generalize to the living.

Data Extraction

The next stage in a systematic review is the extraction of data from the studies to compare them with each other. Clearly it is important to decide on the right things to extract. Smucny and colleagues only included outcomes that were of direct patient relevance, such as cough (presumably dropping indirect outcomes, such as respiratory function tests). Scholten and coworkers looked at the sensitivity, specificity, and other measures of diagnostic accuracy for estimating joint effusion, joint line tenderness, and the McMurray test.

Could the Data Be Combined?

Sometimes data cannot be combined, because the studies they came from were not comparable. This is called heterogeneity, which can be tested for statistically. If present and not readily explicable, it renders any combination of the data (meta-analysis) suspect. In the paper by Smucny and colleagues, differences in the administration of the b-agonist (oral or inhaled), age group (children, adults), and previous illness (most, but not all, were free of previous wheeze or abnormal respiratory function tests) suggested that no data combinations were permissible other than for cough symptoms scores. The paper by Scholten and coworkers found heterogeneity of the results, and this was not easily explicable by any of the factors—study quality, setting, spectrum of disease, prevalence, and year of publication—included in the meta-regressions. Thus, they rightly express caution about interpreting this analysis.

What Did the Results Show? What Does This Mean for Clinical Practice?

Of the 7 trials of acceptable quality in the paper by Smucny and colleagues, neither of the 2 studies of children showed any benefit from b-agonists, but did find an increase in side effects (shakiness). Among the adult studies, 4 of 5 showed benefits for the b-agonists, but at the cost of increased shakiness or tremor in 3 of them. Looking at the size of the benefit, the standardized mean difference (this is the mean difference adjusted by the variance and is a way to combine different measurement scales, eg, measures of cough severity scaled from 0 to 4, 0 to 7, and 0 to 10) of the cough score was slightly worse for children in the b-agonist group than the control group. For adults, at least it was in the direction of benefits for those using b-agonists, but the standardized mean improvement in cough score on successive days varied between 0.05 to 0.17. This is less than the “small” attached to a standardized mean difference of 0.2, with 0.5 being “moderate” and 0.8 being “large.” Since the 95% confidence intervals for all changes cross 0, this is not clinically or statistically significant. Combining the data (Table 4 in that article) does not improve the power sufficiently to reach statistical significance. It is difficult to convert these standardized mean differences into clinically interpretable meaning, but it certainly appears that even if the changes were statistically significant, they would not be clinically significant. Therefore it seems reasonable to conclude that research to date does not support the use of b-agonists for acute cough of upper respiratory infections or acute bronchitis. Perhaps the different trends in children rather than adults were because the adults were more likely to have chronic chest disease (eg, more smokers, more had evidence of reversible airway obstruction).

Were there weaknesses of the paper? First, there was mixed use of oral (5 trials) and inhaled (2 trials) b-agonist. This might be expected to increase the rate of side effects relative to any benefits. Second, there is some suggestion that people who might have chronic chest disease, including asthma, do respond. This is not surprising.

The study by Scholten and coworkers is more complicated. The ability of clinicians to accurately detect a damaged meniscus by evaluating joint effusion, the McMurray test, joint line tenderness, and the Apley compression test is disappointingly poor (Table 2 of that article). These results have been combined where possible and are best understood using Figure 2. Imagine a patient coming in with a knee injury. Based on the history and before doing the McMurray test, you estimate the chance of this patient having meniscus damage is 50%. Reading up from 50% of the “prior probability” of meniscus damage on the x-axis, if the McMurray test result is positive, the probability moves only to 70% (read off the y-axis); if negative, it drops only to 35%. In this situation, none of these tests will satisfactorily rule-in or rule-out the chance of meniscus lesions.

It must be remembered that these studies were all conducted in specialist clinics. This might have 2 effects: (1) the physicians there may be better at interpreting the clinical examination (being more specialized); and (2) there are more patients with meniscus damage. To get a feel for this, imagine you are a family physician with a person with a sore knee. The chance there is meniscus damage might be only 10%; a positive McMurray test result will increase that probability to 27%, while a negative result will decrease it to 3%. A positive result is not very useful, but a negative one might support a decision to delay further testing and take a more conservative approach to the evaluation, since the likelihood of meniscal lesion is so small.

Conclusions

We hope that we have shown that carefully studying systematic reviews is both entertaining and instructive for clinical practice.* (Online tutorial) We have decided to stop treating patients with acute cough with their upper respiratory infections with b-agonists unless we hear wheezing or have a strong impression of previous asthma. Further research is unlikely to change this. We now regard the presence or absence of signs in people with injured knees with considerable skepticism and accept that there is even more uncertainty than we did before but await better quality research.

Two important systematic reviews of the literature appear in this issue of JFP. Smucny and colleagues¹ address the question of whether people with cough following an acute respiratory infection should be routinely offered b-agonists. Scholten and coworkers² ask which clinical examination maneuvers of the knee are accurate for diagnosing meniscus lesions. These questions are important, because the answers might change the way we practice. What do they mean, and what should we make of the increasing number of reviews of the literature? Our commentary is designed to provide a short description of systematic reviews, using these 2 papers as examples. First we will look at the requirements for a good systematic review and then see how these 2 papers matched up.

What characterizes a good systematic review?

Reviews of the literature are designed for clinicians who need to know what information published research can provide for patient care but who do not have the time (or the skills) to search it out for themselves. Systematic reviews are designed to minimize the biases of the reviewer, which are all too easily introduced unless the data coming from the literature are treated with the same objectivity and respect as the data in a primary trial. If certain questions are not addressed, the opinions of the reviewer may unduly influence his conclusions.³

Did the review look at all the evidence?

What does it matter if one study is left out? The answer is that it might introduce bias if the omission is because of the study’s results and not mere chance. Perhaps trials of b-agonists for acute bronchitis were more likely to be published if they were positive than if they were negative. Negative trials might only be found in the “grey literature”—that is, research that may not have been indexed into formal peer-reviewed journals, making it harder to locate-which would give positive papers exaggerated prominence. Omitting negative studies would give the impression that b-agonists were more effective than they really are.

The paper by Smucny and colleagues describes a very adequate electronic search of the literature (including the standard collections of intervention trials, such as The Cochrane Library, MEDLINE, and EMBASE), conference proceedings, and other similar collections. They also wrote to people who might know of unpublished research. This is a very thorough investigation of the grey literature. Scholten and coworkers limited their search to English, French, German, and Dutch, without attempting any grey literature search (although they obtained more information from the authors of one study to clear up some uncertainty). Does this create a potential bias? The answer lies in trying to decide to what extent this might have caused a bias. If Thai research written in Thai was excluded, but Thai research written in English was included,⁴ then there is such a potential. But perhaps most scientific medical papers in Thailand are written in English. A question mark remains.

Were studies properly selected on the criteria of quality?

Papers must be selected on quality rather than anything else. One sure test of objectivity is whether the methods description allows replication of the review by another group. Both these papers describe a very clear process; criteria were set before the process was started. If we go to the JFP Web site, www.jfponline.com, we can check each paper’s quality score. The way individual investigators each scored the papers independently should minimize any systematic bias. The paper by Smucny and colleagues checked each paper for randomization, blinding, and loss of patients from the analysis. These have been shown to be the most important issues to assess in randomized controlled trials (Shulz). Since the Scholten and coworkers paper is about diagnostic accuracy we should look for different items, particularly in the selection of patients, the “gold standard,” and any loss to follow-up of patients. It is always hard to know whether the positive results were really positive. So it is important to have robust gold standards against which the physical examinations for all patients were independently compared. These consisted of arthrotomy, arthroscopy, or magnetic resonance imaging. They sensibly decided to leave out studies on the dead: It would be difficult to know if such studies could generalize to the living.

Data Extraction

The next stage in a systematic review is the extraction of data from the studies to compare them with each other. Clearly it is important to decide on the right things to extract. Smucny and colleagues only included outcomes that were of direct patient relevance, such as cough (presumably dropping indirect outcomes, such as respiratory function tests). Scholten and coworkers looked at the sensitivity, specificity, and other measures of diagnostic accuracy for estimating joint effusion, joint line tenderness, and the McMurray test.

Could the Data Be Combined?

Sometimes data cannot be combined, because the studies they came from were not comparable. This is called heterogeneity, which can be tested for statistically. If present and not readily explicable, it renders any combination of the data (meta-analysis) suspect. In the paper by Smucny and colleagues, differences in the administration of the b-agonist (oral or inhaled), age group (children, adults), and previous illness (most, but not all, were free of previous wheeze or abnormal respiratory function tests) suggested that no data combinations were permissible other than for cough symptoms scores. The paper by Scholten and coworkers found heterogeneity of the results, and this was not easily explicable by any of the factors—study quality, setting, spectrum of disease, prevalence, and year of publication—included in the meta-regressions. Thus, they rightly express caution about interpreting this analysis.

What Did the Results Show? What Does This Mean for Clinical Practice?

Of the 7 trials of acceptable quality in the paper by Smucny and colleagues, neither of the 2 studies of children showed any benefit from b-agonists, but did find an increase in side effects (shakiness). Among the adult studies, 4 of 5 showed benefits for the b-agonists, but at the cost of increased shakiness or tremor in 3 of them. Looking at the size of the benefit, the standardized mean difference (this is the mean difference adjusted by the variance and is a way to combine different measurement scales, eg, measures of cough severity scaled from 0 to 4, 0 to 7, and 0 to 10) of the cough score was slightly worse for children in the b-agonist group than the control group. For adults, at least it was in the direction of benefits for those using b-agonists, but the standardized mean improvement in cough score on successive days varied between 0.05 to 0.17. This is less than the “small” attached to a standardized mean difference of 0.2, with 0.5 being “moderate” and 0.8 being “large.” Since the 95% confidence intervals for all changes cross 0, this is not clinically or statistically significant. Combining the data (Table 4 in that article) does not improve the power sufficiently to reach statistical significance. It is difficult to convert these standardized mean differences into clinically interpretable meaning, but it certainly appears that even if the changes were statistically significant, they would not be clinically significant. Therefore it seems reasonable to conclude that research to date does not support the use of b-agonists for acute cough of upper respiratory infections or acute bronchitis. Perhaps the different trends in children rather than adults were because the adults were more likely to have chronic chest disease (eg, more smokers, more had evidence of reversible airway obstruction).

Were there weaknesses of the paper? First, there was mixed use of oral (5 trials) and inhaled (2 trials) b-agonist. This might be expected to increase the rate of side effects relative to any benefits. Second, there is some suggestion that people who might have chronic chest disease, including asthma, do respond. This is not surprising.

The study by Scholten and coworkers is more complicated. The ability of clinicians to accurately detect a damaged meniscus by evaluating joint effusion, the McMurray test, joint line tenderness, and the Apley compression test is disappointingly poor (Table 2 of that article). These results have been combined where possible and are best understood using Figure 2. Imagine a patient coming in with a knee injury. Based on the history and before doing the McMurray test, you estimate the chance of this patient having meniscus damage is 50%. Reading up from 50% of the “prior probability” of meniscus damage on the x-axis, if the McMurray test result is positive, the probability moves only to 70% (read off the y-axis); if negative, it drops only to 35%. In this situation, none of these tests will satisfactorily rule-in or rule-out the chance of meniscus lesions.

It must be remembered that these studies were all conducted in specialist clinics. This might have 2 effects: (1) the physicians there may be better at interpreting the clinical examination (being more specialized); and (2) there are more patients with meniscus damage. To get a feel for this, imagine you are a family physician with a person with a sore knee. The chance there is meniscus damage might be only 10%; a positive McMurray test result will increase that probability to 27%, while a negative result will decrease it to 3%. A positive result is not very useful, but a negative one might support a decision to delay further testing and take a more conservative approach to the evaluation, since the likelihood of meniscal lesion is so small.

Conclusions

We hope that we have shown that carefully studying systematic reviews is both entertaining and instructive for clinical practice.* (Online tutorial) We have decided to stop treating patients with acute cough with their upper respiratory infections with b-agonists unless we hear wheezing or have a strong impression of previous asthma. Further research is unlikely to change this. We now regard the presence or absence of signs in people with injured knees with considerable skepticism and accept that there is even more uncertainty than we did before but await better quality research.

Two important systematic reviews of the literature appear in this issue of JFP. Smucny and colleagues¹ address the question of whether people with cough following an acute respiratory infection should be routinely offered b-agonists. Scholten and coworkers² ask which clinical examination maneuvers of the knee are accurate for diagnosing meniscus lesions. These questions are important, because the answers might change the way we practice. What do they mean, and what should we make of the increasing number of reviews of the literature? Our commentary is designed to provide a short description of systematic reviews, using these 2 papers as examples. First we will look at the requirements for a good systematic review and then see how these 2 papers matched up.

What characterizes a good systematic review?

Reviews of the literature are designed for clinicians who need to know what information published research can provide for patient care but who do not have the time (or the skills) to search it out for themselves. Systematic reviews are designed to minimize the biases of the reviewer, which are all too easily introduced unless the data coming from the literature are treated with the same objectivity and respect as the data in a primary trial. If certain questions are not addressed, the opinions of the reviewer may unduly influence his conclusions.³

Did the review look at all the evidence?

What does it matter if one study is left out? The answer is that it might introduce bias if the omission is because of the study’s results and not mere chance. Perhaps trials of b-agonists for acute bronchitis were more likely to be published if they were positive than if they were negative. Negative trials might only be found in the “grey literature”—that is, research that may not have been indexed into formal peer-reviewed journals, making it harder to locate-which would give positive papers exaggerated prominence. Omitting negative studies would give the impression that b-agonists were more effective than they really are.

The paper by Smucny and colleagues describes a very adequate electronic search of the literature (including the standard collections of intervention trials, such as The Cochrane Library, MEDLINE, and EMBASE), conference proceedings, and other similar collections. They also wrote to people who might know of unpublished research. This is a very thorough investigation of the grey literature. Scholten and coworkers limited their search to English, French, German, and Dutch, without attempting any grey literature search (although they obtained more information from the authors of one study to clear up some uncertainty). Does this create a potential bias? The answer lies in trying to decide to what extent this might have caused a bias. If Thai research written in Thai was excluded, but Thai research written in English was included,⁴ then there is such a potential. But perhaps most scientific medical papers in Thailand are written in English. A question mark remains.

Were studies properly selected on the criteria of quality?

Papers must be selected on quality rather than anything else. One sure test of objectivity is whether the methods description allows replication of the review by another group. Both these papers describe a very clear process; criteria were set before the process was started. If we go to the JFP Web site, www.jfponline.com, we can check each paper’s quality score. The way individual investigators each scored the papers independently should minimize any systematic bias. The paper by Smucny and colleagues checked each paper for randomization, blinding, and loss of patients from the analysis. These have been shown to be the most important issues to assess in randomized controlled trials (Shulz). Since the Scholten and coworkers paper is about diagnostic accuracy we should look for different items, particularly in the selection of patients, the “gold standard,” and any loss to follow-up of patients. It is always hard to know whether the positive results were really positive. So it is important to have robust gold standards against which the physical examinations for all patients were independently compared. These consisted of arthrotomy, arthroscopy, or magnetic resonance imaging. They sensibly decided to leave out studies on the dead: It would be difficult to know if such studies could generalize to the living.

Data Extraction

The next stage in a systematic review is the extraction of data from the studies to compare them with each other. Clearly it is important to decide on the right things to extract. Smucny and colleagues only included outcomes that were of direct patient relevance, such as cough (presumably dropping indirect outcomes, such as respiratory function tests). Scholten and coworkers looked at the sensitivity, specificity, and other measures of diagnostic accuracy for estimating joint effusion, joint line tenderness, and the McMurray test.

Could the Data Be Combined?

Sometimes data cannot be combined, because the studies they came from were not comparable. This is called heterogeneity, which can be tested for statistically. If present and not readily explicable, it renders any combination of the data (meta-analysis) suspect. In the paper by Smucny and colleagues, differences in the administration of the b-agonist (oral or inhaled), age group (children, adults), and previous illness (most, but not all, were free of previous wheeze or abnormal respiratory function tests) suggested that no data combinations were permissible other than for cough symptoms scores. The paper by Scholten and coworkers found heterogeneity of the results, and this was not easily explicable by any of the factors—study quality, setting, spectrum of disease, prevalence, and year of publication—included in the meta-regressions. Thus, they rightly express caution about interpreting this analysis.

What Did the Results Show? What Does This Mean for Clinical Practice?

Of the 7 trials of acceptable quality in the paper by Smucny and colleagues, neither of the 2 studies of children showed any benefit from b-agonists, but did find an increase in side effects (shakiness). Among the adult studies, 4 of 5 showed benefits for the b-agonists, but at the cost of increased shakiness or tremor in 3 of them. Looking at the size of the benefit, the standardized mean difference (this is the mean difference adjusted by the variance and is a way to combine different measurement scales, eg, measures of cough severity scaled from 0 to 4, 0 to 7, and 0 to 10) of the cough score was slightly worse for children in the b-agonist group than the control group. For adults, at least it was in the direction of benefits for those using b-agonists, but the standardized mean improvement in cough score on successive days varied between 0.05 to 0.17. This is less than the “small” attached to a standardized mean difference of 0.2, with 0.5 being “moderate” and 0.8 being “large.” Since the 95% confidence intervals for all changes cross 0, this is not clinically or statistically significant. Combining the data (Table 4 in that article) does not improve the power sufficiently to reach statistical significance. It is difficult to convert these standardized mean differences into clinically interpretable meaning, but it certainly appears that even if the changes were statistically significant, they would not be clinically significant. Therefore it seems reasonable to conclude that research to date does not support the use of b-agonists for acute cough of upper respiratory infections or acute bronchitis. Perhaps the different trends in children rather than adults were because the adults were more likely to have chronic chest disease (eg, more smokers, more had evidence of reversible airway obstruction).

Were there weaknesses of the paper? First, there was mixed use of oral (5 trials) and inhaled (2 trials) b-agonist. This might be expected to increase the rate of side effects relative to any benefits. Second, there is some suggestion that people who might have chronic chest disease, including asthma, do respond. This is not surprising.

The study by Scholten and coworkers is more complicated. The ability of clinicians to accurately detect a damaged meniscus by evaluating joint effusion, the McMurray test, joint line tenderness, and the Apley compression test is disappointingly poor (Table 2 of that article). These results have been combined where possible and are best understood using Figure 2. Imagine a patient coming in with a knee injury. Based on the history and before doing the McMurray test, you estimate the chance of this patient having meniscus damage is 50%. Reading up from 50% of the “prior probability” of meniscus damage on the x-axis, if the McMurray test result is positive, the probability moves only to 70% (read off the y-axis); if negative, it drops only to 35%. In this situation, none of these tests will satisfactorily rule-in or rule-out the chance of meniscus lesions.

It must be remembered that these studies were all conducted in specialist clinics. This might have 2 effects: (1) the physicians there may be better at interpreting the clinical examination (being more specialized); and (2) there are more patients with meniscus damage. To get a feel for this, imagine you are a family physician with a person with a sore knee. The chance there is meniscus damage might be only 10%; a positive McMurray test result will increase that probability to 27%, while a negative result will decrease it to 3%. A positive result is not very useful, but a negative one might support a decision to delay further testing and take a more conservative approach to the evaluation, since the likelihood of meniscal lesion is so small.

Conclusions

We hope that we have shown that carefully studying systematic reviews is both entertaining and instructive for clinical practice.* (Online tutorial) We have decided to stop treating patients with acute cough with their upper respiratory infections with b-agonists unless we hear wheezing or have a strong impression of previous asthma. Further research is unlikely to change this. We now regard the presence or absence of signs in people with injured knees with considerable skepticism and accept that there is even more uncertainty than we did before but await better quality research.