Leah S. Karliner, MD, MAS

Translation Tool and Language Choice

We selected Google Translate (GT) since it is one of the more commonly used online translation tools and because Google is the most widely used search engine in the United States.13 GT uses statistical translation methodology to convert text, documents, and websites between languages; statistical translation involves the following three steps. First, the translation program recognizes a sentence to translate. Second, it compares the words and phrases within that sentence to the billions of words in its library (drawn from bilingual professionally translated documents, such as United Nations proceedings). Third, it uses this comparison to generate a translation combining the words and phrases deemed most equivalent between the source sentence and the target language. If there are multiple sentences, the program recognizes and translates each independently. As the body of bilingual work grows, the program learns and refines its rules automatically.14 In contrast, in rule‐based translation, a program would use manually prespecified rules regarding word choice and grammar to generate a translation.15 We assessed GT's accuracy translating from English to Spanish because Spanish is the predominant non‐English language spoken in the US.1

Document Selection and Preparation

We selected the instruction manual regarding warfarin use prepared by the Agency for Healthcare Research and Quality (AHRQ) for this accuracy evaluation. We selected this manual,16 written at a 6th grade reading level, because a professional Spanish translation was available (completed by ASET International Service, LLC, before and independently of this study), and because patient educational material regarding warfarin has been associated with fewer bleeding events.17 We downloaded the English document on October 19, 2009 and used the GT website to translate it en bloc. We then copied the resulting Spanish output into a text file. The English document and the professional Spanish translation (downloaded the same day) were both converted into text files in the same manner.

Grading Methodology

We scored the translation chosen using both manual and automated evaluation techniques. These techniques are widely used in the machine translation literature and are explained below.

Manual Evaluation: Evaluators, Domains, Scoring

We recruited three nonclinician, bilingual, nativeSpanish‐speaking research assistants as evaluators. The evaluators were all college educated with a Bachelor's degree or higher and were of Mexican, Nicaraguan, and Guatemalan ancestry. Each evaluator received a brief orientation regarding the project, as well as an explanation of the scores, and then proceeded to the blinded evaluation independently.

We asked evaluators to score sentences on Likert scales along five primary domains: fluency, adequacy, meaning, severity, and preference. Fluency and adequacy are well accepted components of machine translation evaluation,18 with fluency being an assessment of grammar and readability ranging from 5 (Perfect fluency; like reading a newspaper) to 1 (No fluency; no appreciable grammar, not understandable) and adequacy being an assessment of information preservation ranging from 5 (100% of information conveyed from the original) to 1 (0% of information conveyed from the original). Given that a sentence can be highly adequate but drastically change the connotation and intent of the sentence (eg, a sentence that contains 75% of the correct words but changes a sentence from take this medication twice a day to take this medication once every two days), we asked evaluators to assess meaning, a measure of connotation and intent maintenance, with scores ranging from 5 (Same meaning as original) to 1 (Totally different meaning from the original).19 Evaluators also assessed severity, a new measure of potential harm if a given sentence was assessed as having errors of any kind, ranging from 5 (Error, no effect on patient care) to 1 (Error, dangerous to patient) with an additional option of N/A (Sentence basically accurate). Finally, evaluators rated a blinded preference (also a new measure) for either of two translated sentences, ranging from Strongly prefer translation #1 to Strongly prefer translation #2. The order of the sentences was random (eg, sometimes the professional translation was first and sometimes the GT translation was). We subsequently converted this to preference for the professional translation, ranging from 5 (Strongly prefer the professional translation) to 1 (Strongly prefer the GT translation) in order to standardize the responses (Figures 1 and 2).

Figure 1

Figure 2

The overall flow of the study is given in Figure 3. Each evaluator initially scored 20 sentences translated by GT and 10 sentences translated professionally along the first four domains. All 30 of these sentences were randomly selected from the original, 263‐sentence pamphlet. For fluency, evaluators had access only to the translated sentence to be scored; for adequacy, meaning, and severity, they had access to both the translated sentence and the original English sentence. Ten of the 30 sentences were further selected randomly for scoring on the preference domain. For these 10 sentences, evaluators compared the GT and professional translations of the same sentence (with the original English sentence available as a reference) and indicated a preference, for any reason, for one translation or the other. Evaluators were blinded to the technique of translation (GT or professional) for all scored sentences and domains. We chose twice as many sentences from the GT preparations for the first four domains to maximize measurements for the translation technology we were evaluating, with the smaller number of professional translations serving as controls.

Figure 3

After scoring the first 30 sentences, evaluators met with one of the authors (R.R.K.) to discuss and consolidate their approach to scoring. They then scored an additional 10 GT‐translated sentences and 5 professionally translated sentences for the first four domains, and 9 of these 15 sentences for preference, to see if the meeting changed their scoring approach. These sentences were selected randomly from the original, 263‐sentence pamphlet, excluding the 30 evaluated in the previous step.

Automated Machine Translation Evaluation

Machine translation researchers have developed automated measures allowing the rapid and inexpensive scoring and rescoring of translations. These automated measures supplement more time‐ and resource‐intensive manual evaluations. The automated measures are based upon how well the translation compares to one or, ideally, multiple professionally prepared reference translations. They correlate well with human judgments on the domains above, especially when multiple reference translations are used (increasing the number of reference translations increases the variability allowed for words and phrases in the machine translation, improving the likelihood that differences in score are related to differences in quality rather than differences in translator preference).20 For this study, we used Metric for Evaluation of Translation with Explicit Ordering (METEOR), a machine translation evaluation system that allows additional flexibility for the machine translation in terms of grading individual sentences and being sensitive to synonyms, word stemming, and word order.21 We obtained a METEOR score for each of the GT‐translated sentences using the professional translation as our reference, and assessed correlation between this automated measure and the manual evaluations for the GT sentences, with the aim of assessing the feasibility of using METEOR in future work on patient educational material translation.

Outcomes and Statistical Analysis

We compared the scores assigned to GT‐translated sentences for each of the five manually scored domains as compared to the scores of the professionally translated sentences, as well as the impact of word count and sentence complexity on the scores achieved specifically by the GT‐translated sentences, using clustered linear regression to account for the fact that each of the 45 sentences were scored by each of the three evaluators. Sentences were classified as simple if they contained one or fewer clauses and complex if they contained more than one clause.22 We also assessed interrater reliability for the manual scoring system using intraclass correlation coefficients and repeatability. Repeatability is an estimate of the maximum difference, with 95% confidence, between scores assigned to the same sentence on the same domain by two different evaluators;23 lower scores indicate greater agreement between evaluators. Since we did not have clinical data or a gold standard, we used repeatability to estimate the value above which a difference between two scores might be clinically significant and not simply due to interrater variability.24 Finally, we assessed the correlation of the manual scores with those calculated by the METEOR automated evaluation tool using Pearson correlation coefficients. All analyses were conducted using Stata 11 (College Station, TX).

Sentence Description

A total of 45 sentences were evaluated by the bilingual research assistants. The initial 30 sentences and the subsequent, post‐consolidation meeting 15 sentences were scored similarly in all outcomes, after adjustment for word length and complexity, so we pooled all 45 sentences (as well as the 19 total sentence pairs scored for preference) for the final analysis. Average sentence lengths were 14.2 words, 15.5 words, and 16.6 words for the English source text, professionally translated sentences, and GT‐translated sentences, respectively. Thirty‐three percent of the English source sentences were simple and 67% were complex.

Manual Evaluation Scores

Sentences translated by GT received worse scores on fluency as compared to the professional translations (3.4 vs 4.7, P < 0.0001). Comparisons for adequacy and meaning were not statistically significantly different. GT‐translated sentences contained more errors of any severity as compared to the professional translations (39% vs 22%, P = 0.05), but a similar number of serious, clinically impactful errors (severity scores of 3, 2, or 1; 4% vs 2%, P = 0.61). However, one GT‐translated sentence was considered erroneous with a severity level of 1 (Error, dangerous to patient). This particular sentence was 25 words long and complex in structure in the original English document; all three evaluators considered the GT translation nonsensical (La hemorragia mayor, llame a su mdico, o ir a la emergencia de un hospital habitacin si usted tiene cualquiera de los siguientes: Red N, oscuro, caf o cola de orina de color.) Evaluators had no overall preference for the professional translation (3.2, 95% confidence interval = 2.7 to 3.7, with 3 indicating no preference; P = 0.36) (Table 1).

Mediation of Scores by Sentence Length or Complexity

We found that sentence length was not associated with scores for fluency, adequacy, meaning, severity, or preference (P > 0.30 in each case). Complexity, however, was significantly associated with preference: evaluators' preferred the professional translation for complex English sentences while being more ambivalent about simple English sentences (3.6 vs 2.6, P = 0.03).

Interrater Reliability and Repeatability

We assessed the interrater reliability for each domain using intraclass correlation coefficients and repeatability. For fluency, the intraclass correlation was best at 0.70; for adequacy, it was 0.58; for meaning, 0.42; for severity, 0.48; and for preference, 0.37. The repeatability scores were 1.4 for fluency, 0.6 for adequacy, 2.2 for meaning, 1.2 for severity, and 3.8 for preference, indicating that two evaluators might give a sentence almost the same score (at most, 1 point apart from one another) for adequacy, but might have opposite preferences regarding which translation of a sentence was superior.

Correlation with METEOR

Correlation between the first four domains and the METEOR scores were less than in prior studies.21 Fluency correlated best with METEOR at 0.53; adequacy correlated least with METEOR at 0.29. The remaining scores were in‐between. All correlations were statistically significant at P < 0.01 (Table 2).

Translation Tool and Language Choice

We selected Google Translate (GT) since it is one of the more commonly used online translation tools and because Google is the most widely used search engine in the United States.13 GT uses statistical translation methodology to convert text, documents, and websites between languages; statistical translation involves the following three steps. First, the translation program recognizes a sentence to translate. Second, it compares the words and phrases within that sentence to the billions of words in its library (drawn from bilingual professionally translated documents, such as United Nations proceedings). Third, it uses this comparison to generate a translation combining the words and phrases deemed most equivalent between the source sentence and the target language. If there are multiple sentences, the program recognizes and translates each independently. As the body of bilingual work grows, the program learns and refines its rules automatically.14 In contrast, in rule‐based translation, a program would use manually prespecified rules regarding word choice and grammar to generate a translation.15 We assessed GT's accuracy translating from English to Spanish because Spanish is the predominant non‐English language spoken in the US.1

Document Selection and Preparation

We selected the instruction manual regarding warfarin use prepared by the Agency for Healthcare Research and Quality (AHRQ) for this accuracy evaluation. We selected this manual,16 written at a 6th grade reading level, because a professional Spanish translation was available (completed by ASET International Service, LLC, before and independently of this study), and because patient educational material regarding warfarin has been associated with fewer bleeding events.17 We downloaded the English document on October 19, 2009 and used the GT website to translate it en bloc. We then copied the resulting Spanish output into a text file. The English document and the professional Spanish translation (downloaded the same day) were both converted into text files in the same manner.

Grading Methodology

We scored the translation chosen using both manual and automated evaluation techniques. These techniques are widely used in the machine translation literature and are explained below.

Manual Evaluation: Evaluators, Domains, Scoring

We recruited three nonclinician, bilingual, nativeSpanish‐speaking research assistants as evaluators. The evaluators were all college educated with a Bachelor's degree or higher and were of Mexican, Nicaraguan, and Guatemalan ancestry. Each evaluator received a brief orientation regarding the project, as well as an explanation of the scores, and then proceeded to the blinded evaluation independently.

We asked evaluators to score sentences on Likert scales along five primary domains: fluency, adequacy, meaning, severity, and preference. Fluency and adequacy are well accepted components of machine translation evaluation,18 with fluency being an assessment of grammar and readability ranging from 5 (Perfect fluency; like reading a newspaper) to 1 (No fluency; no appreciable grammar, not understandable) and adequacy being an assessment of information preservation ranging from 5 (100% of information conveyed from the original) to 1 (0% of information conveyed from the original). Given that a sentence can be highly adequate but drastically change the connotation and intent of the sentence (eg, a sentence that contains 75% of the correct words but changes a sentence from take this medication twice a day to take this medication once every two days), we asked evaluators to assess meaning, a measure of connotation and intent maintenance, with scores ranging from 5 (Same meaning as original) to 1 (Totally different meaning from the original).19 Evaluators also assessed severity, a new measure of potential harm if a given sentence was assessed as having errors of any kind, ranging from 5 (Error, no effect on patient care) to 1 (Error, dangerous to patient) with an additional option of N/A (Sentence basically accurate). Finally, evaluators rated a blinded preference (also a new measure) for either of two translated sentences, ranging from Strongly prefer translation #1 to Strongly prefer translation #2. The order of the sentences was random (eg, sometimes the professional translation was first and sometimes the GT translation was). We subsequently converted this to preference for the professional translation, ranging from 5 (Strongly prefer the professional translation) to 1 (Strongly prefer the GT translation) in order to standardize the responses (Figures 1 and 2).

Figure 1

Figure 2

The overall flow of the study is given in Figure 3. Each evaluator initially scored 20 sentences translated by GT and 10 sentences translated professionally along the first four domains. All 30 of these sentences were randomly selected from the original, 263‐sentence pamphlet. For fluency, evaluators had access only to the translated sentence to be scored; for adequacy, meaning, and severity, they had access to both the translated sentence and the original English sentence. Ten of the 30 sentences were further selected randomly for scoring on the preference domain. For these 10 sentences, evaluators compared the GT and professional translations of the same sentence (with the original English sentence available as a reference) and indicated a preference, for any reason, for one translation or the other. Evaluators were blinded to the technique of translation (GT or professional) for all scored sentences and domains. We chose twice as many sentences from the GT preparations for the first four domains to maximize measurements for the translation technology we were evaluating, with the smaller number of professional translations serving as controls.

Figure 3

After scoring the first 30 sentences, evaluators met with one of the authors (R.R.K.) to discuss and consolidate their approach to scoring. They then scored an additional 10 GT‐translated sentences and 5 professionally translated sentences for the first four domains, and 9 of these 15 sentences for preference, to see if the meeting changed their scoring approach. These sentences were selected randomly from the original, 263‐sentence pamphlet, excluding the 30 evaluated in the previous step.

Automated Machine Translation Evaluation

Machine translation researchers have developed automated measures allowing the rapid and inexpensive scoring and rescoring of translations. These automated measures supplement more time‐ and resource‐intensive manual evaluations. The automated measures are based upon how well the translation compares to one or, ideally, multiple professionally prepared reference translations. They correlate well with human judgments on the domains above, especially when multiple reference translations are used (increasing the number of reference translations increases the variability allowed for words and phrases in the machine translation, improving the likelihood that differences in score are related to differences in quality rather than differences in translator preference).20 For this study, we used Metric for Evaluation of Translation with Explicit Ordering (METEOR), a machine translation evaluation system that allows additional flexibility for the machine translation in terms of grading individual sentences and being sensitive to synonyms, word stemming, and word order.21 We obtained a METEOR score for each of the GT‐translated sentences using the professional translation as our reference, and assessed correlation between this automated measure and the manual evaluations for the GT sentences, with the aim of assessing the feasibility of using METEOR in future work on patient educational material translation.

Outcomes and Statistical Analysis

We compared the scores assigned to GT‐translated sentences for each of the five manually scored domains as compared to the scores of the professionally translated sentences, as well as the impact of word count and sentence complexity on the scores achieved specifically by the GT‐translated sentences, using clustered linear regression to account for the fact that each of the 45 sentences were scored by each of the three evaluators. Sentences were classified as simple if they contained one or fewer clauses and complex if they contained more than one clause.22 We also assessed interrater reliability for the manual scoring system using intraclass correlation coefficients and repeatability. Repeatability is an estimate of the maximum difference, with 95% confidence, between scores assigned to the same sentence on the same domain by two different evaluators;23 lower scores indicate greater agreement between evaluators. Since we did not have clinical data or a gold standard, we used repeatability to estimate the value above which a difference between two scores might be clinically significant and not simply due to interrater variability.24 Finally, we assessed the correlation of the manual scores with those calculated by the METEOR automated evaluation tool using Pearson correlation coefficients. All analyses were conducted using Stata 11 (College Station, TX).

Sentence Description

A total of 45 sentences were evaluated by the bilingual research assistants. The initial 30 sentences and the subsequent, post‐consolidation meeting 15 sentences were scored similarly in all outcomes, after adjustment for word length and complexity, so we pooled all 45 sentences (as well as the 19 total sentence pairs scored for preference) for the final analysis. Average sentence lengths were 14.2 words, 15.5 words, and 16.6 words for the English source text, professionally translated sentences, and GT‐translated sentences, respectively. Thirty‐three percent of the English source sentences were simple and 67% were complex.

Manual Evaluation Scores

Sentences translated by GT received worse scores on fluency as compared to the professional translations (3.4 vs 4.7, P < 0.0001). Comparisons for adequacy and meaning were not statistically significantly different. GT‐translated sentences contained more errors of any severity as compared to the professional translations (39% vs 22%, P = 0.05), but a similar number of serious, clinically impactful errors (severity scores of 3, 2, or 1; 4% vs 2%, P = 0.61). However, one GT‐translated sentence was considered erroneous with a severity level of 1 (Error, dangerous to patient). This particular sentence was 25 words long and complex in structure in the original English document; all three evaluators considered the GT translation nonsensical (La hemorragia mayor, llame a su mdico, o ir a la emergencia de un hospital habitacin si usted tiene cualquiera de los siguientes: Red N, oscuro, caf o cola de orina de color.) Evaluators had no overall preference for the professional translation (3.2, 95% confidence interval = 2.7 to 3.7, with 3 indicating no preference; P = 0.36) (Table 1).

Mediation of Scores by Sentence Length or Complexity

We found that sentence length was not associated with scores for fluency, adequacy, meaning, severity, or preference (P > 0.30 in each case). Complexity, however, was significantly associated with preference: evaluators' preferred the professional translation for complex English sentences while being more ambivalent about simple English sentences (3.6 vs 2.6, P = 0.03).

Interrater Reliability and Repeatability

We assessed the interrater reliability for each domain using intraclass correlation coefficients and repeatability. For fluency, the intraclass correlation was best at 0.70; for adequacy, it was 0.58; for meaning, 0.42; for severity, 0.48; and for preference, 0.37. The repeatability scores were 1.4 for fluency, 0.6 for adequacy, 2.2 for meaning, 1.2 for severity, and 3.8 for preference, indicating that two evaluators might give a sentence almost the same score (at most, 1 point apart from one another) for adequacy, but might have opposite preferences regarding which translation of a sentence was superior.

Correlation with METEOR

Correlation between the first four domains and the METEOR scores were less than in prior studies.21 Fluency correlated best with METEOR at 0.53; adequacy correlated least with METEOR at 0.29. The remaining scores were in‐between. All correlations were statistically significant at P < 0.01 (Table 2).

Translation Tool and Language Choice

We selected Google Translate (GT) since it is one of the more commonly used online translation tools and because Google is the most widely used search engine in the United States.13 GT uses statistical translation methodology to convert text, documents, and websites between languages; statistical translation involves the following three steps. First, the translation program recognizes a sentence to translate. Second, it compares the words and phrases within that sentence to the billions of words in its library (drawn from bilingual professionally translated documents, such as United Nations proceedings). Third, it uses this comparison to generate a translation combining the words and phrases deemed most equivalent between the source sentence and the target language. If there are multiple sentences, the program recognizes and translates each independently. As the body of bilingual work grows, the program learns and refines its rules automatically.14 In contrast, in rule‐based translation, a program would use manually prespecified rules regarding word choice and grammar to generate a translation.15 We assessed GT's accuracy translating from English to Spanish because Spanish is the predominant non‐English language spoken in the US.1

Document Selection and Preparation

We selected the instruction manual regarding warfarin use prepared by the Agency for Healthcare Research and Quality (AHRQ) for this accuracy evaluation. We selected this manual,16 written at a 6th grade reading level, because a professional Spanish translation was available (completed by ASET International Service, LLC, before and independently of this study), and because patient educational material regarding warfarin has been associated with fewer bleeding events.17 We downloaded the English document on October 19, 2009 and used the GT website to translate it en bloc. We then copied the resulting Spanish output into a text file. The English document and the professional Spanish translation (downloaded the same day) were both converted into text files in the same manner.

Grading Methodology

We scored the translation chosen using both manual and automated evaluation techniques. These techniques are widely used in the machine translation literature and are explained below.

Manual Evaluation: Evaluators, Domains, Scoring

We recruited three nonclinician, bilingual, nativeSpanish‐speaking research assistants as evaluators. The evaluators were all college educated with a Bachelor's degree or higher and were of Mexican, Nicaraguan, and Guatemalan ancestry. Each evaluator received a brief orientation regarding the project, as well as an explanation of the scores, and then proceeded to the blinded evaluation independently.

We asked evaluators to score sentences on Likert scales along five primary domains: fluency, adequacy, meaning, severity, and preference. Fluency and adequacy are well accepted components of machine translation evaluation,18 with fluency being an assessment of grammar and readability ranging from 5 (Perfect fluency; like reading a newspaper) to 1 (No fluency; no appreciable grammar, not understandable) and adequacy being an assessment of information preservation ranging from 5 (100% of information conveyed from the original) to 1 (0% of information conveyed from the original). Given that a sentence can be highly adequate but drastically change the connotation and intent of the sentence (eg, a sentence that contains 75% of the correct words but changes a sentence from take this medication twice a day to take this medication once every two days), we asked evaluators to assess meaning, a measure of connotation and intent maintenance, with scores ranging from 5 (Same meaning as original) to 1 (Totally different meaning from the original).19 Evaluators also assessed severity, a new measure of potential harm if a given sentence was assessed as having errors of any kind, ranging from 5 (Error, no effect on patient care) to 1 (Error, dangerous to patient) with an additional option of N/A (Sentence basically accurate). Finally, evaluators rated a blinded preference (also a new measure) for either of two translated sentences, ranging from Strongly prefer translation #1 to Strongly prefer translation #2. The order of the sentences was random (eg, sometimes the professional translation was first and sometimes the GT translation was). We subsequently converted this to preference for the professional translation, ranging from 5 (Strongly prefer the professional translation) to 1 (Strongly prefer the GT translation) in order to standardize the responses (Figures 1 and 2).

Figure 1

Figure 2

The overall flow of the study is given in Figure 3. Each evaluator initially scored 20 sentences translated by GT and 10 sentences translated professionally along the first four domains. All 30 of these sentences were randomly selected from the original, 263‐sentence pamphlet. For fluency, evaluators had access only to the translated sentence to be scored; for adequacy, meaning, and severity, they had access to both the translated sentence and the original English sentence. Ten of the 30 sentences were further selected randomly for scoring on the preference domain. For these 10 sentences, evaluators compared the GT and professional translations of the same sentence (with the original English sentence available as a reference) and indicated a preference, for any reason, for one translation or the other. Evaluators were blinded to the technique of translation (GT or professional) for all scored sentences and domains. We chose twice as many sentences from the GT preparations for the first four domains to maximize measurements for the translation technology we were evaluating, with the smaller number of professional translations serving as controls.

Figure 3

After scoring the first 30 sentences, evaluators met with one of the authors (R.R.K.) to discuss and consolidate their approach to scoring. They then scored an additional 10 GT‐translated sentences and 5 professionally translated sentences for the first four domains, and 9 of these 15 sentences for preference, to see if the meeting changed their scoring approach. These sentences were selected randomly from the original, 263‐sentence pamphlet, excluding the 30 evaluated in the previous step.

Automated Machine Translation Evaluation

Machine translation researchers have developed automated measures allowing the rapid and inexpensive scoring and rescoring of translations. These automated measures supplement more time‐ and resource‐intensive manual evaluations. The automated measures are based upon how well the translation compares to one or, ideally, multiple professionally prepared reference translations. They correlate well with human judgments on the domains above, especially when multiple reference translations are used (increasing the number of reference translations increases the variability allowed for words and phrases in the machine translation, improving the likelihood that differences in score are related to differences in quality rather than differences in translator preference).20 For this study, we used Metric for Evaluation of Translation with Explicit Ordering (METEOR), a machine translation evaluation system that allows additional flexibility for the machine translation in terms of grading individual sentences and being sensitive to synonyms, word stemming, and word order.21 We obtained a METEOR score for each of the GT‐translated sentences using the professional translation as our reference, and assessed correlation between this automated measure and the manual evaluations for the GT sentences, with the aim of assessing the feasibility of using METEOR in future work on patient educational material translation.

Outcomes and Statistical Analysis

We compared the scores assigned to GT‐translated sentences for each of the five manually scored domains as compared to the scores of the professionally translated sentences, as well as the impact of word count and sentence complexity on the scores achieved specifically by the GT‐translated sentences, using clustered linear regression to account for the fact that each of the 45 sentences were scored by each of the three evaluators. Sentences were classified as simple if they contained one or fewer clauses and complex if they contained more than one clause.22 We also assessed interrater reliability for the manual scoring system using intraclass correlation coefficients and repeatability. Repeatability is an estimate of the maximum difference, with 95% confidence, between scores assigned to the same sentence on the same domain by two different evaluators;23 lower scores indicate greater agreement between evaluators. Since we did not have clinical data or a gold standard, we used repeatability to estimate the value above which a difference between two scores might be clinically significant and not simply due to interrater variability.24 Finally, we assessed the correlation of the manual scores with those calculated by the METEOR automated evaluation tool using Pearson correlation coefficients. All analyses were conducted using Stata 11 (College Station, TX).

Sentence Description

A total of 45 sentences were evaluated by the bilingual research assistants. The initial 30 sentences and the subsequent, post‐consolidation meeting 15 sentences were scored similarly in all outcomes, after adjustment for word length and complexity, so we pooled all 45 sentences (as well as the 19 total sentence pairs scored for preference) for the final analysis. Average sentence lengths were 14.2 words, 15.5 words, and 16.6 words for the English source text, professionally translated sentences, and GT‐translated sentences, respectively. Thirty‐three percent of the English source sentences were simple and 67% were complex.

Manual Evaluation Scores

Sentences translated by GT received worse scores on fluency as compared to the professional translations (3.4 vs 4.7, P < 0.0001). Comparisons for adequacy and meaning were not statistically significantly different. GT‐translated sentences contained more errors of any severity as compared to the professional translations (39% vs 22%, P = 0.05), but a similar number of serious, clinically impactful errors (severity scores of 3, 2, or 1; 4% vs 2%, P = 0.61). However, one GT‐translated sentence was considered erroneous with a severity level of 1 (Error, dangerous to patient). This particular sentence was 25 words long and complex in structure in the original English document; all three evaluators considered the GT translation nonsensical (La hemorragia mayor, llame a su mdico, o ir a la emergencia de un hospital habitacin si usted tiene cualquiera de los siguientes: Red N, oscuro, caf o cola de orina de color.) Evaluators had no overall preference for the professional translation (3.2, 95% confidence interval = 2.7 to 3.7, with 3 indicating no preference; P = 0.36) (Table 1).

Mediation of Scores by Sentence Length or Complexity

We found that sentence length was not associated with scores for fluency, adequacy, meaning, severity, or preference (P > 0.30 in each case). Complexity, however, was significantly associated with preference: evaluators' preferred the professional translation for complex English sentences while being more ambivalent about simple English sentences (3.6 vs 2.6, P = 0.03).

Interrater Reliability and Repeatability

We assessed the interrater reliability for each domain using intraclass correlation coefficients and repeatability. For fluency, the intraclass correlation was best at 0.70; for adequacy, it was 0.58; for meaning, 0.42; for severity, 0.48; and for preference, 0.37. The repeatability scores were 1.4 for fluency, 0.6 for adequacy, 2.2 for meaning, 1.2 for severity, and 3.8 for preference, indicating that two evaluators might give a sentence almost the same score (at most, 1 point apart from one another) for adequacy, but might have opposite preferences regarding which translation of a sentence was superior.

Correlation with METEOR

Correlation between the first four domains and the METEOR scores were less than in prior studies.21 Fluency correlated best with METEOR at 0.53; adequacy correlated least with METEOR at 0.29. The remaining scores were in‐between. All correlations were statistically significant at P < 0.01 (Table 2).

Patient Population and Setting

Our study examined patients admitted to the General Medicine Service at the University of California, San Francisco Medical Center (UCSF) between July 1, 2001 and June 30th, 2003, the time period during which UCSF participated in the Multicenter Hospitalist Trial (MHT) a prospective quasi‐randomized trial of hospitalist care for general medicine patients.13, 14

UCSF Moffitt‐Long Hospital is a 400‐bed urban academic medical center which provides services to the City and County of San Francisco, an ethnically and linguistically diverse area. UCSF employs staff language interpreters in Spanish, Chinese and Russian who travel to its many outpatient clinics, Comprehensive Cancer Center, Children's Hospital, as well as to Moffitt‐Long Hospital upon request; phone interpretation is also available when in‐person interpreters are not available, for off hours needs and for less common languages. During the period of this study there were no specific inpatient guidelines in place for use of interpretation services at UCSF, nor were there any specific interventions targeting LEP or non‐English speaking inpatients.

Patients were eligible for the MHT if they were 18 years of age or older and admitted at random to a hospitalist or non‐hospitalist physician (eg, outpatient general internist attending on average 1‐month/year); a minority of patients were cared for directly by their primary care physician while in the hospital, and were excluded. For purposes of our study, which merged MHT data with hospital administrative data on primary language, we further excluded all admissions for patients for whom primary language was missing (n = 5), whose listing was unknown or other language (n = 78), sign language (n = 3) or whose language was listed but was not one of the included languages (n = 258). Included languages were English, Chinese, Russian and Spanish. Because LOS and cost data were skewed, we excluded those admissions with the top 1% longest stays and the top 1% highest cost (n = 176); these exclusions did not alter the proportion of admissions across language and ethnicity. In addition, we excluded 102 admissions that were missing data on cost and 11 with costs <$500 and which were likely to be erroneous. Our research was approved by the UCSF Institutional Review Board.

Data Sources

We collected administrative data from Transition Systems Inc (TSI, Boston, MA) billing databases at UCSF as part of the MHT. These data include patient demographics, insurance, costs, ICD‐9CM diagnostic codes, admission and discharge dates in Uniform Bill 92 format. Patient mortality information was collected as part of the MHT using the National Death Index.14

Language data were collected from a separate patient‐registration database (STOR) at UCSF. Information on a patient's primary language is entered at the time each patient first registers at UCSF, whether for the index hospitalization or for prior clinic visits, and is based generally on patient self report. As part of our validation step, we cross‐checked 829 STOR language entries against patient reports and found 91% agreement with the majority of the errors classifying non‐English speakers as English‐speakers.

Measures

Admission Characteristics of the Sample

A total of 7023 patients were admitted to the General Medicine service, 5877 (84%) of whom were English‐speakers and 1146 (16%) non‐English‐speakers (Table 1). Overall, half of the admitted patients were women (50%), and the vast majority was insured (93%). The most common principal diagnoses were respiratory and gastrointestinal disorders. Only a small number of non‐English speakers 164 (14%) were recorded in the UCSF Interpreter Services database as having had any interaction with a professional staff interpreter during their hospitalization.

Among English speakers, Whites and African Americans were the most common ethnicities; however, more than 500 admissions were categorized as Asian ethnicity, and more than 600 as patients of other ethnicity. Close to 300 admissions were for Latinos. Among non‐English speakers, Chinese speakers had the largest number of admissions (n = 584), while Spanish and Russian speakers had similar numbers (n = 272 and 290 respectively).

Non‐English speakers were older, more likely to be female, more likely to be insured by Medicare, and more likely to have a higher comorbidity index score. While comorbidity scores were similar among non‐English speakers (Chinese 1.13 1.50; Russian 1.09 1.37; Spanish 1.06 1.30), they differed considerably among English speakers (White 0.94 1.29; African American 1.05 1.40; Asian 1.04 1.45; Latino 0.89 1.23; Other 0.91 1.29).

Hospital Outcome by Language‐Ethnicity Group (Table 2)

When aggregated together, non‐English speakers were somewhat more likely to be dead at 30‐days and have lower cost admissions; however, they did not differ from English speakers on LOS or readmission rates. While differences among disaggregated language‐ethnicity groups were not all statistically significant, English‐speaking Whites had the longest LOS (mean = 4.9 days) and highest costs (mean = $10,530). English‐speaking African Americans, Chinese and Spanish speakers had the highest 30‐day readmission rates; whereas, English‐speaking Latinos and Russian speakers had markedly lower 30‐day readmission rates (2.5% and 6.4%, respectively). Chinese speakers had the highest 30‐day mortality, followed by English speaking Whites and Asians.

We further investigated differences among English speakers to better understand the very high rate of readmission for African Americans and the very low rate for English‐speaking Latinos. African Americans were on average younger than other English speakers (55 19 years vs. 60 21 years; P < 0.001); but, they had higher comorbidity scores than other English speakers (1.05 1.40 vs. 0.94 1.31; P = 0.008), and were more likely to be admitted for non‐malignant blood disorders (eg, sickle cell disease), endocrine disorders (eg, diabetes mellitus), and circulatory disorders (eg, stroke). In contrast, English‐speaking Latinos were also younger than other English speakers (53 21 years vs. 59 20 years; P < 0.001), but they trended toward lower comorbidity scores (0.87 1.23 vs. 0.97 1.33; P = 0.2), and were more likely to be admitted for gastrointestinal and musculoskeletal disorders, and less likely to be admitted for malignancy and endocrine disorders.

Multivariate Analyses: Association of Aggregated and Disaggregated Language‐Ethnicity Groups With Hospital Outcomes (Table 3)

In multivariate models examining aggregated language‐ethnicity groups, non‐English speakers had a trend toward higher odds of readmission at 30‐days post‐discharge than the English‐speaking group (odds ratio [OR], 1.3; 95% confidence interval [CI], 1.0‐1.7). There were no significant differences for LOS, cost, or 30‐day mortality. Compared to English speakers, Chinese and Spanish speakers had 70% and 50% higher adjusted odds of readmission at 30‐days post‐discharge respectively, while Russian speakers' odds of readmission was not increased. Additionally, Chinese speakers had 7% shorter LOS than English‐speakers. There were no significant differences among any of the language‐ethnicity groups for 30‐day mortality. The increased odds of readmission for Chinese and Spanish speakers compared to English speakers was robust to reinclusion of the admissions with the top 1% LOS and top 1% cost.

Multivariate Analyses: Association of Language for Asians and Latinos, and of Ethnicity for English speakers, With Hospital Outcomes (Table 4)

Both Chinese and Spanish speakers had significantly higher odds of 30‐day readmission than their English speaking Asian and Latino counterparts. There were no significant differences in LOS, cost, or 30‐day mortality in this within‐ethnicity analysis. Among English speakers, admissions for patients with Asian ethnicity were 15% shorter and resulted in 9% lower costs than for Whites. While LOS and cost were similar for English‐speaking Latino and White admissions, English‐speaking Latinos had markedly lower odds of 30‐day readmission than their White counterparts. Whereas African‐Americans had 6% shorter LOS, 40% higher odds of readmission and 30% lower odds of mortality at 30‐days than English speaking Whites.

Patient Population and Setting

Our study examined patients admitted to the General Medicine Service at the University of California, San Francisco Medical Center (UCSF) between July 1, 2001 and June 30th, 2003, the time period during which UCSF participated in the Multicenter Hospitalist Trial (MHT) a prospective quasi‐randomized trial of hospitalist care for general medicine patients.13, 14

UCSF Moffitt‐Long Hospital is a 400‐bed urban academic medical center which provides services to the City and County of San Francisco, an ethnically and linguistically diverse area. UCSF employs staff language interpreters in Spanish, Chinese and Russian who travel to its many outpatient clinics, Comprehensive Cancer Center, Children's Hospital, as well as to Moffitt‐Long Hospital upon request; phone interpretation is also available when in‐person interpreters are not available, for off hours needs and for less common languages. During the period of this study there were no specific inpatient guidelines in place for use of interpretation services at UCSF, nor were there any specific interventions targeting LEP or non‐English speaking inpatients.

Patients were eligible for the MHT if they were 18 years of age or older and admitted at random to a hospitalist or non‐hospitalist physician (eg, outpatient general internist attending on average 1‐month/year); a minority of patients were cared for directly by their primary care physician while in the hospital, and were excluded. For purposes of our study, which merged MHT data with hospital administrative data on primary language, we further excluded all admissions for patients for whom primary language was missing (n = 5), whose listing was unknown or other language (n = 78), sign language (n = 3) or whose language was listed but was not one of the included languages (n = 258). Included languages were English, Chinese, Russian and Spanish. Because LOS and cost data were skewed, we excluded those admissions with the top 1% longest stays and the top 1% highest cost (n = 176); these exclusions did not alter the proportion of admissions across language and ethnicity. In addition, we excluded 102 admissions that were missing data on cost and 11 with costs <$500 and which were likely to be erroneous. Our research was approved by the UCSF Institutional Review Board.

Data Sources

We collected administrative data from Transition Systems Inc (TSI, Boston, MA) billing databases at UCSF as part of the MHT. These data include patient demographics, insurance, costs, ICD‐9CM diagnostic codes, admission and discharge dates in Uniform Bill 92 format. Patient mortality information was collected as part of the MHT using the National Death Index.14

Language data were collected from a separate patient‐registration database (STOR) at UCSF. Information on a patient's primary language is entered at the time each patient first registers at UCSF, whether for the index hospitalization or for prior clinic visits, and is based generally on patient self report. As part of our validation step, we cross‐checked 829 STOR language entries against patient reports and found 91% agreement with the majority of the errors classifying non‐English speakers as English‐speakers.

Measures

Admission Characteristics of the Sample

A total of 7023 patients were admitted to the General Medicine service, 5877 (84%) of whom were English‐speakers and 1146 (16%) non‐English‐speakers (Table 1). Overall, half of the admitted patients were women (50%), and the vast majority was insured (93%). The most common principal diagnoses were respiratory and gastrointestinal disorders. Only a small number of non‐English speakers 164 (14%) were recorded in the UCSF Interpreter Services database as having had any interaction with a professional staff interpreter during their hospitalization.

Among English speakers, Whites and African Americans were the most common ethnicities; however, more than 500 admissions were categorized as Asian ethnicity, and more than 600 as patients of other ethnicity. Close to 300 admissions were for Latinos. Among non‐English speakers, Chinese speakers had the largest number of admissions (n = 584), while Spanish and Russian speakers had similar numbers (n = 272 and 290 respectively).

Non‐English speakers were older, more likely to be female, more likely to be insured by Medicare, and more likely to have a higher comorbidity index score. While comorbidity scores were similar among non‐English speakers (Chinese 1.13 1.50; Russian 1.09 1.37; Spanish 1.06 1.30), they differed considerably among English speakers (White 0.94 1.29; African American 1.05 1.40; Asian 1.04 1.45; Latino 0.89 1.23; Other 0.91 1.29).

Hospital Outcome by Language‐Ethnicity Group (Table 2)

When aggregated together, non‐English speakers were somewhat more likely to be dead at 30‐days and have lower cost admissions; however, they did not differ from English speakers on LOS or readmission rates. While differences among disaggregated language‐ethnicity groups were not all statistically significant, English‐speaking Whites had the longest LOS (mean = 4.9 days) and highest costs (mean = $10,530). English‐speaking African Americans, Chinese and Spanish speakers had the highest 30‐day readmission rates; whereas, English‐speaking Latinos and Russian speakers had markedly lower 30‐day readmission rates (2.5% and 6.4%, respectively). Chinese speakers had the highest 30‐day mortality, followed by English speaking Whites and Asians.

We further investigated differences among English speakers to better understand the very high rate of readmission for African Americans and the very low rate for English‐speaking Latinos. African Americans were on average younger than other English speakers (55 19 years vs. 60 21 years; P < 0.001); but, they had higher comorbidity scores than other English speakers (1.05 1.40 vs. 0.94 1.31; P = 0.008), and were more likely to be admitted for non‐malignant blood disorders (eg, sickle cell disease), endocrine disorders (eg, diabetes mellitus), and circulatory disorders (eg, stroke). In contrast, English‐speaking Latinos were also younger than other English speakers (53 21 years vs. 59 20 years; P < 0.001), but they trended toward lower comorbidity scores (0.87 1.23 vs. 0.97 1.33; P = 0.2), and were more likely to be admitted for gastrointestinal and musculoskeletal disorders, and less likely to be admitted for malignancy and endocrine disorders.

Multivariate Analyses: Association of Aggregated and Disaggregated Language‐Ethnicity Groups With Hospital Outcomes (Table 3)

In multivariate models examining aggregated language‐ethnicity groups, non‐English speakers had a trend toward higher odds of readmission at 30‐days post‐discharge than the English‐speaking group (odds ratio [OR], 1.3; 95% confidence interval [CI], 1.0‐1.7). There were no significant differences for LOS, cost, or 30‐day mortality. Compared to English speakers, Chinese and Spanish speakers had 70% and 50% higher adjusted odds of readmission at 30‐days post‐discharge respectively, while Russian speakers' odds of readmission was not increased. Additionally, Chinese speakers had 7% shorter LOS than English‐speakers. There were no significant differences among any of the language‐ethnicity groups for 30‐day mortality. The increased odds of readmission for Chinese and Spanish speakers compared to English speakers was robust to reinclusion of the admissions with the top 1% LOS and top 1% cost.

Multivariate Analyses: Association of Language for Asians and Latinos, and of Ethnicity for English speakers, With Hospital Outcomes (Table 4)

Both Chinese and Spanish speakers had significantly higher odds of 30‐day readmission than their English speaking Asian and Latino counterparts. There were no significant differences in LOS, cost, or 30‐day mortality in this within‐ethnicity analysis. Among English speakers, admissions for patients with Asian ethnicity were 15% shorter and resulted in 9% lower costs than for Whites. While LOS and cost were similar for English‐speaking Latino and White admissions, English‐speaking Latinos had markedly lower odds of 30‐day readmission than their White counterparts. Whereas African‐Americans had 6% shorter LOS, 40% higher odds of readmission and 30% lower odds of mortality at 30‐days than English speaking Whites.

Patient Population and Setting

Our study examined patients admitted to the General Medicine Service at the University of California, San Francisco Medical Center (UCSF) between July 1, 2001 and June 30th, 2003, the time period during which UCSF participated in the Multicenter Hospitalist Trial (MHT) a prospective quasi‐randomized trial of hospitalist care for general medicine patients.13, 14

UCSF Moffitt‐Long Hospital is a 400‐bed urban academic medical center which provides services to the City and County of San Francisco, an ethnically and linguistically diverse area. UCSF employs staff language interpreters in Spanish, Chinese and Russian who travel to its many outpatient clinics, Comprehensive Cancer Center, Children's Hospital, as well as to Moffitt‐Long Hospital upon request; phone interpretation is also available when in‐person interpreters are not available, for off hours needs and for less common languages. During the period of this study there were no specific inpatient guidelines in place for use of interpretation services at UCSF, nor were there any specific interventions targeting LEP or non‐English speaking inpatients.

Patients were eligible for the MHT if they were 18 years of age or older and admitted at random to a hospitalist or non‐hospitalist physician (eg, outpatient general internist attending on average 1‐month/year); a minority of patients were cared for directly by their primary care physician while in the hospital, and were excluded. For purposes of our study, which merged MHT data with hospital administrative data on primary language, we further excluded all admissions for patients for whom primary language was missing (n = 5), whose listing was unknown or other language (n = 78), sign language (n = 3) or whose language was listed but was not one of the included languages (n = 258). Included languages were English, Chinese, Russian and Spanish. Because LOS and cost data were skewed, we excluded those admissions with the top 1% longest stays and the top 1% highest cost (n = 176); these exclusions did not alter the proportion of admissions across language and ethnicity. In addition, we excluded 102 admissions that were missing data on cost and 11 with costs <$500 and which were likely to be erroneous. Our research was approved by the UCSF Institutional Review Board.

Data Sources

We collected administrative data from Transition Systems Inc (TSI, Boston, MA) billing databases at UCSF as part of the MHT. These data include patient demographics, insurance, costs, ICD‐9CM diagnostic codes, admission and discharge dates in Uniform Bill 92 format. Patient mortality information was collected as part of the MHT using the National Death Index.14

Language data were collected from a separate patient‐registration database (STOR) at UCSF. Information on a patient's primary language is entered at the time each patient first registers at UCSF, whether for the index hospitalization or for prior clinic visits, and is based generally on patient self report. As part of our validation step, we cross‐checked 829 STOR language entries against patient reports and found 91% agreement with the majority of the errors classifying non‐English speakers as English‐speakers.

Measures

Admission Characteristics of the Sample

A total of 7023 patients were admitted to the General Medicine service, 5877 (84%) of whom were English‐speakers and 1146 (16%) non‐English‐speakers (Table 1). Overall, half of the admitted patients were women (50%), and the vast majority was insured (93%). The most common principal diagnoses were respiratory and gastrointestinal disorders. Only a small number of non‐English speakers 164 (14%) were recorded in the UCSF Interpreter Services database as having had any interaction with a professional staff interpreter during their hospitalization.

Among English speakers, Whites and African Americans were the most common ethnicities; however, more than 500 admissions were categorized as Asian ethnicity, and more than 600 as patients of other ethnicity. Close to 300 admissions were for Latinos. Among non‐English speakers, Chinese speakers had the largest number of admissions (n = 584), while Spanish and Russian speakers had similar numbers (n = 272 and 290 respectively).

Non‐English speakers were older, more likely to be female, more likely to be insured by Medicare, and more likely to have a higher comorbidity index score. While comorbidity scores were similar among non‐English speakers (Chinese 1.13 1.50; Russian 1.09 1.37; Spanish 1.06 1.30), they differed considerably among English speakers (White 0.94 1.29; African American 1.05 1.40; Asian 1.04 1.45; Latino 0.89 1.23; Other 0.91 1.29).

Hospital Outcome by Language‐Ethnicity Group (Table 2)

When aggregated together, non‐English speakers were somewhat more likely to be dead at 30‐days and have lower cost admissions; however, they did not differ from English speakers on LOS or readmission rates. While differences among disaggregated language‐ethnicity groups were not all statistically significant, English‐speaking Whites had the longest LOS (mean = 4.9 days) and highest costs (mean = $10,530). English‐speaking African Americans, Chinese and Spanish speakers had the highest 30‐day readmission rates; whereas, English‐speaking Latinos and Russian speakers had markedly lower 30‐day readmission rates (2.5% and 6.4%, respectively). Chinese speakers had the highest 30‐day mortality, followed by English speaking Whites and Asians.

We further investigated differences among English speakers to better understand the very high rate of readmission for African Americans and the very low rate for English‐speaking Latinos. African Americans were on average younger than other English speakers (55 19 years vs. 60 21 years; P < 0.001); but, they had higher comorbidity scores than other English speakers (1.05 1.40 vs. 0.94 1.31; P = 0.008), and were more likely to be admitted for non‐malignant blood disorders (eg, sickle cell disease), endocrine disorders (eg, diabetes mellitus), and circulatory disorders (eg, stroke). In contrast, English‐speaking Latinos were also younger than other English speakers (53 21 years vs. 59 20 years; P < 0.001), but they trended toward lower comorbidity scores (0.87 1.23 vs. 0.97 1.33; P = 0.2), and were more likely to be admitted for gastrointestinal and musculoskeletal disorders, and less likely to be admitted for malignancy and endocrine disorders.

Multivariate Analyses: Association of Aggregated and Disaggregated Language‐Ethnicity Groups With Hospital Outcomes (Table 3)

In multivariate models examining aggregated language‐ethnicity groups, non‐English speakers had a trend toward higher odds of readmission at 30‐days post‐discharge than the English‐speaking group (odds ratio [OR], 1.3; 95% confidence interval [CI], 1.0‐1.7). There were no significant differences for LOS, cost, or 30‐day mortality. Compared to English speakers, Chinese and Spanish speakers had 70% and 50% higher adjusted odds of readmission at 30‐days post‐discharge respectively, while Russian speakers' odds of readmission was not increased. Additionally, Chinese speakers had 7% shorter LOS than English‐speakers. There were no significant differences among any of the language‐ethnicity groups for 30‐day mortality. The increased odds of readmission for Chinese and Spanish speakers compared to English speakers was robust to reinclusion of the admissions with the top 1% LOS and top 1% cost.

Multivariate Analyses: Association of Language for Asians and Latinos, and of Ethnicity for English speakers, With Hospital Outcomes (Table 4)

Both Chinese and Spanish speakers had significantly higher odds of 30‐day readmission than their English speaking Asian and Latino counterparts. There were no significant differences in LOS, cost, or 30‐day mortality in this within‐ethnicity analysis. Among English speakers, admissions for patients with Asian ethnicity were 15% shorter and resulted in 9% lower costs than for Whites. While LOS and cost were similar for English‐speaking Latino and White admissions, English‐speaking Latinos had markedly lower odds of 30‐day readmission than their White counterparts. Whereas African‐Americans had 6% shorter LOS, 40% higher odds of readmission and 30% lower odds of mortality at 30‐days than English speaking Whites.