From the Item Bank

The Professional Testing Blog

Key Considerations for Translating Examinations and Pitfalls to Avoid

We recently provided a few broad “considerations” on the topic of translation of exams for cultural adaptation to a client. Translation of an examination, must be done correctly in order to assure fairness. There are many steps required to help assure comparability between native (source) and non-native (target) exam forms.

1. Comparability of content must be reviewed to assure that there are no local differences.

There may be different dialects within a given language. A decision must be made to determine which dialect best fits the program’s needs. Given the expense associated with translating exams, getting this correct up front is essential. One key consideration in making this decision is the dialect candidates are most familiar with.

2. The final version of the translation should be reviewed by bilingual SMEs. The original translation can be sourced to a translation organization, but it still needs to be reviewed and approved by the bilingual SMEs to assure comparability.

Finding bilingual SMEs can be challenging. The SMEs hold the key to “good” translation because we will not know the adequacy of the translation until the test has been administered and statistics have been calculated. Not only should SMEs be strong with respect to grammar and language for two different languages, but they also must be expert in the technical content of the certification program.

3. After forms are created, technical assurances must be met to assure that the passing standards are equivalent. Additional pass rate and technical test reporting activities must be conducted to satisfy accreditation standards.

Steps need to be taken to assure that the translated forms meet the same difficulty level of the original form. Translating items can inadvertently make items either more difficult or less difficult. Regardless of how good the translation is, there is still a threat to the comparability of exam scores by virtue of the actual translation. In addition, if resources permit, a beta administration should be conducted with a convenience sample of examinees to confirm that seat time is the same between the source and target language versions of the exam.

4. A statistical research study called differential item functioning (DIF) must be conducted. For these research studies, the focal (English) and reference groups (translated items) are compared using nonparametric statistics.

Certifying bodies have a responsibility to demonstrate that translated forms and their respective cut-scores are comparable and fair. One approach is to conduct a Differential Item Functioning (DIF) study. This study determines whether translated items differ from their original form using statistics.

5. If significant differences are found on items, based on the DIF analysis, a bilingual SME DIF panel is convened to determine if the differences are due to translation or actual differences in ability.

After the DIF study is completed, a bilingual SME meeting is required to review the results of the study. Any item exhibiting DIF is presented to the SMEs in both languages to determine if there is a translation bias. If it is determined that items perform differentially between examinees of different language versions yet of equal ability, then those items are deemed to be biased and appropriate scoring changes may need to be made, including the passing score and possible rescoring all candidates.  This may result in  reissuing pass/fail reports to candidates affected by the bias.  This is a scenario you want to avoid.

6. As new forms are introduced in the foreign language; the process is repeated. The approach presented above is based on translating existing exam forms. Some organizations may choose to translate the entire item bank. Translating the item bank brings even more challenges.

It is a very expensive and time-consuming process requiring additional psychometric and SME work/expense. Organizations should wait until there is sufficient demand before committing resources.

A word of caution about translation. We have seen a number of organizations consider translating exams based on a perception that it will grow the credential or be relevant for “marketing” purposes.  Translated programs should generate enough revenue from the target language to support the maintenance of translation and the examination.  If the “if you build it, they will come” model doesn’t work, then you may be opening  Pandora’s box. .

For more detailed discussions on the topic of test translation and adaptation, written by Dr. Joy Matthews-Lopez, please visit our blog on our blog at: http://www.proftesting.com/blog/?s=test+adaptation

Categorized in: Industry News, Test Development