Article published In:
TargetVol. 35:1 (2023) ► pp.63–96
An item-based, Rasch-calibrated approach to assessing translation quality
Item-based scoring has been advocated as a psychometrically robust approach to translation quality assessment, outperforming traditional neo-hermeneutic and error analysis methods. The past decade has witnessed a succession of item-based scoring methods being developed and trialed, ranging from calibration of dichotomous items to preselected item evaluation. Despite this progress, these methods seem to be undermined by several limitations, such as the inability to accommodate the multifaceted reality of translation quality assessment and inconsistent item calibration procedures. Against this background, we conducted a methodological exploration, utilizing what we call an item-based, Rasch-calibrated method, to measure translation quality. This new method, built on the sophisticated psychometric model of many-facet Rasch measurement, inherits the item concept from its predecessors, but addresses previous limitations. In this article, we demonstrate its operationalization and provide an initial body of empirical evidence supporting its reliability, validity, and utility, as well as discuss its potential applications.
Article outline
- 1.Introduction
- 2.Literature review
- 2.1The developmental trajectory of item-based assessment
- 2.2Critical evaluation of CDI and PIE
- 2.3Item-based, Rasch-calibrated evaluation
- 3.Method
- 3.1Samples of English-to-Chinese translation
- 3.2Annotators and raters
- 3.3Item selection
- 3.4Item scoring
- 3.5Item calibration
- 3.6Post-hoc questionnaire and interview
- 3.7Data analysis
- 4.Results
- 4.1Reliability evidence
- 4.2Validity evidence
- 4.3Perceived utility
- 5.Discussion
- 6.Conclusion
- Acknowledgements
- Notes
-
References
References (37)
References
Bachman, Lyle F. 2004. Statistical Analyses for Language Assessment. Cambridge: Cambridge University Press.
Bond, Trevor G., and Christine M. Fox. 2015. Applying the Rasch Model: Fundamental Measurement in the Human Sciences. 3rd ed. New York: Routledge.
Campbell, Stuart J. 1991. “Towards a Model of Translation Competence.” Meta 36 (2–3): 329–343.
Colina, Sonia. 2008. “Translation Quality Evaluation: Some Empirical Evidence for a Functionalist Approach.” The Translator 14 (1): 97–134.
Eckes, Thomas. 2015. Introduction to Many-Facet Rasch Measurement: Analyzing and Evaluating Rater-Mediated Assessments. 2nd ed. Frankfurt am Main: Peter Lang.
Eyckmans, June, and Philippe Anckaert. 2017. “Item-based Assessment of Translation Competence: Chimera of Objectivity Versus Prospect of Reliable Measurement.” In Translator Quality – Translation Quality: Empirical Approaches to Assessment and Evaluation, edited by Geoffrey S. Koby and Isabel Lacruz, special issue of Linguistica Antverpiensia 161: 40–56.
Eyckmans, June, Philippe Anckaert, and Winibert Segers. 2009. “The Perks of Norm Referenced Translation Evaluation.” In Testing and Assessment in Translation and Interpreting Studies: A Call for Dialogue between Research and Practice, edited by Claudia V. Angelelli and Holly E. Jacobson, 73–93. Amsterdam: John Benjamins.
Eyckmans, June, Winibert Segers, and Philippe Anckaert. 2012. “Translation Assessment Methodology and the Prospects of European Collaboration.” In Collaboration in Language Testing and Assessment, edited by Dina Tsagari and Ildikó Csépes, 171–184. Frankfurt am Main: Peter Lang.
Green, Rita. 2013. Statistical Analyses for Language Testers. Basingstoke: Palgrave Macmillan.
Han, Chao. 2016. “Investigating Score Dependability in English/Chinese Interpreter Certification Performance Testing: A Generalizability Theory Approach.” Language Assessment Quarterly 13 (3): 186–201.
Han, Chao. 2017. “Using Analytic Rating Scales to Assess English/Chinese Bi-directional Interpretation: A Longitudinal Rasch Analysis of Scale Utility and Rater Behavior.” In Translator Quality – Translation Quality: Empirical Approaches to Assessment and Evaluation, edited by Geoffrey S. Koby and Isabel Lacruz, special issue of Linguistica Antverpiensia 161: 196–215.
Han, Chao. 2019. “A Generalizability Theory Study of Optimal Measurement Design for a Summative Assessment of English/Chinese Consecutive Interpreting.” Language Testing 36 (3): 419–438.
Han, Chao. 2020. “Translation Quality Assessment: A Critical Methodological Review.” The Translator 26 (3): 257–273.
House, Juliane. 2015. Translation Quality Assessment: Past and Present. Abingdon: Routledge.
IBM Corp. 2012. IBM SPSS Statistics for Windows. V. 21.0. Armonk, NY: IBM Corp.
Kockaert, Hendrik J., and Winibert Segers. 2012. “L’assurance qualité des traductions: items sélectionnés et évaluation assistée par ordinateur [Quality assurance of translations: Selected items and computer-assisted evaluation].” Meta 57 (1): 159–176.
Kockaert, Hendrik J., and Winibert Segers. 2017. “Evaluation of Legal Translations: PIE Method (Preselected Items Evaluation).” JoSTrans 271: 148–163.
Lauscher, Susanne. 2000. “Translation Quality Assessment: Where Can Theory and Practice Meet?” The Translator 6 (2): 149–168.
Linacre, John M. 1989. Many-Facet Rasch Measurement. Chicago: MESA Press.
Linacre, John M. 1999. “Investigating Rating Scale Category Utility.” Journal of Outcome Measurement 3 (2): 103–122.
Linacre, John M. 2002. “What Do Infit and Outfit, Mean-Square and Standardized Mean?” Rasch Measurement Transactions 16 (2): 878.
Linacre, John M. 2017. FACETS: Computer Program for Many Faceted Rasch Measurement. V. 3.80.0. Beaverton, OR: Winsteps.
Martínez Mateo, Robert. 2014. “A Deeper Look into Metrics for Translation Quality Assessment (TQA): A Case Study.” Miscelanea 491: 73–93.
Myford, Carol M., and Edward W. Wolfe. 2003. “Detecting and Measuring Rater Effects Using Many-Facet Rasch Measurement: Part I.” Journal of Applied Measurement 4 (4): 386–422.
O’Brien, Sharon. 2012. “Towards a Dynamic Quality Evaluation Model for Translation.” JoSTrans 171: 55–77.
Rasch, Georg. 1980. Probabilistic Models for Some Intelligence and Attainment Tests. Chicago: MESA Press.
Teague, Ben. 1987. “ATA Accreditation and Excellence in Practice.” In Translation Excellence: Assessment, Achievement, Maintenance, edited by Marilyn Gaddis Rose, 21–26. Amsterdam: John Benjamins.
Turner, Barry, Miranda Lai, and Neng Huang. 2010. “Error Deduction and Descriptors – A Comparison of Two Methods of Translation Test Assessment.” Translation & Interpreting 2 (1): 11–23.
Waddington, Christopher. 2001. “Should Translations Be Assessed Holistically or Through Error Analysis?” Hermes 261: 15–38.
Williams, Malcolm. 1989. “The Assessment of Professional Translation Quality: Creating Credibility out of Chaos.” TTR 2 (2): 13–33.
Wind, Stefanie A., and Meghan E. Peterson. 2018. “A Systematic Review of Methods for Evaluating Rating Quality in Language Assessment.” Language Testing 35 (2): 161–192.
Cited by (2)
Cited by two other publications
Gong, Min
2024.
The neural network algorithm-based quality assessment method for university English translation.
Network: Computation in Neural Systems ► pp. 1 ff.
Han, Chao, Xiaolei Lu & Peixin Zhang
This list is based on CrossRef data as of 5 july 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.