Cover not available
Article published In:
Target
Vol. 26:1 (2014) ► pp.98–127

Measuring translation difficulty

An empirical study

| Beijing Foreign Studies University
| Kent State University

The purpose of this study was to find a method to measure difficulty in a translation task. Readability formulas have been suggested to be a useful tool and yet this needs to be empirically tested. In this study, NASA Task Load Index, a multidimensional scale for measuring mental workload, was used to assess the level of translation difficulty for the translator. It was found that a text’s readability only partially accounts for its translation difficulty level. Translation quality score was found to be an unreliable indicator of translation difficulty level, while time-on-task was significantly, but weakly, related to translation difficulty level. A formula was developed to predict a text’s translation difficulty level for a translator by using the translator’s pre-translation rating.

Keywords: translation, difficulty, empirical workload, cognitive, readability, competence, measurement, method
Article outline
  • 1.Introduction
  • 2.Methods
    • 2.1Participants
    • 2.2Materials
      • Test passages
      • Translation ratings
    • 2.3Design and procedure
      • Experimental design
      • Test procedure
      • Grading the translations
      • Calculating interrater reliability
      • Checking for normality
  • 3.Results
    • 3.1Checking NASA-TLX’s reliability
    • 3.2Translation difficulty level and translation quality score
    • 3.3Translation difficulty level and time on task
    • 3.4Translation difficulty level and text readability
    • 3.5Predicting the level of translation difficulty of a text
  • 4.Discussion
    • 4.1Readability formulas and translation difficulty
    • 4.2Good translators and poor translators
    • 4.3Workload and performance
    • 4.4Limitations of this study and suggestions for further study
  • Appendixes
  • References
Published online: 7 March 2014
DOI logo
https://doi.org/10.1075/target.26.1.04sun
References
Anagnostou, Nikolaos K
., and George R.S. Weir 2007 “From Corpus-Based Collocation Frequencies to Readability Measure.” In Texts, Textbooks and Readability , ed. by George R. S. Weir, and Toshiaki Ozasa, 34–48. Glasgow: University of Strathclyde Publishing.Google Scholar
Baker, Mona
(1992) 2011  In Other Words: A Coursebook on Translation . London: Routledge.   DOI logoGoogle Scholar
Bastable, Susan Bacorn
2008  Nurse as Educator: Principles of Teaching and Learning for Nursing Practice . 3rd ed. London: Jones and Bartlett.Google Scholar
Campbell, Stuart
and Sandra Hale 2003 “Translation and Interpreting Assessment in the Context of Educational Measurement.” In Translation Today: Trends and Perspectives , ed. by Gunilla M. Anderman, and Margaret Rogers, 205–224. Clevedon: Multilingual Matters. DOI logoGoogle Scholar
1999
 “What Makes a Text Difficult to Translate?” Available at: [URL] (accessed Nov 1 2009).
Carrell, Patricia L
., and Ulla Connor 1991 “Reading and Writing Descriptive and Persuasive Texts.” The Modern Language Journal 75 (3): 314–324. DOI logoGoogle Scholar
Chall, Jeanne Sternlicht, and Edgar Dale
1995  Readability Revisited: The New Dale-Chall Readability Formula .Cambridge, MA: Brookline Books.Google Scholar
Chen, Peter Y
., and Autumn D. Krauss 2004a “Internal Reliability.” In The Sage Encyclopedia of Social Science Research Methods , ed. by Michael S. Lewis-Beck, Alan Bryman, and Tim Futing Liao, 501–502. ThousandOaks, CA: Sage.Google Scholar
2004b
 “Reliability.” In The Sage Encyclopedia of Social Science Research Methods, ed. by Michael S. Lewis-Beck, Alan Bryman, and Tim Futing Liao, 952–956. ThousandOaks, CA: Sage.   DOI logoGoogle Scholar
Cohen, Jacob
1988  Statistical Power Analysis for the Behavioral Sciences . 2nd ed.Hillsdale, NJ: L. Erlbaum Associates.Google Scholar
Cronbach, Lee J
1951 “Coefficient Alpha and the Internal Structure of Tests.” Psychometrika 16 (3): 297–334. DOI logoGoogle Scholar
Dahl, Östen
2004  The Growth and Maintenance of Linguistic Complexity . Amsterdam: John Benjamins.   DOI logoGoogle Scholar
Davidson, Janet E
., and Robert J. Sternberg eds 2003  The Psychology of Problem Solving . Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Dragsted, B.
2004  Segmentation in Translation and Translation Memory Systems: An Empirical Investigation of Cognitive Segmentation and Effects of Integrating a Tm-System into the Translation Process . PhD diss. Copenhagen Business School.Google Scholar
DuBay, William H
2004 “The Principles of Readability.” Available at: [URL] (accessed November 1 2009).
Englund Dimitrova, Birgitta
2005  Expertise and Explicitation in the Translation Process . Amsterdam: John Benjamins.   DOI logoGoogle Scholar
Feldt, Leonard S
1961 “The Use of Extreme Groups to Test for the Presence of a Relationship.” Psychometrika 26 (3): 307–316. DOI logoGoogle Scholar
Gerloff, Pamela
1988  From French to English: A Look at the Translation Process in Students, Bilinguals, and Professional Translators . PhD. diss. Harvard University.Google Scholar
Green, Samuel B
1991 “How Many Subjects Does It Take to Do a Regression Analysis.” Multivariate Behavioral Research 26 (3): 499–510. DOI logoGoogle Scholar
Hale, Sandra
and Stuart Campbell 2002 “The Interaction between Text Difficulty and Translation Accuracy.” Babel 48 (1): 14–33. DOI logoGoogle Scholar
Hart, Sandra G
., and Lowell E. Staveland 1988 “Development of Nasa-Tlx (Task Load Index): Results of Empirical and Theoretical Research.” In Human Mental Workload , ed. by Peter A. Hancock, and Najmedin Meshkati, 139–183. Amsterdam: North-Holland. DOI logoGoogle Scholar
Higgins, Jennifer, Michael Russell
and Thomas Hoffmann 2005.”Examining the Effect of Computer-Based Passage Presentation on Reading Test Performance.” The Journal of Technology, Learning, and Assessment 41. [URL].
House, Juliane
2009 “Quality.” In Routledge Encyclopedia of Translation Studies , ed. by Mona Baker, and Gabriela Saldanha, 222–225. London: Routledge.Google Scholar
Hoyt, Linda, Margaret E. Mooney, and Brenda Parkes
, eds 2003  Exploring Informational Texts: From Theory to Practice .Portsmouth, NH: Heinemann.Google Scholar
Jääskeläinen, Riitta
1990  Features of Successful Translation Processes: A Think-aloud Protocol Study . Unpublished licentiate thesis, University of Joensuu, Savonlinna School of Translation Studies.Google Scholar
1996
 “Hard Work Will Bear Beautiful Fruit. A Comparison of Two Think-Aloud Protocol Studies.” Meta 41 (1): 60–74. DOI logoGoogle Scholar
1999
  Tapping the Process: An Explorative Study of the Cognitive and Affective Factors Involved in Translating . University of Joensuu Publications in the Humanities 22. Joensuu: University of Joensuu.Google Scholar
Jensen, Kristian T.H.
2009 “Indicators of Text Complexity.” Copenhagen Studies in Language 371: 61–80.Google Scholar
Jex, Henry. R.
1988 “Measuring Mental Workload: Problems, Progress, and Promises.” In Human Mental Workload, Advances in Psychology 52 , ed. by Peter. A. Hancock, and Najmedin Meshkati, 5–38. Amsterdam: North-Holland.   DOI logoGoogle Scholar
Kelley, Truman L
1939 “The Selection of Upper and Lower Groups for the Validation of Test Items.” Journal of Educational Psychology 301: 17–24. DOI logoGoogle Scholar
Kiraly, Donald Charles
1995  Pathways to Translation: From Process to Pedagogy .Kent, OH: Kent State University Press.Google Scholar
Krings, Hans P
2001  Repairing Texts: Empirical Investigations of Machine Translation Post-Editing Processes . Translated by Geoffrey Koby, Gregory Shreve, Katjz Mischerikow, and Sarah Litzer. Kent, OH: Kent State University Press.Google Scholar
Moray, Neville
1979  Mental Workload: Its Theory and Measurement . New York: Plenum Press. DOI logoGoogle Scholar
Moroney, William F
., David W. Biers, F. Thomas Eggemeier, and Jennifer A. Mitchell 1992 “A Comparison of Two Scoring Procedures with the Nasa Task Load Index in a Simulated Flight Task.” In Proceedings of the IEEE 1992 National Aerospace and Electronics Conference - Naecon 1992 , Dayton, 18–22 May 1992, 734–740.   DOI logo
Nord, Christiane
2005  Text Analysis in Translation: Theory, Methodology, and Didactic Application of a Model for Translation-Oriented Text Analysis . 2nd ed. Amsterdam: Rodopi.Google Scholar
Nunnally, Jum C
1978  Psychometric Theory . 2nd ed. New York: McGraw-Hill.Google Scholar
O’Donnell, Robert D
., and F. Thomas Eggemeier 1986 “Workload Assessment Methodology.” In Handbook of Perception and Human Performance. Vol. 2: Cognitive Processes and Performance , ed. by Kenneth R. Boff, Lloyd Kaufman, and James P. Thomas, 42/41–42/49. New York: Wiley.Google Scholar
Paas, Fred G
1992 “Training Strategies for Attaining Transfer of Problem-Solving Skill in Statistics: A Cognitive-Load Approach.” Journal of Educational Psychology 84 (4): 429–434. DOI logoGoogle Scholar
PACTE
2005 “Investigating Translation Competence: Conceptual and Methodological Issues.” Meta 50 (2): 609–619. DOI logoGoogle Scholar
Pallant, Julie
2011  Spss Survival Manual . 4th ed. Crows Nest (Australia): Allen & Unwin.Google Scholar
Pedhazur, Elazar J
1997  Multiple Regression in Behavioral Research: Explanation and Prediction . 3rd ed. United States: Wadsworth/Thomson Learning.Google Scholar
Preacher, Kristopher J
., Derek D. Rucker, Robert C. MacCallum, and W. Alan Nicewander 2005 “Use of the Extreme Groups Approach: A Critical Reexamination and New Recommendations.” Psychological Methods 10 (2): 178–192. DOI logoGoogle Scholar
Pretz, Jean E
., Adam J. Naples, and Robert J. Sternberg 2003 “Recognizing, Defining, and Representing Problems.” In The Psychology of Problem Solving , ed. by Janet E. Davidson, and Robert J. Sternberg, 3–30. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Pym, Anthony
2010  Exploring Translation Theories . London: Routledge.Google Scholar
2003
 “Redefining Translation Competence in an Electronic Age: In Defence of a Minimalist Approach.” Meta 48 (4): 481–497. DOI logoGoogle Scholar
Rayner, Keith, and Alexander Pollatsek
1989  The Psychology of Reading .Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
Shreve, Gregory M
2002 “Knowing Translation: Cognitive and Experiential Aspects of Translation Expertise from the Perspective of Expertise Studies.” In Translation Studies: Perspectives on an Emerging Discipline , ed. by Alessandra Riccardi, 150–173. Cambridge: Cambridge University Press.Google Scholar
Sweller, John, Paul L. Ayres
and Slava Kalyuga 2011  Cognitive Load Theory . New York: Springer. DOI logoGoogle Scholar
Taylor, Wilson L
1953 “‘Cloze Procedure’: A New Tool for Measuring Readability.” Journalism Quarterly 301: 415–433. DOI logoGoogle Scholar
Tirkkonen-Condit, Sonja
1987 “Think-Aloud Protocols in the Study of the Translation Process.” In CDEF 86: Papers from the Conference of Departments of English in Finland , ed. by Heikki Nyyssönen, Riitta Kataja, and Vesa Komulainen, 39–49. Oulu: University of Oulu.Google Scholar
Tsang, Pamela S
2006 “Mental Workload.” In International Encyclopedia of Ergonomics and Human Factors , ed. by Waldemar Karwowski, 809–813. BocaRaton, FL: CRC/Taylor & Francis.Google Scholar
Vidulich, Michael A
1988 “The Cognitive Psychology of Subjective Mental Workload.” In Human Mental Workload , ed. by Peter A. Hancock, and Najmedin Meshkati, 219–229. Amsterdam: North-Holland. DOI logoGoogle Scholar
Wilson, Glenn F
.. and F. Thomas Eggemeier 2006 “Mental Workload Measurement.” In International Encyclopedia of Ergonomics and Human Factors , ed. by Waldemar Karwowski, 814–817. BocaRaton, FL: CRC/Taylor & Francis.Google Scholar
Wilss, Wolfram
1982  The Science of Translation: Problems and Methods . Tübinger Beitraäge zur Linguistik. Tuübingen: G. Narr.Google Scholar
Zamanian, Mostafa, and Pooneh Heydari
2012 “Readability of Texts: State of the Art.” Theory and Practice in Language Studies 2 (1): 43–53. DOI logoGoogle Scholar
Zimmerman, Lynn W
2010  ESL, EFL, and Bilingual Education: Exploring Historical, Sociocultural, Linguistic, and Instructional Foundations .Charlotte, NC: Information Age Pub.Google Scholar
Zipf, George Kingsley
1935  The Psycho-Biology of Language: An Introduction to Dynamic Philology . Boston: Houghton Mifflin.Google Scholar
Cited by

Cited by 28 other publications

A.S. Mohammed, Tawffeek
2023. Translating Islamic Media Discourse from Arabic into English: An Analysis of Translation Process. In Translatology, Translation and Interpretation - Toward a New Scientific Endeavor, DOI logo
Abdel Latif, Muhammad M. M.
2018. Towards a typology of pedagogy-oriented translation and interpreting research. The Interpreter and Translator Trainer 12:3 ► pp. 322 ff. DOI logo
Abdel Latif, Muhammad M. M.
2020. Translation and Interpreting Assessment Research. In Translator and Interpreter Education Research [New Frontiers in Translation Studies, ], ► pp. 61 ff. DOI logo
Angelone, Erik
2018. Chapter 2. Reconceptualizing problems in translation using triangulated process and product data. In Innovation and Expansion in Translation Process Research [American Translators Association Scholarly Monograph Series, XVIII], ► pp. 17 ff. DOI logo
Araghi, Sahar, Alfons Palangkaraya & Elizabeth Webster
2023. The impact of language translation quality on commerce: The example of patents. Journal of International Business Policy DOI logo
Atabekova, Anastasia, Valentina Stepanova, Natalia Udina, Olga Gorbatenko & Tatyana Shoustikova
2017. Should Math Tools and Quantitative Methods be Part of University-based Translator and Interpreter’s Training? Russian Graduates’ Voices in the Focus. EURASIA Journal of Mathematics, Science and Technology Education 13:8 DOI logo
Azari, Razieh, Marziyeh Khalilizadeh Ganjalikhani & Anahita Amirshoja’i
2018. Legislation for patient information leaflets in Iran: Focus on lay-friendliness. Health Promotion Perspectives 8:4 ► pp. 263 ff. DOI logo
Chen, Xia & Jackie Xiu Yan
2024. Examining student translators’ writing and translation products: quality, errors and self-perceived mental workload. The Interpreter and Translator Trainer ► pp. 1 ff. DOI logo
Chmiel, Agnieszka & Nicoletta Spinolo
2022. Testing the impact of remote interpreting settings on interpreter experience and performance. Translation, Cognition & Behavior 5:2 ► pp. 250 ff. DOI logo
Gieshoff, Anne Catherine & Andrea Hunziker Heeb
2023. Cognitive load and cognitive effort. Translation, Cognition & Behavior 6:1 ► pp. 3 ff. DOI logo
Hu, Zhenming, Binghan Zheng & Xiangling Wang
2021. The impact of a metacognitive self-regulation inventory in translator self-training: a pre-post study with English-Chinese translation students. The Interpreter and Translator Trainer 15:4 ► pp. 430 ff. DOI logo
Jia, Yanfang & Sanjun Sun
2023. Man or machine? Comparing the difficulty of human translation versus neural machine translation post-editing. Perspectives 31:5 ► pp. 950 ff. DOI logo
Jiang, Xinlei & Yue Jiang
2020. Effect of dependency distance of source text on disfluencies in interpreting. Lingua 243 ► pp. 102873 ff. DOI logo
Kuang, Huolingxiao & Binghan Zheng
2022. How does interpreting performance correlate with note-taking process, note-taking product and note-reading process? An eye-tracking and pen-recording study. Across Languages and Cultures 23:2 ► pp. 167 ff. DOI logo
Kuang, Huolingxiao & Binghan Zheng
2023. Note-taking effort in video remote interpreting: effects of source speech difficulty and interpreter work experience. Perspectives 31:4 ► pp. 724 ff. DOI logo
Liu, Yanmei, Binghan Zheng & Hao Zhou
2019. Measuring the difficulty of text translation. Target 31:1 ► pp. 125 ff. DOI logo
Prieto Ramos, Fernando & Giorgina Cerutti
2021. Terminology as a source of difficulty in translating international legal discourses: an empirical cross-genre study. International Journal of Legal Discourse 6:2 ► pp. 155 ff. DOI logo
Sun, Sanjun
2019. Measuring Difficulty in Translation and Post-editing: A Review. In Researching Cognitive Processes of Translation [New Frontiers in Translation Studies, ], ► pp. 139 ff. DOI logo
Szarkowska, Agnieszka, Krzysztof Krejtz, Łukasz Dutka & Olga Pilipczuk
2016. Cognitive load in intralingual and interlingual respeaking – a preliminary study. Poznan Studies in Contemporary Linguistics 52:2 DOI logo
Vanroy, Bram, Orphée De Clercq, Arda Tezcan, Joke Daems & Lieve Macken
2021. Metrics of Syntactic Equivalence to Assess Translation Difficulty. In Explorations in Empirical Translation Process Research [Machine Translation: Technologies and Applications, 3], ► pp. 259 ff. DOI logo
Wang, Lulu & Sanjun Sun
2023. Dictating translations with automatic speech recognition: Effects on translators’ performance. Frontiers in Psychology 14 DOI logo
Wang, Yu & Ali Jalalian Daghigh
2023. Cognitive effort in human translation and machine translation post-editing processes. FORUM. Revue internationale d’interprétation et de traduction / International Journal of Interpretation and Translation 21:1 ► pp. 139 ff. DOI logo
Wu, Zhiwei
2019. Text characteristics, perceived difficulty and task performance in sight translation. Interpreting. International Journal of Research and Practice in Interpreting 21:2 ► pp. 196 ff. DOI logo
Yan, Hao, Yi Zhang, Yanqin Feng, Yang Li, Yueting Zhang, Yujun Lee, Maoqing Chen, Zijuan Shi, Yuan Liang, Yuqin Hei & Xu Duan
2024. Assessing mental demand in consecutive interpreting: Insights from an fNIRS study. Acta Psychologica 243 ► pp. 104132 ff. DOI logo
Yang, Chengsong, Yimeng Wang & Na Fan
2022. Are parallel translation tasks parallel in difficulty? an eye-tracking study. Perspectives 30:4 ► pp. 711 ff. DOI logo
Yarbrough, Nükhet D.
2016. Assessment of Creative Thinking Across Cultures Using the Torrance Tests of Creative Thinking (TTCT): Translation and Validity Issues. Creativity Research Journal 28:2 ► pp. 154 ff. DOI logo
Yuan, Rongjie
2022. Material development for beginner student interpreters: how does text structure contribute to the difficulty of consecutive interpreting?. The Interpreter and Translator Trainer 16:1 ► pp. 58 ff. DOI logo
Zhou, Xiangyan, Xiangling Wang & Xiaodong Liu
2022. The impact of task complexity and translating self-efficacy belief on students’ translation performance: Evidence from process and product data. Frontiers in Psychology 13 DOI logo

This list is based on CrossRef data as of 8 april 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.