Cover not available
Article published In:
Target
Vol. 26:1 (2014) ► pp.98–127

Measuring translation difficulty

An empirical study

 | Beijing Foreign Studies University
 | Kent State University
The purpose of this study was to find a method to measure difficulty in a translation task. Readability formulas have been suggested to be a useful tool and yet this needs to be empirically tested. In this study, NASA Task Load Index, a multidimensional scale for measuring mental workload, was used to assess the level of translation difficulty for the translator. It was found that a text’s readability only partially accounts for its translation difficulty level. Translation quality score was found to be an unreliable indicator of translation difficulty level, while time-on-task was significantly, but weakly, related to translation difficulty level. A formula was developed to predict a text’s translation difficulty level for a translator by using the translator’s pre-translation rating.
Keywords: translation, difficulty, empirical workload, cognitive, readability, competence, measurement, method
Article outline
  • 1.Introduction
  • 2.Methods
    • 2.1Participants
    • 2.2Materials
      • Test passages
      • Translation ratings
    • 2.3Design and procedure
      • Experimental design
      • Test procedure
      • Grading the translations
      • Calculating interrater reliability
      • Checking for normality
  • 3.Results
    • 3.1Checking NASA-TLX’s reliability
    • 3.2Translation difficulty level and translation quality score
    • 3.3Translation difficulty level and time on task
    • 3.4Translation difficulty level and text readability
    • 3.5Predicting the level of translation difficulty of a text
  • 4.Discussion
    • 4.1Readability formulas and translation difficulty
    • 4.2Good translators and poor translators
    • 4.3Workload and performance
    • 4.4Limitations of this study and suggestions for further study
  • Appendixes
  • References
Published online: 7 March 2014
DOI logo
https://doi.org/10.1075/target.26.1.04sun
References (58)
References
Anagnostou, Nikolaos K., and George R.S. Weir. 2007. “From Corpus-Based Collocation Frequencies to Readability Measure.” In Texts, Textbooks and Readability , ed. by George R. S. Weir, and Toshiaki Ozasa, 34–48. Glasgow: University of Strathclyde Publishing.Google Scholar
Baker, Mona. (1992) 2011. In Other Words: A Coursebook on Translation . London: Routledge.   DOI logoGoogle Scholar
Bastable, Susan Bacorn. 2008. Nurse as Educator: Principles of Teaching and Learning for Nursing Practice . 3rd ed. London: Jones and Bartlett.Google Scholar
Campbell, Stuart and Sandra Hale. 2003. “Translation and Interpreting Assessment in the Context of Educational Measurement.” In Translation Today: Trends and Perspectives , ed. by Gunilla M. Anderman, and Margaret Rogers, 205–224. Clevedon: Multilingual Matters. DOI logoGoogle Scholar
1999. “What Makes a Text Difficult to Translate?” Available at: [URL] (accessed Nov. 1, 2009).
Carrell, Patricia L., and Ulla Connor. 1991. “Reading and Writing Descriptive and Persuasive Texts.” The Modern Language Journal 75 (3): 314–324. DOI logoGoogle Scholar
Chall, Jeanne Sternlicht, and Edgar Dale. 1995. Readability Revisited: The New Dale-Chall Readability Formula .Cambridge, MA: Brookline Books.Google Scholar
Chen, Peter Y., and Autumn D. Krauss. 2004a. “Internal Reliability.” In The Sage Encyclopedia of Social Science Research Methods , ed. by Michael S. Lewis-Beck, Alan Bryman, and Tim Futing Liao, 501–502. ThousandOaks, CA: Sage.Google Scholar
2004b. “Reliability.” In The Sage Encyclopedia of Social Science Research Methods, ed. by Michael S. Lewis-Beck, Alan Bryman, and Tim Futing Liao, 952–956. ThousandOaks, CA: Sage.   DOI logoGoogle Scholar
Cohen, Jacob. 1988. Statistical Power Analysis for the Behavioral Sciences . 2nd ed.Hillsdale, NJ: L. Erlbaum Associates.Google Scholar
Cronbach, Lee J. 1951. “Coefficient Alpha and the Internal Structure of Tests.” Psychometrika 16 (3): 297–334. DOI logoGoogle Scholar
Dahl, Östen. 2004. The Growth and Maintenance of Linguistic Complexity . Amsterdam: John Benjamins.   DOI logoGoogle Scholar
Davidson, Janet E., and Robert J. Sternberg, eds. 2003. The Psychology of Problem Solving . Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Dragsted, B. 2004. Segmentation in Translation and Translation Memory Systems: An Empirical Investigation of Cognitive Segmentation and Effects of Integrating a Tm-System into the Translation Process . PhD diss. Copenhagen Business School.Google Scholar
DuBay, William H. 2004. “The Principles of Readability.” Available at: [URL] (accessed November 1, 2009).
Englund Dimitrova, Birgitta. 2005. Expertise and Explicitation in the Translation Process . Amsterdam: John Benjamins.   DOI logoGoogle Scholar
Feldt, Leonard S. 1961. “The Use of Extreme Groups to Test for the Presence of a Relationship.” Psychometrika 26 (3): 307–316. DOI logoGoogle Scholar
Gerloff, Pamela. 1988. From French to English: A Look at the Translation Process in Students, Bilinguals, and Professional Translators . PhD. diss. Harvard University.Google Scholar
Green, Samuel B. 1991. “How Many Subjects Does It Take to Do a Regression Analysis.” Multivariate Behavioral Research 26 (3): 499–510. DOI logoGoogle Scholar
Hale, Sandra and Stuart Campbell. 2002. “The Interaction between Text Difficulty and Translation Accuracy.” Babel 48 (1): 14–33. DOI logoGoogle Scholar
Hart, Sandra G., and Lowell E. Staveland. 1988. “Development of Nasa-Tlx (Task Load Index): Results of Empirical and Theoretical Research.” In Human Mental Workload , ed. by Peter A. Hancock, and Najmedin Meshkati, 139–183. Amsterdam: North-Holland. DOI logoGoogle Scholar
Higgins, Jennifer, Michael Russell and Thomas Hoffmann. 2005.”Examining the Effect of Computer-Based Passage Presentation on Reading Test Performance.” The Journal of Technology, Learning, and Assessment 41. [URL].
House, Juliane. 2009. “Quality.” In Routledge Encyclopedia of Translation Studies , ed. by Mona Baker, and Gabriela Saldanha, 222–225. London: Routledge.Google Scholar
Hoyt, Linda, Margaret E. Mooney, and Brenda Parkes, eds. 2003. Exploring Informational Texts: From Theory to Practice .Portsmouth, NH: Heinemann.Google Scholar
Jääskeläinen, Riitta. 1990. Features of Successful Translation Processes: A Think-aloud Protocol Study . Unpublished licentiate thesis, University of Joensuu, Savonlinna School of Translation Studies.Google Scholar
1996. “Hard Work Will Bear Beautiful Fruit. A Comparison of Two Think-Aloud Protocol Studies.” Meta 41 (1): 60–74. DOI logoGoogle Scholar
1999. Tapping the Process: An Explorative Study of the Cognitive and Affective Factors Involved in Translating . University of Joensuu Publications in the Humanities 22. Joensuu: University of Joensuu.Google Scholar
Jensen, Kristian T.H. 2009. “Indicators of Text Complexity.” Copenhagen Studies in Language 371: 61–80.Google Scholar
Jex, Henry. R. 1988. “Measuring Mental Workload: Problems, Progress, and Promises.” In Human Mental Workload, Advances in Psychology 52 , ed. by Peter. A. Hancock, and Najmedin Meshkati, 5–38. Amsterdam: North-Holland.   DOI logoGoogle Scholar
Kelley, Truman L. 1939. “The Selection of Upper and Lower Groups for the Validation of Test Items.” Journal of Educational Psychology 301: 17–24. DOI logoGoogle Scholar
Kiraly, Donald Charles. 1995. Pathways to Translation: From Process to Pedagogy .Kent, OH: Kent State University Press.Google Scholar
Krings, Hans P. 2001. Repairing Texts: Empirical Investigations of Machine Translation Post-Editing Processes . Translated by Geoffrey Koby, Gregory Shreve, Katjz Mischerikow, and Sarah Litzer. Kent, OH: Kent State University Press.Google Scholar
Moray, Neville. 1979. Mental Workload: Its Theory and Measurement . New York: Plenum Press. DOI logoGoogle Scholar
Moroney, William F., David W. Biers, F. Thomas Eggemeier, and Jennifer A. Mitchell. 1992. “A Comparison of Two Scoring Procedures with the Nasa Task Load Index in a Simulated Flight Task.” In Proceedings of the IEEE 1992 National Aerospace and Electronics Conference - Naecon 1992 , Dayton, 18–22 May 1992, 734–740.   DOI logo
Nord, Christiane. 2005. Text Analysis in Translation: Theory, Methodology, and Didactic Application of a Model for Translation-Oriented Text Analysis . 2nd ed. Amsterdam: Rodopi.Google Scholar
Nunnally, Jum C. 1978. Psychometric Theory . 2nd ed. New York: McGraw-Hill.Google Scholar
O’Donnell, Robert D., and F. Thomas Eggemeier. 1986. “Workload Assessment Methodology.” In Handbook of Perception and Human Performance. Vol. 2: Cognitive Processes and Performance , ed. by Kenneth R. Boff, Lloyd Kaufman, and James P. Thomas, 42/41–42/49. New York: Wiley.Google Scholar
Paas, Fred G. 1992. “Training Strategies for Attaining Transfer of Problem-Solving Skill in Statistics: A Cognitive-Load Approach.” Journal of Educational Psychology 84 (4): 429–434. DOI logoGoogle Scholar
PACTE. 2005. “Investigating Translation Competence: Conceptual and Methodological Issues.” Meta 50 (2): 609–619. DOI logoGoogle Scholar
Pallant, Julie. 2011. Spss Survival Manual . 4th ed. Crows Nest (Australia): Allen & Unwin.Google Scholar
Pedhazur, Elazar J. 1997. Multiple Regression in Behavioral Research: Explanation and Prediction . 3rd ed. United States: Wadsworth/Thomson Learning.Google Scholar
Preacher, Kristopher J., Derek D. Rucker, Robert C. MacCallum, and W. Alan Nicewander. 2005. “Use of the Extreme Groups Approach: A Critical Reexamination and New Recommendations.” Psychological Methods 10 (2): 178–192. DOI logoGoogle Scholar
Pretz, Jean E., Adam J. Naples, and Robert J. Sternberg. 2003. “Recognizing, Defining, and Representing Problems.” In The Psychology of Problem Solving , ed. by Janet E. Davidson, and Robert J. Sternberg, 3–30. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Pym, Anthony. 2010. Exploring Translation Theories . London: Routledge.Google Scholar
2003. “Redefining Translation Competence in an Electronic Age: In Defence of a Minimalist Approach.” Meta 48 (4): 481–497. DOI logoGoogle Scholar
Rayner, Keith, and Alexander Pollatsek. 1989. The Psychology of Reading .Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
Shreve, Gregory M. 2002. “Knowing Translation: Cognitive and Experiential Aspects of Translation Expertise from the Perspective of Expertise Studies.” In Translation Studies: Perspectives on an Emerging Discipline , ed. by Alessandra Riccardi, 150–173. Cambridge: Cambridge University Press.Google Scholar
Sweller, John, Paul L. Ayres, and Slava Kalyuga. 2011. Cognitive Load Theory . New York: Springer. DOI logoGoogle Scholar
Taylor, Wilson L. 1953. “‘Cloze Procedure’: A New Tool for Measuring Readability.” Journalism Quarterly 301: 415–433. DOI logoGoogle Scholar
Tirkkonen-Condit, Sonja. 1987. “Think-Aloud Protocols in the Study of the Translation Process.” In CDEF 86: Papers from the Conference of Departments of English in Finland , ed. by Heikki Nyyssönen, Riitta Kataja, and Vesa Komulainen, 39–49. Oulu: University of Oulu.Google Scholar
Tsang, Pamela S. 2006. “Mental Workload.” In International Encyclopedia of Ergonomics and Human Factors , ed. by Waldemar Karwowski, 809–813. BocaRaton, FL: CRC/Taylor & Francis.Google Scholar
Vidulich, Michael A. 1988. “The Cognitive Psychology of Subjective Mental Workload.” In Human Mental Workload , ed. by Peter A. Hancock, and Najmedin Meshkati, 219–229. Amsterdam: North-Holland. DOI logoGoogle Scholar
Wilson, Glenn F.. and F. Thomas Eggemeier. 2006. “Mental Workload Measurement.” In International Encyclopedia of Ergonomics and Human Factors , ed. by Waldemar Karwowski, 814–817. BocaRaton, FL: CRC/Taylor & Francis.Google Scholar
Wilss, Wolfram. 1982. The Science of Translation: Problems and Methods . Tübinger Beitraäge zur Linguistik. Tuübingen: G. Narr.Google Scholar
Zamanian, Mostafa, and Pooneh Heydari. 2012. “Readability of Texts: State of the Art.” Theory and Practice in Language Studies 2 (1): 43–53. DOI logoGoogle Scholar
Zimmerman, Lynn W. 2010. ESL, EFL, and Bilingual Education: Exploring Historical, Sociocultural, Linguistic, and Instructional Foundations .Charlotte, NC: Information Age Pub.Google Scholar
Zipf, George Kingsley. 1935. The Psycho-Biology of Language: An Introduction to Dynamic Philology . Boston: Houghton Mifflin.Google Scholar
Cited by (31)

Cited by 31 other publications

Araghi, Sahar & Alfons Palangkaraya
2024. The link between translation difficulty and the quality of machine translation: a literature review and empirical investigation. Language Resources and Evaluation 58:4 ► pp. 1093 ff. DOI logo
Araghi, Sahar, Alfons Palangkaraya & Elizabeth Webster
2024. The impact of language translation quality on commerce: The example of patents. Journal of International Business Policy 7:2 ► pp. 224 ff. DOI logo
Chen, Xia & Jackie Xiu Yan
2024. Examining student translators’ writing and translation products: quality, errors and self-perceived mental workload. The Interpreter and Translator Trainer 18:3 ► pp. 465 ff. DOI logo
Liu, Xiao & Ying Cui
2024. The impact of text type on Chinese-English translation effort: an investigation with reference to translation entropy. Humanities and Social Sciences Communications 11:1 DOI logo
Yan, Hao, Yi Zhang, Yanqin Feng, Yang Li, Yueting Zhang, Yujun Lee, Maoqing Chen, Zijuan Shi, Yuan Liang, Yuqin Hei & Xu Duan
2024. Assessing mental demand in consecutive interpreting: Insights from an fNIRS study. Acta Psychologica 243 ► pp. 104132 ff. DOI logo
A.S. Mohammed, Tawffeek
2023. Translating Islamic Media Discourse from Arabic into English: An Analysis of Translation Process. In Translatology, Translation and Interpretation - Toward a New Scientific Endeavor, DOI logo
Gieshoff, Anne Catherine & Andrea Hunziker Heeb
2023. Cognitive load and cognitive effort. Translation, Cognition & Behavior 6:1 ► pp. 3 ff. DOI logo
Jia, Yanfang & Sanjun Sun
2023. Man or machine? Comparing the difficulty of human translation versus neural machine translation post-editing. Perspectives 31:5 ► pp. 950 ff. DOI logo
Wang, Lulu & Sanjun Sun
2023. Dictating translations with automatic speech recognition: Effects on translators’ performance. Frontiers in Psychology 14 DOI logo
Wang, Yu & Ali Jalalian Daghigh
2023. Cognitive effort in human translation and machine translation post-editing processes. FORUM. Revue internationale d’interprétation et de traduction / International Journal of Interpretation and Translation 21:1 ► pp. 139 ff. DOI logo
Chmiel, Agnieszka & Nicoletta Spinolo
2022. Testing the impact of remote interpreting settings on interpreter experience and performance. Translation, Cognition & Behavior 5:2 ► pp. 250 ff. DOI logo
Jia, Yanfang & Binghan Zheng
2022. The interaction effect between source text complexity and machine translation quality on the task difficulty of NMT post-editing from English to Chinese: A multi-method study. Across Languages and Cultures 23:1 ► pp. 36 ff. DOI logo
Kuang, Huolingxiao & Binghan Zheng
2022. How does interpreting performance correlate with note-taking process, note-taking product and note-reading process? An eye-tracking and pen-recording study. Across Languages and Cultures 23:2 ► pp. 167 ff. DOI logo
Kuang, Huolingxiao & Binghan Zheng
2023. Note-taking effort in video remote interpreting: effects of source speech difficulty and interpreter work experience. Perspectives 31:4 ► pp. 724 ff. DOI logo
Yang, Chengsong, Yimeng Wang & Na Fan
2022. Are parallel translation tasks parallel in difficulty? an eye-tracking study. Perspectives 30:4 ► pp. 711 ff. DOI logo
Yuan, Rongjie
2022. Material development for beginner student interpreters: how does text structure contribute to the difficulty of consecutive interpreting?. The Interpreter and Translator Trainer 16:1 ► pp. 58 ff. DOI logo
Zhou, Xiangyan, Xiangling Wang & Xiaodong Liu
2022. The impact of task complexity and translating self-efficacy belief on students’ translation performance: Evidence from process and product data. Frontiers in Psychology 13 DOI logo
Hu, Zhenming, Binghan Zheng & Xiangling Wang
2021. The impact of a metacognitive self-regulation inventory in translator self-training: a pre-post study with English-Chinese translation students. The Interpreter and Translator Trainer 15:4 ► pp. 430 ff. DOI logo
Prieto Ramos, Fernando & Giorgina Cerutti
2021. Terminology as a source of difficulty in translating international legal discourses: an empirical cross-genre study. International Journal of Legal Discourse 6:2 ► pp. 155 ff. DOI logo
Vanroy, Bram, Orphée De Clercq, Arda Tezcan, Joke Daems & Lieve Macken
2021. Metrics of Syntactic Equivalence to Assess Translation Difficulty. In Explorations in Empirical Translation Process Research [Machine Translation: Technologies and Applications, 3], ► pp. 259 ff. DOI logo
Jiang, Xinlei & Yue Jiang
2020. Effect of dependency distance of source text on disfluencies in interpreting. Lingua 243 ► pp. 102873 ff. DOI logo
Liu, Yanmei, Binghan Zheng & Hao Zhou
2019. Measuring the difficulty of text translation. Target 31:1 ► pp. 125 ff. DOI logo
Sun, Sanjun
2019. Measuring Difficulty in Translation and Post-editing: A Review. In Researching Cognitive Processes of Translation [New Frontiers in Translation Studies, ], ► pp. 139 ff. DOI logo
Wu, Zhiwei
2019. Text characteristics, perceived difficulty and task performance in sight translation. Interpreting. International Journal of Research and Practice in Interpreting 21:2 ► pp. 196 ff. DOI logo
Abdel Latif, Muhammad M. M.
2018. Towards a typology of pedagogy-oriented translation and interpreting research. The Interpreter and Translator Trainer 12:3 ► pp. 322 ff. DOI logo
Abdel Latif, Muhammad M. M.
2020. Translation and Interpreting Assessment Research. In Translator and Interpreter Education Research [New Frontiers in Translation Studies, ], ► pp. 61 ff. DOI logo
Angelone, Erik
2018. Chapter 2. Reconceptualizing problems in translation using triangulated process and product data. In Innovation and Expansion in Translation Process Research [American Translators Association Scholarly Monograph Series, XVIII], ► pp. 17 ff. DOI logo
Azari, Razieh, Marziyeh Khalilizadeh Ganjalikhani & Anahita Amirshoja’i
2018. Legislation for patient information leaflets in Iran: Focus on lay-friendliness. Health Promotion Perspectives 8:4 ► pp. 263 ff. DOI logo
Atabekova, Anastasia, Valentina Stepanova, Natalia Udina, Olga Gorbatenko & Tatyana Shoustikova
2017. Should Math Tools and Quantitative Methods be Part of University-based Translator and Interpreter’s Training? Russian Graduates’ Voices in the Focus. EURASIA Journal of Mathematics, Science and Technology Education 13:8 DOI logo
Szarkowska, Agnieszka, Krzysztof Krejtz, Łukasz Dutka & Olga Pilipczuk
2016. Cognitive load in intralingual and interlingual respeaking – a preliminary study. Poznan Studies in Contemporary Linguistics 52:2 DOI logo
Yarbrough, Nükhet D.
2016. Assessment of Creative Thinking Across Cultures Using the Torrance Tests of Creative Thinking (TTCT): Translation and Validity Issues. Creativity Research Journal 28:2 ► pp. 154 ff. DOI logo

This list is based on CrossRef data as of 7 november 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.