Cover not available
Article published In:
Translation and Interpreting Studies
Vol. 16:1 (2021) ► pp.101–123

Measuring the usability of machine translation in the classroom context

| Hunan University
| Hunan University
| Hunan University

Usability is a key factor for increasing adoption of machine translation. This study aims to measure the usability of machine translation in the classroom context by comparing translation students’ machine translation post-editing output with their manual translation in two comparable translation tasks. Three dimensions of usability were empirically measured: efficiency, effectiveness, and satisfaction. The findings suggest that machine translation post-editing is more efficient than human translation and post-editing produces fewer errors than human translation. While the types of errors vary, errors in terms of accuracy outnumber those related to fluency. In addition, participants perceive the amount of time and work that is saved when post-editing to be greater benefit than the overall utility of post-editing. Likewise, students report a strong desire to learn post-editing skills in training programs.

Keywords: usability, machine translation, post-editing, human translation
Article outline
  • Introduction
  • Relevant research
    • Usability and its assessment
    • Usability and machine translation
    • Usability assessment framework of MT
  • Methodology
    • Participants
    • Instruments
      • Translog-II
      • Questionnaire
      • Quality assessment metrics
    • Translation tasks
      • Translation direction
      • Text selection
      • Metrics for text difficulty, length, and readability
      • Scenario
      • Data collection and procedure
  • Results
    • Efficiency
    • Effectiveness
    • Error count
    • Error type
    • Satisfaction
  • Discussion
  • Conclusion
  • Acknowledgements
  • Note
  • References
Published online: 25 August 2020
DOI logo
https://doi.org/10.1075/tis.18047.yan
References
Bevan, Nigel, James Carter, and Susan Harker
2015 “ISO 9241-11 revised: What have we learnt about usability since 1998?” In Human Computer Interaction: Design and Evaluation, ed. by Kurosu Masaaki, 143–151. Cham: Springer. DOI logoGoogle Scholar
Bowker, Lynne
2020 “Fit-for-purpose translation.” In The Routledge Handbook of Translation and Technology, ed. by Minako O’Hagan, 453–468. London: Routledge.Google Scholar
Bowker, Lynne and Jairo Buitrago Ciro
2015 “Investigating the usefulness of machine translation for newcomers at the public library.” Translation and Interpreting Studies 10(2): 165–186. DOI logoGoogle Scholar
Cadwell, Patrick, Sharon O’Brien, and Carlos Teixeira
2018 “Resistance and accommodation: Factors for the (non-) adoption of machine translation among professional translators.” Perspectives 26(3): 301–321. DOI logoGoogle Scholar
Carl, Michael
2012 “Translog-II: a program for recording user activity data for empirical reading and writing research.” In The 8th International Conference on Language Resources and Evaluation, ed. by Nicoletta Calzolari, et al., 21–27. Istanbul.Google Scholar
Carl, Michael and Toledo Cristina Báez
2019 “Machine translation errors and the translation process: A study across different languages.” The Journal of Specialised Translation 311: 107–132.Google Scholar
Cohen, Jacob
1988 Statistical Power Analysis for the Behavioral Sciences, 2nd ed. Hillsdale, NJ: Erlbaum.Google Scholar
Daems, Joke, et al.
2016 “The effectiveness of consulting external resources during translation and post-editing of general text types.” In New Directions in Empirical Translation Process, ed. by Carl Michael, Bangalore Srinivas, and Schaeffer Moritz, 111–133. Cham: Springer. DOI logoGoogle Scholar
Daems, Joke, Sonia Vandepitte, Robert Hartsuiker, and Lieve Macken
2017a “Translation methods and experience: A comparative analysis of human translation and post-editing with student and professional translators.” Meta 62(2): 246–270. DOI logoGoogle Scholar
2017b “Identifying the machine translation error types with the greatest impact on post-editing effort.” Frontiers in Psychology 81: 1282. DOI logoGoogle Scholar
Davis, Fred
1989 “Perceived usefulness, perceived ease of use, and user acceptance of information technology.” MIS Quarterly 13(3):319–340. DOI logoGoogle Scholar
Davis, Fred, Richard Bagozzi, and Paul Warshaw
1989 “User acceptance of computer technology: A comparison of two theoretical models.” Management Science 35(8): 982–1003. DOI logoGoogle Scholar
De Almeida, Giselle and Sharon O’Brien
2010 “Analysing post-editing performance: Correlations with years of translation experience.” In Proceedings of the 14th Annual Conference of the European Association for Machine Translation, ed. by François Yvon and Viggo Hansen, 1–8. Saint-Raphaël.Google Scholar
Doherty, Stephen and Dorothy Kenny
2014 “The design and evaluation of a statistical machine translation syllabus for translation students.” The Interpreter and Translator Trainer 8(2): 295–315. DOI logoGoogle Scholar
Doherty, Stephen and Sharon O’Brien
2014 “Assessing the usability of raw machine translated output: A user-centered study using eye tracking.” International Journal of Human-Computer Interaction 30(1): 40–51. DOI logoGoogle Scholar
Ducar, Cynthia and Deborah Houk Schocket
2018 “Machine translation the L2 classroom: pedagogical solutions for making peace with Google Translate.” Foreign Language Annals 51(4): 779–795. DOI logoGoogle Scholar
Fiederer, Rebecca and Sharon O’Brien
2009 “Quality and machine translation: A realistic objective?” The Journal of Specialised Translation 111: 52–74.Google Scholar
Flanagan, Marian and Tina Paulsen Christensen
2014 “Testing post-editing guidelines: How translation trainees interpret them and how to tailor then for translator training purposes.” The Interpreter and Translator Trainer 8(2):257–275. DOI logoGoogle Scholar
Flesch, Rudolf
1948 “A new readability yardstick.” Journal of Applied Psychology 321: 221–223. DOI logoGoogle Scholar
García, Ignacio
2010 “Is machine translation ready yet?” Target 22(1): 7–21. DOI logoGoogle Scholar
2011 “Translating by post-editing: Is it the way forward?” Machine Translation 251: 217–237. DOI logoGoogle Scholar
Germann, Ulrich
2008 “Yawat: Yet another world alignment tool.” In Proceedings of 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies, 20–23. Ohio: Columbus. DOI logoGoogle Scholar
Gile, Daniel
1994 “Methodological aspects of interpretation and translation research.” In Bridge the Gap: Empirical Research in Simultaneous Interpretation, ed. by Sylvie Lambert and Barbara Moser-Mercer, 39–56. Philadelphia, PA: John Benjamins. DOI logoGoogle Scholar
Guerberof Arenas, Ana
2012 “Productivity and Quality in the Post-editing of Outputs from Translation memories and Machine translation.” Ph.D. dissertation. Universitat Rovira i Virgili, Tarragona.Google Scholar
2014 “Correlations between productivity and quality when post-editing in a professional context.” Machine Translation 28 (3–4): 165–186. DOI logoGoogle Scholar
Hansen, Gyde
2008 “The dialogue in translation process research.” In Translation and Cultural Diversity: Selected proceedings of the XVII FIT World Congress, 386–397. Shanghai: Foreign Language Press.Google Scholar
2013 “The translation process as object of research.” In The Routledge Handbook of Translation Studies, ed. by Carmen Millán and Francesca Bartrina, 88–101. London/New York: Routledge.Google Scholar
Harrati, Nouzha, et al.
2016 “Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis.” Computers in Human Behavior 611: 463–471. DOI logoGoogle Scholar
ISO9241-11
2018 “Ergonomics of human-system interaction-Part 11: Usability: Definitions and concepts.” ISO 9241-11.Google Scholar
Jia, Yanfang, Michael Carl, and Xiangling Wang
2019 “How does the post-editing of neural machine translation compare with from-scratch translation? A product and process study.” The Journal of Specialised Translation 311: 60–85.Google Scholar
Kenny, Dorothy
2018 “Sustaining disruption? The transition from statistical to neural machine translation.” Revista Tradumàtica 161: 59–70. DOI logoGoogle Scholar
Kenny, Dorothy and Stephen Doherty
2014 “Statistical machine translation in the translation curriculum: Overcoming obstacles and empowering translators.” The Interpreter and Translator Trainer 8(2): 276–294. DOI logoGoogle Scholar
Kingscott, Geoffrey
2002 “Technical translation and related disciplines.” Perspectives 10(4): 247–255. DOI logoGoogle Scholar
Koponen, Maarit
2010 “Assessing machine translation quality with error analysis.” In Electronic Proceedings of the KaTu Symposium on Translation and Interpreting Studies 41: 1–12.Google Scholar
2015 “How to teach machine translation post-editing? Experiences from a post-editing course.” In 4th Workshop on Post-Editing Technology and Practice (WPTP4), 2–15. Miami: Florida.Google Scholar
2016 “Is machine translation post-editing worth the effort? A survey of research into post-editing and effort.” The Journal of Specialised Translation 251: 131–147.Google Scholar
Kortum, Philip and Frederick Oswald
2017 “The impact of personality on the subjective assessment of usability.” International Journal of Human-Computer Interaction 341: 177–186. DOI logoGoogle Scholar
Krüger, Ralph
2019 “A model for measuring the usability of computer-assisted translation tools.” In Challenging Boundaries: New Approaches to Specialized Communication, ed. by Heike Elisabeth Jüngst, Lisa Link, Klaus Schubert, and Christiane Zehrer, 93–117. Berlin: Frank & Timme.Google Scholar
Lacruz, Isable, Michael Denkowski, and Alon Lavie
2014 “Cognitive demand and cognitive effort in PE.” In Third Workshop on PE Technology and Practice, ed. by Sharon O’Brien, Michel Simard, and Lucia Specia, 73–84. AMTA.Google Scholar
Lewis, James R.
2012 “Usability testing.” In Handbook of Human Factors and Ergonomics, ed. by Gavriel Salvendy, 1267–1312. New York: Wiley. DOI logoGoogle Scholar
Lexile
2007 The Lexile Framework for Reading: Theoretical Framework and Development (Tech. Rep). Durham, NC: MetaMetrics, Inc.Google Scholar
Lommel, Arle, Hans Uszkoreit, and Burchardt Aljoscha
2014 “Multidimensional Quality Metrics (MQM): A framework for declaring and describing translation quality metrics.” Tradumàtica 121: 455–463. DOI logoGoogle Scholar
Mariana, Valerie, Troy Cox, and Alan Melby
2015 “The multidimensional quality metrics (MQM) framework: a new framework for translation quality assessment.” The Journal of Specialised Translation 231: 137–161.Google Scholar
Mellinger, Christopher D.
2017 “Translators and machine translation: Knowledge and skills gaps in translator pedagogy.” The Interpreter and Translator Trainer 11(4): 280–293. DOI logoGoogle Scholar
Mellinger, Christopher D. and Gregory M. Shreve
2016 “Match evaluation and over-editing in a translation memory environment.” In Reembedding Translation Process Research, ed. by Ricardo Muñoz Martín, 132–148. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Mellinger, Christopher D. and Thomas A. Hanson
2017 Quantitative Research Methods in Translation and Interpreting Studies. New York: Routledge.Google Scholar
2018 “Interpreter traits and the relationship with technology and visibility.” Translation and Interpreting Studies 13(3): 366–392. DOI logoGoogle Scholar
MetaMetrics
2018 About Lexile ® Measures for Reading. [URL]. Last accessed 23 May 2020.
Moorkens, Joss
2018 “What to expect from neural machine translation: A practical in-class translation evaluation exercise.” The Interpreter and Translator Trainer 12(4): 375–387. DOI logoGoogle Scholar
Moorkens, Joss, Antonio Toral, Sheila Castilho, and Andy Way
2018 “Translators’ perceptions of literary post-editing using statistical and neural machine translation.” Translation Space 7(2): 240–262. DOI logoGoogle Scholar
O’Brien, Sharon
2004 “Machine translatability and post-editing effort: how do they relate?” Translating and the Computer 261:1–31.Google Scholar
2007 “An empirical investigation of temporal and technical post-editing effort.” Translation and Interpreting Studies 2(1): 83–136. DOI logoGoogle Scholar
2011 “Towards predicting post-editing productivity.” Machine Translation 251: 197–215. DOI logoGoogle Scholar
Plitt, Mirko and François Masselot
2010 “A productivity test of statistical machine translation PE in a typical localization context.” Prague Bulletin of Mathematical Linguistics 931: 7–16. DOI logoGoogle Scholar
Pym, Anthony
2013 “Translation skill-sets in a machine-translation age.” Meta 58(3): 487–503. DOI logoGoogle Scholar
R Core Team
2018 “R: A language and environment for statistical computing.” R Foundation for Statistical Computing. Vienna. [URL]. Last accessed 23 May 2020.
Raita, Eeva and Antti Oulasvira
2011 “Too good to be bad: Favorable product expectations boost subjective usability ratings.” Interacting with Computers 231: 363–371. DOI logoGoogle Scholar
Rossi, Caroline
2017 “Introducing statistical machine translation in translator training: From users and perceptions to course design, and back again.” Tradumàtica 151: 48–62. DOI logoGoogle Scholar
Rossi, Caroline and Jean-Pierre Chevrot
2019 “Uses and perceptions of machine translation at the European Commission.” The Journal of Specialised Translation 311: 201–216.Google Scholar
Sakamoto, Akiko
2019 “Unintended consequences of translation technologies: from project managers’ perspectives.” Perspectives 27(1): 58–73. DOI logoGoogle Scholar
Sánchez-Gijón, Pilar and Olga Torres-Hostench
2014 “MT Post-editing into the mother tongue or into a foreign language? Spanish-to-English MT translation output post-edited by translation trainees.” In Proceedings of the Third Workshop on Post-editing Technology and Practice, ed. by Sharon O’Brien, Michel Simard and Lucia Specia, 5–17. Vancouver.Google Scholar
Shuttleworth, Mark
2002 “Combing MT and TM on a technology-oriented translation masters: aims and perspectives.” In Proceedings of the 6th EAMT Workshop on Teaching Machine Translation, 123–129. Manchester.Google Scholar
Suojanen, Tytti, Kaisa Koskinen, and Tiina Tuominen
2015 User-Centered Translation. London/New York: Routledge.Google Scholar
Temizöz, Ö.
2016 “Postediting machine translation output: subject-matter experts versus professional translators.” Perspectives 24(4): 2–18. DOI logoGoogle Scholar
Temnikova, Irina
2010 “Cognitive evaluation approach for a controlled language PE experiment.” In Proceedings of the 7th International Conference on Language Resources and Evaluation, ed. by Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Mike Rosner and Daniel Tapias, 3485–3490. Valletta.Google Scholar
Tirkkonen-Condit, Sonja
1990 “Professional vs. Non professional translation: A think-aloud protocol study.” In Learning, Keeping and Using Language: Selected papers from the 8th World Congress of Applied Linguistics, ed. by M. A. K. Halliday, John Gibbons, and Howard Nicholas, 381–394. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Thode, Henry
2002 Testing for Normality. New York: Marcel Dekker. DOI logoGoogle Scholar
Trace, Jonathan, Gerriet Janssen, and Valerie Meier
2015 “Measuring the impact of rater negotiation in writing performance assessment.” Language Testing 341: 3–22. DOI logoGoogle Scholar
Van der Heijden, Hans
2004 “User acceptance of hedonic information systems.” MIS Quarterly 28(4): 695–704. DOI logoGoogle Scholar
Wang, Huashu
2018 “The development of translation technology in the era of big data.” In Restructuring Translation Education: Implications from China for the Rest of the World, ed. by Feng Yue, et al., 13–26. Singapore: Springer.Google Scholar
Wu, Jen-Her and Shu-Ching Wang
2005 “What drives mobile commerce? An empirical evaluation of the revised technology acceptance model.” Information & Management 421: 719–729. DOI logoGoogle Scholar
Yamada, Masaru
2019 “The impact of Google neural machine translation on post-editing by student translators.” The Journal of Specialized Translation 311: 87–105.Google Scholar
Yang, Yanxia and Xiangling Wang
2019 “Modeling the intention to use machine translation for student translators: An extension of technology acceptance model.” Computers & Education 1331: 116–126. DOI logoGoogle Scholar
Zaharias, Panagiotis
2009 “Developing a usability evaluation method for e-learning applications: From functional usability to motivation to learn.” International Journal of Human-computer Interaction 25(1): 75–98. DOI logoGoogle Scholar
Zhai, Yuming, Aurélien Max, and Anne Vilnat
2018 “Construction of a multilingual corpus annotated with translation relations.” In Proceedings of the first workshop on linguistic resources for natural language processing, ed. by Peter Machonis, Anabela Barreiro, Kristina Kocijan, and Max Silberztein, 102–111. New Mexico: Santa Fe.Google Scholar