Cover not available
Article published In:
Translation Spaces
Vol. 13:2 (2024) ► pp.265–292

The Machine Translation Post-Editing Annotation System (MTPEAS)

A standardized and user-friendly taxonomy for student post-editing quality assessment

 | UCLouvain
 | UCLouvain
ORCID logo | UCLouvain
Machine translation post-editing quality evaluation has received relatively little attention in translation pedagogy to date. It is a time-consuming process that involves the comparison of three texts (source text, machine translation and student post-edited text) and the systematic identification and correction of students’ edits (or absence thereof) of machine translation (MT) output. There are as yet no widely available, standardized, user-friendly annotation systems for use in translator education. In this article, we address this gap by describing the Machine Translation Post-Editing Annotation System (MTPEAS). MTPEAS includes a taxonomy of seven categories that are presented in easy-to-understand terms: Value-adding edits, Successful edits, Unnecessary edits, Incomplete edits, Error-introducing edits, Unsuccessful edits, and Missing edits. We then assess the robustness of the MTPEAS taxonomy in a pilot study of 30 students’ post-edited texts and offer some preliminary findings on students’ MT error identification and correction skills.
Keywords: machine translation post-editing, post-editing quality evaluation, translator education, post-editing training, annotation system, inter-rater agreement
Article outline
  • 1.Introduction: Post-editing quality evaluation in translator education
  • 2.The Machine Translation Post-Editing Annotation System (MTPEAS)
    • 2.1The MTPEAS annotation system
    • 2.2The MTPEAS taxonomy
    • 2.3The MTPEAS decision tree: A user-friendly, graphical representation of the annotation system
  • 3.Pilot Study: Data and methodology
    • 3.1Data
    • 3.2Methodology
  • 4.Pilot Study: Results and discussion
    • 4.1Inter-rater agreement on segment identification
    • 4.2Inter-rater agreement on MTPEAS annotation
    • 4.3Preliminary findings
  • 5.Concluding remarks
  • Acknowledgments
  • Notes
  • References
Published online: 9 September 2024
DOI logo
https://doi.org/10.1075/ts.24002.bod
References (41)
References
Blain, Frédéric, Jean Senellart, Holger Schwenk, Mirko Plitt, and Johann Roturier. 2011. “Qualitative analysis of post-editing for high quality machine translation.” In Proceedings of Machine Translation Summit XIII: Papers, 164–171.Google Scholar
Castagnoli, Sara, Dragos Ciobanu, Natalie Kübler, Kerstin Kunz, and Alexandra Volanschi. 2011. “Designing a learner translator corpus for training purposes.” In Corpora, language, teaching, and resources: From theory to practice, edited by Natalie Kübler, 221–248. Peter Lang.Google Scholar
Castilho, Sheila, Stephen Doherty, Federico Gaspari, and Joss Moorkens. 2018. “Approaches to Human and Machine Translation Quality Assessment.” In Translation Quality Assessment, edited by Joss Moorkens, Sheila Castilho, Federico Gaspari, and Stephen Doherty, 9–38. Cham: Springer. DOI logoGoogle Scholar
Daems, Joke, Sonia Vandepitte, Robert J. Hartsuiker, and Lieve Macken. 2017. “Translation Methods and Experience: A Comparative Analysis of Human Translation and Post-editing with Students and Professional Translators.” Meta 62 (2): 245–270. DOI logoGoogle Scholar
De Almeida, Giselle. 2013. Translating the post-editor: an investigation of post-editing changes and correlations with professional experience across two Romance languages. PhD Thesis, Dublin City University.
Doherty, Stephen, and Dorothy Kenny. 2014. “The design and evaluation of a Statistical Machine Translation syllabus for translation students.” The Interpreter and Translator Trainer 8 (2): 295–315. DOI logoGoogle Scholar
Doherty, Stephen, Joss Moorkens, Federico Gaspari, and Sheila Castilho. 2018. “On education and training in translation quality assessment.” In Translation Quality Assessment, edited by Joss Moorkens, Sheila Castilho, Federico Gaspari, and Stephen Doherty, 95–106. Cham: Springer. DOI logoGoogle Scholar
EMT Expert Group. 2022. “Competence Framework”. Accessed January 19, 2024. [URL]
Froeliger, Nicolas, Claire Larsonneur, and Giuseppe Sofo (eds). 2023. Traduction humaine et traitement automatique des langues: Vers un nouveau consensus?/Human Translation and Natural Language Processing: Forging a New Consensus? Studi e ricerche 35. Edizioni Ca’ Foscari – Venice University Press: Venice.Google Scholar
Gene, Viveta, and Lucía Guerrero. 2022. “A Common Machine Translation Post-Editing Training Handbook for Academia, Clients, LSPs and Post-Editors.” GALA MTPE Training Special Interest Group.Google Scholar
Ginovart Cid, Clara, and Carme Colominas Ventura. 2021. “The MT post-editing skill set: Course descriptions and educators’ thoughts.” In Translation Revision and Post-editing: Industry Practices and Cognitive Processes, edited by Maarit Koponen, Brian S. Mossop, Isabelle Robert, and Giovanna Schocchera, 226–246. London/New York: Routledge.Google Scholar
Granger, Sylviane, and Marie-Aude Lefer. 2023. “Learner translation corpora: Bridging the gap between learner corpus research and corpus-based translation studies.” International Journal of Learner Corpus Research 9 (1): 1–28. DOI logoGoogle Scholar
. 2021. “ Translation-oriented Annotation System manual (Version 2.0) .” CECL Papers 3. Louvain-la-Neuve: Centre for English Corpus Linguistics/Université catholique de Louvain. Accessed January 19, 2024. [URL]
. 2020. “The Multilingual Student Translation corpus: A resource for translation teaching and research.” Language Resources and Evaluation 541:1183–1199. DOI logoGoogle Scholar
Guerberof, Ana, and Joss Moorkens. 2019. “Machine translation and post-editing training as part of a master’s programme.” The Journal of Specialised Translation 311: 217–238.Google Scholar
Hu, Ke, and Patrick Cadwell. 2016. “A Comparative Study of Post-editing Guidelines.” Baltic J. Modern Computing 4 (2): 346–353.Google Scholar
Huertas Barros, Elsa, and Juliet Vine (eds). 2018. New Perspectives in Assessment in Translator Training. London: Routledge.Google Scholar
Kenny, Dorothy. 2020. “Technology and translator training.” In The Routledge Handbook of Translation and Technology, edited by Minako O’Hagan, 498–515. London: Routledge.Google Scholar
Kenny, Dorothy and Doherty Stephen. 2014. “Statistical machine translation in the translation curriculum: overcoming obstacles and empowering translators.” The Interpreter and Translator Trainer 8 (2): 276–294. DOI logoGoogle Scholar
Konttinen, Kalle, Leena Salmi, and Maarit Koponen. 2021. “Revision and post-editing competences in translation education.” In Translation Revision and Post-editing: Industry Practices and Cognitive Processes, edited by Maarit Koponen, Brian S. Mossop, Isabelle Robert, and Giovanna Schocchera, 187–202. London/New York: Routledge.Google Scholar
Koponen, Maarit. 2015. “How to teach machine translation post-editing? Experiences from a post-editing course.” In Proceedings of 4th Workshop on post-editing technology and practice, Miami, USA, October 30 to November 3, 2015, 2–15. Accessed January 19, 2024. [URL]
Koponen, Maarit and Leena Salmi. 2017. “Post-editing quality: Analysing the correctness and necessity of post-editor corrections.” Linguistica Antverpiensia, New Series: Themes in Translation Studies 161: 137–148.Google Scholar
Koponen, Maarit, Leena Salmi, and Markku Nikulin. 2019. “A product and process analysis of post-editor corrections on neural, statistical and rule-based machine translation output.” Machine Translation 33 (1): 61–90. DOI logoGoogle Scholar
Kübler, Natalie, Alexandra Mestivier, and Mojca Pecman. 2022. “Using comparable corpora for translating and post-editing complex noun phrases in specialized texts. Insights from English-to-French specialized translation.” In Extending the Scope of Corpus-Based Translation Studies, edited by Sylviane Granger and Marie-Aude Lefer, 237–266. London: Bloomsbury. DOI logoGoogle Scholar
Kunilovskaya, Maria. 2015. “How far do we agree on the quality of translation?” English Studies at NBU 1 (1): 18–31. DOI logoGoogle Scholar
Landis, J. Richard, and Koch, Gary G. 1977. “The measurement of observer agreement for categorical data.” Biometrics 331: 159–174. DOI logoGoogle Scholar
Lefer, Marie-Aude, Romane Bodart, Adam Obrusník, and Justine Piette. 2023. The Post-Edit Me! project. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, 493–494.Google Scholar
Lefer, Marie-Aude, Romane Bodart, Justine Piette, and Adam Obrusník. 2024. “MTPE quality evaluation in translator education: the postedit.me app.” EAMT2024, Sheffield.Google Scholar
Lipnevich, Anastasiya A., and Ernesto Panadero. 2021. “A review of feedback models and theories: Descriptions, definitions, and conclusions.” Frontiers in Education 61: 720195. DOI logoGoogle Scholar
Massey, Gary, Maria Piotrowska, and Mariusz Marczak. 2023. “(Re-) profiling T&I education: meeting evolution with innovation”. The Interpreter and Translator Trainer 17 (3):325–331. DOI logoGoogle Scholar
Mellinger, Christopher D. 2017. “Translators and machine translation: knowledge and skills gaps in translator pedagogy.” The Interpreter and Translator Trainer 11 (4): 280–293. DOI logoGoogle Scholar
Nitzke, Jean, Anke Tardel, and Silvia Hansen-Schirra. 2019. “Training the modern translator – the acquisition of digital competencies through blended learning”. The Interpreter and Translator Trainer 13 (3): 292–306. DOI logoGoogle Scholar
O’Brien, Sharon. 2002. “Teaching post-editing: a proposal for course content.” In Proceedings of the 6th EAMT Workshop: Teaching machine translation, Manchester, England, November 14–15, 2002, 99–106. Accessed January 19, 2024. [URL]
Pavlović, Nataša, and Goranka Antunović. 2021. “Towards acquiring post-editing abilities through research-informed practical tasks.” Strani jezici: časopis za primijenjenu lingvistiku 50 (2): 185–205. DOI logoGoogle Scholar
Plaza Lara, Cristina. 2019. “SWOT Analysis of the Inclusion of Machine Translation and Post -Editing in the Master’s Degrees Offered in the EMT Network.” The Journal of Specialised Translation 311: 260–280.Google Scholar
Rico, Celia, and Diana Gonzalez Pastor. 2022. “The Role of Machine Translation in Translation Education: A Thematic Analysis of Translator Educators’ Beliefs.” Translation & Interpreting 14 (1): 177–197. DOI logoGoogle Scholar
Schumacher, Perrine. 2023. La post-édition de traduction automatique en contexte d’apprentissage. PhD Thesis, University of Liège and University of Geneva.
Stasimioti, Maria, and Vilelmini Sosoni. 2021. “Investigating post-editing: A mixed-methods study with experienced and novice translators in the English-Greek language pair.” In Translation, interpreting, cognition: The way out of the box, edited by Tra&Co Group, 79–104. Berlin: Language Science Press. DOI logoGoogle Scholar
Strobl, Carola. 2022. “Revision behaviour of translation students in response to electronic feedback. Insights from a pilot study using Hypal4MUST.” Paper presented at EST conference 2022 , Oslo, Norway, 22–25 June 2022.
Yamada, Masaru. 2019. “The impact of Google neural machine translation on post-editing by student translators.” The Journal of Specialised Translation 31 (1): 87–106.Google Scholar