Cover not available
Article published In:
Methodological Issues in Experimental Research in Audiovisual Translation and Media Accessibility
Edited by Gian Maria Greco, Anna Jankowska and Agnieszka Szarkowska
[Translation Spaces 11:1] 2022
► pp. 89–112

Experimental research in automatic subtitling

At the crossroads between machine translation and audiovisual translation

ORCID logo | Fondazione Bruno Kessler | University of Trento
Recent developments in neural machine translation, and especially speech translation, are gradually but firmly entering the field of audiovisual translation (AVT). Automation in subtitling is extending from a machine translation (MT) component to fully automatic subtitling, which comprises MT, auto-spotting and automatic segmentation. The rise of this new paradigm renders MT-oriented experimental designs inadequate for the evaluation and investigation of automatic subtitling, since they fail to encompass the multimodal nature and technical requirements of subtitling. This paper highlights the methodological gaps to be addressed by multidisciplinary efforts in order to overcome these inadequacies and obtain metrics and methods that lead to rigorous experimental research in automatic subtitling. It presents a review of previous experimental designs in MT for subtitling, identifies their limitations for conducting research under the new paradigm and proposes a set of recommendations towards achieving replicability and reproducibility in experimental research at the crossroads between AVT and MT.
Keywords: Subtitling, Neural Machine Translation, speech translation, evaluation, post-editing
Article outline
  • 1.Introduction
  • 2.From MT for subtitles to automatic subtitling
  • 3.Review of previous experimental approaches in MT for subtitling
    • 3.1Automatic evaluation
    • 3.2Human evaluation
    • 3.3Other experimental aspects
  • 4.Issues and challenges in conducting experimental research in automatic subtitling
    • 4.1Variable number of segments
    • 4.2Auto-spotting and segmentation come into play
    • 4.3Disentangling the variables
  • 5.Experiences and recommendations
    • 5.1Research in automatic subtitling should comprise all dimensions of subtitling
    • 5.2Improved interfaces
    • 5.3Towards reporting standards
    • 5.4Test data and benchmarks
    • 5.5Evaluation should be independent of generation
  • 6.Conclusions
  • Notes
  • References
Published online: 14 June 2022
DOI logo
https://doi.org/10.1075/ts.21021.kar
References (51)
References
Álvarez, Aitor, Marina Balenciaga, Arantza del Pozo, Haritz Arzelus, Anna Matamala, and Carlos-D. Martínez-Hinarejos. 2016. Impact of Automatic Segmentation on the Quality, Productivity and Self-reported Post-editing Effort of Intralingual Subtitles. In Proceedings of the Tenth International Conference on Language Re-sources and Evaluation (LREC’16), Portorož, Slovenia, pp. 3049–3053. European Language Resources Association (ELRA).Google Scholar
Alves, Fabio. 2003. Tradução, cognição e contextualidade: Triangulando a interface processo produto no desempenho de tradutores novos [Translation, cognition and contextualisation: Triangulating the process-product interface in the performance of novice translators]. Translation, cognition D.E.L.T.A. 191, 71–108.Google Scholar
Armstrong, Stephen, Colm Caffrey, and Marian Flanagan. 2006. Translating DVD Subtitles from English-German and English-Japanese Using Example-Based Machine Translation. In MuTra 2006–Audiovisual Translation Scenarios: Conference Proceedings.Google Scholar
American Psychological Association-APA. 2010. Publication Manual of the American Psychological Association (6th edition ed.). Washington: APA.Google Scholar
Aziz, Wilker, Sheila C. M. de Sousa, and Lucia Specia. 2012. Cross-lingual Sentence Compression for Subtitles. In 16th Annual Conference of the European Association for Machine Translation, EAMT, Trento, Italy, pp. 103–110.Google Scholar
Barrault, Loїc, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella Frank. 2018. Findings of the Third Shared Task on Multimodal Machine Translation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, Belgium, Brussels, pp. 304–323. Association for Computational Linguistics. DOI logoGoogle Scholar
Belz, Anya, Simon Mille, and David M. Howcroft. 2020. Disentangling the properties of human evaluation methods: A classification system to support comparability, meta-evaluation and reproducibility testing. In Proceedings of the 13th International Conference on Natural Language Generation, Dublin, Ireland, pp. 183–194. Association for Computational Linguistics.Google Scholar
Bentivogli, Luisa, Nicola Bertoldi, Mauro Cettolo, Marcello Federico, Matteo Negri, and Marco Turchi. 2016. On the Evaluation of Adaptive Machine Translation for Human Post-Editing. IEEE/ACM Transactions on Audio, Speech, and Language Processing 24(2), 388–399. DOI logoGoogle Scholar
Bentivogli, Luisa, Mauro Cettolo, Marcello Federico, and Christian Federmann. 2018. Machine translation human evaluation: An investigation of evaluation based on Post-editing and its relation with Direct Assessment. In Proceedings of the 15th International Workshop on Spoken Language Translation (IWSLT 2018).Google Scholar
Bolaños-García-Escribano, Alejandro, Jorge Díaz-Cintas, and Serenella Massidda. 2021. Latest advancements in audiovisual translation education. The Interpreter and Translator Trainer 15(1), 1–12. DOI logoGoogle Scholar
Burchardt, Aljoscha, Arle Lommel, Lindsay Bywood, Kim Harris, and Maja Popović. 2016. Machine translation quality in an audiovisual context. Target 28(2), 206–221. DOI logoGoogle Scholar
Bywood, Lindsay, Panayota Georgakopoulou, and Thierry Etchegoyhen. 2017. Embracing the threat: machine translation as a solution for subtitling. Perspectives 25(3), 492–508. DOI logoGoogle Scholar
Carroll, Mary and Jan Ivarsson. 1998. Code of Good Subtitling Practice. Simrishamn: TransEdit.Google Scholar
de Sousa, Sheila C. M., Wilker Aziz, and Lucia Specia. 2011. Assessing the Post-Editing Effort for Automatic and Semi-Automatic Translations of DVD Subtitles. In Proceedings of the International Conference Recent Advances in Natural Language Processing, pp. 97–103. Association for Computational Linguistics.Google Scholar
Doddington, George. 2002. Automatic Evaluation of Machine Translation Quality Using N-Gram Co-Occurrence Statistics. In Proceedings of the Second International Conference on Human Language Technology Research, HLT ’02, San Francisco, CA, USA, pp. 138–145. Morgan Kaufmann Publishers Inc. DOI logoGoogle Scholar
Doherty, Stephen. 2016. The impact of translation technologies on the process and product of translation. International Journal of Communication 10 1, 947–969.Google Scholar
Díaz-Cintas, Jorge. 2013. The technology turn in subtitling. In Proceedings from the 5th International Maastricht – Lódź Duo Colloquium on Translation and Meaning, Part 9 1, pp. 119–132.Google Scholar
Etchegoyhen, Thierry, Lindsay Bywood, Mark Fishel, Panayota Georgakopoulou, Jie Jiang, Gerard van Loenhout, Arantza del Pozo, Mirjiam S. Maučec, Anja Turner, and Martin Volk. 2014. Machine Translation for Subtitling: A Large-Scale Evaluation. In Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC), pp. 46–53.Google Scholar
Facebook. 2020. “New Automated Captions Powered by AI”. Last accessed: 30/06/2021. [URL]
Hadley, James, Maja Popović, Haithem Afli, and Andy Way (eds.). 2019. Proceedings of the Qualities of Literary Machine Translation, Dublin, Ireland. European Association for Machine Translation.Google Scholar
Howcroft, David M., Anja Belz, Miruna Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. In Proceedings of the 13th International Conference on Natural Language Generation, Dublin, Ireland, pp. 169–182. Association for Computational Linguistics.Google Scholar
Hu, Ke, Sharon O’Brien, and Dorothy Kenny. 2019. A reception study of machine translated subtitles for MOOCs. Perspectives, 1–18.Google Scholar
Karakanta, Alina, Matteo Negri, and Marco Turchi. 2020a. Is 42 the Answer to Everything in Subtitling-oriented Speech Translation? In Proceedings of the 17th International Conference on Spoken Language Translation, Online, pp. 209–219. Association for Computational Linguistics. DOI logoGoogle Scholar
. 2020b. MuST-Cinema: a Speech-to-Subtitles corpus. In Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France, pp. 3727–3734. European Language Resources Association.Google Scholar
Karamitroglou, Fotios. 1998. A proposed set of subtitling standards in Europe. Translation Journal.Google Scholar
Koponen, Maarit, Umut Sulubacak, Kaisa Vitikainen, and Jörg Tiedemann. 2020. MT for subtitling: User evaluation of post-editing productivity. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation (EAMT 2020), pp. 115–124.Google Scholar
Krings, Hans P. (2001). Repairing texts: Empirical investigations of machine translation post-editing process. The Kent State University Press.Google Scholar
Lison, Pierre and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), Portorož, Slovenia, pp. 923–929. European Language Resources Association (ELRA).Google Scholar
Lo, Chi-kiu. 2019. YiSi – a Unified Semantic MT Quality Evaluation and Estimation Metric for Languages with Different Levels of Available Resources. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), Florence, Italy, pp. 507–513. Association for Computational Linguistics. DOI logoGoogle Scholar
Matusov, Evgeny, Gregor Leusch, Oliver Bender, and Hermann Ney. 2005. Evaluating Machine Translation Output with Automatic Sentence Segmentation. In Proceedings of the Second International Workshop on Spoken Language Translation, Pittsburgh, Pennsylvania, USA.Google Scholar
Matusov, Evgeny, Patrick Wilken, and Yota Georgakopoulou. 2019. Customizing Neural Machine Translation for Subtitling. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), Florence, Italy, pp. 82–93. Association for Computational Linguistics. DOI logoGoogle Scholar
Melero, Maite, Antoni Oliver, and Toni Badia. (2006). Automatic Multilingual Subtitling in the eTITLE Project. In Proceedings of ASLIB Translating and the Computer 28 1.Google Scholar
Moorkens, Joss and Sharon O’Brien. (2017). Assessing User Interface Needs of Post-Editors of Machine Translation. Human Issues in Translation Technology: The IATIS Yearbook, 109–130.Google Scholar
Nikolic, Kristijan and Lindsay Bywood. 2021. Audiovisual Translation: The Road Ahead. Journal of Audiovisual Translation 4(1), 50–70. DOI logoGoogle Scholar
Orero, Pilar, Stephen Doherty, Jan-Louis Kruger, Anna Matamala, Jan Pedersen, Elisa Perego, Pablo Romero-Fresco, Sara Rovira-Esteva, Olga Soler-Vilageliu, and Agnieszka Szarkowska. 2018. Conducting experimental research in audiovisual translation (AVT): A position paper. Journal of Specialised Translation 301, 105–126.Google Scholar
Orrego-Carmona, David, Łukasz Dutka, and Agnieszka Szarkowska. 2018. Using translation process research to explore the creation of subtitles: an eye-tracking study comparing professional and trainee subtitlers. Journal of Specialised Translation 0(30), 150–180.Google Scholar
O’Brien, Sharon. 2021. Translation, human–computer interaction and cognition. In Fabio Alves and Arnt Lykke Jakobsen (eds.), The Routledge Handbook of Translation and Cognition, pp. 376–388. Routledge.Google Scholar
O’Hagan, Minako. 2003. Can language technology respond to the subtitler’s dilemma? – a preliminary study. In Proceedings of the 25th International Conference on Translation and the Computer.Google Scholar
Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, pp. 311–318. Association for Computational Linguistics.Google Scholar
Pedersen, Jan. 2017. The FAR Model: Assessing Quality in Interlingual Subtitling. The Journal of Specialised Translation 281, 210–229.Google Scholar
Piperidis, Stelios, Iason Demiros, and Prokopis Prokopidis. 2005. Infrastructure for a Multilingual Subtitle Generation System. In 9th International Symposium on Social Communication, pp. 24–28.Google Scholar
Popović, Maja. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisbon, Portugal, pp. 392–395. Association for Computational Linguistics. DOI logoGoogle Scholar
Popowich, Fred, Paul McFetridge, Davide Turcato, and Janine Toole. 2000. Machine Translation of Closed Captions. Machine Translation, 311–341. DOI logoGoogle Scholar
Rei, Ricardo, Craig Stewart, Ana C. Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 2685–2702. Association for Computational Linguistics. DOI logoGoogle Scholar
Snover, Matthew, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, Cambridge, Massachusetts, USA, pp. 223–231. Association for Machine Translation in the Americas.Google Scholar
Szarkowska, Agnieszka, Jorge Diaz-Cintas, and Olivia Gerber-Morón. 2020. Quality is in the eye of the stakeholders: what do professional subtitlers and viewers think about subtitling? Universal Access in the Information Society 201, 661–675. DOI logoGoogle Scholar
Tadmor-Ramanovich, Michelle and Nadav Bar. 2019. “On-Device Captioning with Live Caption.” Google AIBlog. Last accessed: 30/06/2021. [URL]
Volk, Martin, Rico Sennrich, Christian Hardmeier, and Frida Tidström. 2010. Machine Translation of TV Subtitles for Large Scale Production. In Ventsislav Zhechev (ed.), Proceedings of the Second Joint EM+/CNGL Workshop ”Bringing MT to the User: Research on Integrating MT in the Translation Industry (JEC’10), Denver, pp. 53–62.Google Scholar
Wang, Weiyue, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney. 2016. CharacTer: Translation Edit Rate on Character Level. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, Berlin, Germany, pp. 505–510. Association for Computational Linguistics. DOI logoGoogle Scholar
Wang, Xin, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. 2019. VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research. In The IEEE International Conference on Computer Vision (ICCV). DOI logoGoogle Scholar
Cited by (6)

Cited by six other publications

Fei, Luo
2024. 2024 IEEE 2nd International Conference on Sensors, Electronics and Computer Engineering (ICSECE), ► pp. 806 ff. DOI logo
Zhang, Yadan, Omer Hassan Ali Mahfoodh & Debbita Ai Lin Tan
2024. A Corpus-based Evaluation of English-Mandarin Cultural References between Fansubbing and Official Subtitling. Journal of Intercultural Communication ► pp. 109 ff. DOI logo
Arnáiz-Uzquiza, Verónica & Paula Igareda González
2023. Traducción Automática y Posedición en TAV: estudio piloto de una práctica en alza. TRANS: Revista de Traductología :27 ► pp. 197 ff. DOI logo
Hrabčáková, Adriána
2023. Some Current Trends and Research Possibilities in the Audiovisual Translation of the 2020s. Language Culture Politics International Journal 1:1/2023 ► pp. 49 ff. DOI logo
Hrabčáková, Adriána
2023. Some Current Trends and Research Possibilities in the Audiovisual Translation of the 2020s. Language Culture Politics International Journal 1:1/2023 ► pp. 49 ff. DOI logo
Jiménez Salazar, Argeniz Britto & Mario Aguilar Fruna
2023. Enfoques actuales de la investigación en la traducción. LATAM Revista Latinoamericana de Ciencias Sociales y Humanidades 4:2 DOI logo

This list is based on CrossRef data as of 30 december 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.