Cover not available
Part of
Innovation and Expansion in Translation Process Research
Edited by Isabel Lacruz and Riitta Jääskeläinen
[American Translators Association Scholarly Monograph Series XVIII] 2018
► pp. 267–294

Chapter 12
Multimodal measurement of cognitive load during subtitle processing

Same-language subtitles for foreign-language viewers

This chapter presents the findings of a multimodal study that combines a suite of methods, namely, eye tracking, electroencephalography, and self-reported psychometrics, to investigate the impact of language and layout on cognitive load in the absence and presence of conventional same-language subtitles and integrated titles (a transcript of the spoken language presented dynamically in line with the film presentation) in an English fiction film viewed by L2 speakers of English. Findings show similar levels of immersion for conventional same-language subtitles and integrated titles, whereas the latter appears to facilitate deeper processing of subtitle contents resulting in reductions in perceived complexity.
Keywords: cognitive load, electroencephalography, dynamic text, eye tracking, integrated titles, multimodal, reading index, split attention, subtitles, transportation, working memory
Article outline
  • 1.Introduction
  • 2.Background
    • 2.1Cognitive load theory
    • 2.2The cognitive processing of subtitles
    • 2.3Integrated titles
  • 3. Method
    • 3.1Measuring cognitive load
      • 3.1.1Subjective measures
      • 3.1.2 Eye tracking measures
      • 3.1.3Electroencephalography measures
    • 3.2Participants
    • 3.3Materials
    • 3.4Procedure
    • 3.5Data processing
    • 3.6Statistical analysis
  • 4.Results
    • 4.1Transportation
    • 4.2Perceived linguistic complexity
    • 4.3Eye tracking measures
    • 4.4Summative EEG data
    • 4.5 Correlational analysis
  • 5.Discussion
  • Notes
  • References
Published online: 1 February 2018
DOI logo
https://doi.org/10.1075/ata.18.12kru
References (67)
References
Antonenko, Pavlo, Fred Paas, and Roland Grabner. 2010. “Using electroencephalography to measure cognitive load.” Educational Psychology Review 22: 425–438. DOI logoGoogle Scholar
Anglin, Gary J., Hossein Vaez, and Kathryn L. Cunningham. 2004. “Visual representations and learning: The role of static and animated graphics.” Handbook of research on educational communications and technology 2: 865–916.Google Scholar
Ayres, Paul, and John Sweller. 2005. “The Split-attention principle in multimedia learning.” In The Cambridge handbook of multimedia learning, ed. by Richard E. Mayer, 135–146. Cambridge, UK: Cambridge University Press. DOI logoGoogle Scholar
Bayram, Servet, and Duygu M. Bayraktar. 2012. “Using eye tracking to study on attention and recall in multimedia learning environments: The effects of design in learning.” World Journal on Educational Technology 4 (2): 81–98.Google Scholar
Ben-Shaul, Nitzan S. 2008. Hyper-narrative interactive cinema: Problems and solutions. Amsterdam: Rodopi.Google Scholar
Bird, Stephen A., and John N. Williams. 2002. “The effect of bimodal input on implicit and explicit memory: An investigation into the benefits of within-language subtitling.” Applied Psycholinguistics 23 (4): 509–533. DOI logoGoogle Scholar
Brown, Andy, Rhia Jones, and Mike Crabb. 2015. “Dynamic subtitles: The user experience.” TVXʹ15 Proceedings of the ACM International Conference on Interactive Experiences for TV and Online Video, 103–112. New York, NY: Association for Computing Machinery. DOI logoGoogle Scholar
Brünken, Roland, Jan L. Plass, and Detlev Leutner. 2003. “Direct measurement of cognitive load in multimedia learning.” Educational Psychologist 38 (1): 53–61. DOI logoGoogle Scholar
Danan, Martine. 2004. “Captioning and subtitling: Undervalued language learning strategies.” Meta: Translators’ Journal 49 (1): 67–77. DOI logoGoogle Scholar
Debue, Nicolas, and Cécile van de Leemput. 2014. “What does germane load mean? An empirical contribution to the Cognitive Load Theory.” Frontiers in Psychology 5: 1099. DOI logoGoogle Scholar
Díaz Cintas, Jorge, and Pablo Muñoz Sánchez. 2006. “Fansubs: Audiovisual translation in an amateur environment.” The Journal of Specialised Translation 6: 37–52.Google Scholar
Delorme, Arnaud, and Scott Makeig. 2004. “EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis.” Journal of Neuroscience Methods 134 (1): 9–21. DOI logoGoogle Scholar
Doherty, Stephen, and Sharon O’Brien. 2014. “Assessing the usability of raw machine translated output: A user-centred study using eye tracking.” International Journal of Human Computer Interaction 30 (1): 40–51. DOI logoGoogle Scholar
Doherty, Stephen, Sharon O’Brien, and Michael Carl. 2010. “Eye tracking as an MT evaluation technique.” Machine Translation 24 (1): 1–13. DOI logoGoogle Scholar
d’Ydewalle, Géry, and Ingrid Gielen. 1992. “Attention allocation with overlapping sound, image, and text.” In Eye movements and visual cognition, ed. by K. Rayner, 415–427. New York: Springer. DOI logoGoogle Scholar
dʹYdewalle, Géry, and Wim De Bruycker. 2007. “Eye movements of children and adults while reading television subtitles.” European Psychologist 12 (3): 196–205. DOI logoGoogle Scholar
Eitel, Alexander, Katharina Scheiter, and Anne Schüler. 2013. “How inspecting a picture affects processing of text in multimedia learning.” Applied Cognitive Psychology 27 (4): 451–461. DOI logoGoogle Scholar
Fox, Wendy. 2012. Integrierte Bildtitel – Eine Alternative zur traditionellen Untertitelung. Am Beispiel der BBC-Serie Being Human. [Integrated titles – An alternative to traditional subtitles. Based on the BBC series Being Human.] Masterʹs Thesis at Johannes Gutenberg University Mainz/Germersheim.Google Scholar
. 2015. Integriti – Integrierte Titel. Ausführlicher Abschlussbericht für das Karl-Steinbuch-Stipendium 2013/14. [Integrated titles – Extensive final report for the Karl Steinbuch scholarship 2013/14.] [URL]
. 2016. “Integrated titles: An improved viewing experience?.” In Eyetracking and Applied Linguistics, ed. by Silvia Hansen-Schirra, and Sambor Grucza, 5–30. Berlin: Language Science Press. DOI logoGoogle Scholar
Foxe, J. J., and Snyder, A. C. 2011. “The role of Alpha-Band brain oscillations as a sensory suppression mechanism during selective attention.” Frontiers in Psychology 2:154. DOI logo.Google Scholar
Garza, Thomas J. 1991. “Evaluating the use of captioned video materials in advanced foreign language learning.” Foreign Language Annals 24 (3): 239–258. DOI logoGoogle Scholar
1996. “The message is the medium: Using video materials to facilitate foreign language performance.” Texas Papers in Foreign Language Education 2 (2): 1–20.Google Scholar
Gerlic, Ivan, and Norbert Jausovec. 2001. “Differences in eeg power and coherence measures related to the type of presentation: Text versus multimedia.” Journal of Educational Computing Research 25 (2): 177–195. DOI logoGoogle Scholar
Gernsbacher, M. A. 2015. “Video captions benefit everyone.” Policy Insights from the Behavioral and Brain Sciences, 2 (1): 195–202. DOI logoGoogle Scholar
Green, Melanie C., and Timothy C. Brock. 2000. “The role of transportation in the persuasiveness of public narratives.” Journal of Personality and Social Psychology 79 (5): 701–721. DOI logoGoogle Scholar
. 2013. “Transport Narrative Questionnaire.” Measurement instrument database for the social science. Retrieved from [URL].
Green, Melanie C., Timothy C. Brock, and Geoff F. Kaufman. 2004. “Understanding media enjoyment: the role of transportation into narrative worlds.” Communication Theory 14 (4): 311–327. DOI logoGoogle Scholar
Holmqvist, Kenneth, Marcus Nyström, Richard Andersson, Richard Dewhurst, Halszka Jarodzka, and Joost van de Weijer. 2011. Eye tracking: A comprehensive guide to methods and measures. Oxford: Oxford University Press.Google Scholar
Irwin, David E. 2004. “Fixation location and fixation duration as indices of cognitive processing.” In The interface of language, vision, and action: Eye movements and the visual world, ed. by John M. Henderson, 105–133. New York, NY: Psychology Press.Google Scholar
Jakobsen, Arnt L., and Kristian H. T. Jensen. 2008. “Eye movement behaviour across four different types of reading tasks.” In Looking at eyes. Eye-tracking studies of reading and translation processing, ed. by Susanne Göpferich, Arnt Lykke Jakobsen, and Inger M Mees, 103–124. Frederiksberg: Samfundslitteratur.Google Scholar
Just, Marcel A., and Patricia A. Carpenter. 1980. “A theory of reading: from eye fixations to comprehension.” Psychological Review 87 (4), 329–354. DOI logoGoogle Scholar
Kalyuga, Slava. 2012. “Instructional benefits of spoken words: A review of cognitive load factors.” Educational Research Review 7 (2): 145–159. DOI logoGoogle Scholar
Klimesch, Wolfgang, Michael Doppelmayr, H. Russegger, T Pachinger, and J. Schwaiger. 1998. “Induced alpha band power changes in the human EEG and attention.” Neuroscience Letters 244: 73–76. DOI logoGoogle Scholar
Kothari, Brij. 2000. “Same language subtitling on Indian television: Harnessing the power of popular culture for literacy.” In Redeveloping communication for social change: Theory, practice and power, ed. by K. Wilkins, 135–146. New York: Rowman and Littlefield.Google Scholar
Kruger, Jan-Louis, and Stephen Doherty. 2016. “Measuring cognitive load in the presence of educational video: towards a multimodal methodology.” Australasian Journal of Educational Technology 32 (6): 19–31.Google Scholar
Kruger, Jan-Louis, Stephen Doherty & Maria Soto-Sanfiel. 2017. “Original language subtitles: Their effects on the native and foreign viewer.” Comunicar 25(50): 23–32. DOI logoGoogle Scholar
Kruger, Jan-Louis, Esté Hefer, and Gordon Matthew. 2013. “Measuring the impact of subtitles on cognitive load: Eye tracking and dynamic audiovisual texts.” In Proceedings of Eye Tracking South Africa, Cape Town, 29–31 August 2013.Google Scholar
Kruger, Jan-Louis, Marìa T. Soto-Sanfiel, Stephen Doherty, and Ronny Ibrahim. 2016. “Towards a cognitive audiovisual translatology: Subtitles and embodied cognition”. In Reembedding translation process research ed. by R. Muñoz. 171–194. Amsterdam: John Benjamins Publishing Company. DOI logoGoogle Scholar
Kruger, Jan-Louis, and Faans Steyn. 2014. “Subtitles and eye tracking: Reading and performance.” Reading Research Quarterly 49 (1): 105–120. DOI logoGoogle Scholar
Kruger, Jan-Louis, Agnieszka Szarkowska, and Izabela Krejtz. 2015. “Subtitles on the moving image: An overview of eye tracking studies.” Refractory: A Journal of Entertainment Media 25. [URL]
Linebarger, Deborah, Jessica T. Piotrowski, and Charles R. Greenwood. 2010. “Onscreen print: The role of captions as a supplemental literacy tool.” Journal of Research in Reading 33 (2): 148–167. DOI logoGoogle Scholar
Lorch, Elizabeth Pugzles, and Victoria J. Castle. 1997. “Preschool childrenʹs attention to television: Visual attention and probe response times.” Journal of Experimental Child Psychology 66 (1): 111–127. DOI logoGoogle Scholar
Markham, Paul L. 1999. “Captioned video-tapes and second language listening word recognition.” Foreign Language Annals 32 (3): 321–328. DOI logoGoogle Scholar
McClarty, Rebecca. 2012. “Towards a multidisciplinary approach in creative subtitling.” Monographs in Translating and Interpreting (MonTI) 4: 133–155.Google Scholar
Marian, Viorica, Henrike K. Blumenfeld, and Margarita Kaushanskaya. 2007. “The Language Experience and Proficiency Questionnaire (LEAP-Q): Assessing language profiles in bilinguals and multilinguals.” Journal of Speech, Language, and Hearing Research 50 (4): 940–967. DOI logoGoogle Scholar
Martin, Stewart. 2014. “Measuring cognitive load and cognition: Metrics for technology-enhanced learning.” Educational Research and Evaluation 20 (7–8): 592–621. DOI logoGoogle Scholar
Mayer, Richard E., 2009. Multimedia learning. Cambridge/New York: Cambridge University Press. DOI logoGoogle Scholar
Moreno, Roxana, and Richard E. Mayer. 2002 “Verbal redundancy in multimedia learning: When reading helps listening.” Journal of Educational Psychology 94 (1), 156–163. DOI logoGoogle Scholar
Nornes, Abé M. 1999. “For an abusive subtitling.” Film Quarterly 52 (3): 17–34. DOI logoGoogle Scholar
Paas, Fred, Juhani E. Tuovinen, Huib Tabbers, and Pascal W. M. Van Gerven. 2003. “Cognitive load measurement as a means to advance cognitive load theory.” Educational Psychologist 38 (1): 63–71. DOI logoGoogle Scholar
Paas, Fred, J. G. Van Merriënboer, and J. J. Adam. 1994. “Measurement of cognitive load in instructional research.” Perceptual and Motor Skills 79 (1): 419–430. DOI logoGoogle Scholar
Papenmeier, Frank, and Markus Huff. 2010. “DynAOI: A tool for matching eye-movement data with dynamic areas of interest in Aanimations and movies.” Behavior Research Method 42: 179–187. DOI logoGoogle Scholar
Perego, Elisa, Fabio Del Missier, Marco Porta, and Mauro Mosconi. 2010. “The cognitive effectiveness of subtitle processing.” Media Psychology 13 (3): 243–272. DOI logoGoogle Scholar
Pfurtscheller, Gert, and F. H. Lopes Da Silva. 1999. “Event-related EEG/MEG synchronization and desynchronization: basic principles.” Clinical Neurophysiology 110 (11): 1842–1857. DOI logoGoogle Scholar
Rayner, Keith. 1998. “Eye movements in reading and information processing: 20 Years of research.” Psychological Bulletin 124 (3): 372–422. DOI logoGoogle Scholar
Ritchie, Guy. 2011. Sherlock Holmes: A game of shadows. Village Roadshow Pictures/Silver Pictures/Wigram Productions/Lin Pictures.Google Scholar
Sasamoto, Ryoko, and Stephen Doherty. 2016. “Towards the optimal use of impact captions on TV programmes.” In Conflict and communication: A changing Asia in a globalising world, ed. by Minako OʹHagan, and Qi Zhang, 210–247. Hauppauge, NY: Nova Science Publishers.Google Scholar
Sasamoto, Ryoko, Minako O’Hagan, and Stephen Doherty. 2017. “Telop, Affect, and Media Design: A Multimodal Analysis of Japanese TV Programs.” Television & New Media 18 (5): 427–440.Google Scholar
Schmeck, Annett, Maria Opfermann, Tamara van Gog, Fred Paas, and Detlev Leutner. 2015. “Measuring cognitive load with subjective rating scales during problem solving: differences between immediate and delayed ratings.” Instructional Science 43 (1): 93–114. DOI logoGoogle Scholar
Sweller, John, Jeroen J. G. van Merriënboer, and Fred Paas. 1998. “Cognitive architecture and instructional design.” Educational Psychology Review 10: 251–296. DOI logoGoogle Scholar
Sweller, John, Paul Ayres, and Slava Kalyuga. 2011. Cognitive Load Theory. New York: Springer. DOI logoGoogle Scholar
Tal-Or, Nurit, and Jonathan Cohen. 2010. “Understanding audience involvement: Conceptualizing and manipulating identification and transportation.” Poetics 38: 402–418. DOI logoGoogle Scholar
Vanderplank, Robert. 1988. “The value of teletext Sub-titles in language learning.” ELT Journal 42 (4): 272–281. DOI logoGoogle Scholar
. 1990. “Paying attention to the words: Practical and theoretical problems in watching television programmes with uni-lingual (CEEFAX) sub-titles.” System 18 (2): 221–234. DOI logoGoogle Scholar
. 2010. “Déjà vu? A decade of research on language laboratories, television and video in language learning.” Language Teaching 43 (1): 1–37. DOI logoGoogle Scholar
Vigário, Ricardo Nuno. 1997. “Extraction of ocular artefacts from EEG using independent component analysis.” Electroencephalography and Clinical Neurophysiology 103 (3): 395–404. DOI logoGoogle Scholar
Cited by (13)

Cited by 13 other publications

Jiang, Qihang & Stephen Doherty
2024. An eye-tracking study on the cognitive processing, reception, and perception of swearword translation in films. Translation Spaces DOI logo
Martínez, Fernando Casanova
2024. Multimodal exploration of the thank God expressive construction and its implications for translation. Translation, Cognition & Behavior 7:1 ► pp. 48 ff. DOI logo
Wang, Ruiqi, Wonse Jo, Dezhong Zhao, Weizheng Wang, Arjun Gupte, Baijian Yang, Guohua Chen & Byung-Cheol Min
2024.  Husformer: A Multimodal Transformer for Multimodal Human State Recognition. IEEE Transactions on Cognitive and Developmental Systems 16:4 ► pp. 1374 ff. DOI logo
Weingärtner, Henrike, Maximiliane Windl, Lewis L. Chuang & Fiona Draxler
2024. Proceedings of the International Conference on Mobile and Ubiquitous Multimedia, ► pp. 235 ff. DOI logo
Allan, Brahim, Ian Kegel, Sri Harish Kalidass, Andriy Kharechko, Michael Milliken, Sally McClean, Bryan Scotney & Shuai Zhang
2022. Towards automatic placement of media objects in a personalised TV experience. Multimedia Systems 28:6 ► pp. 2175 ff. DOI logo
Black, Sharon
2022. Could integrated subtitles benefit young viewers? Children’s reception of standard and integrated subtitles: a mixed methods approach using eye tracking. Perspectives 30:3 ► pp. 503 ff. DOI logo
陈, 登水
2022. Application of Eye-Tracking Technology in Personalized Online Learning. Advances in Social Sciences 11:11 ► pp. 4901 ff. DOI logo
Lee, Ping-Jung, Yeu-Ting Liu & Wen-Ta Tseng
2021. One size fits all? In search of the desirable caption display for second language learners with different caption reliance in listening comprehension. Language Teaching Research 25:3 ► pp. 400 ff. DOI logo
Kim, Hyang-Sook & Kyongseok Kim
2020. Open Captioning as a Means of Communicating Health Information: The Role of Cognitive Load in Processing Entertainment-Education Content. Journal of Broadcasting & Electronic Media 64:3 ► pp. 519 ff. DOI logo
Raffi, Francesca
2020. The Impact of Italian Dubbing on Viewers’ Immersive Experience: An Audience Reception Study. Online Journal of Communication and Media Technologies 10:3 ► pp. e202019 ff. DOI logo
Wang, Jiahui, Pavlo Antonenko, Andreas Keil & Kara Dawson
2020. Converging Subjective and Psychophysiological Measures of Cognitive Load to Study the Effects of Instructor‐Present Video. Mind, Brain, and Education 14:3 ► pp. 279 ff. DOI logo
Doherty, Stephen
2018. Analysing variable relationships and time-course data in eye-tracking studies of translation processes and products. In Eye tracking and multidisciplinary studies on translation [Benjamins Translation Library, 143], ► pp. 71 ff. DOI logo
Kruger, Jan-Louis & Stephen Doherty
2018. Triangulation of online and offline measures of processing and reception in AVT. In Reception Studies and Audiovisual Translation [Benjamins Translation Library, 141], ► pp. 91 ff. DOI logo

This list is based on CrossRef data as of 12 january 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.