Cover not available
Article published In:
Translation, Cognition & Behavior
Vol. 4:1 (2021) ► pp.1–25

Does it help to see the speaker’s lip movements?

An investigation of cognitive load and mental effort in simultaneous interpreting

ORCID logo | Zurich University for Applied Sciences (ZHAW)
Simultaneous interpreting combines auditory and visual information. Within a multitude of visual inputs that interpreters receive, the one from the speaker seems to be particularly important (Bühler 1985; Seubert 2019). One reason might be that lip movements enhance speech perception and might thus reduce cognitive load in simultaneous interpreting and hence, induce lower mental effort. This effect may be even more pronounced when noise is added to the source speech. This study was conducted to investigate cognitive load and mental effort during simultaneous interpreting (a) with and without the ability to see speaker’s lip movements, and (b) with and without interfering noise. A group of listeners was included to control for task-related effects. Mental effort and cognitive load were measured using pupillometry, interpreting accuracy measures, and subjective reports. The facilitation hypothesis for lip movements was not confirmed. However, the pupillometric data suggests that lip movements may increase arousal.
Keywords: simultaneous interpreting, cognitive load, arousal, pupillometry, visual input
Article outline
  • 1.Introduction
    • 1.1Lip movements as visual input in simultaneous interpreting
    • 1.2Tapping into cognitive load and mental effort
  • 2.Materials and methods
    • 2.1Participants
    • 2.2Experimental stimuli
    • 2.3Apparatus
    • 2.4Procedure
  • 3.Analysis and results
    • 3.1Subjective reports
    • 3.2Interpreting accuracy
    • 3.3Pupillometry
  • 4.Discussion and conclusions
    • 4.1What is the effect of lip movements on interpreting and what do pupil sizes tell us?
    • 4.2Limitations
    • 4.3Conclusion
  • Acknowledgments
  • Notes
  • References
Published online: 19 May 2021
DOI logo
https://doi.org/10.1075/tcb.00049.gie
References (75)
References
Anderson, Linda. 1994. “Simultaneous Interpretation: Contextual and Translation Aspects.” In Benjamins Translation Library: Vol. 31. Bridging the Gap: Empirical Research in Simultaneous Interpretation. Edited by S. Lambert and B. Moser-Mercer, 101–248. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Bates, Douglas, Martin Mächler, Ben Bolker & Steve Walker. 2015. “Fitting Linear Mixed-Effects Models Using lme4.” Journal of Statistical Software 67 (1). DOI logoGoogle Scholar
Beatty, Jack. 1982. “Phasic Not Tonic Pupillary Responses Vary With Auditory Vigilance Performance.” Psychophysiology, 19(2): 167–172. DOI logoGoogle Scholar
Bernstein, Lynne E., Edward T. Auer, Sumik Takayanagi. 2004. “Auditory Speech Detection in Noise Enhanced by Lipreading.” Speech Communication, 44(1–4): 5–18. DOI logoGoogle Scholar
Brancazio, Lawrence, Catherine T. Best and Carol A. Fowler. 2006. “Visual Influences on Perception of Speech and Nonspeech Vocal-Tract Events.” Language and Speech, 49(1): 21–53. DOI logoGoogle Scholar
Brown, Robert and Howard. E. Page. 1939. “Pupil Dilatation and Dark Adaptation.” Journal of Experimental Psychology, 25(4): 347–360. DOI logoGoogle Scholar
Bühler, Hildegund. 1985. “Conference Interpreting: A Multichannel Communication Phenomenon.” Meta: Journal des traducteurs, 30(1): 49–54. DOI logoGoogle Scholar
Chapman, C. Richard, Shunichi Oka, David H. Bradshaw, Robert C. Jacobson and Gary W. Donaldson. 1999. “Phasic Pupil Dilation Response to Noxious Stimulation in Normal Volunteers: Relationship to Brain Evoked Potentials and Pain Report.” Psychophysiology, 36(1): 44–52. DOI logoGoogle Scholar
Christensen, Rune H. B. 2015. Ordinal – Regression Models for Ordinal Data. Version 2015.1–21. [URL] (accessed 01 December 2015).
Courtney, Christopher G., Michael E. Dawson, Anne M. Schell, Arvind Iyer, Thomas D. Parsons. 2010. “Better than the Real Thing: Eliciting fear with Moving and Static Computer-generated Stimuli.” International Journal of Psychophysiology, 78(2): 107–114. DOI logoGoogle Scholar
Davies, M. 2008. Word frequency data. Retrieved from The Corpus of Contemporary American English (COCA): [URL] (accessed 19 March 2021).
Díaz-Galaz, Stephanie, Presentacion Padilla Benítez, and M. Teresa Bajo Molina. 2015. “The Role of Advance Preparation in Simultaneous Interpreting: A Comparison of Professional Interpreters and Interpreting Students”. Interpreting 17(1): 1–25. DOI logoGoogle Scholar
Dillinger, Mike. 1990. “Comprehension during Interpreting: What Do Interpreters Know that Bilinguals Don’t?” The Interpreters’ Newsletter, 31: 41–58. Accessed March 03 17, 2021. [URL]
Ehrensberger-Dow, Maureen, Michaela Albl-Mikasa, Katrin Andermatt, Andrea Hunziker Heeb and Caroline Lehr. 2020. “Cognitive Load in Processing ELF: Translators, Interpreters, and Other Multilinguals.” Journal of English as a Lingua Franca 9(2): 217–238. DOI logoGoogle Scholar
Engelhardt, Paul E., Fernanda Ferreira and Elana G. Patsenko. 2010. “Pupillometry Reveals Processing Load during Spoken Language Comprehension.” Quarterly Journal of Experimental Psychology, 63(4): 639–645. DOI logoGoogle Scholar
European Commission. 2009a. “United Airlines rewards fittest people.” Retrieved from Speech Repository: [URL] (accessed 4 February 2021).
European Commission. 2009b. “Disenchantment at work.” Retrieved from Speech Repository: [URL] (accessed 4 February 2021)
. 2012a. “Demographic shift in Europe.” Retrieved from Speech Repository: [URL] (accessed 4 February 2021).
European Commission. 2012b. “Greece in the doldrums.” Retrieved from Speech Repository: [URL] (accessed 4 February 2021).
Fisk, Angus S., Shu K. E. Tam, L. A. Brown, Vladyslav V. Vyazovskiy, David M. Bannerman and Stuart N. Peirson. 2018. “Light and Cognition: Roles for Circadian Rhythms, Sleep, and Arousal.” Frontiers in Neurology, 91: 56. DOI logoGoogle Scholar
Gerver, David. 2002. “The Effects of Source Language Presentation Rate on the Performance of Simultaneous Conference Interpreters.” The Interpreting Studies. Reader. Edited by F. Pöchhacker and M. Shesinger, 52–66. London: Routledge.Google Scholar
. 1974. “The Effects of Noise on the Performance of Simultaneous Interpreters: Accuracy of Performance.” Acta Psychologica, 38(3): 159–167. DOI logoGoogle Scholar
Gieshoff, Anne Catherine. 2021. “The impact of visible lip movements on silent pauses in simultaneous interpreting.” Interpreting. Advanced online publication. DOI logoGoogle Scholar
. 2018. “The Impact of Visual Input on Cognitive Load in Simultaneous Interpreting”. Mainz: Johannes Gutenberg Universität.Google Scholar
Granholm, Eric, Robert F. Asarnow, Andrew J. Sarkin and Karen, L. 1996. “Pupillary Responses Index Cognitive Resource Limitations.” Psychophysiology, 331: 457–461. DOI logoGoogle Scholar
Halverson, Sandra. 2017. “Multimethod approaches.” The Handbook of Translation and Cognition. Edited by J. Schwieter and A. Ferreira. 195–212. Hoboken: Wiley Blackwell. DOI logoGoogle Scholar
Hild, Adelina. 2015. “Discourse Comprehension in Simultaneous Interpreting: The Role of Expertise and Redundancy.” Psycholinguistic and Cognitive Inquiries into Translation and Interpreting. Edited by A. Ferreira and J. W. Schwieter. 67–100. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Hyönä, Jukka, Jorma Tommola, Anna-Maria Alaja. 1995. “Pupil Dilation as a Measure of Processing Load in Simultaneous Interpretation and Other Language Tasks.” The Quarterly Journal of Experimental Psychology Section A, 48(3): 598–612. DOI logoGoogle Scholar
Ivars, Amparo J., Daniel P. Calatayud. 2001. “‘I Failed because I Got Nervous’. Anxiety and Performance in Interpreter Trainees: An Empirical Study.” The Interpreter’s Newsletter, 111: 105–120. Accessed March 17, 2021. [URL]
Jesse, Alexandra, Nick Vrignaud, Michael M. Cohen and Dominic W. Massaro. 2000. “The Processing of Information from Multiple Sources in Simultaneous Interpreting.” Interpreting, 5(2): 95–115. DOI logoGoogle Scholar
Jordan, Timothy R. and Paul Sergeant. 2000. “Effects of Distance on Visual and Audiovisual Speech Recognition.” Language and Speech, 43(1): 107–124. DOI logoGoogle Scholar
Kahneman, Daniel. 1973. Attention and Effort. Englewood Cliffs, N.J.: Prentice-Hall.Google Scholar
Korpal, Paweł. 2016. “Interpreting as a Stressful Activity: Physiological Measures of Stress in Simultaneous Interpreting.” Poznan Studies in Contemporary Linguistics, 52(2): 297–316. DOI logoGoogle Scholar
Kramer, Sophia E., Theo S. Kapteyn, Josst M. Festen, and Dirk J. Kuik. 1997. “Assessing Aspects of Auditory Handicap by Means of Pupil Dilation.” Audiology, 361: 155–164. DOI logoGoogle Scholar
Krejtz, Krzysztof, Andrew T. Duchowski, Anna Niedzielska, Cezary Biele, and Izabela Krejtz. 2018. “Eye Tracking Cognitive Load Using Pupil Diameter and Microsaccades with Fixed Gaze”. PLOS ONE 13(9): e0203629. DOI logoGoogle Scholar
Kuchinsky, Stefanie E., Jayne B. Ahlstrom, Kenneth I. Vaden, Stephanie L. Cute, Larry E. Humes, Judy R. Dubno and Mark A. Eckert. 2013. “Pupil Size Varies with Word Listening and Response Selection Difficulty in Older Adults with Hearing Loss: Pupil Size in Older Adults.” Psychophysiology, 50(1): 23–34. DOI logoGoogle Scholar
Lecumberri, Maria L. G., Martin Cooke and Anne Cutler. 2010. “Non-native Speech Perception in Adverse Conditions: A Review.” Speech Communication, 52(1): 864–886. DOI logoGoogle Scholar
Lee, Jieun. 2008. “Rating Scales for Interpreting Performance Assessment.” The Interpreter and Translator Trainer, 2(2): 165–184. DOI logoGoogle Scholar
Liu, Minhua, Diane L. Schallert, and Patrick J. Carroll. 2004. ‘Working Memory and Expertise in Simultaneous Interpreting’. Interpreting 6(1): 19–42. DOI logoGoogle Scholar
Lowenstein, Otto, Richard Feinberg and Irene Loewenfeld. 1963. “Pupillary Movements During Acute and Chronic Fatigue A New Test for the Objective Evaluation of Tiredness.” Investigative Ophthalmology, 2(2): 138–158.Google Scholar
Massaro, Dominic W. And Michael M. Cohen. 1999. “Speech Perception in Perceivers with Hearing Loss: Synergy of Multiple Modalities.” Journal of Speech, Language, and Hearing Research, 42(1), 21–41. DOI logoGoogle Scholar
Mirman, Daniel. 2014. Growth Curve Analysis and Visualization Using R. Boca Raton, FL: CRC Press.Google Scholar
Moser-Mercer, Barbara, Alexandra Künzli and Marina Korac. 1998. “Prolonged Turns in Interpreting: Effects on Quality, Physiological and Psychological Stress (Pilot study).” Interpreting, 3(1): 47–64. DOI logoGoogle Scholar
Oliva, Manuel and Andrey Anikin. 2018. “Pupil Dilation Reflects the Time Course of Emotion Recognition in Human Vocalizations.” Scientific Reports, 81, 4871. DOI logoGoogle Scholar
Peelle, Jonathan E. and Mitchell S. Sommers. 2015. “Prediction and Constraint in Audiovisual Speech Perception.” Cortex, 681: 169–181. DOI logoGoogle Scholar
Peirce, Jonathan, Jeremy R. Gray, Sol Simpson, Michael MacAskill, Richard Höchenberger, Hiroyuki Sogo, Eric Kastman and Jonas Kristoffer Lindeløv. 2019. “PsychoPy2: Experiments in Behavior made Easy.” Behavior Research Methods, 51(1): 195–203. DOI logoGoogle Scholar
Peirce, Jonathan W. 2007. “PsychoPy – Psychophysics Software in Python.” Journal of Neuroscience Methods, 162(1–2), 8–13. DOI logoGoogle Scholar
R Core Team. 2016. “R: A language and environment for statistical computing.” Version 3.2.4. Retrieved from R Foundation for Statistical Computing: [URL] (accessed 1 March 2016).
Rennert, Sylvie. 2008. “Visual Input in Simultaneous Interpreting.” Meta: journal des traducteurs, 53(1): 204–217. DOI logoGoogle Scholar
Roziner, Ilan & Miriam Shlesinger. 2010. “Much Ado about Something Remote: Stress and Performance in Remote Interpreting.” Interpreting, 12(2): 214–247. DOI logoGoogle Scholar
Seeber, Kilian G. 2017. “Multimodal Processing in Simultaneous Interpreting.” The Handbook of Translation and Cognition. Edited by J. Schwieter and A. Ferreira. 461–475. Hoboken: Wiley Blackwell. DOI logoGoogle Scholar
2015. “Cognitive Load in Simultaneous Interpreting: Measures and Methods.” Interdisciplinarity in translation and interpreting process research. Edited by M. Ehrensberger-Dow, S. Göpferich & S. O’Brien (Eds.). 18–33. DOI logoGoogle Scholar
Seeber, Kilian G. and Dirk Kerzel. 2012. “Cognitive Load in Simultaneous Interpreting: Model Meets Data.” International Journal of Bilingualism, 16(2): 228–242. DOI logoGoogle Scholar
Seeber, Kilian G. 2011. “Cognitive Load in Simultaneous Interpreting.” Interpreting, 13(2): 176–204. DOI logoGoogle Scholar
Setton, Robin. 1999. Simultaneous Interpretation: A Cognitive-Pragmatic Analysis. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Seubert, Sabine. 2019. Visuelle Informationen beim Simultandolmetschen: Eine Eyetracking-Studie. Berlin: Frank & Timme GmbH.Google Scholar
Stachowiak-Szymczak, Katarzyna. 2019. Eye Movements and Gestures in Simultaneous and Consecutive Interpreting. DOI logoGoogle Scholar
Sumby, William H. and Irwin Pollack. 1954. “Visual Contribution to Speech Intelligibility in Noise.” Journal of the Acoustical Society of America, (261): 212–215. DOI logoGoogle Scholar
Tabri, Dollen, Kim M. S. A. Chacra & Tim Pring. 2010. “Speech Perception in Noise by Monolingual, Bilingual and Trilingual Listeners.” International Journal of Language & Communication Disorders, 46(4): 411–422. DOI logoGoogle Scholar
Thomas, Sharon M., Timothy R. Jordan. 2004. “Contributions of Oral and Extraoral Facial Movement to Visual and Audiovisual Speech Perception.” Journal of Experimental Psychology: Human Perception and Performance, 30(5): 873–888. DOI logoGoogle Scholar
Tobii support. 2016, July 20. Personal communication.Google Scholar
Tobii Technology. 2010. Tobii TX300 Eye Tracker. User Manual. Accessed March 17, 2021. [URL]
Tommola, Jorma and Johan Lindholm. 1995. “Experimental Research on Interpreting: Which Dependent Variable?” Topics in interpreting research. Edited by J. Tommola. 121–133. Turku: University of Turku.Google Scholar
Tye-Murray, Nancy, Mitchell Sommers and Brent Spehar. 2007. “Auditory and Visual Lexical Neighborhoods in Audiovisual Speech Perception.” Trends in Amplification, 11(4): 233–241. DOI logoGoogle Scholar
van der Wel, Pauline and Henk van Steenbergen. 2018. “Pupil Dilation as an Index of Effort in Cognitive Control Tasks: A Review.” Psychonomic Bulletin & Review, 25(6): 2005–2015. DOI logoGoogle Scholar
Vatikiotis-Bateson, Eric, Inge-Marie Eigsti, Sumio Yano and Kevin G. Munhall. 1998. “Eye Movement of Perceivers during Audiovisual Speech Perception.” Perception & Psychophysics, 60(6): 926–940. DOI logoGoogle Scholar
von Kriegstein, Katharina, Özgür Dogan, Martina Grüter, Anne-Lise Giraud, Christian A. Kell, Thomas Grüter, Andreas Kleinschmidt and Stefan A. Kiebel. 2008. “Simulation of Talking Faces in the Human Brain Improves Auditory Speech Recognition.” Proceedings of the National Academy of Sciences, 105(18): 6747–6752. DOI logoGoogle Scholar
Wickham, H. 2009. ggplot2: Elegant Graphics for Data Analysis. New York: Springer. DOI logoGoogle Scholar
Wong, Hoe K. and Julien Epps. 2016. “Pupillary Transient Responses to Within-Task Cognitive Load Variation.” Computer Methods and Programs in Biomedicine, 1371: 47–63. DOI logoGoogle Scholar
Yoon, Jinny J. and Helen V. Danesh-Meyer. 2019. “Caffeine and the Eye.” Survey of Ophthalmology, 64(3): 334–344. DOI logoGoogle Scholar
Zekveld, Adriana A., Dirk J. Heslenfeld, Ingrid S. Johnsrude, Niek J. Versfeld and Sophia E. Kramer. 2014. “The Eye as a Window to the Listening Brain: Neural Correlates of Pupil Size as a Measure of Cognitive Listening Load.” NeuroImage, 1011: 76–86. DOI logoGoogle Scholar
Zwischenberger, Cornelia. 2010. “Quality Criteria in Simultaneous Interpreting: An International vs. A National View.” The Interpreters’ Newsletter, 151: 127–142. Accessed 17 March 2021. [URL]
Cited by (6)

Cited by six other publications

Bauer, Robert & Petra Jansen
2024. A short mindfulness induction might increase women’s mental rotation performance. Consciousness and Cognition 123 ► pp. 103721 ff. DOI logo
Gieshoff, Anne Catherine & Michaela Albl-Mikasa
2024. Interpreting accuracy revisited: a refined approach to interpreting performance analysis. Perspectives 32:2 ► pp. 210 ff. DOI logo
Peng, Xuejiao, Xiangling Wang & Guangjiao Chen
2024. Text availability and the speaker’s visibility in simultaneous interpreting: effects on the process, product, and interpreters’ perceptions. Perspectives ► pp. 1 ff. DOI logo
Shao, Zhangminzi & Bart Defrancq
2024. Fundamental frequency as an acoustic mirror of interpreters’ cognitive states. Interpreting. International Journal of Research and Practice in Interpreting DOI logo
Gieshoff, Anne Catherine & Andrea Hunziker Heeb
2023. Cognitive load and cognitive effort. Translation, Cognition & Behavior 6:1 ► pp. 3 ff. DOI logo
Melicherčíková, Miroslava & Soňa Hodáková
2023. Prieniky a odlišnosti v kognitívno-osobnostnom profile a tlmočníckom výkone profesionálov a študentov, DOI logo

This list is based on CrossRef data as of 7 november 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.