Cover not available
Article published In:
Target
Vol. 32:1 (2020) ► pp.37–58

Multimodal processing in simultaneous interpreting with text

Interpreters focus more on the visual than the auditory modality

ORCID logo | Adam Mickiewicz University in Poznań
| University of Silesia in Katowice
| Adam Mickiewicz University in Poznań

The present study focuses on (in)congruence of input between the visual and the auditory modality in simultaneous interpreting with text. We asked twenty-four professional conference interpreters to simultaneously interpret an aurally and visually presented text with controlled incongruences in three categories (numbers, names and control words), while measuring interpreting accuracy and eye movements. The results provide evidence for the dominance of the visual modality, which goes against the professional standard of following the auditory modality in the case of incongruence. Numbers enjoyed the greatest accuracy across conditions possibly due to simple cross-language semantic mappings. We found no evidence for a facilitation effect for congruent items, and identified an impeding effect of the presence of the visual text for incongruent items. These results might be interpreted either as evidence for the Colavita effect (in which visual stimuli take precedence over auditory ones) or as strategic behaviour applied by professional interpreters to avoid risk.

Keywords: simultaneous interpreting, multimodal processing, multisensory processing, incongruence, visual modality
Article outline
  • 1.Introduction
  • 2.Existing research on multimodal processing
    • 2.1Multimodal processing in simultaneous interpreting with text
    • 2.2Cognitive aspects of multimodal processing
  • 3.The present study
    • 3.1Participants
    • 3.2Materials
    • 3.3Procedure
  • 4.Results
  • 5.Discussion
  • 6.Conclusions
  • References
Published online: 21 January 2020
DOI logo
https://doi.org/10.1075/target.18157.chm
References
Braun, Susanne, and Andrea Clarici
1996 “Inaccuracy for Numerals in Simultaneous Interpretation: Neurolinguistic and Neuropsychological Perspectives.” Interpreting 71: 85–102.Google Scholar
Cammoun, Rawdha, Catherine Davies, Konstantin Ivanov, and Boris Naimushin
2009 Simultaneous Interpretation with Text. Is the Text ‘Friend’ or ‘Foe’? Laying Foundations for a Teaching Module.” MA diss. University of Geneva.Google Scholar
Chmiel, Agnieszka
2015 “Przetwarzanie w tłumaczeniu symultanicznym.” In Dydaktyka tłumaczenia ustnego, edited by Agnieszka Chmiel and Przemysław Janikowski, 227–247. Katowice: Stowarzyszenie Inicjatyw Wydawniczych przy współpracy UŚ i UAM.Google Scholar
Colavita, Francis
1974 “Human Sensory Dominance.” Perception & Psychophysics 16 (2): 409–412. DOI logoGoogle Scholar
Dehaene, Stanislas, and Laurent Cohen
2007 “Cultural Recycling of Cortical Maps.” Neuron 56 (2): 384–398. DOI logoGoogle Scholar
Desmet, Bart, Mieke Vandierendonck, and Bart Defrancq
2018 “Simultaneous Interpretation of Numbers and the Impact of Technological Support.” In Interpreting and Technology, edited by Claudio Fantinuoli, 13–27. Berlin: Language Science Press.Google Scholar
Deutsch, Avital, and Shlomo Bentin
2001 “Syntactic and Semantic Factors in Processing Gender Agreement in Hebrew: Evidence from ERPs and Eye Movements.” Journal of Memory and Language 45 (2): 200–224. DOI logoGoogle Scholar
Doehrmann, Oliver, and Marcus Naumer
2008 “Semantics and the Multisensory Brain: How Meaning Modulates Processes of Audio-Visual Integration.” Brain Research 12421 (November): 136–150. DOI logoGoogle Scholar
Gile, Daniel
1997 “Conference Interpreting as a Cognitive Management Problem.” In Cognitive Processes in Translation and Interpreting, edited by Joseph Danks, Gregory Shreve, Stephen Fountain, and Michael McBeath, 196–214. Thousand Oaks, CA: Sage.Google Scholar
2009 Basic Concepts and Models for Interpreter and Translator Training. Revised edition. Amsterdam: John Benjamins. DOI logoGoogle Scholar
2018 “The Effort Models and Gravitational Model Clarifications and Update.” June 1. [URL]
Hirst, Rebecca, Lucy Cragg, and Harriet Allen
2018 “Vision Dominates Audition in Adults but Not Children: A Meta-Analysis of the Colavita Effect.” Neuroscience & Biobehavioral Reviews 941 (November): 286–301. DOI logoGoogle Scholar
Iodice, Rosario, Juan José García Meilán, Juan Carro Ramos, and Jeff A. Small
2018 “Sentence Context and Word-Picture Cued-Recall Paired-Associate Learning Procedure Boosts Recall in Normal and Mild Alzheimer’s Disease Patients.” Behavioural Neurology. DOI logoGoogle Scholar
Ivanov, Konstantin, Kate Davies, and Boris Naimushin
2014 “Teaching Simultaneous Interpretation with Text.” In Взаимопонимание в Многоязычном Мире: Культура, Язык, Перевод. Сборник Статей в Честь Профессора и.с. Алексеевой / Fighting the Fog in Multilingualism. A Festschrift in Honour of Irina S. Alekseeva, edited by Sergey Goncharov and Angelique Antonova, 38–48. St. Petersburg: Herzen University Press.Google Scholar
Janikowski, Przemysław, Agnieszka Chmiel, and Agnieszka Lijewska
In preparation). “Coping with Sensory Overload in Simultaneous Interpreting with Text: Examining the Timecourse with Eye-Ear, Ear-Voice, and Eye-Voice Span Measures.”
Jiménez Ivars, Maria Amparo
1999 La traducción a la vista: Un análisis descriptivo. PhD diss. Universitat Jaume I.Google Scholar
Koppen, Camille, Agnès Alsius, and Charles Spence
2008 “Semantic Congruency and the Colavita Visual Dominance Effect.” Experimental Brain Research 184 (4): 533–546. DOI logoGoogle Scholar
Koppen, Camille, and Charles Spence
2007 “Spatial Coincidence Modulates the Colavita Visual Dominance Effect.” Neuroscience Letters 417 (2): 107–111. DOI logoGoogle Scholar
Korpal, Paweł, and Katarzyna Stachowiak
2015 “The Visual or the Aural: Which Modality Is Dominant in Simultaneous Interpreting?” Conference paper presented at the ICEAL (International Conference on Eyetracking and Applied Linguistics), Warszawa, September 22.
2017 “A Closer Look at Numbers in Simultaneous Interpreting: An Eye-Tracking Study.” Poster presented at the 19th European Conference on Eye Movements, Wuppertal, August 20.
Lamberger-Felber, Heike, and Julia Schneider
2009 “Linguistic Interference in Simultaneous Interpreting with Text: A Case Study.” In Efforts and Models in Interpreting and Translation Research. A Tribute to Daniel Gile (Benjamins Translation Library 80), edited by Gyde Hansen, Andrew Chesterman, and Heidrun Gerzymisch-Arbogast, 215–236. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Lambert, Sylvie
2004 “Shared Attention during Sight Translation, Sight Interpretation and Simultaneous Interpretation.” Meta: Journal des traducteurs / Meta: Translators’ Journal 49 (2): 294–306. DOI logoGoogle Scholar
Laurienti, Paul, Robert Kraft, Joseph Maldjian, Jonathan Burdette, and Mark Wallace
2004 “Semantic Congruence Is a Critical Factor in Multisensory Behavioral Performance.” Experimental Brain Research 158 (4): 405–414. DOI logoGoogle Scholar
Leeson, Loraine
2005 “Making the Effort in Simultaneous Interpreting: Some Considerations for Signed Language Interpreters.” In Topics in Signed Language Interpreting: Theory and Practice, edited by Terry Janzen, 51–68. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Lefcheck, Jonathan, and Robert Freckleton
2016 “PIECEWISESEM: Piecewise Structural Equation Modelling in R for Ecology, Evolution, and Systematics.” Methods in Ecology and Evolution 7 (5): 573–579. DOI logoGoogle Scholar
Lemhöfer, Kristin, and Mirjam Broersma
2012 “Introducing LexTALE: A Quick and Valid Lexical Test for Advanced Learners of English.” Behavior Research Methods 44 (2): 325–43. DOI logoGoogle Scholar
Matusz, Pawel, Hannah Broadbent, Jessica Ferrari, Benjamin Forrest, Rebecca Merkley, and Gaia Scerif
2015 “Multi-Modal Distraction: Insights from Children’s Limited Attention.” Cognition 1361 (March): 156–165. DOI logoGoogle Scholar
Messina, Alessandro
1998 “The Reading Aloud of English Language Texts in Simultaneously Interpreted Conferences.” Interpreting 3 (2): 147–161. DOI logoGoogle Scholar
Nakagawa, Shinichi, and Holger Schielzeth
2013 “A General and Simple Method for Obtaining R2 from Generalized Linear Mixed-Effects Models.” Methods in Ecology and Evolution 4 (2): 133–142. DOI logoGoogle Scholar
Pinochi, Diletta
2009 “Simultaneous Interpretation of Numbers: Comparing German and English to Italian. An Experimental Study.” The Interpreters’ Newsletter 141: 33–57.Google Scholar
Prior, Anat, Michal Katz, Islam Mahajna, and Orly Rubinsten
2015 “Number Word Structure in First and Second Language Influences Arithmetic Skills.” Frontiers in Psychology 61 (March). DOI logoGoogle Scholar
Pöchhacker, Franz
2004 Introducing Interpreting Studies. London: Routledge. DOI logoGoogle Scholar
Pym, Anthony
2004 “Text and Risk in Translation.” In Choice and Difference in Translation: The Specifics of Transfer, edited by Maria Sidiropoulou and Anastasia Papaconstantinou, 27–42. Athens: University of Athens.Google Scholar
Schweda Nicholson, Nancy
1989 “Documentation and Text Preparation for Simultaneous Interpretation.” In Coming of Age: Proceedings of the 30th Annual Conference of the American Translators Association, Oct. 11–15, 1989, Washington, D.C., edited by Deanna Hammond, 163–182. Medford, N.J.: Learned Information Inc.Google Scholar
Seeber, Kilian G.
2016 “Modeling Multimodal Processing in Simultaneous Interpreting.” Poster presented at the 75th anniversary of FTI, Genève, September 30.
Seeber, Kilian
2017 “Multimodal Processing in Simultaneous Interpreting.” In The Handbook of Translation and Cognition, edited by John W. Schwieter and Aline Ferreira, 461–475. Hoboken, NJ: Wiley Blackwell. DOI logoGoogle Scholar
Setton, Robin, and Andrew Dawrant
2016 Conference Interpreting: A Complete Course. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Sile, Dina
2003 “Simultaneous Interpreting with Text: A Suggestion for Teaching Methodology.” In The Third Riga Symposium on Pragmatic Aspects of Translation: Proceedings, edited by Andrejs Veisbergs, 153–162. Riga: University of Latvia, Aarhus School of Business.Google Scholar
Sinnett, Scott, Charles Spence, and Salvador Soto-Faraco
2007 “Visual Dominance and Attention: The Colavita Effect Revisited.” Perception & Psychophysics 69 (5): 673–686. DOI logoGoogle Scholar
Sinnett, Scott, Salvador Soto-Faraco, and Charles Spence
2008 “The Co-occurrence of Multisensory Competition and Facilitation.” Acta Psychologica 128 (1): 153–161. DOI logoGoogle Scholar
Spence, Charles
2009 “Explaining the Colavita Visual Dominance Effect.” In Attention (Progress in Brain Research 176), edited by Narayanan Srinivasan, 245–258. Amsterdam: Elsevier. DOI logoGoogle Scholar
Spence, Charles, Cesare Parise, and Yi-Chuan Chen
2012 “The Colavita Visual Dominance Effect.” In The Neural Bases of Multisensory Processes, edited by Micah M. Murray and Mark T. Wallace, 529–566. Boca Raton, FL: CRC Press. [URL]
Spychała, Joanna Maria
2015 The Impact of Source Text Availability on Simultaneous Interpreting Performance. MA diss. Uniwersytet im. Adama Mickiewicza w Poznaniu.Google Scholar
Stubblefield, Alexandra, Lauryn Jacobs, Yongju Kim, and Paula Goolkasian
2013 “Colavita Dominance Effect Revisited: The Effect of Semantic Congruity.” Attention, Perception, & Psychophysics 75 (8): 1827–1839. DOI logoGoogle Scholar
Sullivan, Brian, Harm Brouwer, Alba Rodriguez, Matthew Crocker, and Pia Knoeferle
2016 “Eye Movements during Linguistic-Visual Conflicts.” Poster presented at The Attentive Listener in the Visual World. 3rd AttLis workshop, Potsdam, March 10.
Whiting, Caroline, Yury Shtyrov, and William Marslen-Wilson
2015 “Real-Time Functional Architecture of Visual Word Recognition.” Journal of Cognitive Neuroscience 27 (2): 246–265. DOI logoGoogle Scholar
Wille, Claudia, and Mirjam Ebersbach
2016 “Semantic Congruency and the (Reversed) Colavita Effect in Children and Adults.” Journal of Experimental Child Psychology 1411 (January): 23–33. DOI logoGoogle Scholar
Zeier, Hans
1997 “Psychophysiological Stress Research.” Interpreting 2 (1–2): 231–249. DOI logoGoogle Scholar
Cited by

Cited by 4 other publications

Kruger, Jan-Louis
2021. Eye tracking. In Handbook of Translation Studies [Handbook of Translation Studies, 5], ► pp. 80 ff. DOI logo
Pöchhacker, Franz
2021. Multimodality in interpreting. In Handbook of Translation Studies [Handbook of Translation Studies, 5], ► pp. 152 ff. DOI logo
Seeber, Kilian G., Laura Keller & Alexis Hervais-Adelman
2020. When the ear leads the eye – the use of text during simultaneous interpretation. Language, Cognition and Neuroscience 35:10 ► pp. 1480 ff. DOI logo
Su, Wenchao
2020. Issues and Approaches to CTIS. In Eye-Tracking Processes and Styles in Sight Translation [New Frontiers in Translation Studies, ], ► pp. 9 ff. DOI logo

This list is based on CrossRef data as of 7 february 2022. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.