Visual processing during computer-assisted consecutive interpreting: Evidence from eye movements

Chen, Sijia; Kruger, Jan-Louis

doi:10.1075/intp.00104.che

Article published In:

Technology
Edited by Franz Pöchhacker and Minhua Liu
[Interpreting 26:2] 2024
► pp. 231–252

Visual processing during computer-assisted consecutive interpreting

Evidence from eye movements

Sijia Chen | Guangdong University of Foreign Studies

Jan-Louis Kruger | Macquarie University

This study investigates the visual processing patterns during computer-assisted consecutive interpreting (CACI). In phase I of the proposed CACI workflow, the interpreter listens to the source speech and respeaks it into speech recognition (SR) software. In phase II, the interpreter produces target speech supported by the SR text and its machine translation (MT) output. A group of students performed CACI with their eye movements tracked. In phase I, the participants devoted the majority of their attention to listening and respeaking, with very limited attention distributed to the SR text. However, a positive correlation was found between the percentage of dwell time on the SR text and the quality of respeaking, which suggests that active monitoring could be important. In phase II, the participants devoted more visual attention to the MT text than to the SR text and engaged in deeper and more effortful processing when reading the MT text. We identified a positive correlation between the percentage of dwell time on the MT text and interpreting quality in the L2–L1 direction but not in the L1–L2 direction. These results contribute to our understanding of computer-assisted interpreting and can provide insights for future research and training in this area.

Keywords: computer-assisted interpreting, consecutive interpreting, eye tracking, speech recognition, machine translation

Article outline

1.Introduction
2.Literature review
- 2.1Eye movements in reading
- 2.2Eye movements in respeaking, sight translation and post-editing
3.The present study
- 3.1Participants
- 3.2Apparatus
- 3.3Procedure
- 3.4Data and analysis
4.Results
- 4.1Visual processing in the respeaking phase
- 4.2Visual processing in the production phase
5.Discussion
6.Conclusion
References

Published online: 5 July 2024

https://doi.org/10.1075/intp.00104.che

References (50)

References

Barr, D. J., Levy, R., Scheepers, C. & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language 68 (3), 255–278.

Bates, D., Mächler, M., Bolker, B. & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software 67 1, 1–48.

Carl, M., Dragsted, B., Elming, J., Hardt, D. & Jakobsen, A. L. (2011). The process of post-editing: A pilot study. In A. L. Jakobsen, M. Carl, B. Sharp & M. Zock (Eds.), Proceedings of the 8th International NLPCS Workshop. Frederiksberg: Samfundslitteratur, 131–142.

Chafe, W. (1985). Linguistic differences produced by differences between speaking and writing. In D. R. Olson, A. Hildyard & N. Torrance (Eds.), Literacy, language, and learning: The nature and consequences of reading and writing. Cambridge: Cambridge University Press, 105–122.

Chen, S. (2017). Note-taking in consecutive interpreting: New data from pen recording. Translation and Interpreting 9 (1), 4–23.

(2020a). The impact of directionality on the process and product in consecutive interpreting between Chinese and English: Evidence from pen recording and eye tracking. The Journal of Specialised Translation 34 1, 100–117.

(2020b). The process of note-taking in consecutive interpreting: A digital pen recording approach. Interpreting 22 (1), 117–139.

Chen, S. & Kruger, J.-L. (2023). The effectiveness of computer-assisted interpreting: A preliminary study based on English–Chinese consecutive interpreting. Translation and Interpreting Studies 18 (3), 399–420.

(under review). A computer-assisted consecutive interpreting mode: Training and evaluation.

Chmiel, A., Janikowski, P. & Cieślewicz, A. (2020). The eye or the ear? Source language interference in sight translation and simultaneous interpreting. Interpreting 22 (2), 187–210.

Chmiel, A. & Lijewska, A. (2019). Syntactic processing in sight translation by professional and trainee interpreters: Professionals are more time-efficient while trainees view the source text less. Target 31 (3), 378–397.

(2022). Reading patterns, reformulation and eye-voice span (IVS) in sight translation. Translation and Interpreting Studies 18 (2), 213–234.

Cop, U., Drieghe, D. & Duyck, W. (2015). Eye movement patterns in natural reading: A comparison of monolingual and bilingual reading of a novel. PLoS ONE 10 (8), e0134008.

Dam, H. V. & Engberg, J. (2006). Assessing accuracy in consecutive interpreting: a comparison of semantic network analyses and intuitive assessments. In C. Heine, K. Schubert & H. Gerzymisch-Arbogast (Eds.), Text and translation: Theory and methodology of translation. Tübingen: Narr Francke Attempto Verlag, 215–234.

Davitti, E. & Sandrelli, A. (2020). Embracing the complexity: A pilot study on interlingual respeaking. Journal of Audiovisual Translation 3 (2), 103–139.

Defrancq, B. & Fantinuoli, C. (2021). Automatic speech recognition in the booth: Assessment of system performance, interpreters’ performances and interactions in the context of numbers. Target 33 (1), 73–102.

Demareva, V. & Edeleva, Y. (2020). Eye-tracking based L2 detection: Universal and specific eye movement patterns in L1 and L2 reading. Procedia Computer Science 169 1, 673–676.

Downie, J. (2023). Where is it all going? Technology, economic pressures and the future of interpreting. In G. Corpas Pastor & B. Defrancq (Eds.), Interpreting technologies – current and future trends. Amsterdam: John Benjamins, 277–301.

Dragsted, B. & Hansen, I. G. (2009). Exploring translation and interpreting hybrids. The case of sight translation. Meta 54 (3), 588–604.

Eskenazi, M. A. (2023). Best practices for cleaning eye movement data in reading research. Behavior Research Methods.

Fantinuoli, C. (2023). Towards AI-enhanced computer-assisted interpreting. In G. Corpas Pastor & B. Defrancq (Eds), Interpreting technologies – current and future trends. Amsterdam: John Benjamins, 46–71.

Gerver, D. (1969/2002). The effects of source language presentation rate on the performance of simultaneous conference interpreters. In F. Pöchhacker & M. Shlesinger (Eds.), The interpreting studies reader. London and New York: Routledge, 53–66.

Gile, D. (2009). Basic concepts and models for interpreter and translator training (revised edition). Amsterdam: John Benjamins.

Han, C. (2018). Latent trait modelling of rater accuracy of formative peer assessment of English-Chinese consecutive interpreting. Assessment & Evaluation in Higher Education 43(6), 979–994.

Ho, C-E. (2017). An integrated eye-tracking study into the cognitive process of English–Chinese sight translation: Impacts of training and experience. PhD dissertation, National Taiwan Normal University.

Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H. & van de Weijer, J. (2011). Eye tracking: A comprehensive guide to methods and measures. Oxford: Oxford University Press.

Huang, C-C. (2011). Tracking eye movements in sight translation. MA thesis, National Taiwan Normal University.

Koglin, A. (2015). An empirical investigation of cognitive effort required to post-edit machine translated metaphors compared to the translation of metaphors. Translation and Interpreting 7 (1), 126–141.

Kuperman, V., Siegelman, N., Schroeder, S., Acartürk, C., Alexeeva, S., Amenta, S. … Usal, K. A. (2022). Text reading in English as a second language: Evidence from the Multilingual Eye-Movements Corpus. Studies in Second Language Acquisition 45 (1), 3–37.

Lijewska, A., Chmiel, A. & Inhoff, A. W. (2022). Stages of sight translation: Evidence from eye movements. Applied Psycholinguistics 43 1, 997–1018.

Liu, M. & Chiu, Y-H. (2009). Assessing source material difficulty for consecutive interpreting: Quantifiable measures and holistic judgment. Interpreting 11 (2), 244–266.

Moorkens, J., O’Brien, S., da Silva, I. A. L., Fonseca, N. B. d. L. & Alves, F. (2015). Correlations of perceived post-editing effort with measurements of actual effort. Machine Translation 29 1, 267–284.

O’Brien, S. (2011). Towards predicting post-editing productivity. Machine Translation 25 (3), 197–215.

Pan, J., Yan, M. & Laubrock, J. (2017). Perceptual span in oral reading: The case of Chinese. Scientific Studies of Reading 21 (3), 254–263.

Parshina, O., Laurinavichyute, A. K. & Sekerina, I. A. (2021). Eye-movement benchmarks in Heritage Language reading. Bilingualism: Language and Cognition 24 (1), 69–82.

Pisani, E. & Fantinuoli, C. (2021). Measuring the impact of automatic speech recognition on number rendition in simultaneous interpreting. In C. Wang & B. Zheng (Eds.), Empirical studies of translation and interpreting. New York: Routledge, 181–197.

R Core Team (2023). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing.

Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin 124 (3), 372–422.

(2009). Eye movements and attention in reading, scene perception, and visual search. The Quarterly Journal of Experimental Psychology 62 (8), 1457–1506.

Rinne, J. O., Tommola, J., Laine, M., Krause, B. J., Schmidt, D., Kaasinen, V., Teräs, M., Sipilä, H. & Sunnari, M. (2000). The translating brain: Cerebral activation patterns during simultaneous interpreting. Neuroscience Letters 294 (2), 85–88.

Romero-Fresco, P. (2011). Subtitling through speech recognition: Respeaking. Manchester: Routledge.

(2015). Respeaking. In F. Pöchhacker (Ed.), Routledge encyclopedia of interpreting studies. Abingdon: Routledge, 350–351.

Romero-Fresco, P. & Martínez, J. (2015). Accuracy rate in live subtitling: The NER model. In R. Baños & J. Díaz-Cintas (Eds.), Audiovisual translation in a global context: Mapping an ever-changing landscape. London: Palgrave Macmillan, 28–50.

Scott, S. D., Sharpe, H., O’Leary, K., Dehaeck, U., Hindmarsh, K., Moore, J. G. & Osmond, M. H. (2009). Court reporters: A viable solution for the challenges of focus group data collection? Qualitative Health Research 19 (1), 140–146.

Shreve, G. M., Lacruz, I. & Angelone, E. (2010). Cognitive effort, syntactic disruption, and visual interference in a sight translation task. In G. M. Shreve & E. Angelone (Eds.). Translation and cognition. Amsterdam and Philadelphia: John Benjamins, 63–84.

Sun, F. (1993). Eye movements in reading Chinese: Paragraphs, single characters and pinyin. In S. F. Wright & R. Groner (Eds.), Facets of dyslexia and its remediation. Amsterdam: North-Holland, 245–255.

Sun, F., Morita, M. & Stark, L. W. (1985). Comparative patterns of reading eye movement in Chinese and English. Perception and Psychophysics 37 1, 502–506.

Szarkowska, A., Dutka, Ł., Szychowska, A. & Pilipczuk, O. (2018). Visual attention distribution in intralingual respeaking: An eye-tracking study. In C. Walker & F. M. Federici (Eds.), Eye tracking and multidisciplinary studies on translation. Amsterdam: John Benjamins, 185–201.

Toral, A., Wieling, M. & Way, A. (2018). Post-editing effort of a novel with statistical and neural machine translation. Frontiers in Digital Humanities 5 1, 1–11.

White, S. J., Warrington, K. L., McGowan, V. A. & Paterson, K. B. (2015). Eye movements during reading and topic scanning: Effects of word frequency. Journal of Experimental Psychology: Human Perception and Performance 41 (1), 233–248.