Article published In:
Pragmatics & CognitionVol. 14:1 (2006) ► pp.53–82
What speakers do and what addressees look at
Visual attention to gestures in human interaction live and on video
This study investigates whether addressees visually attend to speakers’ gestures in interaction and whether attention is modulated by changes in social setting and display size. We compare a live face-to-face setting to two video conditions. In all conditions, the face dominates as a fixation target and only a minority of gestures draw fixations. The social and size parameters affect gaze mainly when combined and in the opposite direction from the predicted with fewer gestures fixated on video than live. Gestural holds and speakers’ gaze at their own gestures reliably attract addressees’ fixations in all conditions. The attraction force of holds is unaffected by changes in social and size parameters, suggesting a bottom-up response, whereas speaker-fixated gestures draw significantly less attention in both video conditions, suggesting a social effect for overt gaze-following and visual joint attention. The study provides and validates a video-based paradigm enabling further experimental but ecologically valid explorations of cross-modal information processing.
Keywords: interaction, Gesture, cross-modal information processing, eye gaze
Cited by
Cited by 86 other publications
Alcaraz Carrión, Daniel & Javier Valenzuela
2021. Distant time, distant gesture: speech and gesture correlate to express temporal distance. Semiotica 2021:241 ► pp. 159 ff. 
Arbona, Eléonore, Kilian G. Seeber & Marianne Gullberg
2023. Semantically related gestures facilitate language comprehension during simultaneous interpreting. Bilingualism: Language and Cognition 26:2 ► pp. 425 ff.
Asalıoğlu, Esma Nur & Tilbe Göksun
2023. The role of hand gestures in emotion communication: Do type and size of gestures matter?. Psychological Research 87:6 ► pp. 1880 ff.
Beattie, Geoffrey, Kate Webster & Jamie Ross
2010. The Fixation and Processing of the Iconic Gestures That Accompany Talk. Journal of Language and Social Psychology 29:2 ► pp. 194 ff.
Ben Chikha, Houssem, Bachir Zoudji & Aïmen Khacharem
2023. An eye-tracking study of how coach's nonverbal communication affects memorization of basketball tactical scenes. International Journal of Sports Science & Coaching 18:6 ► pp. 1964 ff.
Bergmann, Kirsten, Stefan Kopp & Friederike Eyssel
2010. Individualized Gesturing Outperforms Average Gesturing – Evaluating Gesture Production in Virtual Humans. In Intelligent Virtual Agents [Lecture Notes in Computer Science, 6356], ► pp. 104 ff.
Bressem, Jana & Silva H. Ladewig
2011. Rethinking gesture phases: Articulatory features of gestural movement?. Semiotica 2011:184
Brône, Geert & Bert Oben
2018. Chapter 1. Introduction. In Eye-tracking in Interaction [Advances in Interaction Studies, 10], ► pp. 1 ff.
Brône, Geert, Bert Oben, Annelies Jehoul, Jelena Vranjes & Kurt Feyaerts
2017. Eye gaze and viewpoint in multimodal interaction management. Cognitive Linguistics 28:3 ► pp. 449 ff.
Callimaci, Antonello & Anne Fortin
2023. The impact of teacher video presence on student performance and satisfaction in undergraduate accounting courses. Accounting Education ► pp. 1 ff.
Cappellini, Marco & Yu-Yin Hsu
2022. Multimodality in webconference-based language tutoring: An ecological approach integrating eye tracking. ReCALL 34:3 ► pp. 255 ff.
Carlson, Crystal, Steven A. Jacobs, Michelle Perry & Ruth Breckinridge Church
2014. The effect of gestured instruction on the learning of physical causality problems. Gesture 14:1 ► pp. 26 ff.
Clough, Sharice, Caitlin Hilverman, Sarah Brown-Schmidt & Melissa C. Duff
2022. Evidence of Audience Design in Amnesia: Adaptation in Gesture but Not Speech. Brain Sciences 12:8 ► pp. 1082 ff.
Clough, Sharice, Victoria-Grace Padilla, Sarah Brown-Schmidt & Melissa C. Duff
2023. Intact speech-gesture integration in narrative recall by adults with moderate-severe traumatic brain injury. Neuropsychologia 189 ► pp. 108665 ff.
Cooperrider, Kensy
De Coster, Lize, Pablo Sánchez-Herrero, Carlos Aliaga, Miguel A. Otaduy, Jorge López-Moreno & Ana Tajadura-Jiménez
2020. Perceived match between own and observed models’ bodies: influence of face, viewpoints, and body size. Scientific Reports 10:1
Debreslioska, Sandra, Joost van de Weijer & Marianne Gullberg
2019. Addressees Are Sensitive to the Presence of Gesture When Tracking a Single Referent in Discourse. Frontiers in Psychology 10
Dewaele, Jean-Marc & Phoebe Moxsom-Turnbull
2020. Visual cues and perception of emotional intensity among L1 and LX users of English. International Journal of Multilingualism 17:4 ► pp. 499 ff.
Drijvers, Linda, Ole Jensen & Eelke Spaak
2021. Rapid invisible frequency tagging reveals nonlinear integration of auditory and visual information. Human Brain Mapping 42:4 ► pp. 1138 ff.
Drijvers, Linda, Julija Vaitonytė & Asli Özyürek
2019. Degree of Language Experience Modulates Visual Attention to Visible Speech and Iconic Gestures During Clear and Degraded Speech Comprehension. Cognitive Science 43:10
Emmorey, K., R. Thompson & R. Colvin
2008. Eye Gaze During Comprehension of American Sign Language by Native and Beginning Signers. Journal of Deaf Studies and Deaf Education 14:2 ► pp. 237 ff.
Fedorova, O.V. & I.Y. Zherdev
2019. Follow the hands of the interlocutor! (on strategies for the distribution of visual attention). Experimental Psychology (Russia) 12:1 ► pp. 98 ff.
Findeisen, Stefanie, Sebastian Horn & Jürgen Seifried
2019. Lernen durch Videos – Empirische Befunde zur Gestaltung von Erklärvideos. MedienPädagogik: Zeitschrift für Theorie und Praxis der Medienbildung ► pp. 16 ff.
Gironzetti, Elisa, Salvatore Attardo & Lucy Pickering
2016. Smiling, gaze, and humor in conversation. In Metapragmatics of Humor [IVITRA Research in Linguistics and Literature, 14], ► pp. 235 ff.
Gullberg, Marianne & Sotaro Kita
2009. Attention to Speech-Accompanying Gestures: Eye Movements and Information Uptake. Journal of Nonverbal Behavior 33:4 ► pp. 251 ff.
Gurney, Daniel J., Louise R. Ellis & Emily Vardon-Hynard
2016. The saliency of gestural misinformation in the perception of a violent crime. Psychology, Crime & Law 22:7 ► pp. 651 ff.
Gurney, Daniel J., Karen J. Pine & Richard Wiseman
2013. The Gestural Misinformation Effect: Skewing Eyewitness Testimony Through Gesture. The American Journal of Psychology 126:3 ► pp. 301 ff.
Gurney, Daniel John
2015. What's Left Unsaid: How Nonverbal Influence Compares with Verbal Influence. Psychiatry, Psychology and Law 22:3 ► pp. 465 ff.
Hessels, Roy S.
2020. How does gaze to faces support face-to-face interaction? A review and perspective. Psychonomic Bulletin & Review 27:5 ► pp. 856 ff.
Hessels, Roy S., Gijs A. Holleman, Tim H. W. Cornelissen, Ignace T. C. Hooge & Chantal Kemner
2018. Eye contact takes two – autistic and social anxiety traits predict gaze behavior in dyadic interaction. Journal of Experimental Psychopathology 9:2 ► pp. jep.062917 ff.
Holler, Judith, Heather Shovelton & Geoffrey Beattie
2009. Do Iconic Hand Gestures Really Contribute to the Communication of Semantic Information in a Face-to-Face Context?. Journal of Nonverbal Behavior 33:2 ► pp. 73 ff.
Holler, Judith & Katie Wilkin
2011. An experimental investigation of how addressee feedback affects co-speech gestures accompanying speakers’ responses. Journal of Pragmatics 43:14 ► pp. 3522 ff.
Hostetter, Autumn B. & Martha W. Alibali
2019. Gesture as simulated action: Revisiting the framework. Psychonomic Bulletin & Review 26:3 ► pp. 721 ff.
Huangfu, Qian, Hong Li, Sanshan Tang, Jianrong Wang, Qian Liu & Guojun Chen
2022. How teacher enthusiasm affects students’ learning of chemistry declarative knowledge in video lectures. Chemistry Education Research and Practice 23:4 ► pp. 898 ff.
Kandana Arachchige, Kendra Gimhani, Wivine Blekic, Isabelle Simoes Loureiro & Laurent Lefebvre
2021. Covert Attention to Gestures Is Sufficient for Information Uptake. Frontiers in Psychology 12
King, Josiah P. J., Jia E. Loy, Hannah Rohde, Martin Corley & Susana Martinez-Conde
2020. Interpreting nonverbal cues to deception in real time. PLOS ONE 15:3 ► pp. e0229486 ff.
Kirk, Elizabeth, Daniel Gurney, Rebecca Edwards & Chris Dodimead
2015. Handmade Memories: The Robustness of the Gestural Misinformation Effect in Children’s Eyewitness Interviews. Journal of Nonverbal Behavior 39:3 ► pp. 259 ff.
Kita, Sotaro
2009. Cross-cultural variation of speech-accompanying gesture: A review. Language and Cognitive Processes 24:2 ► pp. 145 ff.
Korving, H., M. Hernández & E. De Groot
2016. Look at me and pay attention! A study on the relation between visibility and attention in weblectures. Computers & Education 94 ► pp. 151 ff.
Koumoutsakis, Theodora, Ruth Breckinridge Church, Martha W. Alibali, Melissa Singer & Saba Ayman-Nolley
2016. Gesture in Instruction: Evidence from Live and Video Lessons. Journal of Nonverbal Behavior 40:4 ► pp. 301 ff.
Laparle, Schuyler
2023. Gradient at-issueness and semiotic complexity in gesture: a response. Theoretical Linguistics 49:3-4 ► pp. 261 ff.
Lausberg, Hedda, Eran Zaidel, Robyn F. Cruz & Alain Ptito
2007. Speech-independent production of communicative gestures: Evidence from patients with complete callosal disconnection. Neuropsychologia 45:13 ► pp. 3092 ff.
Leng, Xiaoxue, Fuxing Wang & Richard E. Mayer
2024. Is student learning from a video lecture affected by whether the instructor wears a mask?. Applied Cognitive Psychology 38:1
Letsch, Florian, Doreen Jirak & Stefan Wermter
2019. Localizing salient body motion in multi-person scenes using convolutional neural networks. Neurocomputing 330 ► pp. 449 ff.
Louwerse, Max M. & Adrian Bangerter
2010. Effects of Ambiguous Gestures and Language on the Time Course of Reference Resolution. Cognitive Science 34:8 ► pp. 1517 ff.
McDonough, Kim, Pavel Trofimovich, Libing Lu & Dato Abashidze
2020. Visual cues during interaction: Are recasts different from noncorrective repetition?. Second Language Research 36:3 ► pp. 359 ff.
Nicoladis, Elena, Trevor Luk & Shireen Gill
Ningelgen, Jana & Peter Auer
2017. Is there a multimodal construction based on non-deicticsoin German?. Linguistics Vanguard 3:s1
Nirme, Jens, Magnus Haake, Agneta Gulz & Marianne Gullberg
2020. Motion capture-based animated characters for the study of speech–gesture integration. Behavior Research Methods 52:3 ► pp. 1339 ff.
Nirme, Jens, Birgitta Sahlén, Viveka Lyberg Åhlander, Jonas Brännström & Magnus Haake
2020. Audio-visual speech comprehension in noise with real and virtual speakers. Speech Communication 116 ► pp. 44 ff.
Oben, Bert
2018. Chapter 10. Gaze as a predictor for lexical and gestural alignment. In Eye-tracking in Interaction [Advances in Interaction Studies, 10], ► pp. 233 ff.
OBEN, BERT & GEERT BRÔNE
2015. What you see is what you do: on the relationship between gaze and gesture in multimodal alignment. Language and Cognition 7:4 ► pp. 546 ff.
Obermeier, Christian, Thomas Dolk & Thomas C. Gunter
2012. The benefit of gestures during communication: Evidence from hearing and hearing-impaired individuals. Cortex 48:7 ► pp. 857 ff.
Ouwehand, Kim, Tamara van Gog & Fred Paas
2015. Effects of Gestures on Older Adults' Learning from Video‐based Models. Applied Cognitive Psychology 29:1 ► pp. 115 ff.
Piper, Anne Marie & James D. Hollan
2009. Analyzing Multimodal Communication around a Shared Tabletop Display. In ECSCW 2009, ► pp. 283 ff.
Preisig, Basil C., Noëmi Eggenberger, Dario Cazzoli, Thomas Nyffeler, Klemens Gutbrod, Jean-Marie Annoni, Jurka R. Meichtry, Tobias Nef & René M. Müri
2018. Multimodal Communication in Aphasia: Perception and Production of Co-speech Gestures During Face-to-Face Conversation. Frontiers in Human Neuroscience 12
Preisig, Basil C., Noëmi Eggenberger, Giuseppe Zito, Tim Vanbellingen, Rahel Schumacher, Simone Hopfner, Thomas Nyffeler, Klemens Gutbrod, Jean-Marie Annoni, Stephan Bohlhalter & René M. Müri
2015. Perception of co-speech gestures in aphasic patients: A visual exploration study during the observation of dyadic conversations. Cortex 64 ► pp. 157 ff.
Risko, Evan F. & Alan Kingstone
2015. Attention in the Wild. In The Cambridge Handbook of Applied Perception Research, ► pp. 466 ff.
Rowbotham, Samantha, Donna M. Lloyd, Judith Holler & Alison Wearden
2015. Externalizing the Private Experience of Pain: A Role for Co-Speech Gestures in Pain Communication?. Health Communication 30:1 ► pp. 70 ff.
Sandgren, Olof, Richard Andersson, Joost van de Weijer, Kristina Hansson & Birgitta Sahlén
2012. Timing of gazes in child dialogues: a time‐course analysis of requests and back channelling in referential communication. International Journal of Language & Communication Disorders 47:4 ► pp. 373 ff.
Smith, Whitney Goodrich & Carla L. Hudson Kam
2012. Knowing ‘who she is’ based on ‘where she is’: The effect of co-speech gesture on pronoun comprehension. Language and Cognition 4:2 ► pp. 75 ff.
Sondermann, Christina, Markus Huff & Martin Merkt
2024. Distracted by a talking head? An eye tracking study on the effects of instructor presence in learning videos with animated graphic slides. Learning and Instruction 91 ► pp. 101878 ff.
Sondermann, Christina & Martin Merkt
2023. Like it or learn from it: Effects of talking heads in educational videos. Computers & Education 193 ► pp. 104675 ff.
Sparrow, Karen, Christopher Lind & Willem van Steenbrugge
2020. Gesture, communication, and adult acquired hearing loss. Journal of Communication Disorders 87 ► pp. 106030 ff.
Stam, Gale & Marion Tellier
Streeck, Jürgen
2014. Mutual gaze and recognition. In From Gesture in Conversation to Visible Action as Utterance, ► pp. 35 ff.
Stukenbrock, Anja
2018. Chapter 11. Mobile dual eye-tracking in face-to-face interaction. In Eye-tracking in Interaction [Advances in Interaction Studies, 10], ► pp. 265 ff.
Sülflow, Michael & Marcus Maurer
2019. The Power of Smiling. How Politicians’ Displays of Happiness Affect Viewers’ Gaze Behavior and Political Judgments. In Visual Political Communication, ► pp. 207 ff.
Tsunemoto, Aki, Rachael Lindberg, Pavel Trofimovich & Kim McDonough
2022. VISUAL CUES AND RATER PERCEPTIONS OF SECOND LANGUAGE COMPREHENSIBILITY, ACCENTEDNESS, AND FLUENCY. Studies in Second Language Acquisition 44:3 ► pp. 659 ff.
Valtakari, Niilo V., Ignace T. C. Hooge, Charlotte Viktorsson, Pär Nyström, Terje Falck-Ytter & Roy S. Hessels
2021. Eye tracking in human interaction: Possibilities and limitations. Behavior Research Methods 53:4 ► pp. 1592 ff.
van Gog, Tamara, Ilse Verveer & Lise Verveer
2014. Learning from video modeling examples: Effects of seeing the human model's face. Computers & Education 72 ► pp. 323 ff.
van Nispen, Karin, Kazuki Sekine, Ineke van der Meulen & Basil C. Preisig
2022. Gesture in the eye of the beholder: An eye-tracking study on factors determining the attention for gestures produced by people with aphasia. Neuropsychologia 174 ► pp. 108315 ff.
van Wermeskerken, Margot, Bianca Grimmius & Tamara van Gog
2018. Attention to the model's face when learning from video modeling examples in adolescents with and without autism spectrum disorder. Journal of Computer Assisted Learning 34:1 ► pp. 32 ff.
van Wermeskerken, Margot, Susanna Ravensbergen & Tamara van Gog
2018. Effects of instructor presence in video modeling examples on attention and learning. Computers in Human Behavior 89 ► pp. 430 ff.
van Wermeskerken, Margot & Tamara van Gog
2017. Seeing the instructor's face and gaze in demonstration video examples affects attention allocation but not learning. Computers & Education 113 ► pp. 98 ff.
Vanbellingen, Tim, Rahel Schumacher, Noëmi Eggenberger, Simone Hopfner, Dario Cazzoli, Basil C. Preisig, Manuel Bertschi, Thomas Nyffeler, Klemens Gutbrod, Claudio L. Bassetti, Stephan Bohlhalter & René M. Müri
2015. Different visual exploration of tool-related gestures in left hemisphere brain damaged patients is associated with poor gestural imitation. Neuropsychologia 71 ► pp. 158 ff.
Vulchanova, Mila, Valentin Vulchanov, Isabella Fritz & Evelyn A. Milburn
2019. Language and perception: Introduction to the Special Issue “Speakers and Listeners in the Visual World”. Journal of Cultural Cognitive Science 3:2 ► pp. 103 ff.
Wakefield, Elizabeth, Miriam A. Novack, Eliza L. Congdon, Steven Franconeri & Susan Goldin‐Meadow
2018. Gesture helps learners learn, but not merely by guiding their visual attention. Developmental Science 21:6
Wakefield, Elizabeth M., Miriam A. Novack & Susan Goldin‐Meadow
2018. Unpacking the Ontogeny of Gesture Understanding: How Movement Becomes Meaningful Across Development. Child Development 89:3
Wang, Emily Q. & Anne Marie Piper
2018. Accessibility in Action. Proceedings of the ACM on Human-Computer Interaction 2:CSCW ► pp. 1 ff.
Xia, Lihua, Dongmei Han, Haijian Chen & Yonghui Dai
2023. The effects of information format on learners’ experience. Innovations in Education and Teaching International 60:2 ► pp. 252 ff.
Yeates, Peter, Alice Moult, Janet Lefroy, Jacqualyn Walsh-House, Lorraine Clews, Robert McKinley & Richard Fuller
2020. Understanding and developing procedures for video-based assessment in medical education. Medical Teacher 42:11 ► pp. 1250 ff.
Yuan, Mingze, Jialing Zeng, Aihua Wang & Junjie Shang
2021. Would It Be Better if Instructors Technically Adjust Their Image or Voice in Online Courses? Impact of the Way of Instructor Presence on Online Learning. Frontiers in Psychology 12
Zhang, Han, Kevin F. Miller, Xin Sun & Kai S. Cortina
2020. Wandering eyes: Eye movements during mind wandering in video lectures. Applied Cognitive Psychology 34:2 ► pp. 449 ff.
Özer, Demet & Tilbe Göksun
2020. Gesture Use and Processing: A Review on Individual Differences in Cognitive Resources. Frontiers in Psychology 11
[no author supplied]
2012. References. In The Handbook of Conversation Analysis, ► pp. 741 ff.
This list is based on CrossRef data as of 8 april 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.