Cover not available
Part of
Why Gesture?: How the hands function in speaking, thinking and communicating
Edited by Ruth Breckinridge Church, Martha W. Alibali and Spencer D. Kelly
[Gesture Studies 7] 2017
► pp. 267–284

Chapter 12
Computational gesture research

Studying the functions of gesture in human-agent interaction

 | Bielefeld University
Embodied artificial agents, like humanoid robots or virtual characters, can produce a variety of co-speech gestures in interactive settings. This enables a computational branch of gesture research that offers opportunities for (1) investigating production and comprehension processes through computational cognitive modeling, and (2) studying systematically the effects and functions of gesturing in human-agent interaction. In this chapter we review current approaches to synthesize gestures, and we discuss in detail findings from experiments on the effects of gestures produced by virtual characters or robots. Current evidence suggests that synthetic gesturing has considerable effects on how an artificial agent is perceived and attributed with human-like properties. In contrast, so far, there is little evidence showing that contemporary synthetic gesturing helps human addressees to better understand or retain the information communicated by an agent. We discuss what these findings imply for the potential functions that gesture may serve in improving human-agent interaction, and more generally for our understanding of gesture through computational modeling efforts.
Article outline
  • Introduction
  • Computational approaches to synthetic gesture
    • Realizing gestural movements
    • Determining content and form of co-speech gestures
  • The effects of gesture in human-agent interaction studies
    • Task performance effects
    • Communicative effects
    • Social perception effects
  • Conclusions
  • References
Published online: 26 April 2017
DOI logo
https://doi.org/10.1075/gs.7.13kop
References (52)
References
Anderson, John R., Bothell, Daniel, Byrne, Michael D., Douglass, Scott, Lebiere, Christian, & Qin, Yulin 2004. “An integrated theory of the mind.” Psychological Review 111 (4): 1036–1060. DOI logoGoogle Scholar
Bavelas, Janet B., Chovil, Nicole, Coates, Linda, & Roe, Lori 1995. “Gestures specialized for dialogue.” Personality and Social Psychology Bulletin 21: 394–405. DOI logoGoogle Scholar
Bennewitz, Maren, Faber, Felix, Joho, Dominik, & Behnke, Sven 2007. “Fritz – a humanoid communication robot.” In Proceedings of IEEE Int. Symposium on Robot and Human Interactive Communication (RO-MAN 2007), 1072–1077. DOI logoGoogle Scholar
Bergmann, Kirsten 2012. The Production of Co-Speech Iconic Gestures: Empirical Study and Computational Simulation with Virtual Agents. PhD. Thesis, University of Bielefeld.Google Scholar
Bergmann, Kirsten, & Kopp, Stefan 2006. “Verbal or visual? How information is distributed across speech and gesture in spatial dialog).” In Proceedings of the 10th Workshop on the Semantics and Pragmatics of Dialogue, 90–97.Google Scholar
2010. “Modeling the production of coverbal iconic gestures by learning Nayesian decision networks.” Applied Artificial Intelligence 24 (6): 530–551. DOI logoGoogle Scholar
2009. “Increasing expressiveness of virtual agents – autonomous generation of speech and gesture for spatial description tasks.” In Proceedings of the 8th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2009), K. Decker, J. Sichman, G. Sierra, & C. Castelfranchi (eds), 361–368.Google Scholar
Bergmann, Kirsten, Eyssel, Friederike, & Kopp, Stefan 2012. “A second chance to make a first impression? How appearance and nonverbal behavior affect perceived warmth and competence of virtual agents over time.” In Proceedings of Intelligent Virtual Agents (IVA 2012), 126–138. Berlin, Heidelberg: Springer. DOI logoGoogle Scholar
Bergmann, Kirsten, Kopp, Stefan, & Eyssel, Friederike 2010. “Individualized gesturing outperforms average gesturing – evaluating gesture production in virtual humans.” In Proceedings of Intelligent Virtual Agents (IVA 2010), 104–117. Berlin, Heidelberg: Springer. DOI logoGoogle Scholar
Bickmore, Timothy, & Cassell, Justine 2005. “Social dialogue with embodied conversational agents.” In Advances in Natural Multimodal Dialogue Systems, J. C. J. van Kuppevelt, L. Dybkjær, & N. O. Bernsen, (eds) 23–54. New York: Springer. DOI logoGoogle Scholar
Billard, Aude, Calinon, Sylvain, Dillmann, Rüdiger, & Schaal, Stefan 2008. “Robot programming by demonstration.” In Springer Handbook of Robotics, B. Siciliano & O. Khatib (eds) 1371–1394. Berlin, Heidelberg: Springer. DOI logoGoogle Scholar
Blascovich, Jim, Loomis, Jack, Beall, Aandrew C., Swinth, Kimberly R., Hoyt, Crystal L., & Bailenson, Jeremy N. 2002. “Immersive virtual environment technology as a methodological tool for social psychology.” Psychological Inquiry 13 (2): 103–124. DOI logoGoogle Scholar
Bohus, Dan, & Horvitz, Eric 2010. “Facilitating multiparty dialog with gaze, gesture, and speech.” In Proceedings of ICMI-MLMI'10, 1–8. Beijing, China. DOI logoGoogle Scholar
Bremner, Paul, Pipe, Anthony, Melhuish, Chris, Fraser, Mike, & Subramanian, Sriram 2009. “Conversational gestures in human-robot interaction.” In Proceedings of the 2009 IEEE International Conference on Systems, Man and Cybernetics, 1645–1649. DOI logoGoogle Scholar
Breslow, Leonard A., Harrison, Anthony M., & Trafton, J. Gregory. 2010. “Linguistic spatial gestures.” In Proceedings of the 10th International Conference on Cognitive Modeling (ICCM), D. D. Salvucci & G. Gunzelmann (eds), 13–18.Google Scholar
Buisine, Stéphanie, & Martin, Jean-Claude 2007. “The effects of speech – gesture cooperation in animated agents’ behavior in multimedia presentations.” Interacting with Computers 19 (4): 484–493. DOI logoGoogle Scholar
Cassell, Justine, & Thorisson, Kristinn R. 1999. “The power of a nod and a glance: envelope vs. emotional feedback in animated conversational agents.” Applied Artificial Intelligence: 1–20.Google Scholar
Cassell, Justine, Stone, Matthew, & Yan, Hao 2000. “Coordination and context-dependence in the generation of embodied conversation.” In Proceedings of Int. Conference on Natural Language Generation (INLG 2000), 1–8.Google Scholar
Chiu, Chung-Cheng, & Marsella, Stacy 2011. “How to train your avatar: A data driven approach to gesture generation.” In Proceedings of (IVA 2011), 127–140. Berlin, Heidelberg: Springer.Google Scholar
Church, Ruth B., Garber, Philip, & Rogalski, Katheryn 2007. “The role of gesture in memory and social communication.” Gesture 7 (2): 137–158. DOI logoGoogle Scholar
Cuddy, Amy J. C., Glick, Peter, & Beninger, Anna 2011. “The dynamics of warmth and competence judgments, and their outcomes in organizations.” Research in Organizational Behavior 31: 73–98. DOI logoGoogle Scholar
Dehn, Doris M., & Van Mulken, Susanne 2000. “The impact of animated interface agents: a review of empirical research.” International Journal of Human-Computer Studies 52 (1): 1–22. DOI logoGoogle Scholar
DeSteno, David, Breazeal, Cynthia, Frank, Robert H., Pizarro, David, Baumann, Jolie, Dickens, Leah, & Lee, Jin Joo 2012. “Detecting trustworthiness of novel partners in economic exchange.” Psychological Science 23 (12): 1549–1556. DOI logoGoogle Scholar
Gielniak, Michael J., & Thomaz, Andrea L. 2012. “Enhancing interaction through exaggerated motion synthesis.” In Proceedings of the 7th Annual ACM/IEEE International Conference on Human-Robot Interaction (HRI), 375–382. New York: ACM Press.Google Scholar
Hartmann, Björn, Mancini, Maurizio, & Pelachaud, Catherine 2006. “Implementing expressive gesture synthesis for embodied conversational agents.” In Gesture in Human-Computer Interaction and Simulation, S. Gibet, N. Courty, & J.-F. Kamp (eds), 188–199, Springer. DOI logoGoogle Scholar
Hostetter, Autumn B., & Alibali, Martha W. 2007. “Raise your hand if you’re spatial.” Gesture 7 (1): 73–95. DOI logoGoogle Scholar
2008. “Visible embodiment: Gestures as simulated action.” Psychonomic Bulletin & Review 15 (3): 495–514. DOI logoGoogle Scholar
2011. “Cognitive skills and gesture-speech redundancy.” Gesture 11 (1): 40–60. DOI logoGoogle Scholar
Kendon, Adam 2004. Gesture: Visible Action as Utterance. Cambridge University Press. DOI logoGoogle Scholar
Kita, Sotaro, & Davies, Thomas S. 2009. “Competing conceptual representations trigger co-speech representational gestures.” Language and Cognitive Processes 24 (5): 761–775. DOI logoGoogle Scholar
Kita, Sotaro, & Özyürek, Asli 2003. “What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking.” Journal of Memory and Language 48: 16–32. DOI logoGoogle Scholar
Kopp, Stefan 2014. “Gestures, postures, gaze and movements in computer science: embodied agents.” In Body-Language-Communication: An International Handbook on Multimodality in Human Interaction, C. Müller, A. Cienki, E. Fricke, S. Ladewig, D. McNeill, & J. Bressem (eds), 1948–1954. Mouton De Gruyter.Google Scholar
Kopp, Stefan, & Wachsmuth, Ipke 2004. “Synthesizing multimodal utterances for conversational agents.” Computer Animation and Virtual Worlds 15 (1): 39–52. DOI logoGoogle Scholar
Kopp, Stefan, Bergmann, Kirsten, & Kahl, Sebastian 2013. “A spreading-activation model of the semantic coordination of speech and gesture.” In Proceedings of the 35th Annual Meeting of the Cognitive Science Society (CogSci 2013), 823–828.Google Scholar
Kopp, Stefan 2017. “Using cognitive models to understand multimodal processes the case for speech and gesture production.” In Handbook of Multimodal-Multisensor Interfaces, S. Oviatt, B. Schuller, P. Cohen & A. Krueger (eds). ACM Books, Morgan Claypool.DOI logoGoogle Scholar
Krämer, Nicole C., Tietz, Bernd, & Bente, Gary 2003. Effects of embodied interface agents and their gestural activity (pp. 292–300). Proceedings of Intelligent Virtual Agents (IVA 2003), Berlin, Heidelberg: Springer.Google Scholar
Lee, Jina, & Marsella, Stacy 2006. “Nonverbal behavior generator for embodied conversational agents.” In Proceedings of Intelligent Virtual Agents (IVA 2006), 243–255. Berlin, Heidelberg: Springer. DOI logoGoogle Scholar
Macedonia, Manuela, Bergmann, Kirsten, & Roithmayr, Friederich 2014. “Imitation of a pedagogical agent’s gestures enhances memory for words in second language.” Science Journal of Education 2 (5): 162–169. DOI logoGoogle Scholar
Macedonia, Manuela, Müller, Karsten, & Friederici, Angela D. 2010. “The impact of iconic gestures on foreign language word learning and its neural substrate.” Human Brain Mapping 32 (6): 982–998. DOI logoGoogle Scholar
Mayer, Richard E., & DaPra, C.Scott 2012. “An embodiment effect in computer-based learning with animated pedagogical agents.” Journal of Experimental Psychology: Applied 18 (3): 239–252.Google Scholar
Moshkina, Lilia, Trickett, Susan, & Trafton, J. Gregory 2014. “Social engagement in public places.” In Proceedings of ACM/IEEE International Conference on Human-Robot Interaction (HRI), 382–389. New York: ACM Press.Google Scholar
Neff, Michael, Kipp, Michael, Albrecht, Irene, & Seidel, Hans-Peter 2008. “Gesture modeling and animation based on a probabilistic re-creation of speaker style.” ACM Transactions on Graphics 27 (1): 1–24. DOI logoGoogle Scholar
Ng-Thow-Hing, Victor, Pengcheng Luo, & Okita, S. 2010. “Synchronized gesture and speech production for humanoid robots.” In Proceedings of the 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, 4617–4624. DOI logoGoogle Scholar
Niewiadomski, Radoslaw, Demeure, Virginie, & Pelachaud, Catherine 2010. “Warmth, competence, believability and virtual agents.” In Proceedings of Intelligent Virtual Agents (IVA 2010), 272–285. Berlin, Heidelberg: Springer. DOI logoGoogle Scholar
Okuno, Yusuke, Kanda, Takayuki, Imai, Michita, Ishiguro, Hiroshi, & Hagita, Norihiro 2009. “Providing route directions.” In Proceedings of the 4th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 53–60. New York: ACM Press. DOI logoGoogle Scholar
Rehm, Matthias, & André, Elisabeth 2007. “More than just a friendly phrase: Multimodal aspects of polite behavior in agents.” In Conversational Informatics, 69–84. Chichester, UK: John Wiley & Sons. DOI logoGoogle Scholar
Ruttkay, Zsofia 2007. “Presenting in style by virtual humans.” In Proceedings of the COST Action International Conference on Verbal and Nonverbal Communication Behaviours, 23–36. DOI logoGoogle Scholar
Salem, Maha, Eyssel, Friederike, Rohlfing, Katharina, Kopp, Stefan, & Joublin, Frank 2013. “To err is human(-like): Effects of robot gesture on perceived anthropomorphism and likability.” International Journal of Social Robotics 5 (3): 313–323. DOI logoGoogle Scholar
Salem, Maha, Kopp, Stefan, Wachsmuth, Ipke, Rohlfing, Katharina, & Joublin, Frank 2012. “Generation and evaluation of communicative robot gesture.” International Journal of Social Robotics 4 (2): 201–217. DOI logoGoogle Scholar
Sidner, Candace L. & Lee, Christopher 2003. “Engagement rules for human-robot collaborative interactions.” In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics (ICSMC), 3957–3962.Google Scholar
Stone, Matthew, DeCarlo, Doug, Oh, Insuk, Rodriguez, Christian, Stere, Adrian, Lees, Alyssa, & Bregler, Chris 2004. “Speaking with hands: Creating animated conversational characters from recordings of human performance.” ACM Transactions on Graphics 23 (3): 506–513. DOI logoGoogle Scholar
Winograd, Terry 1973. “A Process model of language understanding.” In Computer Models of Thought and Language, R. C. Schank & K. M. Colby (eds), 152–186. San Francisco.Google Scholar
Cited by (7)

Cited by seven other publications

Bressem, Jana
2024. Systems of Gesture Coding and Annotation. In The Cambridge Handbook of Gesture Studies, ► pp. 158 ff. DOI logo
Mittelberg, Irene & Jennifer Hinnell
2024. Iconicity, Schematicity, and Representation in Gesture. In The Cambridge Handbook of Gesture Studies, ► pp. 56 ff. DOI logo
Robrecht, Amelie Sophie, Hendric Voss, Lisa Gottschalk & Stefan Kopp
2024. Proceedings of the ACM International Conference on Intelligent Virtual Agents, ► pp. 1 ff. DOI logo
Kelly, Spencer D. & Quang‐Anh Ngo Tran
2023. Exploring the Emotional Functions of Co‐Speech Hand Gesture in Language and Communication. Topics in Cognitive Science DOI logo
Wang, Xi & Fangfei Lv
2022. Review of Galhano-Rodrigues, Galvão & Cruz-Santos (2019): Recent perspectives on gesture and multimodality. Gesture 21:2-3 ► pp. 365 ff. DOI logo
Nirme, Jens, Magnus Haake, Agneta Gulz & Marianne Gullberg
2020. Motion capture-based animated characters for the study of speech–gesture integration. Behavior Research Methods 52:3 ► pp. 1339 ff. DOI logo
Romero Andonegi, Asier, Irati de Pablo Delgado, Aintzane Etxebarria Lejarreta & Ainara Romero Andonegi
2019. Dynamic processes of intermodal coordination in the ontogenesis of language. Gesture 18:1 ► pp. 57 ff. DOI logo

This list is based on CrossRef data as of 5 january 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.