Article published In:
GestureVol. 17:1 (2018) ► pp.65–97
Teasing apart listener-sensitivity
The role of interaction
Using a repetition paradigm, in which speakers describe the same event to a sequence of listeners, we analyze the degree of
reduction in representational gestures. We find that when listener feedback, both verbal and non-verbal, is minimal and unvarying,
speakers steadily reduce their motoric commitment in repeated gestures across tellings without regard to the novelty of the
information to the listener. Within this specific condition, we interpret the result to coincide with the view that gestures
primarily serve as a part of speech production rather than a communicative act. Importantly, we propose that gestural sensitivity
to the listener derives from an interaction between interlocutors, rather than simple modeling of the listener’s state of
knowledge in the mind of the speaker alone.
Article outline
- Introduction
- Objectives
- Theories of gesture production
- For the speaker: The Listener-neutral explanation
- For the listener: Sources of listener-sensitivity
- The Repetition Effect
- The Repetition Effect in speech
- The Repetition Effect in gesture
- Experiment
- Methods
- The speakers
- The stimulus
- Recording
- Procedures
- Experimental conditions and hypotheses
- The confederates
- Coding
- Predictors for gesture size and analysis
- Results
- An example of the results
- Preliminary discussion
- A perceptual judgment follow-up study
- Participants and stimuli
- Procedures
- Predictors for gestural effort and analysis
- Results
- Preliminary discussion
- General discussion
- Clarifying the notion of listener-sensitivity
- The Speaker-neutral explanation: Why the reduction?
- Conclusion and future directions
-
Acknowledgements
- Notes
-
References
References (111)
References
Alibali, Martha W., Lucia M. Flevares, & Susan Goldin-Meadow (1997). Assessing knowledge conveyed in gesture: Do teachers have the upper hand? Journal of Educational Psychology, 89 (1), 183–193.
Alibali, Martha W., Dana C. Heath, & Heather J. Myers (2001). Effects of visibility between speaker and listener on gesture production: Some gestures are meant to be seen. Journal of Memory and Language, 44 (2), 169–188.
Alibali, Martha W., Sotaro Kita, & Amanda J. Young (2000). Gesture and the process of speech production: We think, therefore we gesture. Language and cognitive processes, 15 (6), 593–613.
Allwood, Jens & Loredana Cerrato (2003). A study of gestural feedback expressions. In Proceedings of the First Nordic Symposium on Multimodal Communication (pp. 7–22). Copenhagen: CST, Center for Sprogteknologi.
Anderson, Anne H. & Barbara Howard (2002). Referential form and word duration in video-mediated and face-to-face dialogues. In Johan Bos, Mary Ellen Foster, & Colin Matheson (Eds.), Proceedings of EDILOG 2002. Edinburgh: Cognitive Science Centre, University of Edinburgh.
Ariel, Mira (1990). Accessing noun phrase antecedents. London: Routledge.
Ariel, Mira (1988). Referring and accessibility, Journal of Linguistics, 241, 65–87.
Aylett, Matthew & Alice Turk (2004). The smooth signal redundancy hypothesis: A functional explanation for relationships between redundancy, prosodic prominence, and duration in spontaneous speech. Language and Speech, 471, 31–56.
Aylett, Matthew & Alice Turk (2006). Language redundancy predicts syllabic duration and the spectral characteristics of vocalic syllable nuclei. The Journal of the Acoustical Society of America, 119 (5), 3048–3058.
Baker, Rachel E. & Ann R. Bradlow (2009). Variability in word duration as a function of probability, speech style, and prosody. Language and Speech, 521, 391–413.
Bard, Ellen G., Anne H. Anderson, Catherine Sotillo, Matthew Aylett, Gwyneth Doherty-Sneddon, & Alison Newlands (2000). Controlling the intelligibility of referring expressions in dialogue. Journal of Memory and Language, 421, 1–22.
Bard, Ellen G. & Matthew Aylett (2005). Referential form, duration, and modelling the listener in spoken dialogue. In John C. Trueswell & Michael K. Tanenhaus (Eds.), Approaches to studying world-situated language use: Bridging the language-as-product and language-as-action traditions (pp. 173–191). Cambridge: MIT Press.
Bates, Douglas (2013). Linear mixed model implementation in lme4. University of Wisconsin–Madison.
Bavelas, Janet & Nicole Chovil (2000). Visible acts of meaning an integrated message model of language in face-to-face dialogue. Journal of Language and social Psychology, 19 (2), 163–194.
Bavelas, Janet, Jennifer Gerwing, Chantelle Sutton, & Danielle Prevost (2008). Gesturing on the telephone: Independent effects of dialogue and visibility. Journal of Memory and Language, 58 (2), 495–520.
Beattie, Geoffrey & Rima Aboudan (1994). Gestures, pauses and speech: An experimental investigation of the effects of changing social context on their precise temporal relationships. Semiotica, 99 (3–4), 239–272.
Beattie, Geoffrey & Jane Coughlan (1999). An experimental investigation of the role of iconic gestures in lexical access using the tip‐of‐the‐tongue phenomenon. British Journal of Psychology, 90 (1), 35–56.
Beattie, Geoffrey & Heather Shovelton (2005). Why the spontaneous images created by the hands during talk can help make TV advertisements more effective. British Journal of Psychology, 96 (1), 21–37.
Bell, Alan, Jason M. Brenier, Michelle Gregory, Cynthia Girand, & Dan Jurafsky (2009). Predictability effects on durations of content and function words in conversational English. Journal of Memory and Language, 601, 92–111.
Bybee, Joan (2001). Phonology and language use. Cambridge: Cambridge University Press.
Bybee, Joan (2002). Word frequency and context of use in the lexical diffusion of phonetically conditioned sound change. Language Variation and Change, 141, 261–290.
Chafe, Wallace (1976). Givenness, contrastiveness, definiteness, subjects, topics and point of view. In Charles N. Li (Ed.), Subject and topic (pp.25–55). New York: Academic Press.
Chawla, Purnima & Robert M. Krauss (1994). Gesture and speech in spontaneous and rehearsed narratives. Journal of Experimental Social Psychology, 301, 580–601.
Chovil, Nicole (1991). Discourse-oriented facial displays in conversation. Research on Language and Social Interaction, 251, 163–194.
Clark, Herbert H. & Thomas B. Carlson (1982). Hearers and speech acts. Language, 58 (2), 332–373.
Clark, Herbert H. & Catherine R. Marshall (1981). Definite reference and mutual knowledge. In Aravind K. Joshi, Bonnie Webber, & Ivan A. Sag (Eds.), Elements of discourse understanding (pp. 10–63). Cambridge: Cambridge University Press.
Clark, Herbert H. & Edward F. Schaefer (1989). Contributing to discourse. Cognitive Science, 13 (2), 259–294.
Clark, Herbert H. & Deanna Wilkes-Gibbs (1986). Referring as a collaborative process. Cognition, 22 (1), 1–39.
Cohen, Akiba A. & Randall P. Harrison (1973). Intentionality in the use of hand illustrators in face-to-face communication situations. Journal of Personality and Social Psychology, 28 (2), 276–279.
Du Bois, John W. (1980) Beyond definiteness: The trace of identity in discourse. In Wallace Chafe (Ed.), The pear stories: Cognitive, cultural, and linguistic aspects of narrative production (pp. 203–274). Norwood, NJ: Ablex.
Fisher, Cynthia & Hisayo Tokura (1995). The given-new contract in speech to infants. Journal of Memory and Language, 341, 287–310.
Fowler, Carol A. (1988). Differential shortening of repeated context words produced in various communicative contexts. Language and Speech, 311, 307–319.
Fowler, Carol A. & Jonathan Housum (1987). Talkers’ signaling of “new” and “old” words in speech and listeners’ perception and use of the distinction. Journal of Memory and Language, 26 (5), 489–504.
Fowler, Carol A., Elena T. Levy, & Julie M. Brown (1997). Reductions of spoken words in certain discourse contexts. Journal of Memory and Language, 37 (1), 24–40.
Frank, Austin F. & T. Florian Jaeger (2008). Speaking rationally: Uniform information density as an optimal strategy for language production. In Bradley C. Love, Kenneth M. McRae, & Vladimir M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 939–944). Austin, TX: Cognitive Science Society.
Freeman, Valerie (2014). Hyperarticulation as a signal of stance. Journal of Phonetics, 451, 1–11.
Frick-Horbury, Donna & Robert E. Guttentag (1998). The effects of restricting hand gesture production on lexical retrieval and free recall. The American Journal of Psychology, 111 (1), 43–62.
Galati, Alexia & Susan E. Brennan (2010). Attenuating repeated information: For the speaker, or for the addressee? Journal of Memory and Language, 621, 35–51.
Galati, Alexia & Susan E. Brennan (2013). Speakers adapt gestures to addressees’ knowledge: implications for models of co-speech gesture. Language, Cognition and Neuroscience, 29 (4), 435–451.
Gerwing, Jennifer & Janet Bavelas (2013). The social interactive nature of gestures: Theory, assumptions, methods, and findings. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva S. Ladewig, David McNeill, & Sedinha Teßendorf (Eds). Body – language – communication: An international handbook on multimodality in human interaction (Vol. 11, pp. 821–836). Berlin: Walter de Gruyter.
Goodwin, Charles (1981). Conversational organization: Interaction between speakers and hearers. New York: Academic Press.
Hilliard, Caitlin & Susan W. Cook (2015, June 29). Bridging gaps in common ground: Speakers design their gestures for their listeners. Journal of Experimental Psychology: Learning, Memory, and Cognition. Advance online publication.
Hoetjes, Marieke, Ruud Koolen, Martjin Goudbeek, Emiel Krahmer, & Marc Swerts (2011). GREEBLES Greeble greeb. On reduction in speech and gesture in repeated references. In Proceedings of the 33rd annual meeting of the Cognitive Science Society (CogSci), July 2011, Boston, Massachusetts (pp. 3250–3255). Austin, TX: Cognitive. Science Society.
Hoetjes, Marieke, Ruud Koolen, Martjin Goudbeek, Emiel Krahmer, & Marc Swerts (2015). Reduction in gesture during the production of repeated references. Journal of Memory and Language, 79–801, 1–17.
Hoetjes, Marieke, Emiel Krahmer, & Marc Swerts (2014). On what happens in speech and gesture when communication is unsuccessful. Proceedings of the 36th annual meeting of the Cognitive Science Society (CogSci), July 2014, Québec City, Canada (pp. 2369–2374). Austin, TX: Cognitive. Science Society.
Holler, Judith & Rachel Stevens (2007). The effect of common ground on how speakers use gesture and speech to represent size information. Journal of Language and Social Psychology, 26(1), 4–27.
Holler, Judith & Katie Wilkin (2009). Communicating common ground: how mutually shared knowledge influences the representation of semantic information in speech and gesture in a narrative task. Language and Cognitive Processes, 241, 267–289.
Holler, Judith & Katie Wilkin (2011). An experimental investigation of how addressee feedback affects co-speech gestures accompanying speakers’ responses. Journal of Pragmatics, 43 (14), 3522–3536.
Horton, William S. & Boaz Keysar (1996). When do speakers take into account common ground? Cognition, 59 (1), 91–117.
Hostetter, Autumn B. & Martha W. Alibali (2008). Visible embodiment: Gestures as simulated action. Psychonomic Bulletin and Review, 15 (3), 495–514.
Hostetter, Autumn B. & Martha W. Alibali (2010). Language, gesture, action! A test of the Gesture as Simulated Action framework. Journal of Memory and Language, 63 (2), 245–257.
Isaacs, Ellen A. & Herbert H. Clark (1987). References in conversation between experts and novices. Journal of Experimental Psychology: General, 116 (1), 26–37.
Iverson, Jana M. & Susan Goldin‐Meadow (1997). What’s communication got to do with it? Gesture in children blind from birth. Developmental Psychology, 33 (3), 453–467.
Iverson, Jana M. & Susan Goldin‐Meadow (2001). The resilience of gesture in talk: Gesture in blind speakers and listeners. Developmental Science, 4 (4), 416–422.
Jacobs, Naomi & Alan Garnham (2007). The role of conversational hand gestures in a narrative task. Journal of Memory and Language, 56 (2), 291–303.
Jaeger, Florian T. (2010). Redundancy and reduction: Speakers manage syntactic information density. Cognitive Psychology, 611, 23–62.
Jelec, Anna & Dorota Jaworska (2014). Thoughts on the table: Gesture as a tool for thinking in blind and visually impaired children. Yearbook of the Poznan Linguistic Meeting, 1 (1), 73–88.
Kapatsinski, Vsevolod (2010). Frequency of use leads to automaticity of production: Evidence from repair in conversation. Language & Speech, 53 (1), 71–105.
Kendall, Tyler (2013). Speech rate, pause and sociolinguistic variation: studies in corpus sociophonetics. New York: Palgrave Macmillan.
Kendon, Adam (1994). Do gestures communicate? A review. Research on Language and Social Interaction, 27 (3), 175–200.
Kendon, Adam (2004). Gesture: Visible action as utterance. Cambridge, UK: Cambridge University Press.
Kimbara, Irene (2008). Gesture form convergence in joint description. Journal of Nonverbal Behavior, 32 (2), 123–131.
Kita, Sotaro (1993). Language and thought interface: A study of spontaneous gestures and Japanese mimetics. Unpublished PhD Dissertation. University of Chicago.
Kita, Sotaro (2000). How representational gestures help speaking. In David McNeill (Ed.), Language and gesture: Window into thought and action (pp. 162–185). Cambridge, UK: Cambridge University Press.
Kita, Sotaro & Aslı Özyürek (2003). What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language, 48 (1), 16–32.
Krauss, Robert M. (1998). Why do we gesture when we speak? Current Directions in Psychological Science, 71, 54–60.
Krauss, Robert M., William Apple, Nancy Morency, Charlotte Wenzel, & Ward Winton (1981). Verbal, vocal, and visible factors in judgments of another’s affect. Journal of Personality and Social Psychology, 40 (2), 312–320.
Krauss, Robert M., Yihsiu Chen, & Purnima Chawla (1996). Nonverbal behavior and nonverbal communication: What do conversational hand gestures tell us? Advances in Experimental Social Psychology, 281, 389–450.
Krauss, Robert M., Yihsiu Chen, & Rebecca F. Gottesman (2000). Lexical gestures and lexical access: a process model. In David McNeill (Ed.), Language and gesture: Window into thought and action (pp. 261–283). Cambridge, UK: Cambridge University Press.
Krauss, Robert M., Robert A. Dushay, Yihsiu Chen, & Frances Rauscher (1995). The communicative value of conversational hand gesture. Journal of Experimental Social Psychology, 31 (6), 533–552.
Krauss, Robert M., Palmer Morrel-Samuels, & Christina Colasante (1991). Do conversational hand gestures communicate? Journal of Personality and Social Psychology, 61 (5), 743–754.
Kuhlen, Anna K. & Susan E. Brennan (2013). Language in dialogue: When confederates might be hazardous to your data. Psychonomic Bulletin and Review, 201, 54–72.
Kuhlen, Anna K. & Mandana Seyfeddinipur (2007). From speaker to speaker: Repeated gestures across speakers. Presentation held at Berlin Gesture Center Colloquium, Berlin.
Lam, Tuan Q. & Duane G. Watson (2010). Repetition is easy: Why repeated referents have reduced prominence. Memory and Cognition, 381, 1137–1146.
Levelt, Willem J. (1992). Accessing words in speech production: Stages, processes and representations. Cognition, 42 (1), 1–22.
Lewis, David (1969). Convention: A philosophical study. Cambridge, MA: Harvard University Press.
Lindblom, Björn (1990). Explaining phonetic variation: A sketch of H&H theory. In William J. Hardcastle & Alain Marchal (Eds), Speech production and speech modeling (pp. 403–439). Dordrecht: Kluwer.
Masson-Carro, Ingrid, Martijn Goudbeek, & Emiel Krahmer (2013). The influence of cognitive load on repeated references in speech and gesture. Proceedings of the Tilburg Gesture Research Meeting (TiGeR 2013). June 2013, Tilburg, The Netherlands.
Masson-Carro, Ingrid, Martijn Goudbeek, & Emiel Krahmer (2014). On
the automaticity of reduction in dialogue: Cognitive load and repeated multimodal
references. In Paul Bello et al. (Eds.), Proceedings of the 36th Annual Conference of the
Cognitive Science
Society (pp. 976–981). Austin, TX: Cognitive Science Society.
McNeill, David (1992). Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Press.
McNeill, David (2005). Gesture and thought. Chicago: Chicago University Press.
Mol, Lisette, Emiel Krahmer, Alfons Maes, & Marc Swerts (2009a). Communicative gestures and memory load. In Niels Taatgen & Hedderik van Rijn (Eds.), Proceedings of the 31st Annual Conference of the Cognitive Science Society (pp. 1569–1574). Austin, TX: Cognitive Science Society.
Mol, Lisette, Emiel Krahmer, Alfons Maes, & Marc Swerts (2012). Adaptation in gesture: Converging hands or converging minds? Journal of Memory and Language, 661, 249–264.
Morrel-Samuels, Palmer & Robert M. Krauss (1992). Word familiarity predicts temporal asynchrony of hand gestures and speech. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18 (3), 615–623.
Özyürek, Aslı (2002). Do speakers design their cospeech gestures for their addressees? The effects of addressee location on representational gestures. Journal of Memory and Language, 46 (4), 688–704.
Parrill, Fey (2010). The hands are part of the package: Gesture, common ground, and information packaging. In Sally Rice & John Newman (Eds.), Empirical and experimental methods in cognitive/functional research (pp. 285–302). Stanford: CSLI Publications.
Pederson, Eric (2014). Listener head gestures and the co-construction of narrative timing. Talk given at the meeting of the International Society for Gesture Studies (ISGS 6), San Diego.
Pluymaekers, Mark, Mirjam Ernestus, & R. Harald Baayen (2005). Articulatory planning is continuous and sensitive to informational redundancy. Phonetica, 621, 146–159.
R Development Core Team (2014). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing. Available from: [URL]
Rauscher, Frances H., Robert M. Krauss, & Yihsui Chen (1996). Gesture, speech, and lexical access: The role of lexical movements in speech production. Psychological Science, 7(4), 226–231.
Rimé, Bernard (1982). The elimination of visible behaviour from social interactions: Effects on verbal, nonverbal and interpersonal variables. European journal of Social Psychology, 12 (2), 113–129.
de Ruiter, Jan (1995). Why do people gesture at the telephone? In Proceedings of the CLS opening Academic Year 1995–1996 (pp. 49–56). Nijmegen: Center for Language Studies.
Schelling, Thomas C. (1960). The strategy of conflict. Cambridge; MA: Harvard University Press.
Schneider, Walter, Amy Eschman, & Anthony Zuccolotto (2012). E-Prime user’s guide. Pittsburgh: Psychology Software Tools, Inc.
Singer, Melissa A. & Susan Goldin-Meadow (2005). Children learn when their teacher’s gestures and speech differ. Psychological Science, 16 (2), 85–89.
Sloetjes, Han & Peter Wittenburg (2008, May). Annotation by category: ELAN and ISO DCR. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008).
Stave, Matthew & Eric Pederson (2015). Is the listener really listening? Exploring the effect of verbal and gestural speaker cues on backchanneling. In David C. Noelle, Rick Dale, Anne Warlaumont, Jeff Yoshimi, Teenie Matlock, Carolyn Jennings, & Paul P. Maglio (Eds.), Proceedings of the 37th Annual Meeting of the Cognitive Science Society (pp. 3088–3095). Austin, TX: Cognitive Science Society.
Tannen, Deborah (2007). Talking voices: Repetition, dialogue, and imagery in conversational discourse. New York: Cambridge University Press.
Vajrabhaya, Prakaiwan & Vsevolod Kapatsinski (2011). There is more to the story: First-mention lengthening in Thai interactive discourse. In Proceedings of the International Congress of Phonetic Sciences (Vol. 171, pp. 2050–2053).
Vajrabhaya, Prakaiwan & Vsevolod Kapatsinski (2013). Modeling the listener? What resets acoustic durations of repeated English words. The Journal of the Acoustical Society of America, 134 (5), 4203–4203.
Vajrabhaya, Prakaiwan & Eric Pederson (2012). Have I told you this before? Reduction in magnitude over repetition in co-speech gestures. Poster presented at the meeting of the International Society for Gesture Studies (ISGS 5), Lund, Sweden.
Vajrabhaya, Prakaiwan & Eric Pederson (2013). Repetition vs. listener accommodation: A case study of co-speech gesture in retellings. Poster presented at the meeting of New Ways of Analyzing Variation, Pittsburgh, PA.
Vajrabhaya, Prakaiwan & Eric Pederson (2015). “The baking stick thing”: Automatization of co-speech gesture during lexical access. In David C. Noelle, Rick Dale, Anne Warlaumont, Jeff Yoshimi, Teenie Matlock, Carolyn Jennings, & Paul P. Maglio (Eds.), Proceedings of the 37th Annual Meeting of the Cognitive Science Society (p. 3101–3107). Austin, TX: Cognitive Science Society.
Wagner, Petra, Zofia Malisz, & Stefan Kopp (2014). Gesture and speech in interaction: an overview. Speech Communication, 571, 209–232.
Cited by (2)
Cited by two other publications
Cuffari, Elena Clare
2024.
Gesture and Intersubjectivity. In
The Cambridge Handbook of Gesture Studies,
► pp. 599 ff.
Holler, Judith, Janet Bavelas, Jonathan Woods, Mareike Geiger & Lauren Simons
2022.
Given-New Effects on the Duration of Gestures and of Words in Face-to-Face Dialogue.
Discourse Processes 59:8
► pp. 619 ff.
This list is based on CrossRef data as of 7 september 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.