Cover not available
Article published In:
Interaction and Iconicity in the Evolution of Language
Edited by Stefan Hartmann, Michael Pleyer, James Winters and Jordan Zlatev
[Interaction Studies 18:3] 2017
► pp. 465–488

Multimodal-first or pantomime-first?

Communicating events through pantomime with and without vocalization

| Lund University |
| Nicolaus Copernicus University | Nicolaus Copernicus University |
| Nicolaus Copernicus University | Nicolaus Copernicus University |
| Lund University | Lund University |

A persistent controversy in language evolution research has been whether language emerged in the gestural-visual or in the vocal-auditory modality. A “dialectic” solution to this age-old debate has now been gaining ground: language was fully multimodal from the start and remains so to this day. In this paper, we show this solution to be too simplistic and outline a more specific theoretical proposal, which we designate as pantomime-first. To decide between the multimodal-first and pantomime-first alternatives, we review several lines of interdisciplinary evidence and complement it with a cognitive-semiotic experiment. In the study, the participants saw – and then matched to hand-drawn images – recordings of short transitive events enacted by 4 actors in two conditions: visual (only body movement) and multimodal (body movement accompanied by nonlinguistic vocalization). Significantly, the matching accuracy was greater in the visual than the multimodal condition, though a follow-up experiment revealed that the emotional profiles of the events enacted in the multimodal condition could be reliably detected from the sound alone. We see these results as supporting the proposed pantomime-first scenario.

Keywords: bodily mimesis, pantomime, multimodal origins, multimodality, language origins, language evolution, gesture
Article outline
  • 1.Introduction
  • 2.A “multimodal” origin of language?
    • 2.1Definitional issues: What is linguistic multimodality?
    • 2.2Theoretical issues: What are the alternatives?
    • 2.3Empirical issues: How do we decide?
  • 3.Pantomiming events with and without vocalization: An experimental study
    • 3.1Rationale and hypotheses
    • 3.2Materials
    • 3.3Participants and procedure
    • 3.4Results
    • 3.5Discussion
  • 4.Conclusions
  • Acknowledgements
  • Notes
  • References
Published online: 8 December 2017
DOI logo
https://doi.org/10.1075/is.18.3.08zla
References
Aboitiz, F.
(2012) Gestures, vocalizations, and memory in language origins. Frontiers in Evolutionary Neuroscience, 4(2), 1–15. DOI logoGoogle Scholar
Andrén, M.
(2010) Children's gestures between 18 and 30 months. Lund: Media Tryck.Google Scholar
Arbib, M.
(2005) From monkey-like action recognition to human language: An evolutionary framework for neurolinguistics. Behavioral and brain sciences, 281, 105–168. DOI logoGoogle Scholar
(2012) How the brain got language. Oxford: Oxford University Press. DOI logoGoogle Scholar
Armstrong, D. F., Stokoe, W. C., & Wilcox, S. E.
(1995) Gesture and the nature of language. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Armstrong, D. F., & Wilcox, S. E.
(2007) The gestural origin of language. Oxford: Oxford University Press. DOI logoGoogle Scholar
Brown, J. E.
(2012) The evolution of symbolic communication: An embodied perspective. PhD Thesis, University of Edinburgh.
Burling, R.
(2005) The talking ape: How language evolved. Oxford: Oxford University Press.Google Scholar
Call, J., & Tomasello, M.
(2007) The gestural communication of apes and monkeys. London: Lawrence Erlbaum.Google Scholar
Chandler, D.
(2007) Semiotics: The basics (2nd ed.). London: Routledge. DOI logoGoogle Scholar
Clay, Z., & Zuberbühler, K.
(2014) Vocal communication and social awareness in chimpanzees and bonobos: Insights from studies of vocal communication. In D. Dor, C. Knight, & J. Lewis (Eds.), The social origins of language (pp. 141–156). Oxford: Oxford University Press. DOI logoGoogle Scholar
Collins, C.
(2013) Paleopoetics: The evolution of the literary imagination. New York: Columbia University Press.Google Scholar
Corballis, M. C.
(2003) From mouth to hand: Gesture, speech, and the evolution of right-handedness. Behavioral and Brain Sciences, 26(2), 199–208. DOI logoGoogle Scholar
Donald, M.
(1991) Origins of the modern mind: Three stages in the evolution of human culture. Cambridge, MA: Harvard University Press.Google Scholar
Dor, D.
(2015) The instruction of imagination: Language as a social communication technology. Oxford: Oxford University Press. DOI logoGoogle Scholar
Dunbar, R.
(1996) Grooming, gossip and the evolution of language. London: Faber & Faber.Google Scholar
Fay, N., Arbib, M., & Garrod, D.
(2013) How to bootstrapp a human communication system. Cognitive Science, 371, 1356–1367. DOI logoGoogle Scholar
Fay, N., Lister, C. J., Ellison, T. M., & Goldin-Meadow, S.
(2014) Creating a communication system from scratch: gesture beats vocalization hands down. Frontiers in Psychology, 5, 354. DOI logoGoogle Scholar
Fitch, W. T.
(2010) The evolution of language. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Galantucci, B., & Garrod, S.
(2010) Experimental semiotics: A new approach for studying the emergence and the evolution of human communication. Interaction Studies, 111, 1–13. DOI logoGoogle Scholar
(2011) Experimental semiotics: A review. Frontiers in Human Neuroscience, 5(11). DOI logoGoogle Scholar
Gardner, R. A., & Gardner, B. T.
(1969) Teaching sign language to a chimpanzee. Science 1651, 664–672. DOI logoGoogle Scholar
Hewes, G. W.
(1973) Primate communication and the gestural origin of language. Current Anthropology, 14(1/2), 5–24. DOI logoGoogle Scholar
Hobaiter, C. & Byrne, R. W.
(2012) Gesture use in consortship: Wild chimpanzees’ use of gesture for an ‘evolutionarily urgent’ purpose. In S. Pika & K. Liebal (Eds.), Developments in primate gesture research (pp. 129–146). Amsterdam: John Benjamins. DOI logoGoogle Scholar
Johansson, S.
(2005) Origins of language: Constraints on hypotheses. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Kemmerer, D., Chandrasekaran, B., & Tranel, D.
(2007) A case of impaired verbalization but preserved gesticulation of motion events. Cognitive neuropsychology, 24(1), 70–114. DOI logoGoogle Scholar
Kendon, A.
(2004) Gesture: Visible action as utterance. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
(2012) Language and kinesic complexity. Gesture, 121, 308–326. DOI logoGoogle Scholar
(2014a) Semiotic diversity in utterance production and the concept of “language”. Philosophical Transactions of the Royal Society B, 369(1651), 20130293. DOI logoGoogle Scholar
(2014b) The 'poly-modalic' nature of utterances and its implication. In D. Dor, C. Knight, & D. Lewis (Eds.), The social origins of language (pp. 67–76). Oxford: OUP.Google Scholar
Kita, S., & Özyurek, A.
(2003) What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language, 481, 16–32. DOI logoGoogle Scholar
Landau, B., & Gleitman, L. R.
(2009) Language and experience: Evidence from the blind child. Cambridge, MA: Harvard University Press.Google Scholar
Levinson, S. C., & Holler, J.
(2014) The origin of human multi-modal communication. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences, 3691, 201303302. DOI logoGoogle Scholar
Lock, A., & Zukow-Goldring, P.
(2010) Preverbal communication. In J. Bremner & T. Wachs (Eds.), The Wiley-Blackwell handbook of infant development (pp. 394–425). Oxford: Willey-Blackwell. DOI logoGoogle Scholar
MacNeilage, P. F.
(2008) The origin of speech. Oxford: Oxford University Press.Google Scholar
McNeill, D.
(1985) So you think gestures are nonverbal? Psychological Review, 921, 350–371. DOI logoGoogle Scholar
(2005) Gesture and thought. Chicago: University of Chicago Press. DOI logoGoogle Scholar
(2012) How language began: Gesture and speech in human evolution. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Meir, I., Aronoff, M., Börstell, C., Hwang, S., Ilkbasaran, D., Kastner, I., Lepic, R., Ben Basat, A. L., Padden, C., & Sandler, W.
(2017) The effect of being human and the basis of grammatical word order: Insights from novel communication systems and young sign languages. Cognition, 1581, 189–207. DOI logoGoogle Scholar
Mithen, S.
(2005) The singing Neanderthals: The origins of music, language, mind and body. London: Weidenfeld and Nicholson.Google Scholar
Murillo, E., & Belinchón, M.
(2012) Gestural-vocal coordination. Gesture, 121, 16–39. DOI logoGoogle Scholar
Paus, T.
(2001) Primate anterior cingulate cortex: Where motor control, drive and cognition interface. Nature Reviews Neuroscience, 2(6), 417–424. DOI logoGoogle Scholar
Pika, S.
(2008) What is the nature of gestural communication in great apes? The shared mind: Perspectives on intersubjectivity (pp. 165–186). Amsterdam: John Benjamins. DOI logoGoogle Scholar
Schel, A. M., Townsend, S. W., Machanda, Z., Zuberbühler, K., & Slocombe, K. E.
(2013) Chimpanzee alarm call production meets key criteria for intentionality. PloS One, 8(10), e76674. DOI logoGoogle Scholar
See, A.
(2014) Reevaluating chimpanzee vocal signals: Toward a multimodal account of the origins of human communication. In M. Pina & N. Gontier (Eds.), The evolution of social communication in primates (pp. 195–215). New York: Springer. DOI logoGoogle Scholar
Sonesson, G.
(1997) The ecological foundations of iconicity. Semiotics around the world: Synthesis in diversity (pp. 739–742). Berlin: Mouton de Gruyter.Google Scholar
Stokoe, W. C.
(1991) Semantic phonology. Sign Language Studies, 711, 107–114. DOI logoGoogle Scholar
Taglialatela, J. P., Russell, J. L., Schaeffer, J. A., & Hopkins, W. D.
(2011) Chimpanzee vocal signaling points to a multimodal origin of human language. PloS One, 6(4), e18852. DOI logoGoogle Scholar
Tomasello, M.
(2008) The origins of human communication. Cambridge, MA: MIT Press. DOI logoGoogle Scholar
Vigliocco, G., Perniss, P., & Vinson, D.
(2014) Language as a multimodal phenomenon: implications for language learning, processing and evolution. Philosophical Transactions of the Royal Society B, 369(1651), 20130292. DOI logoGoogle Scholar
Volterra, V., Caselli, M., Caprici, O., & Pizzuto, E.
(2005) Gesture and the emergence and development of language. In M. Tomasello & D. Slobin (Eds.), Beyond nature-nurture: Essays in honor of Elisabeth Bates (pp. 3–40). Mahwah, NJ: Lawrence Erlbaum.Google Scholar
Wacewicz, S., Zywiczynski, P., & Orzechowski, S.
(2016) Visible movements of the orofacial area. Evidence for gestural or multimodal theories of language evolution? Gesture 15(2), 251–284 DOI logoGoogle Scholar
Wittgenstein, L.
(1953) Philosophical investigations. Oxford: Blackwell.Google Scholar
Zlatev, J.
(1997) Situated embodiment: Studies in the emergence of spatial meaning. Stockholm: Gotab.Google Scholar
(2008) From proto-mimesis to language: Evidence from primatology and social neuroscience. Journal of Physiology – Paris, 1021, 137–152. DOI logoGoogle Scholar
(2012) Cognitive semiotics: An emerging field for the transdisciplinary study of meaning. Public Journal of Semiotics, 41, 2–24. DOI logoGoogle Scholar
(2014) Bodily mimesis and the transition to speech. In M. Pina & N. Gontier (Eds.), The evolution of social communication in primates (pp. 165–178). New York: Springer. DOI logoGoogle Scholar
Zywiczynski, P., & Wacewicz, S.
(2015) Ewolucja języka. W stronę hipotez gesturalnych. Toruń: Wydawnictwo Naukowe Uniwersytetu Mikołaja Kopernika.Google Scholar
Zywiczynski, P., Wacewicz, S., & Sibierska, M.
(2016) Defining pantomime for language evolution research Topoi. An International Review of Philosophy [in press]. DOI logoGoogle Scholar
Cited by

Cited by 20 other publications

Adornetti, Ines, Francesco Ferretti, Alessandra Chiera, Slawomir Wacewicz, Przemysław Żywiczyński, Valentina Deriu, Andrea Marini, Rita Magni, Laura Casula, Stefano Vicari & Giovanni Valeri
2019. Do Children With Autism Spectrum Disorders Understand Pantomimic Events?. Frontiers in Psychology 10 DOI logo
Brown, Steven
2024. Chapter 6. The pantomimic origins of the narrative arts. In Perspectives on Pantomime [Advances in Interaction Studies, 12], ► pp. 139 ff. DOI logo
Brown, Steven, Emma Mittermaier, Tanishka Kher & Paul Arnold
2019. How Pantomime Works: Implications for Theories of Language Origin. Frontiers in Communication 4 DOI logo
Delliponti, Angelo, Renato Raia, Giulia Sanguedolce, Adam Gutowski, Michael Pleyer, Marta Sibierska, Marek Placiński, Przemysław Żywiczyński & Sławomir Wacewicz
2023. Experimental Semiotics: A Systematic Categorization of Experimental Studies on the Bootstrapping of Communication Systems. Biosemiotics 16:2 ► pp. 291 ff. DOI logo
Ferretti, Francesco & Ines Adornetti
2021. Persuasive conversation as a new form of communication inHomo sapiens. Philosophical Transactions of the Royal Society B: Biological Sciences 376:1824 DOI logo
Holler, Judith
2022. Visual bodily signals as core devices for coordinating minds in interaction. Philosophical Transactions of the Royal Society B: Biological Sciences 377:1859 DOI logo
Macuch Silva, Vinicius, Judith Holler, Asli Ozyurek & Seán G. Roberts
2020. Multimodality and the origin of a novel communication system in face-to-face interaction. Royal Society Open Science 7:1 ► pp. 182056 ff. DOI logo
Markov, Ilia, Kseniia Kharitonova & Elena L. Grigorenko
2023. Language: Its Origin and Ongoing Evolution. Journal of Intelligence 11:4 ► pp. 61 ff. DOI logo
Olteanu, Alin
2021. Multimodal Modeling: Bridging Biosemiotics and Social Semiotics. Biosemiotics 14:3 ► pp. 783 ff. DOI logo
Placiński, Marek & Przemysław Żywiczyński
2023. Modality effect in interactive alignment: Differences between spoken and text-based conversation. Lingua 293 ► pp. 103592 ff. DOI logo
Placiński, Marek, Przemysław Żywiczyński, Theresa Matzinger, Marta Sibierska, Monika Boruta-Żywiczyńska, Anna Szala & Sławomir Wacewicz
2023. Evolution of Pantomime in Dyadic Interaction. A Motion Capture Study. Journal of Language Evolution DOI logo
Rasenberg, Marlou, Asli Özyürek, Sara Bögels & Mark Dingemanse
2022. The Primacy of Multimodal Alignment in Converging on Shared Symbols for Novel Referents. Discourse Processes 59:3 ► pp. 209 ff. DOI logo
Sibierska, Marta, Monika Boruta-Żywiczyńska, Przemysław Żywiczyński & Sławomir Wacewicz
2022. What’s in a mime?. Interaction Studies. Social Behaviour and Communication in Biological and Artificial Systems 23:2 ► pp. 289 ff. DOI logo
Sibierska, Marta, Przemysław Żywiczyński, Jordan Zlatev, Joost van de Weijer & Monika Boruta-Żywiczyńska
2023. Constraints on communicating the order of events in stories through pantomime. Journal of Language Evolution 8:1 ► pp. 18 ff. DOI logo
Sonesson, Göran H.
2022. The Phenomenology of Semiosis: Approaches to the Gap between the Encyclopaedia and the Porphyrian Tree Spanned by Sedimentation. Philosophies 7:5 ► pp. 114 ff. DOI logo
Zywiczynski, Przemyslaw, Nathalie Gontier & Slawomir Wacewicz
2017. The evolution of (proto-)language: Focus on mechanisms. Language Sciences 63 ► pp. 1 ff. DOI logo
Żywiczyński, Przemysław
2019. How research on language evolution contributes to linguistics. Yearbook of the Poznan Linguistic Meeting 5:1 ► pp. 1 ff. DOI logo
Żywiczyński, Przemysław, Marta Sibierska, Sławomir Wacewicz, Joost van de Weijer, Francesco Ferretti, Ines Adornetti, Alessandra Chiera & Valentina Deriu
2021. Evolution of conventional communication. A cross-cultural study of pantomimic re-enactments of transitive events. Language & Communication 80 ► pp. 191 ff. DOI logo
Żywiczyński, Przemysław & Sławomir Wacewicz
2022. From Body to Language: Gestural and Pantomimic Scenarios of Language Origin in the Enlightenment. Topoi 41:3 ► pp. 539 ff. DOI logo
Żywiczyński, Przemysław, Sławomir Wacewicz & Casey Lister
2021. Pantomimic fossils in modern human communication. Philosophical Transactions of the Royal Society B: Biological Sciences 376:1824 DOI logo

This list is based on CrossRef data as of 31 march 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

video

Item 14. boy-push-man