Based on an analysis accounting for the whole body as a possible articulator in the depiction of actions, this chapter argues for an expansion of the notion of ‘character viewpoint gestures’ to a notion of ‘multimodal action depiction from a character viewpoint’. Our study shows that speakers may deploy only single articulators, providing a semantically reduced depiction of the action, or they may deploy more bodily articulators and give a semantically rich picture of the event narrated. Our findings suggest a continuum of semiotic complexity, capturing the range of bodily involvement from less pantomimic (single articulators involved) to pantomimic depictions (more articulators involved) of actions. The paper closes by discussing our observations with respect to the notion of ‘constructed action’ and ‘role shift’ in sign languages and by giving some general remarks on the multimodal analysis of narrations.
Article outline
1.Introduction
2.Analyzing multimodal narratives
3.Exemplification of the method
Step 1Description of articulators involved in the character viewpoint depiction
Step 2Meaning analysis of body movements
Step 2.1Analysis of meaning conveyed by each articulator involved in the character-viewpoint depictions
Step 2.2Analysis of the meaning in the verbal and interactive context
Step 3Determination of degree of semiotic complexity
4.Ways of expression action in multimodal narrations
4.1Character viewpoint depiction involving a single articulator
4.2Character viewpoint depiction involving two articulators
4.3Character viewpoint depiction involving several articulators
5.Continuum of semiotic complexity
6.Concluding thoughts: Iconicity of action depiction and constructed action in gesture and sign language
2001How gesture viewpoint influences what information decoders receive from iconic gestures. In Oralite et gestualite: Interactions et comportements multimodaux dans la communication, C. Cave, I. Guaitella & S. Santi (eds), 283–287. Paris: L’Harmattan.
Beattie, G. & Shovelton, H.
2002An experimental investigation of some properties of individual iconic gestures that mediate their communicative power. British Journal of Psychology 93(2): 179–192.
Birdwhistell, R.
1970Kinesics and Context. Essays on Body, Motion, Communication. Philadelphia PA: University of Pennsylvania Press.
Bressem, J.
2013A linguistic perspective on the notation of form features in gestures. In C. Mülleret al. (eds), 1079–1098.
Bressem, J.
2015Repetition als Mittel der Musterbildung bei redebegleitenden Gesten. In Handbuch Satz, Äußerung, Schema, C. Dürscheid & J. G. Schneider (eds), 422–442. Berlin: Walter de Gruyter.
Bressem, J., Ladewig, S. H. & Müller, C.
2013Linguistic Annotation System for Gestures (LASG). In Mülleret al.. (eds), 1098–1125.
Bressem, J. & Ladewig, S. H.
2011Rethinking gesture phases – Articulatory features of gestural movement?Semiotica 184(1–4): 53–91.
Bressem, J. & Müller, C.
2014The family of AWAY-gestures: Negation, refusal and negative assessement. In Mülleret al.. (eds), 1592–1605.
1934Sprachtheorie. Die Darstellungsfunktion der Sprache. Stuttgart: Fischer.
Casey, S. & Emmorey, K.
2009Co-speech gesture in bimodal bilinguals. Language and Cognitive Processes 24(2): 290–312.
Cassell, J. & McNeill, D.
1991Gesture and the poetics of prose. Poetics Today 12(3): 375–404.
Cormier, K., Quinto-Pozos, D., Sevcikova, Z. & Schembri, A.
2012Lexicalisation and de-lexicalisation processes in sign languages: Comparing depicting constructions and viewpoint gestures. Language and Communication 32: 329–348.
Davis, M.
1979Laban analysis of nonverbal communication. In Nonverbal Communication: Readings with Commentary, S. Weitz (ed.), 182–206. Oxford: OUP.
Dudis, P. G
2004Body partitioning and real-space blends. Cognitive Linguistics 15(2): 223–238.
Dudis, P. G.
2007Types of depiction in ASL. Ms, Gallaudet University. [URL]
Duncan, S.
2005Gesture in signing: A case study in Taiwan Sign Language. Language and Linguistics 6(2): 279–318.
Eibl-Eibesfeldt, I.
1971Transcultural patterns of ritualized contact behavior. In Behavior and Environment. The Use of Space by Animals and Men, A. H. Esser (ed.), 238–246. New York NY: Plenum.
Ekman, P., Sorenson, E. R. & Friesen, W.
1969Pan-cultural elements in facial displays of emotions. Science 164(3875): 86–88.
Ekman, P., Friesen, W. & Hager, J. C.
2002Facial Action Coding System. Salt Lake City UT: Research Nexus.
Frey, S., Hirsbrunner, H. P., Pool, J. & Daw, W.
1981Das Berner System zur Untersuchung nonverbaler Interaktion: I. Die Erhebung des Rohdatenprotokolls; II. Die Auswertung von Zeitreihen visuell-auditiver Information. In Methoden der Analyse von Face-to-Face-Situationen, P. Winkler (ed.), 203–268. Stuttgart: Metzler.
Goodwin, C.
1981Conversational organization: Interaction between speakers and hearers. New York: Academic Press New York.
Gumperz, J.
1992Contextualization revisited. In The Contextualization of Language [Pragmatics & Beyond New Series 22], P. Auer & A. di Luzio (eds), 39–54. Amsterdam: John Benjamins.
Gülich, E.
1970Makrosyntax der Gliederungssignale im gesprochenen Französisch. Munich: Fink.
Gülich, E. & Hausendorf, H.
2000Vertextungsmuster Narration. In Text-und Gesprächslinguistik/Linguistics of Text and Conversation, K. Brinker, G. Antos, W. Heinemann & S. F. Sager (eds), 369–385. Berlin: Walter de Gruyter.
Günthner, S.
1996The prosodic contextualization of moral work: An analysis of reproaches in ‘why’-formats. In Prosody in Conversation, E. Couper-Kuhlen & M. Selting (eds), 271–302. Cambridge: CUP.
Harrison, S.
2009The expression of negation through grammar and gesture. In Studies in Language and Cognition, J. Zlatev, M. Andren, M. J. Falck & C. Lundmark (eds), 405–409. Cambridge: CUP.
Harrison, S.
2014Head shakes: Variation in form, function, and cultural distribution of a head movement related to “no”. In C. Mülleret al.. (eds), 1496–1501.
Hermann, A. & Pendzich, N.-K.
2014Nonmanual gestures in sign languages. In C. Mülleret al.. (eds), 2149–2163.
Herrmann, A. & Steinbach, M.
2012Quotation in sign languages. In Quotatives: Cross-linguistic and Cross-disciplinary Perspectives [Converging Evidence in Language and Communication Research 15], I. van Alphen & I. Buchstaller (eds), 203–228. Amsterdam: John Benjamins.
Jakobson, R.
1966Selected writings. Berlin: Mouton de Gruyter.
Keim, I.
1993Sprachvariation als konstitutives Merkmal eines sozialen Stils. Am Beispiel einer innerstädtischen Welt „Kleiner Leute“. Soziolinguistik. Der Deutschunterricht 3: 44–56.
Kendon, A.
1967Some functions of gaze-direction in social interaction. Acta Psychologica 26: 22–63.
Kendon, A.
1980Gesture and speech: Two aspects of the process of utterance. In Nonverbal Communication and Language, M. R. Key (ed.), 207–227. The Hague: Mouton.
Kendon, A.
1992Some recent work from Italy on quotable gestures (emblems). Journal of Linguistic Anthropology 2(1): 92–108.
Kendon, A.
2000Language and gesture: Unity or duality. In Language and Gesture, D. McNeill (ed.), 47–63. Cambridge: CUP.
Kendon, A
2004Gesture: Visible Action as Utterance. Cambridge: CUP.
Kendon, A. & Ferber, A.
1973A description of some human greetings. In Comparative Ecology and Behavior of Primates, R. P. Michael & J. H. Crook (eds), 591–668. New York NY: Academic Press.
Kita, S. & Özyürek, A.
2003What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language 48: 16–32.
Knapp, M., Hall, J. & Horgan, T.
2013Nonverbal Communication in Human Interaction. Boston MA: Wadsworth Cengage Learning.
Ladewig, S. H.
2010Beschreiben, suchen und auffordern – Varianten einer rekurrenten Geste. Sprache und Literatur 41(105): 89–111.
Ladewig, S. H.
2011Putting the cyclic gesture on a cognitive basis. CogniTextes 6. [URL] (June 2013).
Ladewig, S. H.
2014cCreating multimodal utterance. The linear integration of gestures into speech. In: Müller, C.et al. (eds), 1662–1677.
Ladewig, S. H. & Bressem, J.
2013New insights into the medium hand – Discovering structures in gestures based on the four parameters of sign language. Semiotica 197: 203–231.
2014aRecurrent gestures. In C. Mülleret al.. (eds), 1558–1575.
Ladewig, S. H
2014bThe cyclic gesture. In C. Mülleret al.. (eds), 1605–1618.
Langacker, R.
2008Cognitive Grammar: A Basic Introduction. Oxford: OUP.
Lausberg, H. & Sloetjes, H.
2009Coding gestural behavior with the NEUROGES – ELAN system. Behavioral Research Methods 41(3): 841–849.
van Loon, E., Pfau, R. & Steinbach, M.
2014The grammaticalization of gestures in sign languages. In C. Mülleret al. (eds), 2133–2149.
Mandel, M.
1977Iconic devices in American Sign Language. In On the other hand. New perspectives on American Sign Language, L. A. Friedman (ed.) New York: Academic Press
Mayerthaler, W.
1980Ikonismus in der Morphologie. Zeitschrift für Semiotik 2: 19–37.
McClave, E. Z.
2000Linguistic functions of head movements in the context of speech. Journal of Pragmatics 32: 855–878.
McNeill, D.
1992Hand and Mind. What Gestures Reveal about Thought. Chicago IL: University of Chicago Press.
McNeill, D.
2005Gesture and Thought. Chicago IL: University of Chicago Press.
Mittelberg, I.
2006Metaphor and Metonymy in Language and Gesture: Discoursive Evidence for Multimodal Models of Grammar. PhD dissertation, Cornell University. Ann Arbor MI: ProQuest/UMI. [URL]
Mittelberg, I.
2014Gestures and iconicity. In C. Mülleret al.. (eds), 1712–1732.
Mittelberg, I. & Waugh, L.
2014Gestures and metonymy. In C. Mülleret al.. (eds), 1747–1766.
Müller, C.
1998Redebegleitende Gesten: Kulturgeschichte, Theorie,
Sprachvergleich. Berlin: Arno Spitz.
Müller, C.
2004Forms and uses of the Palm Up Open Hand. A case of a gesture family? In Semantics and Pragmatics of Everyday Gestures, C. Müller & R. Posner (eds), 233–256.Berlin: Weidler Verlag
Müller, C
2008Metaphors. Dead and Alive, Sleeping and Waking. A Dynamic View. Chicago, University of Chicago Press.
Müller, C.
2010Wie Gesten bedeuten. Eine kognitiv-linguistische und sequenzanalytische Perspektive. Sprache und Literatur 41(105): 37–68.
Müller, C.
2013Gestures as a medium of expression: The linguistic potential of gestures. In C. Mülleret al.. (eds), 202–217.
Müller, C.
2014aGestural modes of representation as techniques of depiction. In Mülleret al.. (eds), 1677–1687. Berlin: De Gruyter Mouton.
Müller, C.
2014bGesture as “deliberate expressive movement.” In From Gesture in Conversation to Visible Action as Utterance: Essays in Honor of Adam Kendon, M. Seyfeddinipur & M. Gullberg (eds), 127–152. Amsterdam: John Benjamins.
Müller, C., Bressem, J. & Ladewig S. H.
2013Towards a grammar of gesture: A form-based view. In Mülleret al.. (eds), 707–733.
C. Müller, C., Ladewig, S. H. & Bressem, J.
2013Gestures and speech from a linguistic perspective: A new field and its history. In C. Mülleret al.. (eds), 55–81.
Müller, C., Ladewig S. H. & Teßendorf S
2010Structures of meaning and reference – Investigating gestures of the concrete and abstract. Paper presented at the 4th Conference of the International Society for Gesture Studies, Frankfurt/Oder, Germany.
Müller, C. & Tag, S.
2010The dynamics of metaphor: Foregrounding and activating metaphoricity in conversational interaction. Cognitive Semiotics 6: 85–120.
Müller, C., Cienki, A, Fricke, E., Ladewig, S. H., McNeill, D. & Teßendorf, S.
(eds)2013Body – Language – Communication: An International Handbook on Multimodality in Human Interaction. Volume 1. Berlin, Boston: De Gruyter Mouton.
Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Jana Bressem
(eds)2014Body – Language – Communication: An International Handbook on Multimodality in Human Interaction. Volume 2. Berlin, Boston: De Gruyter: Mouton.
2010Viewpoint in speech-gesture integration: Linguistic structure, discourse structure, and event structure. Language and Cognitive Processes 25(5): 650–668.
Peirce, C. S.
1960Collected Papers of Charles Sanders Peirce (1931–1958), Vol. I: Principles of Philosophy, Vol. II: Elements of Logic, C. Hartshorne & P. Weiss (eds). Cambridge MA: The Belknap of Harvard University Press.
Pfau, R. & Steinbach, M.
2006Modality-independent and modality specific aspects of grammaticalization in sign languages. Linguistics in Potsdam 24: 3–98. [URL]
Pörings, R. & Schmitz, U.
2003Sprache und Sprachwissenschaft. Eine kognitiv orientierte Einführung. Tübingen: Gunter Narr.
Quasthoff, U.
1979Konversationelle Erzählungen im Sozialamt: Ihr Einsatz als kommunikative Strategien. In Soziologische Analysen. Referate aus den Veranstaltungen der Sektionen der deutschen Gesellschaft für Soziologie und der ad-hoc-Gruppen beim 19. Deutschen Soziologentag, R. Mackensen & F. Sagebiel (eds), 515–534. Berlin: Deutsche Gesellschaft für Soziologie (DGS).
Quasthoff, U.
2000Erzählen als interaktive Gesprächsstruktur. In Text- und Gesprächslinguistik/Linguistics of Text and Conversation, K. Brinker, G. Antos, W. Heinemann & S. F. Sager (eds), 1293–1309. Berlin: Walter de Gruyter.
Quinto-Pozos, D.
2007Why does constructed action seem obligatory? An analysis of “classifiers” and the lack of articulator-referent correspondence. Sign Language Studies 7(4): 458.
Quinto-Pozos, D.
2014Enactment as a (signed) language communicative strategy. In C. Mülleret al.. (eds), 2163–2170.
Quinto-Pozos, D. & Parrill, F.
2012Comparing viewpoint strategies used by co-speech gesturers and signers. Paper presented at the 5th Conference of the International Society for Gesture Studies, Lund, Sweden.
Selting, M.
1995Sprechstile als Kontextualisierungshinweise. In Stilfragen. Jahrbuch 1994 des Instituts für deutsche Sprache, G. Stickel (ed.), 225–256. Berlin: Walter de Gruyter.
Selting, M., Auer, P., Barth-Weingarten, D., Bergmann, J., Bergmann, P., Birkner, K., Couper-Kuhlen, E., Deppermann, E., Gilles, P., Günther, S., Hartung, M., Kern, F., Mertzlufft, C., Meyer, C., Morek, M., Oberzaucher, F., Peter, L., Quasthoff, U. M., Schütte, W., Stukenbrock A. & Uhmann, S.
2004Meta-discursive gestures from Iran: Some uses of the ‘Pistol Hand’. In The Semantics and Pragmatics of Everyday Gestures, C. Müller & R. Posner (eds), 205–216. Berlin: Weidler.
2012Introduction: Viewpoint and perspective in language and gesture, from the Ground down. In Viewpoint in Language: A Multimodal Perspective, B. Dancygier & E. Sweetser (eds), 1–22. Cambridge: CUP.
Sweetser, E. & Stec, K.
2013Managing multiple viewpoints: Coordinating embedded perspective in multimodal narratives.
Paper presented at the ICLC-12, Edmonton, Canada.
Taub, S.
2001Language from the Body: Iconicity and Metaphor in American Sign Language. Cambridge: CUP.
Teßendorf, S.
2014Pragmatic and metaphoric gestures – Combining functional with cognitive approaches. In C. Mülleret al.. (eds), 1540–1558.
Teßendorf, S.
2016Actions as sources of gestures. In Beyond Language Boundaries: Multimodal Use in Multilingual Contexts, M. Fernández-Villanueva & K. Jungbluth (eds), 34–54. Berlin, Boston: De Gruyter
2007Routes from gesture to language. In Verbal and Signed Languages: Comparing Structures, Constructs and Methodologies, E. Pizzuto, P. Pientrandrea & R. Simone (eds), 107–137. Berlin: Walter de Gruyter.
Wilcox, S., Rossini P. & Antinoro E. P.
2010Grammaticalization in sign languages. In Sign Languages, D. Brentari (ed.), 332–355. Cambridge: CUP.
Wilcox, S. & Xavier A. N.
2013A framework for unifying spoken language, signed language, and gesture. Todas as Letras-Revista de Língua e Literatura 15(1): 88–110.
2021. On the diachronic aspect of gestural communication. Languages and Modalities 1 ► pp. 95 ff.
Vandenitte, Sébastien
2022. Making Referents Seen and Heard Across Signed and Spoken Languages: Documenting and Interpreting Cross-Modal Differences in the Use of Enactment. Frontiers in Psychology 13
Zhang, Icy (Yunyi), Tina Izad & Erica A. Cartmill
2024. Embodying Similarity and Difference: The Effect of Listing and Contrasting Gestures During U.S. Political Speech. Cognitive Science 48:3
This list is based on CrossRef data as of 22 april 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.