An interdisciplinary model: Bridging the gap between image and metaphor through cross-modal iconicity

Elleström, Lars

doi:10.1075/ill.15.10ell

Part of

Dimensions of Iconicity
Edited by Angelika Zirker, Matthias Bauer, Olga Fischer and Christina Ljungberg
[Iconicity in Language and Literature 15] 2017
► pp. 167–190

Bridging the gap between image and metaphor through cross-modal iconicity

An interdisciplinary model

Lars Elleström | Linnaeus University

Our minds are capable of perceiving similarities not only within the same but also across different sensory areas and different cognitive domains. Iconicity is representation based on similarity, and cross-modal iconicity, which is an extremely widespread phenomenon, should be understood as iconicity that crosses the borders of different kinds of material, spatiotemporal, and sensorial modes, and, furthermore, the border between sensory structures and cognitive configurations. For instance, a visual entity may resemble and thus iconically represent something that is auditory or abstractly cognitive. The aim of this semiotic study, substantially based on empirical findings in psychological, cognitive, and neurological research, is to suggest a general theoretical framework for conceptualizing cross-modal iconicity and relating different kinds of mono-modal and cross-modal iconicity to each other. Its chief argument is that perception and conception of images and metaphors should be understood as the two extremes in a continuum of iconic representation where cross-modal iconicity bridges the apparent gap between mono-modal, sensory-based iconicity and cognitive iconicity. It offers an outline of material, spatiotemporal, and sensorial modes and their interrelations; a thorough account of cross-modal iconicity; a conceptual structure for charting degrees of similarity and iconicity with the aid of cross-modality; an array of examples illustrating the continuum of iconicity from image to metaphor; and a brief discussion of the notion of image schema as an explanatory factor for cross-modal iconicity.

Article outline

1.Introduction
2.Material, spatiotemporal, and sensorial modes
3.Cross-modal iconicity
4.Degrees of similarity and iconicity
5.From image to metaphor
6.Image schemas
7.Conclusion
References

Published online: 8 September 2017

https://doi.org/10.1075/ill.15.10ell

References

Ahlner, F. & Zlatev, J

2010 Cross-modal iconicity: A cognitive semiotic approach to sound symbolism. Sign Systems Studies 38: 298–348.

Anderson, E. R

1998 A Grammar of Iconism. Madison NJ: Farleigh Dickinson University Press.

Bankieris, K. & Simner, J

2015 What is the link between synaesthesia and sound symbolism? Cognition 136: 186–195.

Bauer, M. & Ernst, C

2010 Diagrammatik: Einführung in ein kultur- und medienwissenschaftliches Forschungsfeld. Bielefeld: Transcript.

Benovsky, J

2012 Photographic representation and depiction of temporal extension. Inquiry 55(2): 194–213.

Bremner, A. J., Caparos, S., Davidoff, J., Fockert, J. de, Linnell, K. J. & Spence, C

2013 “Bouba” and “Kiki” in Namibia? A remote culture make similar shape–sound matches, but different shape–taste matches to Westerners. Cognition 126: 165–172.

Brochard, R., Tassin, M. & Zagar, D

2013 Got rhythm … for better and for worse: Cross-modal effects of auditory rhythm on visual word recognition. Cognition 127: 214–219.

Cacciari, C

2008 Crossing the senses in metaphorical language. In The Cambridge Handbook of Metaphor and Thought, R. W. Gibbs (ed.), 425–443. Cambridge: CUP.

Cienki, A

2005 Image schemas and gesture. In From Perception to Meaning: Image Schemas in Cognitive Linguistics, B. Hampe (ed.), 421–441. Berlin: Mouton de Gruyter.

Colapietro, V

2011 Image, diagram, and metaphor: Unmined resource and unresolved questions. In Semblance and Signification [Iconicity in Language & Literarure 10], P. Michelucci, O. Fischer & C. Ljungberg (eds), 157–171. Amsterdam: John Benjamins.

Cowles, J. T

1935 An experimental study of the pairing of certain auditory and visual stimuli. Journal of Experimental Psychology 18(4): 461–469.

Eco, U

1976 A Theory of Semiotics. Bloomington IN: Indiana University Press.

Elleström, L

2010a The modalities of media: A model for understanding intermedial relations. In Media Borders, Multimodality and Intermediality, L. Elleström (ed.), 11–48. Houndmills: Palgrave Macmillan.

2010b Iconicity as meaning miming meaning, and meaning miming form. In Signergy [Iconicity in Language and Literature 9], J. Conradie, R. Johl, M. Beukes, O. Fischer & C. Ljungberg (eds), 73–100. Amsterdam: John Benjamins.

2013 Spatiotemporal aspects of iconicity. In Iconic Investigations [Iconicity in Language and Literature 12], L. Elleström, O. Fischer & C. Ljungberg (eds), 95–117. Amsterdam: John Benjamins.

2014 Material and mental representation: Peirce adapted to the study of media and arts. The American Journal of Semiotics 30(1–2): 83–138.

Forthcoming. A medium-centered model of communication. Semiotica.

Forceville, C

2009 Non-verbal and multimodal metaphor in a cognitivist framework: Agendas for research. In Multimodal Metaphor, C. Forceville & E. Urios-Aparisi (eds), 19–42. Berlin: Mouton de Gruyter.

Fryer, L., Freeman, J. & Pring, L

2014 Touching words is not enough: How visual experience influences haptic–auditory associations in the “Bouba–Kiki” effect. Cognition 132: 164–173.

Goswami, U

2001 Analogical reasoning in children. In The Analogical Mind: Perspectives from Cognitive Science, D. Gentner, K. J. Holyoak & B. N. Kokinov (eds), 437–470. Cambridge MA: The MIT Press.

Johnson, M

1987 The Body in the Mind: The Bodily Basis of Meaning, Imagination, and Reason. Chicago IL: University of Chicago Press.

Karwoski, T. F., Odbert, H. S. & Osgood, C. E

1942 Studies in synesthetic thinking, II: The role of form in visual responses to music. Journal of General Psychology 26: 199–222.

Köhler, W

1929 Gestalt Psychology. New York NY: Horace Liveright.

Kress, G. & van Leeuwen, T

2001 Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Hodder Arnold.

Martino, G. & Marks, L. E

2000 Cross-modal interaction between vision and touch: The role of synesthetic correspondence. Perception 29: 745–754.

Morris, C

1971 Writings on the General Theory of Signs. The Hague: Mouton.

Mossbridge, J. A., Grabowecky, M. & Suzuki, S

2011 Changes in auditory frequency guide visual–spatial attention. Cognition 121: 133–139.

Nöth, W

2001 Semiotic foundations of iconicity in language and literature. In The Motivated Sign [Iconicity in Language and Literature 2], O. Fischer & M. Nänny (eds), 17–28. Amsterdam: John Benjamins.

Odbert, H. S., Karwoski, T. F. & Eckerson, A. B

1942 Studies in synesthetic thinking: I. Musical and verbal associations of color and mood. Journal of General Psychology 26: 153–173.

Peirce, C. S

1932 Collected Papers of Charles Sanders Peirce, II: Elements of Logic, C. Hartshorne & P. Weiss (eds). Cambridge MA: Harvard University Press.

Perlman, M., Tanner, J. E. & King, B. J

2012 A mother gorilla’s variable use of touch to guide her infant: Insights into iconicity and the relationship between gesture and action. In Developments in Primate Gesture Research [Gesture Studies 6], S. Pika & K. Liebal (eds), 55–72. Amsterdam: John Benjamins.

Perniss, P., Thompson, R. L. & Vigliocco, G

2010 Iconicity as a general property of language: Evidence from spoken and signed languages. Frontiers in Psychology 1: 227.

Perniss, P. & Vigliocco, G

2014 The bridge of iconicity: From a world of experience to the experience of language. Philosophical Transactions of the Royal Society: Biological Sciences 369.

Perry, L. K., Perlman, M. & Lupyan, G

2015 Iconicity in English and Spanish and its relation to lexical category and age of acquisition. PLoS One 10(9).

Popova, Y

2005 Image schemas and verbal synaesthesia. In From Perception to Meaning: Image Schemas in Cognitive Linguistics, B. Hampe (ed.), 395–419. Berlin: Mouton de Gruyter.

Ramachandran, V. S. & Hubbard, E. M

2005 The emergence of the human mind: Some clues from synesthesia. In Synesthesia: Perspectives From Cognitive Neuroscience, N. Sagiv & L. C. Robertson (eds), 147–190. Oxford: OUP.

Röder, B., Pagel, B. & Heed, T

2013 The implicit use of spatial information develops later for crossmodal than for intramodal temporal processing. Cognition 126: 301–306.

Rohrer, T

2005 Image schemata in the brain. In From Perception to Meaning: Image Schemas in Cognitive Linguistics, B. Hampe (ed.), 165–196. Berlin: Mouton de Gruyter.

Smith J. D., Flemming, T. M., Boomer, J., Beran, M. J. & Church, B. A

2013 Fading perceptual resemblance: A path for rhesus macaques (Macaca mulatta) to conceptual matching? Cognition 129: 598–614.

Smith, L. B. & Heise, D

1992 Perceptual similarity and conceptual structure. In Percepts, Concepts and Categories, B. Burns (ed.), 233–272. Amsterdam: Elsevier.

Spence, C

2011 Crossmodal correspondences: A tutorial review. Attention, Perception, & Psychophysics 73(4): 971–995.

Stein, B. E., Spence, C. & Calvert, G

2004 The Handbook of Multisensory Processes. Cambridge MA: The MIT Press.

Sutherland, C. A. M., Thut, G. & Romei, V

2014 Hearing brighter: Changing in-depth visual perception through looming sounds. Cognition 132: 312–323.

Sweeny, T. D., Guzman-Martinez, E., Ortega, L., Grabowecky, M. & Suzuki, S

2012 Sounds exaggerate visual shape. Cognition 124: 194–200.

Tabakowska, E

2003 Iconicity and literary translation. In From Sign to Signing [Iconicity in Language and Literature 3], W. G. Müller & O. Fischer (eds), 361–376. Amsterdam: John Benjamins.

2005 Iconicity as a function of point of view. In Outside-In – Inside-Out [Iconicity in Language and Literature 4], C. Maeder, O. Fischer & W. J. Herlofsky (eds), 375–387. Amsterdam: John Benjamins.

Usnadze, D

1924 Ein experimenteller Beitrag zum Problem der psychologischen Grundlagen der Namengebung. Psychologische Forschung 5(1): 24–43.

Yildirim, I. & Jacobs, R. A

2013 Transfer of object category knowledge across visual and haptic modalities: Experimental and computational studies. Cognition 126: 135–148.

Zbikowski, L. M

2008 Metaphor and music. In The Cambridge Handbook of Metaphor and Thought, R. W. Gibbs (ed.), 502–524. Cambridge: CUP.

Ziembowicz, M., Nowak, A. & Winkielman, P

2013 When sounds look right and images sound correct: Cross-modal coherence enhances claims of pattern presence. Cognition 129: 273–278.

Cited by

Cited by 9 other publications

Order by:

Alarauhio, Juha-Pekka, Tiina Räisänen, Jarkko Toikkanen & Riikka Tumelius

2022. Introduction: Multimodality and Intermediality in the North. In Shaping the North Through Multimodal and Intermedial Interaction [Arctic Encounters, ], ► pp. 1 ff.

Bateman, John A

2019. Transmediality and the End of Disembodied Semiotics. International Journal of Semiotics and Visual Rhetoric 3:2 ► pp. 1 ff.

Bonifazi, Anna

2024. Cross-modal iconicity in songs about weeping. Semiotica 2024:256 ► pp. 1 ff.

Elleström, Lars

2022. Symbolicity, language, and mediality. Semiotica 2022:247 ► pp. 1 ff.

Elleström, Lars & Filip Cieślak

2019. Identifying, Construing, and Bridging over Media Borders. Tekstualia 3:58 ► pp. 21 ff.

Fröhlich, Marlen, Christine Sievers, Simon W. Townsend, Thibaud Gruber & Carel P. van Schaik

2019. Multimodal communication and language origins: integrating gestures and vocalizations. Biological Reviews 94:5 ► pp. 1809 ff.

Lugea, Jane

2018. The year’s work in stylistics 2017. Language and Literature: International Journal of Stylistics 27:4 ► pp. 329 ff.

Pethő, Ágnes

2018. A Good Concept Should Be both Very Concrete and Very Abstract.”. Acta Universitatis Sapientiae, Film and Media Studies 15:1 ► pp. 191 ff.

Salmose, Niklas

2020. Sensorial Aesthetics: Cross-Modal Stylistics in Modernist Fiction. Text Matters: A Journal of Literature, Theory and Culture :10 ► pp. 321 ff.

This list is based on CrossRef data as of 12 april 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.