Cover not available
Part of
Dimensions of Iconicity
Edited by Angelika Zirker, Matthias Bauer, Olga Fischer and Christina Ljungberg
[Iconicity in Language and Literature 15] 2017
► pp. 167–190

Bridging the gap between image and metaphor through cross-modal iconicity

An interdisciplinary model

 | Linnaeus University
Our minds are capable of perceiving similarities not only within the same but also across different sensory areas and different cognitive domains. Iconicity is representation based on similarity, and cross-modal iconicity, which is an extremely widespread phenomenon, should be understood as iconicity that crosses the borders of different kinds of material, spatiotemporal, and sensorial modes, and, furthermore, the border between sensory structures and cognitive configurations. For instance, a visual entity may resemble and thus iconically represent something that is auditory or abstractly cognitive. The aim of this semiotic study, substantially based on empirical findings in psychological, cognitive, and neurological research, is to suggest a general theoretical framework for conceptualizing cross-modal iconicity and relating different kinds of mono-modal and cross-modal iconicity to each other. Its chief argument is that perception and conception of images and metaphors should be understood as the two extremes in a continuum of iconic representation where cross-modal iconicity bridges the apparent gap between mono-modal, sensory-based iconicity and cognitive iconicity. It offers an outline of material, spatiotemporal, and sensorial modes and their interrelations; a thorough account of cross-modal iconicity; a conceptual structure for charting degrees of similarity and iconicity with the aid of cross-modality; an array of examples illustrating the continuum of iconicity from image to metaphor; and a brief discussion of the notion of image schema as an explanatory factor for cross-modal iconicity.
Article outline
  • 1.Introduction
  • 2.Material, spatiotemporal, and sensorial modes
  • 3.Cross-modal iconicity
  • 4.Degrees of similarity and iconicity
  • 5.From image to metaphor
  • 6.Image schemas
  • 7.Conclusion
  • References
Published online: 8 September 2017
DOI logo
https://doi.org/10.1075/ill.15.10ell
References (50)
References
Ahlner, F. & Zlatev, J. 2010. Cross-modal iconicity: A cognitive semiotic approach to sound symbolism. Sign Systems Studies 38: 298–348. DOI logoGoogle Scholar
Anderson, E. R. 1998. A Grammar of Iconism. Madison NJ: Farleigh Dickinson University Press.Google Scholar
Bankieris, K. & Simner, J. 2015. What is the link between synaesthesia and sound symbolism? Cognition 136: 186–195. DOI logoGoogle Scholar
Bauer, M. & Ernst, C. 2010. Diagrammatik: Einführung in ein kultur- und medienwissenschaftliches Forschungsfeld. Bielefeld: Transcript. DOI logoGoogle Scholar
Benovsky, J. 2012. Photographic representation and depiction of temporal extension. Inquiry 55(2): 194–213. DOI logoGoogle Scholar
Bremner, A. J., Caparos, S., Davidoff, J., Fockert, J. de, Linnell, K. J. & Spence, C. 2013. “Bouba” and “Kiki” in Namibia? A remote culture make similar shape–sound matches, but different shape–taste matches to Westerners. Cognition 126: 165–172. DOI logoGoogle Scholar
Brochard, R., Tassin, M. & Zagar, D. 2013. Got rhythm … for better and for worse: Cross-modal effects of auditory rhythm on visual word recognition. Cognition 127: 214–219. DOI logoGoogle Scholar
Cacciari, C. 2008. Crossing the senses in metaphorical language. In The Cambridge Handbook of Metaphor and Thought, R. W. Gibbs (ed.), 425–443. Cambridge: CUP. DOI logoGoogle Scholar
Cienki, A. 2005. Image schemas and gesture. In From Perception to Meaning: Image Schemas in Cognitive Linguistics, B. Hampe (ed.), 421–441. Berlin: Mouton de Gruyter. DOI logoGoogle Scholar
Colapietro, V. 2011. Image, diagram, and metaphor: Unmined resource and unresolved questions. In Semblance and Signification [Iconicity in Language & Literarure 10], P. Michelucci, O. Fischer & C. Ljungberg (eds), 157–171. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Cowles, J. T. 1935. An experimental study of the pairing of certain auditory and visual stimuli. Journal of Experimental Psychology 18(4): 461–469. DOI logoGoogle Scholar
Eco, U. 1976. A Theory of Semiotics. Bloomington IN: Indiana University Press. DOI logoGoogle Scholar
Elleström, L. 2010a. The modalities of media: A model for understanding intermedial relations. In Media Borders, Multimodality and Intermediality, L. Elleström (ed.), 11–48. Houndmills: Palgrave Macmillan. DOI logoGoogle Scholar
. 2010b. Iconicity as meaning miming meaning, and meaning miming form. In Signergy [Iconicity in Language and Literature 9], J. Conradie, R. Johl, M. Beukes, O. Fischer & C. Ljungberg (eds), 73–100. Amsterdam: John Benjamins. DOI logoGoogle Scholar
. 2013. Spatiotemporal aspects of iconicity. In Iconic Investigations [Iconicity in Language and Literature 12], L. Elleström, O. Fischer & C. Ljungberg (eds), 95–117. Amsterdam: John Benjamins. DOI logoGoogle Scholar
. 2014. Material and mental representation: Peirce adapted to the study of media and arts. The American Journal of Semiotics 30(1–2): 83–138. DOI logoGoogle Scholar
. Forthcoming. A medium-centered model of communication. Semiotica.
Forceville, C. 2009. Non-verbal and multimodal metaphor in a cognitivist framework: Agendas for research. In Multimodal Metaphor, C. Forceville & E. Urios-Aparisi (eds), 19–42. Berlin: Mouton de Gruyter. DOI logoGoogle Scholar
Fryer, L., Freeman, J. & Pring, L. 2014. Touching words is not enough: How visual experience influences haptic–auditory associations in the “Bouba–Kiki” effect. Cognition 132: 164–173. DOI logoGoogle Scholar
Goswami, U. 2001. Analogical reasoning in children. In The Analogical Mind: Perspectives from Cognitive Science, D. Gentner, K. J. Holyoak & B. N. Kokinov (eds), 437–470. Cambridge MA: The MIT Press.Google Scholar
Johnson, M. 1987. The Body in the Mind: The Bodily Basis of Meaning, Imagination, and Reason. Chicago IL: University of Chicago Press.Google Scholar
Karwoski, T. F., Odbert, H. S. & Osgood, C. E. 1942. Studies in synesthetic thinking, II: The role of form in visual responses to music. Journal of General Psychology 26: 199–222. DOI logoGoogle Scholar
Köhler, W. 1929. Gestalt Psychology. New York NY: Horace Liveright.Google Scholar
Kress, G. & van Leeuwen, T. 2001. Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Hodder Arnold.Google Scholar
Martino, G. & Marks, L. E. 2000. Cross-modal interaction between vision and touch: The role of synesthetic correspondence. Perception 29: 745–754. DOI logoGoogle Scholar
Morris, C. 1971. Writings on the General Theory of Signs. The Hague: Mouton. DOI logoGoogle Scholar
Mossbridge, J. A., Grabowecky, M. & Suzuki, S. 2011. Changes in auditory frequency guide visual–spatial attention. Cognition 121: 133–139. DOI logoGoogle Scholar
Nöth, W. 2001. Semiotic foundations of iconicity in language and literature. In The Motivated Sign [Iconicity in Language and Literature 2], O. Fischer & M. Nänny (eds), 17–28. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Odbert, H. S., Karwoski, T. F. & Eckerson, A. B. 1942. Studies in synesthetic thinking: I. Musical and verbal associations of color and mood. Journal of General Psychology 26: 153–173. DOI logoGoogle Scholar
Peirce, C. S. 1932. Collected Papers of Charles Sanders Peirce, II: Elements of Logic, C. Hartshorne & P. Weiss (eds). Cambridge MA: Harvard University Press.Google Scholar
Perlman, M., Tanner, J. E. & King, B. J. 2012. A mother gorilla’s variable use of touch to guide her infant: Insights into iconicity and the relationship between gesture and action. In Developments in Primate Gesture Research [Gesture Studies 6], S. Pika & K. Liebal (eds), 55–72. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Perniss, P., Thompson, R. L. & Vigliocco, G. 2010. Iconicity as a general property of language: Evidence from spoken and signed languages. Frontiers in Psychology 1: 227. DOI logoGoogle Scholar
Perniss, P. & Vigliocco, G. 2014. The bridge of iconicity: From a world of experience to the experience of language. Philosophical Transactions of the Royal Society: Biological Sciences 369. DOI logoGoogle Scholar
Perry, L. K., Perlman, M. & Lupyan, G. 2015. Iconicity in English and Spanish and its relation to lexical category and age of acquisition. PLoS One 10(9). DOI logoGoogle Scholar
Popova, Y. 2005. Image schemas and verbal synaesthesia. In From Perception to Meaning: Image Schemas in Cognitive Linguistics, B. Hampe (ed.), 395–419. Berlin: Mouton de Gruyter. DOI logoGoogle Scholar
Ramachandran, V. S. & Hubbard, E. M. 2005. The emergence of the human mind: Some clues from synesthesia. In Synesthesia: Perspectives From Cognitive Neuroscience, N. Sagiv & L. C. Robertson (eds), 147–190. Oxford: OUP.Google Scholar
Röder, B., Pagel, B. & Heed, T. 2013. The implicit use of spatial information develops later for crossmodal than for intramodal temporal processing. Cognition 126: 301–306. DOI logoGoogle Scholar
Rohrer, T. 2005. Image schemata in the brain. In From Perception to Meaning: Image Schemas in Cognitive Linguistics, B. Hampe (ed.), 165–196. Berlin: Mouton de Gruyter. DOI logoGoogle Scholar
Smith J. D., Flemming, T. M., Boomer, J., Beran, M. J. & Church, B. A. 2013. Fading perceptual resemblance: A path for rhesus macaques (Macaca mulatta) to conceptual matching? Cognition 129: 598–614. DOI logoGoogle Scholar
Smith, L. B. & Heise, D. 1992. Perceptual similarity and conceptual structure. In Percepts, Concepts and Categories, B. Burns (ed.), 233–272. Amsterdam: Elsevier.Google Scholar
Spence, C. 2011. Crossmodal correspondences: A tutorial review. Attention, Perception, & Psychophysics 73(4): 971–995. DOI logoGoogle Scholar
Stein, B. E., Spence, C. & Calvert, G. 2004. The Handbook of Multisensory Processes. Cambridge MA: The MIT Press.Google Scholar
Sutherland, C. A. M., Thut, G. & Romei, V. 2014. Hearing brighter: Changing in-depth visual perception through looming sounds. Cognition 132: 312–323. DOI logoGoogle Scholar
Sweeny, T. D., Guzman-Martinez, E., Ortega, L., Grabowecky, M. & Suzuki, S. 2012. Sounds exaggerate visual shape. Cognition 124: 194–200. DOI logoGoogle Scholar
Tabakowska, E. 2003. Iconicity and literary translation. In From Sign to Signing [Iconicity in Language and Literature 3], W. G. Müller & O. Fischer (eds), 361–376. Amsterdam: John Benjamins. DOI logoGoogle Scholar
. 2005. Iconicity as a function of point of view. In Outside-In – Inside-Out [Iconicity in Language and Literature 4], C. Maeder, O. Fischer & W. J. Herlofsky (eds), 375–387. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Usnadze, D. 1924. Ein experimenteller Beitrag zum Problem der psychologischen Grundlagen der Namengebung. Psychologische Forschung 5(1): 24–43. DOI logoGoogle Scholar
Yildirim, I. & Jacobs, R. A. 2013. Transfer of object category knowledge across visual and haptic modalities: Experimental and computational studies. Cognition 126: 135–148. DOI logoGoogle Scholar
Zbikowski, L. M. 2008. Metaphor and music. In The Cambridge Handbook of Metaphor and Thought, R. W. Gibbs (ed.), 502–524. Cambridge: CUP. DOI logoGoogle Scholar
Ziembowicz, M., Nowak, A. & Winkielman, P. 2013. When sounds look right and images sound correct: Cross-modal coherence enhances claims of pattern presence. Cognition 129: 273–278. DOI logoGoogle Scholar
Cited by (9)

Cited by nine other publications

Bonifazi, Anna
2023. Cross-modal iconicity in songs about weeping. Semiotica 0:0 DOI logo
Alarauhio, Juha-Pekka, Tiina Räisänen, Jarkko Toikkanen & Riikka Tumelius
2022. Introduction: Multimodality and Intermediality in the North. In Shaping the North Through Multimodal and Intermedial Interaction [Arctic Encounters, ], ► pp. 1 ff. DOI logo
Elleström, Lars
2022. Symbolicity, language, and mediality. Semiotica 2022:247 ► pp. 1 ff. DOI logo
Salmose, Niklas
2020. Sensorial Aesthetics: Cross-Modal Stylistics in Modernist Fiction. Text Matters: A Journal of Literature, Theory and Culture :10 ► pp. 321 ff. DOI logo
Bateman, John A
2019. Transmediality and the End of Disembodied Semiotics. International Journal of Semiotics and Visual Rhetoric 3:2 ► pp. 1 ff. DOI logo
Elleström, Lars & Filip Cieślak
2019. Identifying, Construing, and Bridging over Media Borders. Tekstualia 3:58 ► pp. 21 ff. DOI logo
Fröhlich, Marlen, Christine Sievers, Simon W. Townsend, Thibaud Gruber & Carel P. van Schaik
2019. Multimodal communication and language origins: integrating gestures and vocalizations. Biological Reviews 94:5 ► pp. 1809 ff. DOI logo
Lugea, Jane
2018. The year’s work in stylistics 2017. Language and Literature: International Journal of Stylistics 27:4 ► pp. 329 ff. DOI logo
Pethő, Ágnes
2018. A Good Concept Should Be both Very Concrete and Very Abstract.”. Acta Universitatis Sapientiae, Film and Media Studies 15:1 ► pp. 191 ff. DOI logo

This list is based on CrossRef data as of 19 november 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.