Cover not available
Part of
Less Frequently Used Research Methodologies in Applied Linguistics
Edited by A. Mehdi Riazi
[Research Methods in Applied Linguistics 6] 2024
► pp. 49–67

Chapter 4
Multimodality

A systemic-functional semiotic perspective

| National Institute of Education
| National Institute of Education
| National Institute of Education

Multimodality examines how language and other resources (e.g., images and gestures) are integrated to make meaning for communication. This chapter aims to introduce two approaches to multimodality from a systemic-functional semiotic perspective, namely social semiotics and systemic functional multimodal discourse analysis (SF-MDA), and discuss their applications in applied linguistics. Specifically, this chapter begins with an overview of the theoretical underpinnings of the two approaches, focusing on their shared functionalist origins and the origin’s theoretical and methodological implications. Next, the chapter discusses the affordance of social semiotics and SF-MDA and highlights data collection procedures and analytical processes in applied linguistic studies informed by the two approaches. Finally, critiques of the two approaches are addressed with multimodal research facilitated by eye-tracking technology.

Article outline
  • 1.Introduction
  • 2.Theoretical foundations
    • 2.1Multimodality as a functionalist social semiotic inquiry
    • 2.2Social semiotics and systemic functional multimodal discourse analysis (SF-MDA)
  • 3.Methodological orientation
    • 3.1Principles and affordances
    • 3.2Types of research questions addressed by social semiotics and SF-MDA
    • 3.3Procedures of data collection and analysis
      • 3.3.1Data collection
      • 3.3.2Data analysis
    • 3.4Research ethics
  • 4.Systemic-functional semiotics: Critiques and responses
  • 5.Conclusions
  • References
Published online: 5 January 2024
DOI logo
https://doi.org/10.1075/rmal.6.04che
References
Amundrud, T.
(2018) Applying multimodal research to the tertiary foreign language classroom: Looking at gaze. In H. D. S. Joyce & S. Feez (Eds), Multimodality across classrooms: Learning about and through different modalities (pp. 160–177). Routledge. DOI logoGoogle Scholar
Bateman, J.
(2014) Text and image: A critical introduction to the visual/verbal divide. Routledge. DOI logoGoogle Scholar
Bateman, J. A.
(2018) Towards critical multimodal discourse analysis: A response to Ledin and Machin. Critical Discourse Studies, 15(5), 1–9. DOI logoGoogle Scholar
Bateman, J. A., Beckmann, A., & Varela, R. I.
(2018) From empirical studies to visual narrative organization: Exploring page composition. In A. Dunst, J. Laubrock, & J. Wildfeuer (Eds.), Empirical comics research: Digital, multimodal, and cognitive methods (pp.127–153). Routledge. DOI logoGoogle Scholar
Bateman, J., Wildfeuer, J., & Hiippala, T.
(2017) Multimodality: Foundations, research and analysis. Walter de Gruyter. DOI logoGoogle Scholar
Bezemer, J., & Kress, G.
(2008) Writing in multimodal texts: A social semiotic account of designs for learning. Written Communication, 25(2), 166–195. DOI logoGoogle Scholar
(2016) Multimodality, learning and communication: A social semiotic frame. Routledge.Google Scholar
Bezemer, J., & Mavers, D.
(2011) Multimodal transcription as academic practice: A social semiotic perspective. International Journal of Social Research Methodology, 13(3), 191–206. DOI logoGoogle Scholar
Boeriis, M., & Holsanova, J.
(2012) Tracking visual segmentation: Connecting semiotic and cognitive perspectives. Visual Communication, 11(3), 259–281. DOI logoGoogle Scholar
Bowcher, W. L., & Zhang, Z.
(2020) Pointing at words: Gestures, language and pedagogy in elementary literacy classrooms in China. Linguistics and Education, 55, 100779. DOI logoGoogle Scholar
Butler, C.
(2006) On functionalism and formalism: A reply to Newmeyer. Functions of Language, 13(2), 97–227. DOI logoGoogle Scholar
Chapelle, C. A.
(2016) Teaching culture in introductory foreign language textbooks. Palgrave Macmillan. DOI logoGoogle Scholar
Chen, Y.
(2022) Salient visual foci on human faces in viewers’ engagement with advertisements: Eye-tracking evidence and theoretical implications. Multimodality and Society, 2(1), 3–22. DOI logoGoogle Scholar
Christie, F.
(2002) Classroom discourse analysis: A functional perspective. Continuum.Google Scholar
Conklin, K., Pellicer-Sánchez, A., & Carrol, G.
(2018) Eye-tracking: A guide for applied linguistics research. Cambridge University Press. DOI logoGoogle Scholar
Cowan, K.
(2014) Multimodal transcription of video: examining interaction in early years classrooms. Classroom Discourse, 5(1), 6–21. DOI logoGoogle Scholar
Darvin, R.
(2020) Creativity and criticality: Reimagining narratives through translanguaging and transmediation. Applied Linguistics Review, 11(4), 581–606. DOI logoGoogle Scholar
Dik, S. C., & Hengeveld, K.
(Eds.) (1997) The theory of functional grammar, Part 1: The structure of the clause. Walter de Gruyter.Google Scholar
Feng, W. D.
(2019) Infusing moral education into English language teaching: An ontogenetic analysis of social values in EFL textbooks in Hong Kong. Discourse: Studies in the Cultural Politics of Education, 40(4), 458–473. DOI logoGoogle Scholar
Flewitt, R.
(2006) Using video to investigate preschool classroom interaction: Education research assumptions and methodological practices. Visual Communication, 5, 25–50. DOI logoGoogle Scholar
Godfroid, A.
(2019) Eye tracking in second language acquisition and bilingualism: A research synthesis and methodological guide (1st ed.). Routledge. DOI logoGoogle Scholar
Halliday, M. A. K.
(1978) Language as social semiotic: The social interpretation of language and meaning. Hodder Education.Google Scholar
Halliday, M. A. K., & Matthiessen, C.
(2004) An introduction to functional grammar (3rd ed.). Edward Arnold.Google Scholar
Holsanova, J.
(2014) Reception of multimodality: Applying eye tracking methodology in multimodal research. In C. Jewitt (Ed.), The Routledge handbook of multimodal analysis (pp. 285–296). Routledge.Google Scholar
Holsanova, J., Rahm, H., & Holmqvist, K.
(2006) Entry points and reading paths on newspaper spreads: Comparing a semiotic analysis with eye-tracking measurements. Visual Communication, 55(1), 65–93. DOI logoGoogle Scholar
Hood, S. E.
(2011) Body language in face-to-face teaching: A focus on textual and interpersonal meaning. In E. A. Thompson, M. Stenglin, & S. Dreyfus (Eds), Semiotic margins: Meaning in multimodalities (pp. 31–52). Continuum.Google Scholar
Jewitt, C.
(2009) Different approaches to multimodality. In C. Jewitt (Ed.). The Routledge handbook of multimodal analysis (pp. 28–39). Routledge.Google Scholar
(2014a) An introduction to multimodality. In C. Jewitt (Ed.). The Routledge handbook of multimodal analysis (pp.15–30). Routledge.Google Scholar
(2014b) Different approaches to multimodality. In C. Jewitt (Ed.). The Routledge handbook of multimodal analysis (pp. 31–43). Routledge.Google Scholar
Jewitt, C., Bezemer, J., & O’Halloran, K.
(2016) Introducing multimodality. Routledge. DOI logoGoogle Scholar
Jiang, L., Yang, M., & Shu, Y.
(2020) Chinese ethnic minority students’ investment in English learning empowered by digital multimodal composing. TESOL Quarterly, 54(4), 954–979. DOI logoGoogle Scholar
Joo, S. J., Chik, A., & Djonov, E.
(2020) The construal of English as a global language in Korean EFL textbooks for primary school children. Asian Englishes, 22(1), 68–84. DOI logoGoogle Scholar
Knoblauch, H., Schnettler, B., Raab, J., & Soeffner, H.-G.
(Eds.) (2009) Video analysis: Methodology and methods; qualitative audiovisual data analysis in sociology (2nd ed.). Peter Lang.Google Scholar
Korhonen, V.
(2010) Dialogic literacy: A sociocultural literacy learning approach. In A. Lloyd & T. Sanna (Eds.), Practising information literacy (pp. 211–226). Chandos. DOI logoGoogle Scholar
Kress, G.
(2000) Design and transformation: New theories of meaning. In B. Cope & M. Kalantzis (Eds.), Multiliteracies: Literacy learning and the design of social futures (pp. 149–158). Routledge.Google Scholar
(2010) Multimodality: A social semiotic approach to contemporary communication. Routledge.Google Scholar
(2014) What is mode? In C. Jewitt (Ed.), The Routledge handbook of multimodal analysis (pp.60–75). Routledge.Google Scholar
Kress, G., Jewitt, C., Bourne, J., Franks, A., Hardcastle, J., Jones, K., & Reid, E.
(2005) English in urban classrooms: A multimodal perspective on teaching and learning. Falmer.Google Scholar
Kress, G., & Selander, S.
(2012) Multimodal design, learning and cultures of recognition. The Internet and Higher Education, 15(4), 265–268. DOI logoGoogle Scholar
Kress, G., & van Leeuwen, T.
(2001) Multimodal discourse: The modes and media of contemporary communication. Edward Arnold.Google Scholar
(2006) Reading images: The grammar of visual design (2nd ed.). Routledge. DOI logoGoogle Scholar
Ledin, P., & Machin, D.
(2018) Doing critical discourse studies with multimodality: From metafunctions to materiality. Critical Discourse Studies, 15(1), 1–17. DOI logoGoogle Scholar
Lemke, J.
(1998) Multiplying meaning: Visual and verbal semiotics in scientific text. In J. R. Martin & R. Vell (Eds.), Reading science: Critical and function (pp. 87–113). Routledge.Google Scholar
Lim, F. V.
(2021a) Designing learning with embodied teaching: Perspectives from multimodality. Routledge.Google Scholar
(2021b) Investigating intersemiosis: A systemic functional multimodal discourse analysis of the relationship between language and gesture in classroom discourse. Visual Communication, 20(1), 34–58. DOI logoGoogle Scholar
Lim, F. V., O’Halloran, K. L., & Podlasov, A.
(2012) Spatial pedagogy: Mapping meanings in the use of classroom space. Cambridge Journal of Education, 42(2), 235–251. DOI logoGoogle Scholar
Miller, S. M., & McVee, M. B.
(Eds.) (2013) Multimodal composing in classrooms: Learning and teaching for the digital world. Routledge. DOI logoGoogle Scholar
Mills, K. A.
(2015) Literacy theories for the digital age: Social, critical, multimodal, spatial, material and sensory lenses. Multilingual Matters. DOI logoGoogle Scholar
New London Group
(1996) A pedagogy of multiliteracies: Designing social futures. Harvard Educational Review, 66(1), 60–92. DOI logoGoogle Scholar
Norris, S.
(2004) Analyzing multimodal interaction: A methodological framework. Routledge. DOI logoGoogle Scholar
O’Halloran, K. L.
(2005) Mathematical discourse: Language, symbolism and visual images. Continuum.Google Scholar
(2008) Systemic functional-multimodal discourse analysis (SF-MDA): Constructing ideational meaning using language and visual imagery. Visual Communication, 7(4), 443–475. DOI logoGoogle Scholar
(2011) Multimodal discourse analysis. In K. Hyland & B. Paltridge (Eds.), Companion to discourse (pp. 120–137). Continuum.Google Scholar
(Ed.) (2004) Multimodal discourse analysis: Systemic-functional perspectives. Continuum.Google Scholar
O’Halloran, K. L., & Lim, F. V.
(2014) Systemic functional multimodal discourse analysis. In S. Norris & C. Maier (Eds.), Texts, images and interactions: A Reader in multimodality (pp. 135–154). De Gruyter. DOI logoGoogle Scholar
O’Halloran, K. L., Podlasov, A., Chua, A., & Marissa, K. L. E.
(2012) Interactive software for multimodal analysis. Visual Communication, 11(3), 363–381. DOI logoGoogle Scholar
O’Toole, M.
(2011) The language of displayed art (2nd ed.). Routledge.Google Scholar
Pirini, J., Matelau-Doherty, T., & Norris, S.
(2018) Multimodal analysis. In A. Phakiti, P. De Costa, L. Plonsky, & S. Starfield, (Eds.), The Palgrave handbook of applied linguistics research methodology (pp. 639–658). Palgrave Macmillan. DOI logoGoogle Scholar
Ranker, J.
(2017) The role of signifier differences, associations, and combinations in creative digital video composing: Making meaning with gestures, objects, actions, and speech. Pedagogies: An International Journal, 12(2), 196–218. DOI logoGoogle Scholar
Stivers, T., & Sidnell, J.
(2005) Introduction: Multimodal interaction. Semotica, 156, 1–20. DOI logoGoogle Scholar
Teo, P., & Zhu, J.
(2018) A multimodal analysis of affect and attitude education in China’s English textbooks. Multimodal Communication, 7(1), 20170014. DOI logoGoogle Scholar
Thomas, M. A.
(2020) Formalism and functionalism in linguistics: The engineer and the collector. Routledge.Google Scholar
Van Leeuwen, T.
(2005) Introducing social semiotics. Routledge.Google Scholar
Weninger, C.
(2021) Multimodality in critical language textbook analysis. Language, Culture & Curriculum, 34(2), 133–146. DOI logoGoogle Scholar
Weninger, C. & Kiss, T.
(2013) Culture in English as a Foreign Language (EFL) textbooks: A semiotic approach. TESOL Quarterly, 47(4), 694–716. DOI logoGoogle Scholar
Wigham, C., & Satar, M.
(2021) Multimodal (inter)action analysis of task instructions in language teaching via videoconferencing: A case study. ReCALL, 33(3), 195–213. DOI logoGoogle Scholar
Wittenburg, P., Brugman, H., Russel, A., Klassmann, A., & Sloetjes, H.
(2006, May 24–26). ELAN: A professional framework for multimodality research [Conference presentation]. The International Conference on Language Resources and Evaluation, Genoa, Italy. [URL]
Xiong, T. & Peng, Y.
(2021) Representing culture in Chinese as a second language textbooks: A critical social semiotic approach. Language, Culture & Curriculum, 34(2), 163–182. DOI logoGoogle Scholar