Cover not available
Article published In:
ITL - International Journal of Applied Linguistics
Vol. 173:1 (2022) ► pp.2–17

Multimodal reading and second language learning

ORCID logo | University College London
Most of the texts that second language learners engage with include both text (written and/or spoken) and images. The use of images accompanying texts is believed to support reading comprehension and facilitate learning. Despite their widespread use, very little is known about how the presentation of multiple input sources affects the attentional demands and the underlying cognitive processes involved. This paper provides a review of research on multimodal reading, with a focus on attentional processing. It first introduces the relevant theoretical frameworks and empirical evidence provided in support of the use of pictures in reading. It then reviews studies that have looked at the processing of text and pictures in first and second language contexts. Based on this review, main gaps in research and future research directions are identified. The discussion provided in this paper aims at advancing research on multimodal reading in a second language. Achieving a better understanding of the underlying cognitive processes in multimodal reading is crucial to inform pedagogical practices and to develop theoretical accounts of second language multimodal reading.
Keywords: multimodality, second language learning, reading, eye-tracking, cognitive processes, multimedia learning, imagery
Article outline
  • Introduction
  • Multimodality in reading
  • Multimodal reading in second language learning
  • Cognitive processing of multimodal texts
  • Research gaps and future directions
    • Purposes of reading
    • Relationship between attentional allocation and learning
    • Image-Text relationship
  • Conclusion
  • References
Published online: 1 February 2022
DOI logo
https://doi.org/10.1075/itl.21039.pel
References (70)
References
Abraham, P., & Farías, M. (2017). Reading with eyes wide open: Reflections on the impact of multimodal texts on second language reading. Íkala, Revista de Lenguaje y Cultura, 22 (1), 57–70. DOI logoGoogle Scholar
Alemdag, E., & Cagiltay, K. (2018). A systematic review of eye tracking research on multimedia learning. Computers & Education, 125 1, 413–428. DOI logoGoogle Scholar
Bezemer, J., & Kress, G. (2008). Writing in multimodal texts: A social semiotic account of designs for learning. Written Communication, 25 (2), 166–195. DOI logoGoogle Scholar
Bisson, M-J., W. Van Heuven, K. Conklin & R. Tunney. (2015). The role of verbal and pictorial information in multimodal incidental acquisition of foreign language vocabulary. Quarterly Journal of Experimental Psychology 68 1, 1306–1326. DOI logoGoogle Scholar
Boerma, I. E., Mol, S. E., & Jolles, J. (2016). Reading pictures for story comprehension requires mental imagery skills. Frontiers in Psychology, 7 (1630), 1–10. DOI logoGoogle Scholar
Buck, G. (2001). Assessing listening. New York, NY: Cambridge University Press. DOI logoGoogle Scholar
Center, Y., Freeman, L., Robertson, G., & Outhred, L. (1999). The effect of visual imagery training on the reading and listening comprehension of low listening comprehenders in year 2. Journal of Research in Reading 22 1, 241–256. DOI logoGoogle Scholar
Chang, Y., & Choi, S. (2014). Effects of seductive details evidenced by gaze duration. Neurobiology of Learning and Memory, 109 1, 131–138. DOI logoGoogle Scholar
Chen, S. C., Hsiao, M. S., & She, H. C. (2015). The effects of static versus dynamic 3D representations on 10th grade students’ atomic orbital mental model construction: Evidence from eye movement behaviors. Computers in Human Behavior, 53 1, 169–180. DOI logoGoogle Scholar
Clark, J. M., & Paivio, A. (1991). Dual coding theory and education. Educational Psychology Review, 3 1, 149–210. DOI logoGoogle Scholar
Eitel, A., & Scheiter, K. (2015). Picture or text first? Explaining sequence effects when learning with pictures and text. Educational Psychology Review, 27 1, 153–180. DOI logoGoogle Scholar
Elley, W. B. & F. Mangubhai. (1983). The impact of reading on second language learning. Reading Research Quarterly 19 (1), 53–67. DOI logoGoogle Scholar
Evans, M. A. & J. Saint-Aubin. (2005). What children are looking at during shared storybook reading: Evidence from eye movement monitoring. Psychological Science 16 1, 913–920. DOI logoGoogle Scholar
Grabe, W. (2006). Areas of research that influence L2 reading instruction. In Usó-Juan, E. & Martínez-Flor, A. (Eds.), Current trends in the development and teaching of the four language skills (pp. 279–301). New York: Mouton de Gruyter. DOI logoGoogle Scholar
(2009). Reading in a second language: Moving from theory to practice. Cambridge: Cambridge University Press.Google Scholar
Godfroid, A., Boers, F., & Housen, A. (2013). An eye for words: Gauging the role of attention in incidental L2 vocabulary acquisition by means of eye-tracking. Studies in Second Language Acquisition, 35 (3), 483–517. DOI logoGoogle Scholar
Guijarro, A. J. M., & Sanz, M. J. P. (2009). In interaction of image and verbal text in a picture book. A multimodal and systemic functional study. In Ventola, E., & Guijarro, A. J. M. (Eds.), The world told and the world shown (pp. 107–123). London: Palgrave Macmillan.Google Scholar
Hannus, M., & Hyönä, J. (1999). Utilization of illustrations during learning of science textbook passages among low- and high-ability children. Contemporary Educational Psychology, 24 (2), 95–123. DOI logoGoogle Scholar
Hill, D. (2013). Graded readers. ELT Journal 67 (1), 85–125. DOI logoGoogle Scholar
Holsanova, J. (2014). Reception of multimodality: Applying eye tracking methodology in multimodal research. In C. Jewitt (Ed.), The Routledge Handbook of Multimodal Analysis (pp. 287–298). New York: Routledge.Google Scholar
Johnson, C. I., & Mayer, R. E. (2012). An eye movement analysis of the spatial contiguity effect in multimedia learning. Journal of Experimental Psychology: Applied, 18 1, 178–191. DOI logoGoogle Scholar
Justice, L. M., Skibbe, L., Canning, A., & Lankford, C. (2005). Pre-schoolers, print and storybooks: An observational study using eye movement analysis. Journal of Research in Reading, 28 1, 229–243. DOI logoGoogle Scholar
Kress, G. (2000). Multimodality: A social semiotic approach to communication. London: Routledge.Google Scholar
Kress, G. & van Leeuwen, T. (2021). Reading images: The grammar of visual design. New York: Routledge.Google Scholar
Lee, H., & Mayer, R. (2018). Fostering learning from instructional video in a second language. Applied Cognitive Psychology, 32 1, 648–654. DOI logoGoogle Scholar
Lee, M., & Révész, A. (2018). Promoting grammatical development through textually enhanced captions: An eye-tracking study. The Modern Language Journal, 102 1, 557–577. DOI logoGoogle Scholar
Makransky, G., Terkildsen, T. S., & Mayer, R. E. (2019). Role of subjective and objective measures of cognitive processing during learning in explaining the spatial contiguity effect. Learning and Instruction, 61 1, 23–34. DOI logoGoogle Scholar
Mason, L., Pluchino, P., Tornatora, M. C., & Ariasi, N. (2013a). An eye-tracking study of learning science text with concrete and abstract illustrations. Journal of Experimental Education, 81 1, 356–384. DOI logoGoogle Scholar
Mason, L., Tornatora, M. C., & Pluchino, P. (2013b). Do fourth graders integrate text and picture in processing and learning from an illustrated science text? Evidence from eye-movement patterns. Computers and Education, 60 1, 95–109. DOI logoGoogle Scholar
(2015). Integrative processing of verbal and graphical information during re-reading predicts learning from illustrated text: An eye movement study. Reading and Writing, 28 1, 851–872. DOI logoGoogle Scholar
Mayer, R. E. (2001). Multimedia learning. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
(2009). Multimedia learning (2nd ed.). Cambridge: Cambridge University Press. DOI logoGoogle Scholar
(2014a). Introduction to multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed.) (pp. 1–24). Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Mayer, R. (2014b). Cognitive theory of multimedia learning. In R. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 43–71). Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Mayer, R. E. (2021). Multimedia learning (3rd ed.). Cambridge: Cambridge University Press.Google Scholar
Mayer, R. E., & Anderson, R. B. (1991). Animations need narrations: An experimental test of a dual-coding hypothesis. Journal of Educational Psychology, 83 (4), 484–490. DOI logoGoogle Scholar
Mayer, R. E., Howarth, J. T., Kaplan, M., & Hanna, S. (2018). Applying the segmenting principle to online geography slideshow lessons. Educational Technology Research and Development, 66 (3), 563–577. DOI logoGoogle Scholar
Mayer, R. E., & Moreno, R. (1998). A split-attention effect in multimedia learning: Evidence for dual processing systems in working memory. Journal of Educational Psychology, 90 (2), 312–320. DOI logoGoogle Scholar
Mills, K., & Unsworth, L. (2017). Multimodal literacy. Oxford research encyclopedia of education. Retrieved 22 Jul. 2021, from: [URL]
Montero Perez, M. (2020). Multimodal input in SLA research. Studies in Second Language Acquisition, 42 (3), 653–663. DOI logoGoogle Scholar
Montero Perez, M., Van Den Noortgate, W., & Desmet, P. (2013). Captioned video for L2 listening and vocabulary learning: A meta-analysis. System, 41 1, 720–739. DOI logoGoogle Scholar
Montero Perez, M., Peters, E., & Desmet, P. (2015). Enhancing vocabulary learning through captioned video: an eye-tracking study. The Modern Language Journal, 99 (2), 308–28. DOI logoGoogle Scholar
Moreno, R., & Mayer, R. E. (1999). Multimedia-supported metaphors for meaning making in mathematics. Cognition and Instruction, 17 (3), 215–248. DOI logoGoogle Scholar
(2002a). Learning science in virtual reality multimedia environments: Role of methods and media. Journal of Educational Psychology, 94 (3), 598–610. DOI logoGoogle Scholar
(2002b). Verbal redundancy in multimedia learning: When reading helps listening. Journal of Educational Psychology, 94 (1), 156–163. DOI logoGoogle Scholar
Omaggio, A. C. (1979). Pictures and second language comprehension: Do they help? Foreign Language Annals, 12 1, 107–116. DOI logoGoogle Scholar
Paivio, A. (1986). Mental representations: A dual coding approach. Oxford: Oxford University Press.Google Scholar
(2006). Mind and its evolution: A dual coding approach. Lawrence Erlbaum.Google Scholar
Pellicer-Sánchez, A. (2016). Incidental L2 vocabulary acquisition from and while reading: An eye-tracking study. Studies in Second Language Acquisition, 38 1, 97–130. DOI logoGoogle Scholar
Pellicer-Sánchez, A., Conklin, K., Rodgers, M. P., & Parente, F. (2021), The Effect of Auditory Input on Multimodal Reading Comprehension: An Examination of Adult Readers’ Eye Movements. The Modern Language Journal, 105(4), 936–956: DOI logoGoogle Scholar
Pellicer-Sánchez, A., Tragant, E., Conklin, K., Rodgers, M., Llanes, A., & Serrano, R. (2018). L2 reading and reading-while-listening in multimodal learning conditions: An eye-tracking study. ELT Research Papers, 18 (01), 1–28. London: British Council.Google Scholar
Pellicer-Sánchez, A., Tragant, E., Conklin, K., Rodgers, M., Serrano, R., & Llanes, A. (2020). Young learners’ processing of multimodal input and its impact on reading comprehension: An eye-tracking study. Studies in Second Language Acquisition, 42 (3), 577–598. DOI logoGoogle Scholar
Peters, E., & Muñoz, C. (2020). Introduction to the special issue: Language learning from multimodal input. Studies in Second Language Acquisition, 42 (3), 489–497. DOI logoGoogle Scholar
Plass, J. L., Chun, D. M., Mayer, R. E., & Leutner, D. (1998). Supporting visual and verbal learning preferences in a second language multimedia learning environment. Journal of Educational Psychology, 90 1, 25–36. DOI logoGoogle Scholar
Puimège, E., Montero Perez, M., & Peters, E. (2021). Promoting L2 acquisition of multiword units through textually enhanced audiovisual input: An eye-tracking study. Second Language Research. Online First. DOI logoGoogle Scholar
Rodgers, M. P. H., & Webb, S. (2017). The effects of captions on EFL learners’ comprehension of English-language television programs. Calico Journal, 34 1, 20–38. DOI logoGoogle Scholar
Rodgers, M. P. H. & Webb, S. (2019). Incidental vocabulary learning through viewing television. ITL-International Journal of Applied Linguistics, 171 (2), 191–220. DOI logoGoogle Scholar
Roy-Charland, A., Saint-Aubin, J., & Evans, M. A. (2007). Eye movements in shared book reading with children from kindergarten to grade 4. Reading and Writing, 20 1, 909–931. DOI logoGoogle Scholar
Schmidt-Weigand, F. (2011). Does animation amplify the modality effect–or is there any modality effect at all? Zeitschrift für Padagogische Psychologie, 25 (4), 245–256. DOI logoGoogle Scholar
Schmidt-Weigand, F., Kohnert, A., & Glowalla, U. (2010). A closer look at split visual attention in system-and self-paced instruction in multimedia learning. Learning and Instruction, 20 (2), 100–110. DOI logoGoogle Scholar
Schnotz, W., & Baadte, C. (2008). Domain learning versus language learning with multimedia. In M. Farías & K. Obilinovic (Eds.), Aprendizaje multimodal/Multimodal learning (pp. 21–49). Santiago de Chile: Publifahu USACH.Google Scholar
Seburn, T. (2017). Learner-sourced visuals for deeper text engagement and conceptual comprehension. In K. Donaghy & D. Xerri (Eds.), The image in English language teaching (pp. 79–88). Malta: Gutenberg Press.Google Scholar
Serrano, R., & Pellicer-Sánchez, A. (2019). Young L2 learners’ online processing of information in a graded reader during reading-only and reading-while-listening conditions: A study of eye movements. Applied Linguistics Review. Advance online publication. DOI logoGoogle Scholar
Takacs, Z. K., & Bus, A. G. (2016). Benefits of motion in animated storybooks for children’s visual attention and story comprehension. An eye-tracking study. Frontiers in Psychology, 7 1, 1–12. DOI logoGoogle Scholar
Tang, G. (1992). The effect of graphic representation of knowledge structures on ESL reading comprehension. Studies in Second Language Acquisition, 14 1, 177–195. DOI logoGoogle Scholar
Tragant, E., & Pellicer-Sánchez, A. (2019). Young learners’ engagement with multimodal exposure: An eye tracking study. System, 80 1, 212–223. DOI logoGoogle Scholar
Unsworth, L. (2006). Towards a metalanguage for multiliteracies education: Describing the meaning – making resources of language-image interaction. English Teaching, 5 (1), 55–76.Google Scholar
(2014). Multimodal reading comprehension: Curriculum expectations and large-scale literacy testing practices. Pedagogies: An International Journal, 9 (1), 26–44. DOI logoGoogle Scholar
Winke, P., Gass, S., & Sydorenko, T. (2010). The effects of captioning videos used for foreign language listening activities. Language Learning & Technology, 14 1, 65–86.
CrossRef DOI logo with hyperlink to permanent DOI
Google Scholar
Wright, A. (2010). Pictures for language learning. Cambridge: Cambridge University Press.Google Scholar
Cited by (7)

Cited by seven other publications

Avello, Daniela & Carmen Muñoz
2024. The development of L1 and L2 reading skills from captioned video viewing in primary school EFL learners. In Audiovisual input and second language learning [Language Learning & Language Teaching, 61], ► pp. 25 ff. DOI logo
Vanbuel, Marieke & Bart Deygers
2024. Assessing Receptive Skills in an Adult L2 Learner Population with Diverse Educational Backgrounds. Language Assessment Quarterly 21:4-5 ► pp. 379 ff. DOI logo
Cárdenas-Claros, Mónica Stella, Tetyana Sydorenko, Elizabeth Huntley & Maribel Montero Perez
2023. Teachers’ voices on multimodal input for second or foreign language learning. Language Teaching Research DOI logo
Marzban, Sheida, Habil Gyöngyi Fábián & Béla Weiss
2023. 2023 Symposium on Eye Tracking Research and Applications, ► pp. 1 ff. DOI logo
Reynolds, Barry Lee
2023. Introduction: A Scoping Review of Second Language Vocabulary Learning in the Wild. In Vocabulary Learning in the Wild, ► pp. 1 ff. DOI logo
Reynolds, Barry Lee, Xiaowen (Serina) Xie & Quy Huynh Phu Pham
2023. The potentials for incidental vocabulary acquisition from listening to computer science academic lectures: a higher education corpus-based case study from Macau. Frontiers in Psychology 14 DOI logo
Reynolds, Barry Lee, Yanwen Cui, Chian-Wen Kao & Nathan Thomas
2022. Vocabulary Acquisition through Viewing Captioned and Subtitled Video: A Scoping Review and Meta-Analysis. Systems 10:5 ► pp. 133 ff. DOI logo

This list is based on CrossRef data as of 8 january 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.