Cover not available
Article published In:
Text and Context Revisited Within a Multimodal Framework
Edited by Yves Gambier and Olli Philippe Lautenbacher
[Babel 70:1/2] 2024
► pp. 138–163

Towards a corpus-based approach to graphic elements in creative subtitling

A case study of the YouTube channel “Apenjie with Dawang”

| The Hong Kong Polytechnic University
This article explores how a corpus-based approach allows us to describe and analyze the multimodal complexity of graphic elements in creative subtitling. To this end, the article focuses on a YouTube channel, Apenjie with Dawang, featuring a dog and its owner. This channel’s subtitling strategies were experimental with multiple graphic elements (colors, positions, font sizes, and emojis). Informed by a social semiotic approach to multimodality, a corpus of 1,155 coupled pairs of Chinese-English subtitles was annotated for modal shifts and metafunctional shifts. Some major findings include: (a) emojis were much more likely to be added to the target subtitles for the animals than for the humans; (b) speaker-identifying graphic elements (color and position) were lost in the target subtitles, but emojis were systematically added to mark animals as the speakers; (c) the addition of emojis evoked complementary-interpersonal meanings, suggesting that the subtitlers might have prioritized audience engagement over textual fidelity; (d) although the target subtitles used fewer graphic elements, the semiotic meanings could be similar or complementary to those of the source subtitles. Based on these findings, the article also discusses the opportunities and challenges of a corpus-based approach to graphic elements in creative subtitling.
Keywords: creative subtitling, corpus, graphic elements, multimodality, metafunction, mode
Article outline
  • 1.Introduction
  • 2.Research background
    • 2.1A social semiotic approach to multimodality
    • 2.2A corpus-based approach to subtitles
  • 3.The study
    • 3.1Corpus
    • 3.2Coding framework
    • 3.3Coding and analytical procedures
  • 4.Results
    • 4.1Within-group comparisons
    • 4.2Inter-group comparisons
  • 5.Discussion
    • 5.1Graphic elements in creative subtitling
    • 5.2Opportunities and challenges of a corpus-based approach
  • 6.Conclusion
  • Notes
  • References
Published online: 15 December 2023
DOI logo
https://doi.org/10.1075/babel.00382.wu
References (77)
References
Primary sources
Video links for the examples in Figure 3.
Example 1: [URL] (2:44)
Example 2: [URL] (1:20)
Example 3: https://www.youtube.com/watch?v=G_rOvmYpwZ8 (0:12)
Example 4: [URL] (0:05)
Secondary sources
Agresti, Alan. 2013. Categorical Data Analysis. Hoboken: John Wiley & Sons.Google Scholar
Baños, Rocío, Silvia Bruti, and Serenella Zanotti. 2013. “Corpus Linguistics and Audiovisual Translation: In Search of an Integrated Approach.” Perspectives 21 (4): 483–490. DOI logoGoogle Scholar
Bassnett, Susan, Lawrence Venuti, Jan Pedersen, and Ivana Hostová. 2022. “Translation and Creativity in the 21st Century.” World Literature Studies 14 (1): 3–17. DOI logoGoogle Scholar
Bateman, John. 2008. Multimodality and Genre: A Foundation for the Systematic Analysis of Multimodal Documents. London: Palgrave Macmillan. DOI logoGoogle Scholar
Bateman, John A. 2014. “Using Multimodal Corpora for Empirical Research.” In The Routledge Handbook of Multimodal Analysis, edited by Carey Jewitt, 238–252. London: Routledge.Google Scholar
Beckman, Rachel. 2008. “An Out-of-Character Role for Subtitles.” Washington Post, 14 November. [URL]
Bernardini, Silvia, and Adriano Ferraresi. 2022. “Corpus Linguistics.” In The Routledge Handbook of Translation and Methodology, edited by Federico Zanettin and Christopher Rundle, 207–222. London: Routledge. DOI logoGoogle Scholar
Brookes, Gavin, and Tony McEnery. 2020. “Corpus Linguistics.” In The Routledge Handbook of English Language and Digital Humanities, edited by Svenja Adolphs and Dawn Knight, 378–404. London: Routledge. DOI logoGoogle Scholar
Bruti, Silvia. 2020. “Corpus Approaches and Audiovisual Translation.” In The Palgrave Handbook of Audiovisual Translation and Media Accessibility, edited by Łukasz Bogucki and Mikołaj Deckert, 381–395. Cham: Palgrave Macmillan. DOI logoGoogle Scholar
Bywood, Lindsay. 2019. “Testing The Retranslation Hypothesis for Audiovisual Translation: The Films of Volker Schlöndorff Subtitled into English.” Perspectives 27 (6): 815–832. DOI logoGoogle Scholar
Chen, Yuping. 2019. Translating Film Subtitles into Chinese: A Multimodal Study. Singapore: Springer. DOI logoGoogle Scholar
Cohn, Neil. 2013. “Beyond Speech Balloons and Thought Bubbles: The Integration of Text and Image.” Semiotica 1971: 35–63. DOI logoGoogle Scholar
Dahne, Annie, and Roberta Piazza. 2020. “‘Subtitles Have to Become My Ears Not My Eyes’ Pragmatic-Stylistic Choices behind Closed Captions for the Deaf and Hard of Hearing: The Example of Breaking Bad .” In Telecinematic Stylistics, edited by Christian Hoffmann and Monika Kirner-Ludwig, 285–306. London: Bloomsbury. DOI logoGoogle Scholar
Danesi, Marcel. 2017. The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London: Bloomsbury. DOI logoGoogle Scholar
De Linde, Zoé, and Neil Kay. 1999. “Processing Subtitles and Film Images: Hearing vs Deaf Viewers.” The Translator 5 (1): 45–60. DOI logoGoogle Scholar
Díaz Cintas, Jorge. 2018. “‘Subtitling’s a Carnival’: New Practices in Cyberspace.” The Journal of Specialised Translation 301: 127–149.Google Scholar
Díaz Cintas, Jorge, and Aline Remael. 2021. Subtitling: Concepts and Practices. London: Routledge.Google Scholar
Dwyer, Tessa. 2017. Speaking in Subtitles: Revaluing Screen Translation. Edinburgh: Edinburgh University Press.Google Scholar
Field, Andy. 2018. Discovering Statistics Using IBM SPSS Statistics, 5th ed. Los Angeles: Sage.Google Scholar
Foerster, Anna. 2010. “Towards a Creative Approach in Subtitling: A Case Study.” In New Insights into Audiovisual Translation and Media Accessibility, edited by Jorge Díaz Cintas, Anna Matamala, and Josélia Neves, 81–98. Amsterdam: Rodopi. DOI logoGoogle Scholar
Fox, Wendy. 2016. “Integrated Titles: An Improved Viewing Experience.” In Eyetracking and Applied Linguistics, edited by Silvia Hansen-Schirra and Sambor Grucza, 5–29. Berlin: Language Science Press.Google Scholar
. 2018. Can Integrated Titles Improve the Viewing Experience? Investigating the Impact of Subtitling on the Reception and Enjoyment of Film Using Eye Tracking and Questionnaire Data. Berlin: Language Science Press.Google Scholar
Gambier, Yves. 2003. “Introduction: Screen Transadaptation: Perception and Reception. Special Issue on Screen Translation.” The Translator 9 (2): 171–190. DOI logoGoogle Scholar
. 2013. “The Position of Audiovisual Translation Studies.” In The Routledge Handbook of Translation Studies, edited by Carmen Millán and Francesca Bartrina, 45–59. London: Routledge. DOI logoGoogle Scholar
Guillot, Marie-Noëlle. 2019. “Subtitling on the Cusp of its Futures.” In The Routledge Handbook of Audiovisual Translation, edited by Luis Pérez-González, 31–47. London: Routledge.Google Scholar
Halliday, Michael Alexander Kirkwood. 1978. Language as Social Semiotic: The Social Interpretation of Language and Meaning. London: Edward Arnold.Google Scholar
. 2001. “Towards a Theory of Good Translation.” In Exploring Translation and Multilingual Text Production: Beyond Content, edited by Erich Steiner and Colin Yallop, 13–18. Berlin: Mouton de Gruyter. DOI logoGoogle Scholar
Halliday, Michael Alexander Kirkwood, and Christian M. I. M. Matthiessen. 2014. Halliday’s Introduction to Functional Grammar. London: Routledge. DOI logoGoogle Scholar
Hart, Christopher. 2020. “Multimodal Discourse Analysis.” In Researching Discourse: A Student Guide, edited by Christopher Hart, 143–179. London: Routledge. DOI logoGoogle Scholar
Jewitt, Carey, Jeff Bezemer, and Kay O’Halloran. 2016. Introducing Multimodality. London: Routledge. DOI logoGoogle Scholar
Kress, Gunther. 2010. Multimodality: A Social Semiotic Approach to Contemporary Communication. London: Routledge.Google Scholar
. 2014. “What Is a Mode?” In The Routledge Handbook of Multimodal Analysis, edited by Carey Jewitt, 2nd ed., 60–75. London: Routledge.Google Scholar
. 2020. “Transposing Meaning: Translation in a Multimodal Semiotic Landscape.” In Translation and Multimodality: Beyond Words, edited by Monica Boria, Ángeles Carreres, María Noriega-Sánchez, and Marcus Tomalin, 24–48. London: Routledge.Google Scholar
Kress, Gunther, and Theo van Leeuwen. 2006. Reading Images: The Grammar of Visual Design, 2nd ed. London: Routledge. DOI logoGoogle Scholar
Kruger, Jan-Louis. 2021. “Translation, Multimodality and Cognition.” In The Routledge Handbook of Translation and Cognition, edited by Fabio Alves and Arnt Lykke Jakobsen, 433–444. London: Routledge.Google Scholar
Levshina, Natalia. 2017. “Online Film Subtitles as A Corpus: An N-Gram Approach.” Corpora 12 (3): 311–338. DOI logoGoogle Scholar
Logi, Lorenzo, and Michele Zappavigna. 2021. “A Social Semiotic Perspective on Emoji: How Emoji and Language Interact to Make Meaning in Digital Messages.” New Media & Society 25 (12): 3222–3246. DOI logoGoogle Scholar
Malamatidou, Sofia. 2018. Corpus Triangulation: Combining Data and Methods in Corpus-Based Translation Studies. London: Routledge.Google Scholar
Matthiessen, Christian M. I. M. 2021. “Translation, Multilingual Text Production and Cognition Viewed in Terms of Systemic Functional Linguistics.” In The Routledge Handbook of Translation and Cognition, edited by Fabio Alves and Arnt Lykke Jakobsen, 517–544. London: Routledge.Google Scholar
McClarty, Rebecca. 2012. “Towards A Multidisciplinary Approach in Creative Subtitling.” MonTI: Monografías de traducción e interpretación 41: 133–153. DOI logoGoogle Scholar
. 2014. “In Support of Creative Subtitling: Contemporary Context and Theoretical Framework.” Perspectives 22 (4): 592–606. DOI logoGoogle Scholar
McEnery, Tony, and Andrew Hardie. 2012. Corpus Linguistics: Method, Theory and Practice. Cambridge: Cambridge University Press.Google Scholar
Miles, Matthew B., and A. Michael Huberman. 1994. Qualitative Data Analysis: An Expanded Sourcebook. 2 ed. Thousand Oaks, CA: Sage.Google Scholar
Mubenga, Kajingulu. 2009. “Towards a Multimodal Pragmatic Analysis of Film Discourse in Audiovisual Translation.” Meta 54 (3): 466–484. DOI logoGoogle Scholar
Nornes, Abé Mark. 1999. “For an Abusive Subtitling.” Film Quarterly 52 (3): 17–34. DOI logoGoogle Scholar
Pavesi, Maria. 2019. “Corpus-Based Audiovisual Translation Studies: Ample Room for Development.” In The Routledge Handbook of Audiovisual Translation, edited by Luis Pérez-González, 315–333. London: Routledge.Google Scholar
. 2022. “Corpora and The Language of Films: Exploring Dialogue in English and Italian.” In The Routledge Handbook of Corpus Linguistics, 2nd ed., edited by Anne O’Keeffe and Michael J. McCarthy, 547–561. London: Routledge. DOI logoGoogle Scholar
Pedersen, Jan. 2011. Subtitling Norms for Television: An Exploration Focussing on Extralinguistic Cultural References. Amsterdam and Philadelphia: John Benjamins Publishing Company. DOI logoGoogle Scholar
Perego, Elisa. 2019. “Into the Language of Museum Audio Descriptions: A Corpus-Based Study.” Perspectives 27 (3): 333–349. DOI logoGoogle Scholar
Pérez-González, Luis. 2007. “Appraising Dubbed Conversation: Systemic Functional Insights into the Construal of Naturalness in Translated Film Dialogue.” The Translator 13 (1): 1–38. DOI logoGoogle Scholar
. 2014. Audiovisual Translation: Theories, Methods and Issues. London: Routledge. DOI logoGoogle Scholar
Pinnow, Rachel J. 2015. “A Patriot is Respectful: (Re-)Examining the Architecture of Ideology in Educational Contexts.” In International Handbook of Semiotics, edited by Peter Pericles Trifonas, 1249–1269. New York: Springer. DOI logoGoogle Scholar
Pinto, Sara Ramos, and Aishah Mubaraki. 2020. “Multimodal Corpus Analysis of Subtitling: The Case of Non-Standard Varieties.” Target 32 (3): 389–419. DOI logoGoogle Scholar
Remael, Aline, and Nina Reviers. 2020. “Media Accessibility and Accessible Design.” In The Routledge Handbook of Translation and Technology, edited by Minako O’Hagan, 482–497. London: Routledge.Google Scholar
Reviers, Nina. 2018. “Studying the Language of Dutch Audio Description: An Example of a Corpus-Based Analysis.” Translation and Translanguaging in Multilingual Contexts 41: 178–202. DOI logoGoogle Scholar
Romero-Fresco, Pablo. 2009. “Naturalness in the Spanish Dubbing Language: A Case of Not-so-close Friends,” Meta 54 (1), 49–72. DOI logoGoogle Scholar
Secară, Alina. 2011. “RU Ready 4 New Subtitles? Investigating the Potential of Social Translation Practices and Creative Spellings.” Linguistica Antverpiensia, New Series–Themes in Translation Studies 101: 153–171.Google Scholar
Soffritti, Marcello. 2019. “Multimodal Corpora in Audiovisual Translation Studies.” In The Routledge Handbook of Audiovisual Translation, edited by Luis Pérez-González, 334–349. London: Routledge.Google Scholar
Sun, Juhyung, Sarah Lasser, and Sun Kyong Lee. 2022. “Understanding Emojis: Cultural Influences in Interpretation and Choice of Emojis.” Journal of International and Intercultural Communication 16 (3): 246–261.Google Scholar
Taylor, Christopher. 2003. “Multimodal Transcription in the Analysis, Translation and Subtitling of Italian Films.” The Translator 9 (2): 191–205. DOI logoGoogle Scholar
. 2013. “Multimodality and Audiovisual Translation.” In Handbook of Translation Studies, vol. 41, edited by Yves Gambier and Luc van Doorslaer, 98–104. Amsterdam and Philadelphia: John Benjamins Publishing Company. DOI logoGoogle Scholar
. 2016. “The Multimodal Approach in Audiovisual Translation.” Target 28 (2): 222–236. DOI logoGoogle Scholar
. 2020. “Multimodality and Intersemiotic Translation.” In The Palgrave Handbook of Audiovisual Translation and Media Accessibility, edited by Łukasz Bogucki and Mikołaj Deckert, 83–99. Cham: Palgrave Macmillan. DOI logoGoogle Scholar
Thompson, Paul. 2022. “Building A Specialised Audiovisual Corpus.” In The Routledge Handbook of Corpus Linguistics (2nd edition), edited by Anne O’Keeffe and Michael J. McCarthy, 75–88. London: Routledge. DOI logoGoogle Scholar
Tirkkonen-Condit, Sonja, and Jukka Mäkisalo. 2007. “Cohesion in Subtitles: A Corpus-Based Study.” Across Languages and Cultures 8 (2): 221–230. DOI logoGoogle Scholar
Tognini-Bonelli, Elena. 2001. Corpus Linguistics at Work. Amsterdam and Philadelphia: John Benjamins Publishing Company. DOI logoGoogle Scholar
Toury, Gideon. 2012. Descriptive Translation Studies and Beyond, rev. ed. Amsterdam and Philadelphia: John Benjamins Publishing Company. DOI logoGoogle Scholar
Tuominen, Tiina, Catalina Jiménez Hurtado, and Anne Ketola. 2018. “Why Methods Matter: Approaching Multimodality in Translation Research.” Linguistica Antverpiensia: New Series–Themes in Translation Studies 171: 1–21.Google Scholar
Valentini, Cristina. 2007. “Developing AVT Corpora for a Quantitative Approach to Language Transfer in Cinematic Products.” In Selected Papers of the CETRA Research Seminar in Translation Studies 2006, edited by Francis Mus, 1–18. Leuven: CETRA.Google Scholar
Wu, Zhiwei, and Zhuojia Chen. 2022. “Towards a Corpus-Driven Approach to Audiovisual Translation (AVT) Reception: A Case Study of YouTube Viewer Comments.” The Journal of Specialised Translation 381: 128–154.Google Scholar
Zanettin, Federico. 2012. Translation-Driven Corpora: Corpus Resources for Descriptive and Applied Translation Studies. Manchester: St. Jerome.Google Scholar