A systematic review of reliability in corpus-based metadiscourse studies

Basma Bouziri and Ruth Breeze
Université de Jendouba | Université de Sfax | Universidad de Navarra

Metadiscourse has been a major focus of research over the last twenty-five years, attracting methodological approaches from the areas of textual pragmatics and discourse studies, many of which are supported by corpus linguistics. A major challenge in corpus-based discourse studies, however, is subjectivity, which may affect their quality and undermine their methodological rigor. To reduce subjectivity and guarantee consistency, assessing reliability of coding is essential. This study advocates combining quantitative with qualitative approaches to reliability. We argue that this mixed-method approach will provide a better assessment of reliability. To this aim, this methodological synthesis surveyed research covering empirical corpus-based studies on metadiscourse published in indexed and peer-reviewed journals. One major finding is that most studies did not report conducting any reliability measure. Issues in reliability accounts were also identified for those that did. Another major finding is a pervasive lack of transparency and comprehensiveness in reliability reports. Recommendations for enhancing reliability are listed.

Publication history

Date received: 6 January 2025

Date accepted: 2 October 2025

Published online: 5 May 2026

Table of contents

In this section, we provide a description of the state of the art as far as reliability measurements in general and reliability in discourse and pragmatic studies in particular are concerned. We then move to advocate for the need for a mixed-method approach to reliability that will enable a holistic account of reliability in corpus-based metadiscourse studies.

Full-text access is restricted to subscribers. Log in to obtain additional credentials. For subscription information see Subscription & Price. Direct PDF access to this article can be purchased through our e-platform.

References

Ahmed, Abdelhamid M., and Xiao Zhang
2023 “Students’ Voice in L2 English Writing: A Systematic Review of Literature.” Ampersand 10. Google Scholar logo with link to Google Scholar
Belur, Jyoti, Lisa Tompson, Amy Thornton, and Miranda Simon
2021 “Interrater Reliability in Systematic Review Methodology: Exploring Variation in Coder Decision-Making.” Sociological Methods & Research 50 (2): 837–65. Google Scholar logo with link to Google Scholar
Bouziri, Basma
2019 “A Corpus-Assisted Genre Analysis of the Tunisian Lecture Corpus: Focus on Metadiscourse.” PhD dissertation. Université Catholique de Louvain.
2020 “Topic Signaling Metadiscourse Devices in the Tunisian Lecture Corpus.” ESP Today 8 (2): 227–49. Google Scholar logo with link to Google Scholar
2021 “A Tripartite Interpersonal Model for Investigating Metadiscourse in Academic Lectures.” Applied Linguistics 42 (5): 970–89. WOS:000727808600008. Google Scholar logo with link to Google Scholar
Callies, Marcus
2015 “Learner Corpus Methodology.” In The Cambridge Handbook of Learner Corpus Research, edited by Sylviane Granger, Gaetanelle Gilquin, and Fanny Meunier, 35–55. Cambridge University Press. Google Scholar logo with link to Google Scholar
Chong, Sin Wang, and Luke Plonsky
2023 “A Typology of Secondary Research in Applied Linguistics.” Applied Linguistics Review 15 (4): 1569–94. Google Scholar logo with link to Google Scholar
Chong, Sin Wang, and Luke Plonsky
2024 “Research Synthesis in Language Education.” In Reference Module in Social Sciences. Elsevier. Google Scholar logo with link to Google Scholar
Crismore, Avon, Raija Markkanen, and Margaret S. Steffensen
1993 “Metadiscourse in Persuasive Writing: A Study of Texts Written by American and Finnish University Students.” Written Communication 10 (1): 39–71. Google Scholar logo with link to Google Scholar
Feng, Guangchao Charles
2015 “Mistakes and How to Avoid Mistakes in Using Intercoder Reliability Indices.” Methodology 11 (1): 13–22. Google Scholar logo with link to Google Scholar
Freelon, Deen
2010 “ReCal: Intercoder Reliability Calculation as a Web Service.” International Journal of Internet Science 5 (1): 20–33. http://​dfreelon​.org​/utils​/recalfront​/recal2/
Grisot, Cristina
2017 “A Quantitative Approach to Conceptual, Procedural and Pragmatic Meaning: Evidence from Inter-Annotator Agreement.” Journal of Pragmatics 117 (August): 245–63. Google Scholar logo with link to Google Scholar
Hober, Nicole, Tülay Dixon, and Tove Larsson
2023 “Towards Increased Reliability and Transparency in Projects with Manual Linguistic Coding.” Corpora 18 (2): 245–58. Google Scholar logo with link to Google Scholar
Hovy, E., and Julia Lavid
2010 “Towards a ‘Science’ of Corpus Annotation: A New Methodological Challenge for Corpus Linguistics.” International Journal of Translation 22 (1).Google Scholar logo with link to Google Scholar
Hunston, Susan
1999 “Evaluation and the Planes of Discourse: Status and Value in Persuasive Texts.” In Evaluation in Text: Authorial Stance and the Construction of Discourse, edited by Geoff Thompson, and Susan Hunston, 176–207. Oxford University Press.Google Scholar logo with link to Google Scholar
Hyland, Ken
2005 Metadiscourse: Exploring Interaction in Writing. Continuum Discourse Series.Google Scholar logo with link to Google Scholar
Landis, J. Richard, and Gary G. Koch
1977 “An Application of Hierarchical Kappa-type Statistics in the Assessment of Majority Agreement among Multiple Observers.” Biometrics 33 (2): 363–374.Google Scholar logo with link to Google Scholar
Larsson, Tove, Magali Paquot, and Luke Plonsky
2020 “Inter-Rater Reliability in Learner Corpus Research: Insights from a Collaborative Study on Adverb Placement.” International Journal of Learner Corpus Research 6 (2): 237–51. Google Scholar logo with link to Google Scholar
Liu, Yanhua, and Guangwei Hu
2021 “Mapping the Field of English for Specific Purposes (1980–2018): A Co-Citation Analysis.” English for Specific Purposes 61: 97–116. Google Scholar logo with link to Google Scholar
Norouzian, Reza
2021 “Interrater Reliability in Second Language Meta-Analyses: The Case of Categorical Moderators.” Studies in Second Language Acquisition 43 (4): 896–915. Google Scholar logo with link to Google Scholar
Paltridge, Brian, and Aek Phakiti
eds. 2015 Research Methods in Applied Linguistics: A Practical Resource. Bloomsbury Academic.Google Scholar logo with link to Google Scholar
Pearson, William S., and Esmaeel Abdollahzadeh
2023 “Metadiscourse in Academic Writing: A Systematic Review.” Lingua 293. Google Scholar logo with link to Google Scholar
Perez Aguilar, Marta, and Elisabeth Macia Arno
2002 “Metadiscourse in Lecture Comprehension: Does It Really Help Foreign Language Learners?” Atlantis 24 (1): 7–21.Google Scholar logo with link to Google Scholar
Plonsky, Luke, and Frederick Oswald
2015 “Meta-Analyzing Second Language Research.” In Methodological Synthesis in Quantitative L2 Research: A Review of Reviews and a Case Study of Exploratory Factor Analysis. Routledge, 106–128. Google Scholar logo with link to Google Scholar
Plonsky, Luke, and Deirdre J. Derrick
2016 “A Meta-Analysis of Reliability Coefficients in Second Language Research.” The Modern Language Journal 100 (2): 538–53. Google Scholar logo with link to Google Scholar
Rodwell, Mary K.
1998 Social Work Constructivist Research. Garland Publishing.Google Scholar logo with link to Google Scholar
Schröter, Thorsten, and Agnieszka Jablonska Eklöf
2020 “Assessment in English for Young Learners in Sweden: Guidelines, Challenges and Coping Strategies.” Educare 3 (October): 1–45. Google Scholar logo with link to Google Scholar
Spooren, Wilbert, and Liesbeth Degand
2010 “Coding Coherence Relations: Reliability and Validity.” Corpus Linguistics and Linguistic Theory 6 (2): 241–266. Google Scholar logo with link to Google Scholar
Swales, John M.
2019 “The Futures of EAP Genre Studies: A Personal Viewpoint.” Journal of English for Academic Purposes 38 (March): 75–82. Google Scholar logo with link to Google Scholar
Wood, Margaret, Tove Larsson, Luke Plonsky, Scott Sterling, Merja Kytö, and Katherine Yaw
2024 Addressing Questionable Research Practices in Applied Linguistics: A Practical Guide. Applied Linguistics Press.Google Scholar logo with link to Google Scholar
 
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue