Article published In:
Functions of LanguageVol. 25:2 (2018) ► pp.229–258
A stepwise method for annotating appraisal
Despite a growing awareness of methodological issues, the literature on appraisal has not so far provided adequate answers to some of the key challenges involved in reliably identifying and classifying evaluative language expressions. This article presents a stepwise method for the manual annotation of appraisal in text that is designed to optimize reliability, replicability and transparency. The procedure consists of seven steps, from the creation of a context-specific annotation manual to the statistical analysis of the quantitative data derived from the manually-performed annotations. By presenting this method, the article pursues the twofold purpose of (i) providing a practical tool that can facilitate more reliable, replicable and transparent analyses, and (ii) fostering a discussion of the best practices that should be observed when manually annotating appraisal.
Article outline
- 1.Introduction
- 2.Challenges in annotating appraisal
- 2.1Challenges in identifying appraisal
- 2.2Challenges in classifying appraisal
- 2.3Reliability, replicability, transparency
- 3.The stepwise annotation method
- 3.1General principles
- Principle 1.All choices should be accounted for
- Principle 2.The annotation guidelines should be tested and refined until maximum reliability is achieved
- Principle 3.Reliability should always be assessed, and reliability scores reported and discussed
- 3.2An outline of the steps
- 3.2.1Step 1: Define the scope of the project
- 3.2.2Step 2: Select and configure an annotation tool
- 3.2.3Step 3: Draft a context-specific annotation manual
- 3.2.4Step 4: Assess reliability
- 3.2.5Step 5: Refine the annotation manual
- 3.2.6Step 6: Annotate the corpus
- 3.2.7Step 7: Analyze the results
- 3.1General principles
- 4.Conclusion
- Acknowledgements
- Notes
-
References
References
References
Artstein, Ron & Massimo Poesio
2008 Inter-coder agreement for computational linguistics. Computational Linguistics 34(4). 555–596. 
Baayen, R. Harald
2008 Analyzing linguistic data: A practical introduction to Statistics using R. Cambridge: CUP.
Bartalesi Lenzi, Valentina, Giovanni Moretti & Rachele Sprugnoli
2012 CAT: the CELCT Annotation Tool. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk & Stelios Piperidis (eds.), Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC12), Istanbul: European Language Resources Association (ELRA). Available online at: [URL] [Accessed 22 September 2015]
Bednarek, Monika
2006 Evaluation in media discourse: Analysis of a newspaper corpus. London: Continuum.
2008 Emotion talk across corpora. Basingstoke: Palgrave.
2014 An astonishing season of destiny! Evaluation in blurbs used for advertising TV series. In Geoff Thompson & Laura Alba-Juez (eds.), 197–220.
Ben-Aaron, Diana
2005 Given and news: Evaluation in newspaper stories about national anniversaries. Text & Talk 25(5). 691–718.
Carretero, Marta & Maite Taboada
2014 Graduation within the scope of Attitude in English and Spanish consumer reviews of books and movies. In Geoff Thompson & Laura Alba-Juez (eds.), 221–239.
Cohen, Jacob
1960 A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1). 37–46.
Cozby, Paul & Scott Bates
2011 Methods in behavioral research, 11th edn. New York, NY: McGraw-Hill.
Dice, Lee R.
1945 Measures of the amount of ecologic association between species. Ecology 26(3). 297–302.
Divjak, Dagmar
2006 Ways of intending: Delineating and structuring near synonyms. In Stefan Th. Gries & Anatol Stefanowitsch (eds.), Corpora in cognitive linguistics: Corpus-based approaches to syntax and lexis, 19–56. Berlin: Mouton.
Don, Alexanne
2007 An approach to the analysis of textual identity through profiles of evaluative disposition. In Proceedings of the Australian Systemic Functional Linguistics Association 2007 Conference. Available at: [URL]. [Accessed 22 September 2015]
Fuoli, Matteo
2012 Assessing social responsibility: A quantitative analysis of Appraisal in BP’s and IKEA’s social reports. Discourse & Communication 6(1). 55–81.
Fuoli, Matteo & Dylan Glynn
2013 Computer-assisted manual annotation of evaluative language expressions: Bridging discourse and corpus approaches. Paper presented at the Evaluative Language and Corpus Linguistics Workshop
, Lancaster University, 22 July 2013.
Fuoli, Matteo & Charlotte Hommerberg
2015 Optimizing transparency, reliability and replicability: Annotation principles and inter-coder agreement in the quantification of evaluative expressions. Corpora 10(3). 315–349.
Fuoli, Matteo & Carita Paradis
2014 A model of trust-repair discourse. Journal of Pragmatics 741. 52–69.
Geeraerts, Dirk, Stefan Grondelaers & Peter Bakema
1994 The structure of lexical variation. Berlin: Mouton.
Glynn, Dylan
2009 Polysemy, syntax, and variation: A usage-based method for Cognitive Semantics. In Vyvyan Evans & Stéphanie Pourcel (eds.), New Directions in Cognitive Linguistics, 77–104. Amsterdam: Benjamins.
Gries, Stefan Th.
1999 Particle movement: A cognitive and functional approach. Cognitive Linguistics 10(2). 105–146.
2013 Statistics for linguistics with R: A practical introduction. Berlin: Mouton.
Hommerberg, Charlotte & Alexanne Don
2015 Appraisal and the language of wine appreciation: A critical discussion of the potential of the Appraisal framework as a tool to analyse specialised genres. Functions of Language 22(2). 161–191.
Hood, Susan
2004 Appraising research: Taking a stance in academic writing. Sydney: University of Technology PhD thesis.
Hunston, Susan
2004 Counting the uncountable: Problems of identifying evaluation in a text and in a corpus. In Alan Partington, John Morley & Louann Haarman (eds.), Corpora and Discourse, 157–188. Bern: Peter Lang.
2011 Corpus approaches to evaluation: Phraseology and evaluative language. New York, NY: Taylor & Francis.
Kirk, Jerome & Marc L. Miller
1986 Reliability and validity in qualitative research. Beverly Hills, CA: Sage.
Krippendorf, Klaus
2004 Content Analysis: An introduction to its methodology. Thousand Oaks, CA: Sage.
Lipovsky, Caroline
2008 Constructing affiliation and solidarity in job interviews. Discourse & Communication 2(4). 411–432.
2011
It’s really a great presentation!: Appraising candidates in job interviews. Linguistics and the Human Sciences 4(2). 161–185.
2013 Negotiating ones expertise through appraisal in CVs. Linguistics and the Human Sciences 8(3). 307–333.
Mackay, James & Jean Parkinson
2009
My very own mission impossible: An APPRAISAL analysis of student teacher reflections on a design and technology project. Text & Talk 29(6). 729–753.
Macken-Horarik, Mary & Anne Isaac
2014 Appraising Appraisal. In Geoff Thompson & Laura Alba-Juez (eds.), 67–92.
Marshall, Christine, Ralph Adendorff & Vivian de Klerk
2010 The role of APPRAISAL in the NRF Rating System: An analysis of Judgement and Appreciation in peer reviewers’ reports. Southern African Linguistics and Applied Language Studies 27(4). 391–412.
Martin, J. R.
2000 Beyond exchange: Appraisal systems in English. In Susan Hunston & Geoff Thompson (eds.), Evaluation in text: Authorial stance and the construction of discourse, 142–175. Oxford: OUP.
Martin, J. R. & David Rose
2003 Working with discourse: Meaning beyond the clause. London: Continuum.
Martin, J. R. & Peter R. R. White
2005 The language of evaluation: Appraisal in English. London: Palgrave Macmillan.
Mauranen, Anna & Marina Bondi
2003 Evaluative language use in academic discourse. Journal of English for Academic Purposes 2(4). 269–271.
O’Donnell, Mick
2008 Demonstration of the UAM CorpusTool for text and image annotation. Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Demo Session. 13–16.
2012 Appraisal analysis and the computer. Revista Canaria de Estudios Ingleses 651. 115–130.
2014 Exploring identity through Appraisal Analysis: A corpus annotation methodology. Linguistics and the Human Sciences 9(1). 95–116.
Page, Ruth E.
2003 An analysis of APPRAISAL in childbirth narratives with special consideration of gender and storytelling style. Text & Talk 23(2). 211–237.
Paradis, Carita
2001 Adjectives and boundedness. Cognitive Linguistics 12(1). 47–64.
Paradis, Carita, Joost van de Weijer, Caroline Willners & Magnus Lindgren
2012 Evaluative polarity of antonyms. Lingue e linguaggio 11(2). 199–214.
Pounds, Gabrina
2010 Attitude and subjectivity in Italian and British hardnews reporting: The construction of a culture-specific ‘reporter’ voice. Discourse Studies 12(1). 106–137.
2011
This property offers much character and charm: Evaluation in the discourse of online property advertising. Text & Talk 31(2). 195–220.
Pustejovsky, James & Amber Stubbs
2012 Natural language annotation for machine learning. Sebastopol, CA: O’Reilly Media.
Read, Jonathon & John Carroll
2012 Annotating expressions of Appraisal in English. Language Resources and Evaluation 46(3). 421–447.
Ryshina-Pankova, Marianna
2014 Exploring argumentation in course-related blogs through ENGAGEMENT. In Geoff Thompson & Laura Alba-Juez (eds.), 281–302.
Santamaría-García, Carmen
2014 Evaluative discourse and politeness in university students’ communication through social networking sites. In Geoff Thompson & Laura Alba-Juez (eds.), 387–411.
Taboada, Maite & Marta Carretero
2012 Contrastive analyses of evaluation in text: Key issues in the design of an annotation system for ATTITUDE applicable to consumer reviews in English and Spanish. Linguistics and the Human Sciences 6(1–3). 275–295.
Taboada, Maite, Marta Carretero & Jennifer Hinnell
2014 Loving and hating the movies in English, German and Spanish. Languages in Contrast 14(1). 127–161.
Thompson, Geoff
2014 AFFECT and emotion, target-value mismatches, and Russian dolls: Refining the APPRAISAL model. In Geoff Thompson & Laura Alba-Juez (eds.), 47–66.
Thompson, Geoff & Laura Alba-Juez
Thompson, Geoff & Susan Hunston
2000 Evaluation: An introduction. In Susan Hunston & Geoff Thompson (eds.), Evaluation in text: Authorial stance and the construction of discourse, 1–27. Oxford: OUP.
Cited by
Cited by 40 other publications
Alwohaibi, Hala Abdulrahman & Hesham Suleiman Alyousef
2023. An Investigation of Engagement Resources in the World Cup 2022 Newspaper Articles in the Arab World Media. Journal of Contemporary Language Research 2:2 ► pp. 118 ff. 
Andersson, Marta
2022. ‘So many “virologists” in this thread!’. Pragmatics. Quarterly Publication of the International Pragmatics Association (IPrA) 32:4 ► pp. 489 ff.
Benítez-Castro, Miguel-Ángel & Encarnación Hidalgo-Tenorio
Bisiada, Mario
2018. A cross-linguistic analysis of the ‘homework’ metaphor in German and English political discourse. Discourse & Society 29:6 ► pp. 609 ff.
Carretero, Marta
2023. ‘Noticeably Better / Visibly Shaken’: The Use of Four Adverbs of High Perceivability with Meanings of Attitude. <i>WORD</i> 69:2 ► pp. 117 ff.
Cheng, Shi
2023. A review of interpersonal metafunction studies in systemic functional linguistics (2012–2022). Journal of World Languages 0:0
Cibulskienė, Jurga
2023. Scepticism voiced through extended metaphors. Metaphor and the Social World 13:2 ► pp. 197 ff.
Domingo, Cecilia
2023. INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION, ► pp. 731 ff.
Forceville, Charles & Natalia Sánchez-Querubín
2022. Relevance theory perspectives on web-mediated communication. In Dynamism in Metaphor and Beyond [Metaphor in Language, Cognition, and Communication, 9], ► pp. 325 ff.
FUOLI, MATTEO, JEANNETTE LITTLEMORE & SARAH TURNER
2022. Sunken ships and screaming banshees: metaphor and evaluation in film reviews. English Language and Linguistics 26:1 ► pp. 75 ff.
Fusari, Sabrina
2019. “Logically, We Quite Agree with the IARC”: Negotiating Interpersonal Meaning in a Corpus of Scientific Texts. In Specialized Discourses and Their Readerships [The M.A.K. Halliday Library Functional Linguistics Series, ], ► pp. 25 ff.
Geng, Yifan & Sue Wharton
2019. How do thesis writers evaluate their own and others’ findings? An appraisal analysis and a pedagogical intervention. English for Specific Purposes 56 ► pp. 3 ff.
Glynn, Dylan
2022. Chapter 8. Emergent categories. In Analogy and Contrast in Language [Human Cognitive Processing, 73], ► pp. 245 ff.
Hansson, Sten, Ruth Page & Matteo Fuoli
2022. Discursive Strategies of Blaming: The Language of Judgment and Political Protest Online. Social Media + Society 8:4 ► pp. 205630512211387 ff.
Hidalgo-Downing, Laura & Paula Pérez-Sobrino
2024. Developing an annotation protocol for evaluative stance and metaphor in discourse: theoretical and methodological considerations. Text & Talk 44:2 ► pp. 197 ff.
Hidalgo-Downing, Laura, Paula Pérez-Sobrino, Laura Filardo-Llamas, Carmen Maíz-Arévalo, Begoña Núñez-Perucha, Alfonso Sánchez-Moya & Julia T. Williams Camus
2024. A protocol for the annotation of evaluative stance and metaphor across four discourse genres. Revista Española de Lingüística Aplicada/Spanish Journal of Applied Linguistics
HIDALGO-TENORIO, ENCARNACIÓN & MIGUEL-ÁNGEL BENÍTEZ-CASTRO
2021. The Language of Evaluation in the Narratives by the Magdalene Laundries Survivors: The Discourse of Female Victimhood. Applied Linguistics 42:2 ► pp. 315 ff.
Jing, Yi & Qingxin Xu
2023. Danmu video as the resemiotised: An attitudinal analysis of Home Visit. Media, Culture & Society 45:4 ► pp. 753 ff.
Kolhatkar, Varada, Hanhan Wu, Luca Cavasso, Emilie Francis, Kavan Shukla & Maite Taboada
2020. The SFU Opinion and Comments Corpus: A Corpus for the Analysis of Online News Comments. Corpus Pragmatics 4:2 ► pp. 155 ff.
Lamb, David A. & Thora Tenbrink
2022. Evaluating Jesus and other “heroes”. Language, Context and Text. The Social Semiotics Forum 4:2 ► pp. 227 ff.
Liu, Yufeng & Dechao Li
2022. The US-China battle over Coronavirus in the news media: Metaphor transfer as a representation of stance mediation. Discourse & Society 33:4 ► pp. 456 ff.
Lu, Rickey
2024. Audience design and pragmatic conceptions of moves and upvotes during advice-giving on Reddit. Journal of Pragmatics 219 ► pp. 30 ff.
Makki, Mohammad & Andrew S. Ross
2023. “We were cocked & loaded to retaliate”. Journal of Language Aggression and Conflict 11:1 ► pp. 1 ff.
Mavis Ho, Nga-Ki
2021. Transcreation in marketing: a corpus-based study of persuasion in optional shifts from English to Chinese. Perspectives 29:3 ► pp. 426 ff.
Page, Ruth
2019. Self-denigration and the mixed messages of ‘ugly’ selfies in Instagram. Internet Pragmatics 2:2 ► pp. 173 ff.
Paternoster, Annick
2019. Chapter 3. Politeness and evaluative adjectives in Italian turn-of-the-century etiquette books (1877–1914). In Politeness in Nineteenth-Century Europe [Pragmatics & Beyond New Series, 299], ► pp. 107 ff.
Ross, Andrew S. & David Caldwell
2020. ‘Going negative’: An APPRAISAL analysis of the rhetoric of Donald Trump on Twitter. Language & Communication 70 ► pp. 13 ff.
Shrestha, Prithvi N.
2022. Examining evaluative language used in assessment feedback on business students’ academic writing. Assessing Writing 54 ► pp. 100664 ff.
Simaki, Vasiliki, Eleni Seitanidi & Carita Paradis
2022. Evaluating stance annotation of Twitter data. Research in Corpus Linguistics 11:1 ► pp. 53 ff.
Sun, Jing & Zhenqian Liu
2023. Evaluation Mechanism of Political Discourse: A Holistic Approach. Journal of Psycholinguistic Research 52:6 ► pp. 2143 ff.
Sun, Shuyi & Peter Crosthwaite
2022. “Establish a niche” vianegation: A corpus-based study ofnegationwithin the Move 2 sections of PhD thesis introductions. Open Linguistics 8:1 ► pp. 189 ff.
Sun, Shuyi Amelia & Peter Crosthwaite
2022. “The findings might not be generalizable”: Investigating negation in the limitations sections of PhD theses across disciplines. Journal of English for Academic Purposes 59 ► pp. 101155 ff.
Sun, Shuyi Amelia, Feng Kevin Jiang & Yanhua Liu
2024. “Maybe, but probably not”: A cross-disciplinary study of negation in Three Minute Thesis presentations. English for Specific Purposes 74 ► pp. 117 ff.
Trnavac, Radoslava & Nele Põldvere
2024. Investigating Appraisal and the Language of Evaluation in Fake News Corpora. Corpus Pragmatics
Trnavac, Radoslava & Maite Taboada
2023. Engagement and constructiveness in online news comments in English and Russian. Text & Talk 43:2 ► pp. 235 ff.
van Driel, Martine
2021. Chapter 8. Evaluating news events. In Style and Reader Response [Linguistic Approaches to Literature, 36], ► pp. 143 ff.
Van Herck, Rebecca, Sofie Decock & Bernard De Clerck
2020. “Can you send us a PM please?” Service recovery interactions on social media from the perspective of organizational legitimacy. Discourse, Context & Media 38 ► pp. 100445 ff.
Vilar Lluch, Sara
2022. Redefining attitude for studying explicit and indirect evaluations of human behaviour. Functions of Language 29:2 ► pp. 199 ff.
Watanabe, Hideo
2023. The discursive construction of a conflict: a case of disputed islands in the East China Sea. Text & Talk 43:3 ► pp. 333 ff.
Yu, Wei
2023. Negotiation of justice: the discursive construction of attitudinal positioning in bilingual legal judgments of HKSAR v KWAN WAN KI
. International Journal of Legal Discourse 8:2 ► pp. 299 ff.
This list is based on CrossRef data as of 29 march 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.