Article published In:
PragmaticsVol. 21:3 (2011) ► pp.453–471
Rater variation in the assessment of speech acts
This study addresses variability among native speaker raters who evaluated pragmatic performance of learners of English as a foreign language. Using a five-point rating scale, four native English speakers of mixed cultural background (one African American, one Asian American, and two Australians) assessed the appropriateness of two types of speech acts (requests and opinions) produced by 48 Japanese EFL students. To explore norms and the reasoning behind the raters’ assessment practice, individual introspective verbal interviews were conducted. Eight students’ speech act productions (64 speech acts in total) were selected randomly, and the raters were asked to rate each speech act and then explain their rating decision. Interview data revealed similarities and differences in their use of pragmatic norms and social rules in evaluating appropriateness.
Keywords: Assessment, Rater variation, Pragmatic competence, Speech acts
Available under the Creative Commons Attribution-NonCommercial (CC BY-NC) 4.0 license.
References
Austin, J.L
(1962) How to do things with words. Cambridge: Harvard University Press.
Bachman, Lyle, and Adrian Palmer
(1996) Language Testing in Practice: Designing and Developing Useful Language Tests. Oxford: Oxford University Press.
Brown, Annie
(2000) An investigation of rater's orientation in awarding scores in the IELTS interview. In R. Tulloch (ed.), IELTS Research Report 31: 49-84.
(2003) Interviewer variation and the co-construction of speaking proficiency. Language Testing 201: 1-25. 
(2005) Interviewer Variability in Oral Proficiency Interviews. Frankfurt am Main: Peter Lang.
Brown, James D
(2001) Six types of pragmatics tests in two different contexts. In K. Rose & G. Kasper (eds.), Pragmatics in Language Teaching. Cambridge: Cambridge University Press, pp. 301-325.
Clark, H.H
(1979) Responding to indirect speech acts. Cognitive Psychology 111: 430-477.
Cohen, Andrew
(1994) Assessing Language Ability in the Classroom. Rowley, MS: Newbury House.
Cohen, Andrew, and Elite Olshtain
(1981) Developing a measure of socio-cultural competence: The case of apology. Language Learning 311: 113-134.
Davies, Alan
(2003) The native speaker in applied linguistics. In A. Davies & K. Elder (eds.), Handbook of Applied Linguistics. New York: Blackwell, pp. 431-450.
Ducassee, Ana M
(2009) Assessing paired orals: Raters' orientation to interaction. Language Testing 261: 423-443.
Green, A
(1998) Verbal Protocol Analysis in Language Testing Research: A Handbook (vol.51). Cambridge: Cambridge University Press.
Hoffman-Hicks, Sheila
(1992) Linguistic and pragmatic competence: Their relationship in the overall competence of the language learner. In L.F. Bouton & Y. Kachru (eds.), Pragmatics and Language Learning Monograph Series Vol.31.Urbana-Champaign, IL: University of Illinois, pp. 66-80.
Hudson, Thom, Emily Detmer, and James D. Brown
(1995) Developing Prototypic Measures of Cross- Cultural Pragmatics (Technical Report No.7). Honolulu, HI: University of Hawai'I at Manoa, Second Language Teaching & Curriculum Center.
Johnson, Jeff, and Gad Lim
(2009) The influence of rater language background on writing performance assessment. Language Testing 261: 485-505.
Levinson, Stephen
Liu, Jianda
(2006) Measuring Interlanguage Pragmatic Knowledge of EFL Learners. Frankfurt: Peter Lang.
May, Lyn
(2007) Interaction in a paired speaking test: The rater's perspective. Ph.D. dissertation: The University of Melbourne.
(2009) Co-constructed interaction in a paired speaking test: The rater's perspective. Language Testing 261: 397-421.
McNamara, Tim, and Tom Lumley
(1997) The effect of interlocutor and assessment mode variables in overseas assessments of speaking skills in occupational settings. Language Testing 141: 140-156.
Mey, Jacob
Roever, Carston
(2005) Testing EFL Pragmatics. Frankfurt: Peter Lang.
O'Loughlin, Kieran
(2002) The impact of gender in oral proficiency testing. Language Testing 191: 169-192.
Poliitt, Alastair, and Neil Murray
(1996) What do raters really pay attention to? In M. Milanovic & N. Saville (eds.), Performance Testing Cognition and Assessment: Selected papers from the 15th Language Testing Research Colloquium. Cambridge: Cambridge University Press, pp. 74-91.
Taguchi, Naoko
(2006) Analysis of appropriateness in a speech act of request in L2 English. Pragmatics 161: 513-535. BoP
Thomas, Jenny
Sasaki, Miyuki
(1998) Investigating EFL students' production of speech acts: A comparison of production questionnaires and role-plays. Journal of Pragmatics 301: 457-484. BoP
BoPWalter, Scott
(2007) A conversation-analytic hermeneutic rating protocol to assess L2 oral pragmatic competence. Language Testing 241: 155-183.
Yamashita, Sayoko
(1996) Six Measures of JSL Pragmatics (Technical report #14). Honolulu, HI: University of Hawai'i at Manoa, Second Language Teaching & Curriculum Center.
(2001) Using pictures for research in pragmatics-eliciting pragmatic strategies by picture response tests. In T. Hudson & J.D. Brown (eds.), A Focus on Language Test Development: Expanding the Language Proficiency Construct Across a Variety of Tests. (Technical Report #21). University of Hawaii, Second language Teaching and Curriculum Center, pp. 35-56.
Yoshitake, S
(1997) Measuring interlanguage pragmatic competence of Japanese students of English as a foreign language: A multi-test framework evaluation. Unpublished Ph.D. thesis, Columbia Pacific University, Novata, CA.
Cited by
Cited by 23 other publications
Ajabshir, Zahra Fakher
2019. The effect of synchronous and asynchronous computer-mediated communication (CMC) on EFL learners' pragmatic competence. Computers in Human Behavior 92 ► pp. 169 ff. 
Alemi, Minoo, Zohreh R. Eslami & Atefeh Rezanejad
2014. Rating EFL Learners’ Interlanguage Pragmatic Competence by Non-Native English Speaking Teachers. Procedia - Social and Behavioral Sciences 98 ► pp. 171 ff.
Arnold, Nike
2020. Community, Group Dynamics, Identity, and Anxiety in Learning an Additional Language. In The Cambridge Introduction to Applied Linguistics, ► pp. 71 ff.
Beaulieu, Suzie, Fanny Forsberg Lundell & Javier Bejarano
2022. Interlocutors’ judgment of Lx conventional expressions: An exploratory study. Intercultural Pragmatics 19:5 ► pp. 597 ff.
Cheng, Dongmei
2022. “Communication is a two-way street”. Pragmatics. Quarterly Publication of the International Pragmatics Association (IPrA) ► pp. 1 ff.
Susan Conrad, Alissa Hartig & Lynn Santelmann
2020. The Cambridge Introduction to Applied Linguistics,
Dixon, Tülay & Daniel H. Dixon
2024. A simple methodology for identifying speech act tendencies. Applied Pragmatics 6:1 ► pp. 60 ff.
Gesuato, Sara & Victoriya Trubnikova
2022. Assessing Pragmatic Skills in Speech Act Production and Perception. In Handbook of Research on Policies and Practices for Assessing Inclusive Teaching and Learning [Advances in Educational Marketing, Administration, and Leadership, ], ► pp. 211 ff.
Hernandez, Todd A.
2016. Acquisition of L2 Spanish requests in short-term study abroad. Study Abroad Research in Second Language Acquisition and International Education 1:2 ► pp. 186 ff.
Hernández, Todd A.
2021. Explicit instruction for the development of L2 Spanish pragmatic ability during study abroad. System 96 ► pp. 102395 ff.
Ishihara, Noriko
2013. Teacher-based assessment of L2 Japanese pragmatics: Classroom applications. In Assessing Second Language Pragmatics, ► pp. 124 ff.
Kostromitina, Maria & Yongzhi Miao
2024. Listener perception of appropriateness of L1 and L2 refusals in English. Studies in Second Language Learning and Teaching
Kusevska, Marija, Biljana Ivanovska, Nina Daskalovska & Tatjana Ulanska
2016. Profiling Pragmatic Competence of Foreign Language Learners. Romanian Journal of English Studies 13:1 ► pp. 79 ff.
Li, Shuai, Naoko Taguchi & Feng Xiao
2019. Variations in Rating Scale Functioning in Assessing Speech Act Production in L2 Chinese. Language Assessment Quarterly 16:3 ► pp. 271 ff.
Li, Shuai, Ting Wen, Xian Li, Yali Feng & Chuan Lin
2023. Comparing holistic and analytic marking methods in assessing speech act production in L2 Chinese. Language Testing 40:2 ► pp. 249 ff.
Lu, Xiaofei & Brody Bluemel
2020. Automated Assessment of Language. In The Cambridge Introduction to Applied Linguistics, ► pp. 86 ff.
Omar, Fouad Rashid & Özge Razı
2022. Impact of instruction based on movie and TV series clips on EFL learners’ pragmatic competence: Speech acts in focus. Frontiers in Psychology 13
Pinto, Derrin
2019. Shifting responsibilities: Student e-mail excuses and how faculty perceive them. Lingua 222 ► pp. 53 ff.
SHLEYKİNA, Galina
2021. Pragmatic failure revisited: Jaworski’s (1994) study in a new light. Eurasian Journal of Applied Linguistics ► pp. 303 ff.
Sickinger, Pawel & Katrin Renkwitz
2021. How Good Are Native Speakers at Evaluating Pragmatic Competence? Native Speaker Assessments of Learner Status and Appropriate Pragmatic Behaviour in English. Langages N° 222:2 ► pp. 113 ff.
Sonnenburg-Winkler, Sunni L., Zohreh R. Eslami & Ali Derakhshan
2020. Rater variation in pragmatic assessment: The impact of the linguistic background on peer-assessment and self-assessment. Lodz Papers in Pragmatics 16:1 ► pp. 67 ff.
Tajeddin, Zia & Minoo Alemi
2014. Criteria and Bias in Native English Teachers’ Assessment of L2 Pragmatic Appropriacy: Content and FACETS Analyses. The Asia-Pacific Education Researcher 23:3 ► pp. 425 ff.
Tajeddin, Zia & Minoo Alemi
2020. Pragmatics and Good Language Teachers. In Lessons from Good Language Teachers, ► pp. 189 ff.
This list is based on CrossRef data as of 13 april 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.