Testing the malleability of teachers’ judgments of second language speech
This study examined whether a negative social bias can influence how teachers evaluate second language (L2)
speech. Twenty-eight teachers of L2 German from Western Canada – 14 native speakers (NSs) and 14 proficient non-native speakers
(NNSs) – rated recordings of 24 adult L2 learners of German across five speech dimensions (accentedness, comprehensibility,
vowel/consonant accuracy, intonation, flow) using 1,000-point scales. Immediately before rating, half of NS and NNS teachers heard
critical comments about undergraduate German students’ language skills, while the other half heard no biasing comments. Under
negative bias, while the NNS teachers provided favorable evaluations across all five measures, NS teachers followed suit for only
intonation and flow, downgrading L2 speakers’ accentedness, comprehensibility, and vowel/consonant accuracy. Findings call into
question the relative stability of L2 speech ratings and highlight the importance of social context and teacher status as native
versus non-native speakers of the target language in assessments of L2 speaking performance.
Keywords: German, social attitudes, speech rating, accent bias, linguistic stereotyping, second language, speech assessment, teacher status, accentedness, comprehensibility, fluency
Article outline
- 1.Introduction
- 2.Background literature
- 2.1Listener biases and their origins
- 2.2Listener-specific factors
- 2.3The current study
- 3.Method
- 3.1Teacher raters
- 3.2Speech materials
- 3.3Rating procedure
- 3.4Data analysis
- 4.Results
- 5.Discussion
- 5.1L2 speech ratings and listener bias
- 5.2Sources of teacher bias
- 5.3Relationship between teacher status and negative bias varies across speech dimensions
- 5.4Uneven baselines in the absence of negative bias
- 6.Implications and conclusion
- Acknowledgements
- Notes
-
References
References (72)
References
Anderson-Hsieh, J., Johnson, R., & Koehler, K. (1992). The relationship between native speaker judgments of nonnative pronunciation and deviance in segmentals, prosody, and syllable structure. Language Learning, 421, 529–555. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Beukeboom, C. J., & Semin, G. R. (2006). How mood turns on language. Journal of Experimental Social Psychology, 421, 553–566. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Bodenhausen, G. V., Mussweiler, T., Gabriel, S., & Moreno, K. N. (2001). Affective influences on stereotyping and intergroup relations. In J. P. Forgas (Ed.), Handbook of affect and social cognition (pp. 319–343). Mahwah, NJ: Lawrence Erlbaum.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Bongaerts, T., van Summeren, C., Planken, B., & Schils, E. (1997). Age and ultimate attainment in the pronunciation of a foreign language. Studies in Second Language Acquisition, 191, 447–465. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Boughton, Z. (2006). When perception isn’t reality: Accent identification and perceptual dialectology in French. Journal of French Language Studies, 161, 277–304. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Bradlow, A. R., & Bent, T. (2008). Perceptual adaptation to non-native speech. Cognition, 1061, 707–729. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Brown, A. (1995). The effect of rater variables in the development of an occupation-specific language performance test. Language Testing, 121, 1–15. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Bryk, A. S., & Schneider, B. (2003). Trust in schools: A core resource for school reform. Educational Leadership, 601, 40–45.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Buckingham, L. (2014). Attitudes to English teachers’ accents in the Arabian Gulf. International Journal of Applied Linguistics, 241, 50–73. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Carey, M. D., Mannell, R. H., & Dunn, P. K. (2011). Does a rater’s familiarity with a candidate’s pronunciation affect the rating in oral proficiency interviews? Language Testing, 281, 201–219. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Cargile, A. C., & Giles, H. (1997). Understanding language attitudes: Exploring listener affect and identity. Language & Communication, 171, 195–217. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Chiba, R., Matsuura, H., & Yamamoto, A. (1995). Japanese attitudes toward English accents. World Englishes, 141, 77–86. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
D’Onofrio, A. (2018). Controlled and automatic perceptions of a sociolinguistic marker. Language Variation and Change, 301, 261–285. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
D’Onofrio, A. (forthcoming). Sociolinguistic signs as cognitive representations. In L. Hall-Lew, E. Moore, & R. J. Podesva (Eds.), Social meaning and linguistic variation. Cambridge, UK: Cambridge University Press.
Dalton-Puffer, C., Kaltenböck, G., & Smit, U. (1997). Learner attitudes and L2 pronunciation in Austria. World Englishes, 161, 115–128. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Derwing, T. M., & Munro, M. (2013). The development of L2 oral language skills in two L1 groups: A 7-year study. Language Learning, 631, 163–185. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Derwing, T. M., Rossiter, M. J., Munro, M. J., & Thomson, R. I. (2004). Second language fluency: Judgments on different tasks. Language Learning, 541, 655–679. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Eskenazi, M. (2009). An overview of spoken language technology for education. Speech Communication, 511, 832–844. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Fayer, J. M., & Krasinski, E. (1987). Native and nonnative judgments of intelligibility and irritation. Language Learning, 371, 313–326. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Fazio, R. H., Sanbonmatsu, D. M., Powell, M. C., & Kardes, F. R. (1986). On the automatic activation of attitudes. Journal of Personality and Social Psychology, 501, 229–238. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Foote, J. A., & Trofimovich, P. (2018). Is it because of my language background? A study of language background influence on comprehensibility judgments. Canadian Modern Language Review, 741, 253–278. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Ford, C. E. (1984). The influence of speech variety on teachers’ evaluation of students with comparable academic ability. TESOL Quarterly, 181, 25–40. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Galinsky, A. D., & Moskowitz, G. B. (2000). Perspective-taking: Decreasing stereotype expression, stereotype accessibility, and in-group favoritism. Journal of Personality and Social Psychology, 781, 708–724. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Goethe Institut (2004). Einstufungstest [Placement test]. [URL]
Hansen Edwards, J. G., Zampini, M. L., & Cunningham, C. (2018). The accentedness, comprehensibility, and intelligibility of Asian Englishes. World Englishes, 371, 538–557. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Hansen Edwards, J. G., Zampini, M. L., & Cunningham, C. (2019). Listener judgments of speaker and speech traits of varieties of Asian English. Journal of Multilingual and Multicultural Development, 401, 691–706. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Hansen, K., Rakić, T., & Steffens, M. C. (2014). When actions speak louder than words: Preventing discrimination of nonstandard speakers. Journal of Language and Social Psychology, 331, 68–77. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
He, D., & Miller, L. (2011). English teacher preference: The case of China’s non-English-major students. World Englishes, 301, 428–443. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Hu, G., & Lindemann, S. (2009). Stereotypes of Cantonese English, apparent native/non-native status, and their effect on non-native English speakers’ perception. Journal of Multilingual and Multicultural Development, 301, 253–269. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Hu, G., & Su, J. (2015). The effect of native/non-native information on non-native listeners’ comprehension. Language Awareness, 241, 273–281. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Hundt, M., Zipp, L., & Huber, A. (2015). Attitudes in Fiji towards varieties of English. World Englishes, 341, 688–707. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Isaacs, T., & Thomson, R. I. (2013). Rater experience, rating scale length, and judgments of L2 pronunciation: Revisiting research conventions. Language Assessment Quarterly, 101, 135–159. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kang, O. (2012). Impact of rater characteristics and prosodic features of speaker accentedness on ratings of international teaching assistants’ oral performance. Language Assessment Quarterly, 91, 249–269. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kang, O., & Rubin, D. L. (2009). Reverse linguistic stereotyping: Measuring the effect of listener expectations on speech evaluation. Journal of Language and Social Psychology, 281, 441–456. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kawakami, K., Phills, C. E., Greenwald, A. G., Simard, D., Pontiero, J., Brnjas, A., & Dovidio, J. F. (2012). In perfect harmony: Synchronizing the self to activated social categories. Journal of Personality and Social Psychology, 1021, 562–575. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Lambert, W. E., Hodgson, R. C., Gardner, R. C., & Fillenbaum, S. (1960). Evaluational reactions to spoken languages. The Journal of Abnormal and Social Psychology, 601, 44–51. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Lappin-Fortin, K., & Rye, B. J. (2014). The use of pre-/posttest and self-assessment tools in a French pronunciation course. Foreign Language Annals, 471, 300–320. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Lindemann, S. (2003). Koreans, Chinese or Indians? Attitudes and ideologies about non-native English speakers in the United States. Journal of Sociolinguistics, 71, 348–364. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Major, R. C., Fitzmaurice, S. F., Bunta, F., & Balasubramanian, C. (2002). The effects of non-native accents on listening comprehension: Implications for ESL assessment. TESOL Quarterly, 361, 173–190. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Montgomery, C. (2007). Northern English dialects: A perceptual approach (Unpublished PhD dissertation). University of Sheffield, Sheffield, United Kingdom.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Munro, M. J., Derwing, T. M., & Morton, S. L. (2006). The mutual intelligibility of L2 speech. Studies in Second Language Acquisition, 281, 111–131. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Norton, B., & Toohey, K. (2011). Identity, language learning, and social change. Language Teaching, 441, 412–446. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
O’Brien, M. G. (2014). L2 learners’ assessments of accentedness, fluency, and comprehensibility of native and nonnative German speech. Language Learning, 641, 715–748. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Pantos, A. J., & Perkins, A. W. (2013). Measuring implicit and explicit attitudes toward foreign accented speech. Journal of Language and Social Psychology, 321, 3–20. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Reid, K. T., Trofimovich, P., & O’Brien, M. G. (2019). Social attitudes and speech ratings: Effects of positive and negative bias on multiage listeners’ judgments of second language speech. Studies in Second Language Acquisition, 411, 419–442. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Rose, R. L. (2017). Differences in second language speech fluency ratings: Native versus nonnative listeners. In L. Degand et al. (Eds.), Proceedings of the International Conference “Fluency & Disfluency Across Languages and Language Varieties” (pp. 101–103). Louvain-la-Neuve, Belgium: Université catholique de Louvain.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Rossiter, M. J. (2009). Perceptions of L2 fluency by native and non-native speakers of English. Canadian Modern Language Review, 651, 395–412. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Rubin, D. L. (1992). Nonlanguage factors affecting undergraduates’ judgments of nonnative English-speaking teaching assistants. Research in Higher Education, 331, 51–531. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Ryan, E. B., Carranza, M. A., & Moffie, R. W. (1977). Reactions toward varying degrees of accentedness in the speech of Spanish-English bilinguals. Language and Speech, 201, 267–273. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Ryan, E. B., Hewstone, M., & Giles, H. (1984). Language and intergroup attitudes. In J. R. Eiser (Ed.), Attitudinal judgment (pp. 135–158). New York: Springer. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Sokolov, C. (2014). Self-evaluation of rater bias in written composition assessment. Linguistica, 541, 261–275. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Spillane, J. P. (2006). Distributed leadership. San Francisco, CA: Jossey-Bass. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Spillane, J. P., Hallett, T., & Diamond, J. B. (2003). Forms of capital and the construction of leadership: Instructional leadership in urban elementary schools. Sociology of Education, 761, 1–17. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Steele, C. M., & Aronson, J. (1995). Stereotype threat and the intellectual test performance of African Americans. Journal of Personality and Social Psychology, 691, 797–811. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Stewart, M. A., Ryan, E. G., & Giles, H. (1985). Accent and social class effects on status and solidarity evaluations. Personality and Social Psychology Bulletin, 111, 98–105. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Tajfel, H. (1972). Experiments in a vacuum. In J. Israel & H. T. Triandis (Eds.), The context of social psychology: A critical assessment (pp. 69–119). London, UK: Academic Press.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Isaacs, T., & Trofimovich, P. (2012). Deconstructing comprehensibility: Identifying the linguistic influences on listeners’ L2 comprehensibility ratings. Studies in Second Language Acquisition, 341, 475–505. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Visser, P. S., & Cooper, J. (2007). Attitude change. The Sage handbook of social psychology (pp. 197–218). Thousand Oaks, CA: Sage. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Vissers, C. T. W. M., Virgillito, D., Fitzgerald, D. A., Speckens, A. E. M., Tendolkar, I., Van Oostrom, I., & Chwilla, D. J. (2010). The influence of mood on the processing of syntactic anomalies: Evidence from P600. Neuropsychologia, 481, 3521–3531. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Watson, K., & Clark, L. (2015). Exploring listeners’ real-time reactions to regional accents. Language Awareness, 241, 38–59. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Wheelan, S., & Kaeser, R. M. (1997). The influence of task type and designated leaders on developmental patterns in groups. Small Group Research, 281, 94–121. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Winke, P. (2013). Rating oral language. In C. A. Chapelle (Ed.), The encyclopedia of applied linguistics (pp. 1–7). Hoboken, NJ: Blackwell. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Winke, P., & Gass, S. (2013). The influence of second language experience and accent familiarity on oral proficiency rating: A qualitative investigation. TESOL Quarterly, 471, 762–789. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Winke, P., Gass, S., & Myford, C. (2013). Raters’ L2 background as a potential source of bias in rating oral performance. Language Testing, 301, 231–252. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Xu, W., Wang, Y., & Case, R. E. (2010). Chinese attitudes towards varieties of English: A pre-Olympic examination. Language Awareness, 191, 249–260. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Yao, Z., Saito, K., Trofimovich, P., & Isaacs, T. (2013). Z-Lab [Computer software]. [URL]
Cited by (3)
Cited by three other publications
Brown, Bianca, Botagoz Tusmagambet, Valentino Rahming, Chun-Ying Tu, Michael B. DeSalvo & Seth Wiener
2023.
Searching for the “native” speaker: A preregistered conceptual replication and extension of Reid, Trofimovich, and O’Brien (2019).
Applied Psycholinguistics 44:4
► pp. 475 ff.
![DOI logo](//benjamins.com/logos/doi-logo.svg)
Taylor Reid, Kym, Pavel Trofimovich & Mary Grantham O’Brien
2022.
The Malleability of Listener Judgments of Second Language Speech. In
Theoretical and Practical Developments in English Speech Assessment, Research, and Training [
Second Language Learning and Teaching, ],
► pp. 11 ff.
![DOI logo](//benjamins.com/logos/doi-logo.svg)
Trofimovich, Pavel, Oguzhan Tekin & Kim McDonough
This list is based on CrossRef data as of 5 july 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.