Article published In:
ConcentricVol. 46:1 (2020) ► pp.1–20
Hearing faces and seeing accents?
The factors behind the multisensory speech perception of native Mandarin speakers
In this paper, the McGurk effect displayed by native Mandarin Speakers is examined in the light of reaction time (RT) and response types. Two within-subject factors, FACE and ACCENT, and one between-subject factor, English Proficiency, were incorporated in the experiment. The results showed that FACE and ACCENT, but not English Proficiency, had effects on the participants’ RT and response types. When a foreign ACCENT was dubbed onto a familiar FACE, the RT was the longest, and the McGurk effect was most likely to be found. Other kinds of McGurk stimuli composition did not receive different RT but induced different response types. When the FACE was foreign, regardless of the ACCENT, the participants tended to respond with perceptive illusion. The author concluded that the expectations of the perceiver influenced the use of multisensory integration and thus the longer RT and the appearance of the McGurk effect.
Keywords: the McGurk Effect, multisensory speech perception, reaction time, response types, cognitive effort
Article outline
- 1.The McGurk effect (multisensory integration)
- 1.1MSI and the McGurk effect of the second language speech perception
- 1.2Research purposes and questions
- 2.Methodology
- 2.1Participants
- 2.2Materials
- 2.2.1Raw materials preparation
- 2.2.2Accent analysis
- 2.2.3Materials normalization
- 2.3Procedure
- 2.4Data analysis
- 2.4.1Reaction time (RT)
- 2.4.2Response type
- 3.Results
- 3.1Reaction time
- 3.2Response types
- 4.Discussion
- 5.Limitations
- 6.Conclusion
- Acknowledgements
- Notes
-
References
References
Beauchamp, Michael S.
Bovo, Roberto, Andrea Ciorba, Silvano Prosser, and Alessandro Martini
2009 The McGurk phenomenon in Italian listeners. Acta Otorhinolaryngologica Italica 29.4:203–208.
Burnham, Denis, and Susanna Lau
1998 The effect of tonal information on auditory reliance in the McGurk effect. Paper presented at the AVSP’98 International Conference on Auditory-Visual Speech Processing, University of New South Wales, Sydney.
Chen, Yuchun, and Valérie Hazan
2007 Developmental factor in auditory-visual speech perception-the McGurk effect in Mandarin-Chinese and English Speakers. Paper presented at the AVSP2007 International Conference on Auditory-Visual Speech Processing, Kasteel Groenendaal, Hilvarenbeek, Netherlands.
Fuster-Duran, Angela
1996 Perception of conflicting audio-visual speech: An examination across Spanish and German. Speechreading by Humans and Machines: Models, Systems, and Applications, ed. by David G. Stork and Marcus E. Hennecke, 135–143. Berlin: Springer. 
Hirata, Yukari, and Spencer D. Kelly
2010 Effects of lips and hands on auditory learning of second-language speech sounds. Journal of Speech, Language, and Hearing Research 53.2:298–310.
Kotz, Sonja A., and Kerrie Elston-Güttler
2004 The role of proficiency on processing categorical and associative information in the L2 as revealed by reaction times and event-related brain potentials. Journal of Neurolinguistics 17.2–3:215–235.
Magnotti, John F., Debshila Basu Mallick, Guo Feng, Bin Zhou, Wen Zhou, and Michael S. Beauchamp
2015 Similar frequency of the McGurk effect in large samples of native Mandarin Chinese and American English speakers. Experimental Brain Research 233.9:2581–2586.
McGurk, Harry, and John MacDonald
1976 Hearing lips and seeing voices. Nature 2641:746–748.
Munro, Murray J., and Tracey M. Derwing
1995 Processing time, accent, and comprehensibility in the perception of native and foreign-accented speech. Language and Speech 38.3:289–306.
Nath, Audrey R., and Michael S. Beauchamp
2012 A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion. NeuroImage 59.1:781–787.
Navarra, Jordi, and Salvador Soto-Faraco
2007 Hearing lips in a second language: Visual articulatory information enables the perception of second language sounds. Psychological Research 71.1:4–12.
Marian, Viorica
2009 Audio-visual integration during bilingual language processing. The Bilingual Mental Lexicon: Interdisciplinary Approaches, ed. by Aneta Pavlenko, 52–78. Bristol, Buffalo & Toronto: Multilingual Matters.
Sams, Mikko, Petri Manninen, Veikko Surakka, Pia Helin, and Riitta Kättö
1998 McGurk effect in Finnish syllables, isolated words, and words in sentences: Effects of word meaning and sentence context. Speech Communication 26.1–2: 75–87.
Sekiyama, Kaoru
1997 Cultural and linguistic factors in audiovisual speech processing: The McGurk effect in Chinese subjects. Perception & Psychophysics 59.1:73–80.
Sekiyama, Kaoru, and Yoh’ichi Tohkura
1991 McGurk effect in non-English listeners: Few visual effects for Japanese subjects hearing Japanese syllables of high auditory intelligibility. The Journal of the Acoustical Society of America 90.4:1797–1805.
Sueyoshi, Ayano, and Debra M. Hardison
2005 The role of gestures and facial cues in second language listening comprehension. Language Learning 55.4:661–699.
Tona, Risa, Yasushi Naito, Saburo Moroto, Rinko Yamamoto, Keizo Fujiwara, Hiroshi Yamazaki, Shogo Shinohara, and Masahiro Kikuchi
2015 Audio-visual integration during speech perception in prelingually deafened Japanese children revealed by the McGurk effect. International Journal of Pediatric Otorhinolaryngology 79.12:2072–2078.
Tse, Chi-Shing, and Jeanette Altarriba
2012 The effects of first- and second-language proficiency on conflict resolution and goal maintenance in bilinguals: Evidence from reaction time distributional analyses in a Stroop task. Bilingualism: Language and Cognition 15.3:663–676.