Article published In:
Chinese Language and DiscourseVol. 11:2 (2020) ► pp.169–197
Incoherence in L2 writing
A comparison of expert insider and non-expert outsider ratings
Current knowledge of the functional dimensions (e.g., coherence) of L2 written performance is mainly based on expert readers’ views. Non-expert native readers’ perspectives of L2 written productions are not often examined, which prevents a comprehensive and objective understanding of how an L2 writer’s performance may be perceived and evaluated by the target discourse community. Studies to date also lack clear findings of both the types of incoherence phenomena that may exist in L2 texts and the factors that may contribute to such incoherencies. The current study investigates how expert and non-expert native readers evaluate incoherence in L2 Chinese argumentative essays, as well as how their evaluations may differ quantitatively or qualitatively. The findings reveal that although expert readers marked incoherence in L2 writing significantly more frequently than non-expert readers did, expert and non-expert readers displayed similar patterns and tendencies in their judgments pertaining to the frequency with which incoherence instances appeared in the essays and in the severity of the incoherence.
Keywords: coherence, cohesion, argumentative essay, L2 Chinese, expert rater, non-expert rater
Article outline
- 1.Introduction
- 2.Literature review
- 2.1Conceptualizations of L2 written coherence
- 2.2Expert and non-expert raters’ judgments of L2 written coherence
- 2.3Development of L2 written coherence
- 3.The present study
- 3.1Dataset
- 3.2Raters
- 3.3Rating
- 3.4Analysis
- 4.Results
- 4.1Judgments of frequency and severity of incoherence by the expert and non-expert raters
- 4.2Reasons for incoherence by the expert and non-expert raters
- 4.3Inter-rater relationships
- 5.Discussion
- 5.1Judgments of frequency and severity of incoherence by the expert and non-expert raters
- 5.2Reasons for incoherence by the expert and non-expert raters
- 6.Conclusion and implications
-
Reference
References
Bardovi-Harlig, K.
1990 Pragmatic word order in English composition. In U. Connor & A. Johns (Eds.), Coherence in writing: Research and pedagogical perspectives (pp. 43–65). Alexandria, VA: TESOL.
Carrell, P.
1982 Cohesion is not coherence. TESOL Quarterly, 161, 479–488. 
Chen, Y., & Baker, P.
2016 Investigating criterial discourse features across second language development: Lexical bundles in rated learner essays, CEFR B1, B2 and C1. Applied Linguistics, 371, 849–880.
Chiang, S.
2003 The importance of cohesive conditions to perceptions of writing quality at the early stages of foreign language learning. System, 311, 471–484.
Connor, U.
1990 Linguistic/rhetorical measures for international persuasive student writing. Research in the Teaching of English, 241, 67–87.
Crossley, S., Kyle, K., & McNamara, D.
2016a The tool for the automatic analysis of text cohesion (TAACO): Automatic assessment of local, global, and text cohesion. Behavior Research Methods, 481, 1227–1237.
2016b The development and use of cohesive devices in L2 writing and their relations to judgments of essay quality. Journal of Second Language Writing, 321, 1–16.
Crossley, S., & McNamara, D.
2011 Shared features of L2 writing: Intergroup homogeneity and text classification. Journal of Second Language Writing, 201, 271–285.
2012 Predicting second language writing proficiency: The roles of cohesion and linguistic sophistication. Journal of Research in Reading, 35, 2, 115–135.
Crossley, S., Yang, H. S., & McNamara, D.
2014 What’s so simple about simplified texts? A computational and psycholinguistic investigation of text comprehension and text processing. Reading in a Foreign Language, 26(1), 92–113.
Cumming, A.
1990 Expertise in evaluating second language compositions. Language Testing, 71, 31–51.
Eckes, T.
2008 Rater types in writing performance assessments: A classification approach to rater variability. Language Testing, 25(2), 155–185.
Ferris, D.
1994 Lexical and syntactic features of ESL writing by students at different levels of L2 proficiency. TESOL Quarterly, 281, 414–420.
Guo, L., Crossley, S., & McNamara, D.
2013 Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study. Assessing Writing, 181, 218–238.
Halliday, M., & Hasan, R.
1976 Cohesion in English. London, UK: Longman.
Harman, R.
2013 Literary intertextuality in genre-based pedagogies: Building lexical cohesion in fifth-grade L2 writing. Journal of Second Language Writing, 221, 125–140.
Hyland, K.
2005 Metadiscourse: Exploring interaction in writing. London, UK: Continuum.
Jiang, W.
2013 Measurements of development in L2 written production: The case of L2 Chinese. Applied Linguistics, 341, 1–24.
Kang, Y. J.
2005 Written narratives as an index of L2 competence in Korean EFL learners. Journal of Second Language Writing, 141, 259–279.
Kennedy, C., & Thorp, D.
2007 A corpus-based investigation of linguistic responses to an IELTS academic writing task. In L. Taylor & P. Falvey (Eds), IELTS collected papers: Research in speaking and writing assessment (pp. 316–377). Cambridge, UK: Cambridge University Press.
Knoch, U.
2007 ‘Little coherence, considerable strain for reader’: A comparison between two rating scales for the assessment of coherence. Assessing Writing, 12(2), 108–128.
Kormos, J.
2011 Task complexity and linguistic and discourse features of narrative writing performance. Journal of Second Language Writing, 201, 148–161.
Kuiken, F., & Vedder, I.
2014 Rating written performance: What do raters do and why? Language Testing, 311, 329–348.
2017 Functional adequacy in L2 writing: Towards a new rating scale. Language Testing, 341, 321–336.
Kuiken, F., Vedder, I., & Gilabert, R.
2010 Communicative adequacy and linguistic complexity in L2 writing. In I. Bartning, M. Martin & I. Vedder (Eds.), Communicative proficiency and linguistic development: Intersections between SLA and language testing research (pp. 81–99). Eurosla: European Second Language Association.
Li, S.
2014 The gap in the use of lexical cohesive devices in writing between native Chinese speakers and second language users. Journal of the Chinese Language Teachers Association, 49(3), 25–47.
Li, T., & Wharton, S.
2012 Metadiscourse repertoire of L1 Mandarin undergraduates writing in English: A cross-contextual, cross-disciplinary study. Journal of English for Academic Purposes, 111, 345–356.
Liu, M., & Braine, G.
2005 Cohesive features in argumentative writing produced by Chinese undergraduates. System, 331, 623–636.
McCulley, G.
1985 Writing quality, coherence, and cohesion. Research in the Teaching of English, 191, 269–282.
McCutchen, D.
1996 A capacity theory of writing: Working memory in composition. Educational Psychology Review, 81, 299–325.
McNamara, D.
2001 Reading both high and low coherence texts: Effects of text sequence and prior knowledge. Canadian Journal of Experimental Psychology, 551, 51–62.
McNamara, D., Crossley, S., & McCarthy, P.
2010 Linguistic features of writing quality. Written Communication, 271, 57–86.
McNamara, D., Crossley, S., & Roscoe, R.
2013 Natural language processing in an intelligent writing strategy tutoring system. Behavior Research Methods, 451, 499–515.
McNamara, D., & Kintsch, W.
1996 Learning from texts: Effects of prior knowledge and text coherence. Discourse Processes, 22(3), 247–288.
McNamara, D., Kintsch, E., Songer, N., & Kintsch, W.
1996 Are good texts always better? Interactions of text coherence, background knowledge, and levels of understanding in learning from text. Cognition and Instruction, 141, 1–43.
Neuner, J.
1987 Cohesive ties and chains in good and poor freshman essays. Research in the Teaching of English, 211, 92–105.
O’Reilly, T., & McNamara, D.
2007 Reversing the reverse cohesion effect: Good texts can be better for strategic, high-knowledge readers. Discourse Processes, 431, 121–152.
Reid, J.
1992 A computer text analysis of four cohesion devices in English discourse by native and nonnative writers. Journal of Second Language Writing, 11, 79–107.
Schoonen, R.
2005 Generalizability of writing scores: an application of structural equation modeling. Language Testing, 22(1), 1–30.
Schoonen, R., Vergeer, M., & Eiting, M.
1997 The assessment of writing ability: Expert readers versus lay readers. Language Testing, 14(2), 157–184.
Tannen, D.
1984 Spoken and written narrative in English and Greek. In D. Tannen (Ed.). Coherence in spoken and written discourse (pp. 21–41). Norwood, NJ: Ablex.
Weigle, S. C.
2007 Teaching writing teachers about assessment. Journal of Second Language Writing, 16(3), 194–209.
Witte, S., & Faigley, L.
1981 Coherence, cohesion, and writing ability. College Composition and Communication, 321, 189–204.
Yang, C.
2013 Textual conjunctives and topic-fronting devices in CFL learners’ written summaries. Journal of the Chinese Language Teachers Association, 48(1), 71–89.
Yang, W., & Sun, Y.
2012 The use of cohesive devices in argumentative writing by Chinese EFL learners at different proficiency levels. Linguistics and Education, 231, 31–48.