Analyzing the linguistic complexity of German learner language in a reading comprehension task
Using proficiency classification to investigate short answer data, cross-data generalizability, and the impact of linguistic
analysis quality
While traditionally linguistic complexity analysis of learner language is mostly based on essays, there is increasing
interest in other task types. This is crucial for obtaining a broader empirical basis for characterizing language proficiency and highlights
the need to advance our understanding of how task and learner properties interact in shaping the linguistic complexity of learner
productions. It also makes it important to determine which complexity measures generalize well across which tasks.
In this paper, we investigate the linguistic complexity of answers to reading comprehension questions written by foreign
language learners of German at the college level. Analyzing the corpus with computational linguistic methods identifying a wide range of
complexity features, we explore which linguistic complexity analyses can successfully be performed for such short answers, how learner
proficiency impacts the results, how generalizable they are across different contexts, and how the quality of the underlying analysis
impacts the results.
Article outline
- 1.Introduction
- 2.Related work
- 3.This study
- 4.Data
- 4.1CREG-29K
- 4.2CREG-KU, CREG-OSU, and CREG-7K
- 4.3CREG-104
- 4.3.1Manual annotation of learner language and target hypotheses
- 5.Automatic complexity analysis
- 5.1Feature description
- Lexical complexity
- Morphological complexity
- Phrasal complexity
- Clausal complexity
- Discourse complexity
- Language use
- Human processing
- Surface measures
- 5.2System description
- 6.Determining German L2 proficiency using linguistic complexity analysis
- 6.1Course-level classification
- 6.1.1Set-up of study 1
- 6.1.2Results of study 1
- 6.2Generalizability of complexity modeling
- 6.2.1Set-up of study 2
- 6.2.2Results of study 2
- 7.Performance of complexity models on learner language
- 7.1Accuracy of NLP analysis
- 7.1.1Set-up of study 3.1
- 7.1.2Results of study 3.1
- 7.2Effect on linguistic complexity analysis
- 7.2.1Set-up of study 3.2
- 7.2.2Results of study 3.2
- 7.3Effect on proficiency classification
- 7.3.1Set-up of study 3.3
- 7.3.2Results of study 3.3
- 8.Discussion
- 9.Conclusion
- Acknowledgements
- Notes
-
References
References (68)
References
Alexopoulou, T., Michel, M., Murakami, A., & Meurers, D. (2017). Task effects on linguistic complexity and accuracy: A large-scale learner corpus analysis employing natural language processing techniques. Language Learning, 671, 181–209. 

Biber, D., Gray, B., & Staples, S. (2016). Predicting patterns of grammatical complexity across language exam task types and proficiency levels. Applied Linguistics, 37(5), 639–668. 

Björkelund, A., Bohnet, B., Hafdell, L., & Nugues, P. (2010). A high-performance syntactic and semantic dependency parser. In Demonstration volume of the 23rd COLING (pp. 23–27). Beijing.
Bohnet, B., & Nivre, J. (2012). A transition-based system for joint part-of-speech tagging and labeled non-projective dependency parsing. In Proceedings of the 2012 joint conference on EMNLP and computational natural language learning (pp. 1455–1465). Jeju Island, Korea: Association for Computational Linguistics.
Brants, S., Dipper, S., Hansen, S., Lezius, W., & Smith, G. (2002). The TIGER treebank. In Proceedings of the workshop on treebanks and linguistic theories. Sozopol.
Brants, T., Skut, W., & Uszkoreit, H. (1999). Syntactic annotation of a German newspaper corpus. In Proceedings of the ATALA treebank workshop. Paris.
Brezina, V., & Pallotti, G. (2019). Morphological complexity in written l2 texts. Second Language Research, 35(1), 99–119. 

Brown, C., Snodgrass, T., Kemper, S. J., Herman, R., & Covington, M. A. (2008). Automatic measurement of propositional idea density from part-of-speech tagging. Behavior Research Methods, 40(2), 540–545. 

Brysbaert, M., Buchmeier, M., Conrad, M., Jacobs, A. M., Bölte, J., & Böhl, A. (2011). The word frequency effect: A review of recent developments and implications for the choice of frequency estimates in German. Experimental Psychology, 581, 412–424. 

Caines, A., & Buttery, P. (2017). The effect of task and topic on opportunity of use in learner corpora. In Learner corpus research: New perspectives and applications. London: Bloomsbury.
Chen, D., & Manning, C. (2014). A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 conference on EMNLP (pp. 740–750). Doha, Qatar.
Crossley, S. A. (2020). Linguistic features in writing quality and development: An overview. Journal of Writing Research, 11(3), 415–443. 

Crossley, S. A., Skalicky, S., & Dascalu, M. (2019). Moving beyond classic readability formulas: new methods and new models. Journal of Research in Reading, 42(3–4), 541–561. 

Crossley, S. A., Weston, J. L., Sullivan, S. T. M., & McNamara, D. S. (2011). The development of writing proficiency as a function of grade level: A linguistic analysis. Written Communication, 28(3), 282–311. 

De Clercq, B., & Housen, A. (2019). The development of morphological complexity: A cross-linguistic study of L2 French and English. Second Language Research Special Issue on Linguistic Complexity, 35(1), 71–97.
Dell’Orletta, F., Montemagni, S., & Venturi, G. (2014). Assessing document and sentence readability in less resourced languages and across textual genres. Recent Advances in Automatic Readability Assessment and Text Simplification. Special issue of the International Journal of Applied Linguistics, 165(2), 163–193.
Díaz-Negrillo, A., Meurers, D., Valera, S., & Wunsch, H. (2010). Towards interlanguage POS annotation for effective learner corpora in SLA and FLT. Language Forum, 36(1–2), 139–154.
Duden. (2009). Deutsche Grammatik (4th ed., Vol. 41). Dudenverlag.
Ellis, N. C. (2002). Frequency effecs in language processing. A review with implications for theories of implicit and explicit language acquisition. Studies in Second Language Acquisition, 24(2), 143–188. 

Ellis, R. (2003). Task-based language learning and teaching. Oxford, UK: Oxford University Press.
François, T., & Fairon, C. (2012). An “AI readability” formula for French as a foreign language. In Proceedings of the 2012 joint conference on EMNLP and computational natural language learning.
Galasso, S. (2014). Exploring textual cohesion characteristics for German readability classification (Bachelor Thesis in Computational Linguistics). Department of Linguistics, University of Tübingen. ([URL])
Geertzen, J., Alexopoulou, T., & Korhonen, A. (2013). Automatic linguistic annotation of large scale L2 databases: The EF-Cambridge open language database (EFCAMDAT). In Proceedings of the 31st SLRF. Cascadilla Press.
Gibson, E. (2000). The dependency locality theory: A distance-based theory of linguistic complexity. In A. Marantz, Y. Miyashita, & W. O’Neil (Eds.), Image, language, brain: papers from the first mind articulation project symposium (pp. 95–126). MIT.
Goldhahn, D., Eckart, T., & Quasthoff, U. (2012). Building large monolingual dictionaries at the leipzig corpora collection: From 100 to 200 languages. Proceedings of the 8th International Language Ressources and Evaluation, 759–765.
Hamp, B., & Feldweg, H. (1997). GermaNet – a lexical-semantic net for German. In Proceedings of ACL workshop automatic information extraction and building of lexical semantic resources for NLP applications. Madrid.
Hancke, J. (2013). Automatic prediction of CEFR proficiency levels based on linguistic features of learner language (Unpublished master’s thesis). Department of Linguistics, University of Tübingen.
Hancke, J., Vajjala, S., & Meurers, D. (2012). Readability classification for German using lexical, syntactic, and morphological features. In Proceedings of the 24th COLING (pp. 1063–1080). Mumbay, India.
Heister, J., Würzner, K.-M., Bubenzer, J., Pohl, E., Hanneforth, T., Geyken, A., & Kliegl, R. (2011). dlexDB – eine lexikalische Datenbank für die psychologische und linguistische Forschung. Psychologische Rundschau, 621, 10–20. 

Höhle, T. N. (1986). Der Begriff ‘Mittelfeld’. Anmerkungen über die Theorie der topologischen Felder. In A. Schöne (Ed.), Kontroversen alte und neue. Akten des VII. Internationalen Germanistenkongresses Göttingen 1985 (pp. 329–340). Tübingen: Niemeyer. (Bd. 3)
Housen, A., De Clercq, B., Kuiken, F., & Vedder, I. (2019). Multiple approaches to complexity in second language research. Second Language Research. Special Issue on Linguistic Complexity, 35(1), 2–31.
Housen, A., & Kuiken, F. (2009). Complexity, accuracy and fluency in second language acquisition. Applied Linguistics, 30(4), 461–473. 

Housen, A., Kuiken, F., & Vedder, I. (2012). Complexity, accuracy and fluency: Definitions, measurement and research. In A. Housen, F. Kuiken, & I. Vedder (Eds.), Dimensions of L2 performance and proficiency (pp. 1–20). John Benjamins. 

Hunt, K. W. (1965). A synopsis of clause-to-sentence length factors. The English Journal, 54(4), 300+305-309. 

Lavalley, R., Berkling, K., & Stüker, S. (2015). Preparing children’s writing database for automated processing. In Proceedings of the workshop on language teaching, learning and technology at speech and language technologies in education (pp. 9–15).
Lüdeling, A. (2008). Mehrdeutigkeiten und Kategorisierung: Probleme bei der Annotation von Lernerkorpora. In M. Walter & P. Grommes (Eds.), Fortgeschrittene Lernervarietäten: Korpuslinguistik und Zweispracherwerbsforschung (pp. 119–140). Tübingen: Max Niemeyer Verlag.
Lüdeling, A., Walter, M., Kroymann, E., & Adolphs, P. (2005). Multi-level error annotation in learner corpora. In Proceedings of corpus linguistics. Birmingham.
McCarthy, P. M. (2005). An assessment of the range and usefulness of lexical diversity measures and the potential of the measure of textual, lexical diversity (MTLD) (Unpublished doctoral dissertation). University of Memphis.
McCarthy, P. M., & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods, 42(2), 381–392. 

Meurers, D. (2005). On the use of electronic corpora for theoretical linguistics. case studies from the syntax of German. Lingua, 115(11), 1619–1639. 

Meurers, D. (2015). Learner corpora and natural language processing. In S. Granger, G. Gilquin, & F. Meunier (Eds.), The cambridge handbook of learner corpus research (pp. 537–566). Cambridge University Press. 

Meurers, D. (2020). Natural language processing and language learning. In C. A. Chapelle (Ed.), The concise encyclopedia of applied linguistics (pp. 817–831). Oxford: Wiley.
Meurers, D., & Dickinson, M. (2017). Evidence and interpretation in language learning research: Opportunities for collaboration with computational linguistics. Language Learning, 67(2). 

Michel, M., Murakami, A., Alexopoulou, T., & Meurers, D. (2019). Effects of task type on morphosyntactic complexity across proficiency: Evidence from a large learner corpus of A1 to C2 writings. Instructed Second Language Acquisition, 31, 124–152. 

Ott, N., & Ziai, R. (2010). Evaluating dependency parsing performance on German learner language. In M. Dickinson, K. Müürisep, & M. Passarotti (Eds.), Proceedings of the ninth international workshop on treebanks and linguistic theories (Vol. 91, pp. 175–186). Tartu, Estonia: Tartu University Press. [URL]
Petrov, S., & Klein, D. (2007). Improved inference for unlexicalized parsing. In Proceedings of the NAACL main conference (pp. 404–411). Rochester, New York.
Pilán, I., Vajjala, S., & Volodina, E. (2015). A readable read: Automatic assessment of language learning materials based on linguistic complexity. In Proceedings of CICLING 2015.
Reis, M. (2001). Bilden Modalverben im Deutschen eine syntaktische Klasse? In R. Müller & M. Reis (Eds.), Modalität und Modalverben im Deutschen. Hamburg: Helmut Buske. (Linguistische Berichte – Sonderhefte)
Seeker, W., & Kuhn, J. (2012). Making ellipses explicit in dependency conversion for a German treebank. In Proceedings of the 8th international conference on language resources and evaluation (pp. 3132–3139). Istanbul, Turkey.
Shain, C., van Schijndel, M., Futrell, R., Gibson, E., & Schuler, W. (2016). Memory access during incremental sentence processing causes reading time latency. In Proceedings of the workshop on computational linguistics for linguistic complexity (p. 49–58). Osaka.
Staples, S., Egbert, J., Biber, D., & Gray, B. (2016). Academic writing development at the university level: Phrasal and clausal complexity across level of study, discipline, and genre. Written Communication, 33(2), 149–183. 

Tagliamonte, S. A. (2011). Variationist sociolinguistics: Change, observation, interpretation. John Wiley & Sons.
Telljohann, H., Hinrichs, E., & Kübler, S. (2004). The TüBa-D/Z treebank: Annotating German with a context-free backbone. In Proceedings of the fourth LREC. Lissabon.
Tharwat, A. (2018). Classification assessment methods. Applied Computing and Informatics.
Thielen, C., Schiller, A., Teufel, S., & Stöckert, C. (1999). Guidelines für das Tagging deutscher Textkorpora mit STTS (Tech. Rep.). Stuttgart/Tübingen: Institut für Maschinelle Sprachverarbeitung Stuttgart and Seminar für Sprachwissenschaft Tübingen.
Vajjala, S., & Meurers, D. (2012). On improving the accuracy of readability classification using insights from second language acquisition. In Proceedings of the seventh BEA workshop (pp. 163–173).
Weiss, Z. (2015). More linguistically motivated features of language complexity in readability classification of German textbooks: Implementation and evaluation (Bachelor’s Thesis). Department of Linguistics, University of Tübingen. ([URL])
Weiss, Z. (2017). Using measures of linguistic complexity to assess German L2 proficiency in learner corpora under consideration of task-effects (Unpublished master’s thesis). University of Tübingen, Germany. ([URL])
Weiss, Z., & Meurers, D. (2018). Modeling the readability of German targeting adults and children: An empirically broad analysis and its cross-corpus validation. In Proceedings of the 27th COLING. Santa Fe, New Mexico, USA. [URL]
Weiss, Z., & Meurers, D. (2019a). Analyzing linguistic complexity and accuracy in academic language development of German across elementary and secondary school. In Proceedings of the 14th BEA workshop. Florence, Italy. 

Weiss, Z., & Meurers, D. (2019b). Broad linguistic modeling is beneficial for German L2 proficiency assessment. In A. Abel, A. Glaznieks, V. Lyding, & L. Nicolas (Eds.), Widening the scope of learner corpus research. Selected papers from the fourth learner corpus research conference. Louvain-La-Neuve: Presses Universitaires de Louvain.
Wöllstein, A. (2014). Topologisches Satzmodell (2nd ed.). Heidelberg: Winter.
Yoon, H.-J. (2017). Linguistic complexity in L2 writing revisited: Issues of topic, proficiency, and construct multidimensionality. System, 661, 130–141. 

Yoon, H.-J., & Polio, C. (2016). The linguistic development of students of English as a second language in two written genres. TESOL Quarterly, 275–301.
Ziai, R. (2018). Short answer assessment in context: The role of information structure (Unpublished doctoral dissertation). Eberhard-Karls Universität Tübingen.
Ziegler, N. (2018). Pre-task planning in L2 text-chat: Examining learners’ process and performance. Language Learning & Technology, 22(3), 193–213.
Cited by (3)
Cited by three other publications
Shin, Gyu-Ho, Boo Kyung Jung & Seongmin Mun
2024.
Transformer-based text similarity and second language proficiency: A case of written production by learners of Korean.
Natural Language Processing Journal 6
► pp. 100060 ff.

Möhring, Jupp & Monika Ruszó
2023.
Schreibkompetenz als Variable des Studienerfolgs von internationalen Studienanfänger*innen an deutschen Hochschulen. In
IDT 2022: *mit.sprache.teil.haben Band 1: Mit Sprache handeln,
► pp. 315 ff.

Weiss, Zarah, Kim Lange-Schubert, Barbara Geist & Detmar Meurers
2022.
Sprachliche Komplexität im Unterricht.
Zeitschrift für germanistische Linguistik 50:1
► pp. 159 ff.

This list is based on CrossRef data as of 17 october 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.