Cover not available
Part of
Research Methods in the Study of L2 Writing Processes
Edited by Rosa M. Manchón and Julio Roca de Larios
[Research Methods in Applied Linguistics 5] 2023
► pp. 247–268

Chapter 12
Investigating cognitive processes during writing tests

Methodological considerations when triangulating data from eye tracking, keystroke logging, and stimulated recalls

| Universität Innsbruck
| Universität Innsbruck
| Universität Innsbruck

The purpose of this chapter is to critically discuss some of the key issues when investigating writing processes for the purposes of foreign language assessment research – a branch of research that currently tends to triangulate synchronous observational data (e.g., eye tracking, keystroke logging) with asynchronous data from stimulated verbal recalls or text analysis. We will discuss a range of methodological considerations that should be taken into account when researching foreign language writing processes in the context of language tests and beyond. We will exemplify and critically discuss key issues related to three of the methods predominantly used in this strand of research: eye tracking, keystroke logging, and stimulated recalls. We will illustrate these issues and decision-making processes at the various stages of research by critically reflecting on the lessons learned from two research projects of this kind conducted by the authors.

Article outline
  • Introduction
    • Rationales and aims of process-tracing methods in language testing research
    • Overview of research programme
  • Methodological decisions taken, challenges experienced, and solutions adopted
    • Systematising elicitation procedures for stimulated recalls
    • Defining pause length thresholds
    • Interpreting keystroke-logging data
    • Interpreting stimulated recall data
    • Comparing writing processes across languages
  • Conclusion
  • References
Published online: 23 October 2023
DOI logo
https://doi.org/10.1075/rmal.5.12gug
References
Barkaoui, K.
(2015) Test takers’ writing activities during the TOEFL iBT® writing tasks: A stimulated recall study. In TOEFL iBT®Research Report TOEFL iBT – 25. DOI logoGoogle Scholar
(2016) Examining the cognitive processes engaged by APTIS writing Task 4 on paper and on the computer. In British Council (Ed.), ARAGs research reports online AR-G/2016/1. Retrieved on 27 April 2023 from [URL]
(2019) What can L2 writers’ pausing behavior tell us about their L2 writing processes? Studies in Second Language Acquisition, 41(3), 529–554. DOI logoGoogle Scholar
Bereiter, C., & Scardamalia, M.
(1987) The psychology of written composition. Routledge.Google Scholar
Bowles, M. A.
(2018) Introspective verbal reports: Think-alouds and stimulated recall. In A. Phakiti, P. De Costa, L. Plonsky, & S. Starfield (Eds.), The Palgrave handbook of applied linguistics research methodology (pp. 339–358). Palgrave. DOI logoGoogle Scholar
Brunfaut, T., & McCray, G.
(2015) Looking into test-takers’ cognitive processes whilst completing reading tasks: A mixed-method eye-tracking and stimulated recall study. British Council (Ed.) ARAGs research reports online AR-G/2015/001. Retrieved on 27 April 2023 from [URL]Google Scholar
Conijn, R., Roeser, J. & van Zaanen, M.
(2019) Understanding the keystroke log: The effect of writing task on keystroke features. Reading and Writing, 32, 2353–2374. DOI logoGoogle Scholar
Conklin, K., Pellicer-Sánchez, A., & Carrol, G.
(2018) Eye-tracking: A guide for applied linguistics research. Cambridge University Press. DOI logoGoogle Scholar
Council of Europe
(2001) Common European Framework of Reference for Languages: Learning, teaching, assessment. Cambridge University Press. Retrieved on 27 April 2023 from [URL]
(2020) Common European Framework of Reference for Languages: Learning, teaching, assessment: Companion volume with new descriptors. Retrieved on 27 April 2023 from [URL]
Eberharter, K., Guggenbichler, E., Holzknecht, F., Schwarz, V., Zehentner, M., & Kremmel, B.
(2020) Probing the cognitive validity of the Linguaskill writing component: Exploring test takers’ cognitive processes using eye tracking and keystroke logging. (Unpublished report). Cambridge Assessment English Funded Research Programme Report Series (Round 9).Google Scholar
Field, J.
(2004) Psycholinguistics: The key concepts. Routledge. DOI logoGoogle Scholar
Flower, L., & Hayes, J. R.
(1981) A cognitive process theory of writing. College Composition and Communication, 32(4), 365–387. DOI logoGoogle Scholar
Galbraith, D., & Vedder, I.
(2019) Methodological advances in investigating L2 writing processes: Challenges and perspectives. Studies in Second Language Acquisition, 41(3), 633–645. DOI logoGoogle Scholar
Gass, S. M., & Mackey, A.
(2016) Stimulated recall methodology in second language research (2nd ed.). Routledge.Google Scholar
Guggenbichler, E.
(2020) Investigating writing across foreign languages: Same Matura task, different processes? (Unpublished MA thesis). Universität Innsbruck.
Kellogg, R. T.
(1996) A model of working memory in writing. In C. M. Levy & S. Ransdell (Eds.), The science of writing: Theories, methods, individual differences and applications (pp. 57–71). Lawrence Erlbaum Associates.Google Scholar
Knospe, Y.
(2017) Writing in a third language a study of upper secondary students’ texts, writing processes and metacognition (Doctoral dissertation). University of Antwerp & Umeå University. [URL]
Leijten, M., Van Horenbeeck, E., & Van Waes, L.
(2019) Analysing keystroke logging data from a linguistic perspective. In E. Lindgren & K. P. H. Sullivan (Eds.), Observing writing: Insights from keystroke logging and handwriting (pp. 71–95). Brill. DOI logoGoogle Scholar
Leijten, M., & Van Waes, L.
(2013) Keystroke logging in writing research: Using Inputlog to analyze and visualize writing processes. Written Communication, 30(3), 358–392. DOI logoGoogle Scholar
Leijten, M., Van Waes, L., & Van Horenbeeck, E.
(2015) Analyzing writing process data: A linguistic perspective. In G. Cislaru (Ed.), Writing(s) at the crossroads: The process – product interface (pp. 277–302). John Benjamins. DOI logoGoogle Scholar
Lindgren, E., & Sullivan, K. P. H.
(2006) Analysing online revision. In K.P.H. Sullivan & E. Lindgren (Eds.), Computer keystroke logging and writing: Methods and applications (pp. 157–188). Elsevier. DOI logoGoogle Scholar
López-Serrano, S., Roca de Larios, J., & Manchón, R. M.
(2019) Language reflection fostered by individual L2 writing tasks: Developing a theoretically motivated and empirically based coding system. Studies in Second Language Acquisition, 41(3), 503–527. DOI logoGoogle Scholar
Lu, X., & Révész, A.
(2021) Revising in a non-alphabetic language: The multi-dimensional and dynamic nature of online revisions in Chinese as a second language. System, 100, 1–13. DOI logoGoogle Scholar
Michel, M., Révész, A., Lu, X., Kourtali, N. -E., Lee, M., & Borges, L.
(2020) Investigating L2 writing processes across independent and integrated tasks: A mixed-methods study. Second Language Research, 36(3), 277–304. DOI logoGoogle Scholar
Murphy, L., & Roca de Larios, J.
(2010) Searching for words: One strategic use of the mother tongue by advanced Spanish EFL writers. Journal of Second Language Writing, 19(2), 61–81. DOI logoGoogle Scholar
Reichelt, M., Lefkowitz, N., Rinnert, C., & Schultz, J. M.
(2012) Key issues in foreign language writing. Foreign Language Annals, 45(1), 22–41. DOI logoGoogle Scholar
Révész, A., Kourtali, N. -E., & Mazgutova, D.
(2017) Effects of task complexity on L2 writing behaviors and linguistic complexity: Task complexity and L2 writing. Language Learning, 67(1), 208–241. DOI logoGoogle Scholar
Révész, A., Michel, M., & Lee, M.
(2017) Investigating IELTS Academic Writing Task 2: Relationships between cognitive writing processes, text quality, and working memory. In IELTS, British Council, idp IELTS Australia, & Cambridge English (Eds.), IELTS Research Reports Online Series 2017/3. Retrieved on 27 April 2023 from [URL]
(2019) Exploring second language writers’ pausing and revision behaviours. Studies in Second Language Acquisition, 41(3), 605–631. DOI logoGoogle Scholar
Roca de Larios, J., Manchón, R. M., & Murphy, L.
(2006) Generating text in native and foreign language writing: A temporal analysis of problem-solving formulation processes. The Modern Language Journal, 90(1), 100–114. DOI logoGoogle Scholar
Roca de Larios, J., Marín, J., & Murphy, L.
(2001) A temporal analysis of formulation processes in L1 and L2 writing. Language Learning, 51(3), 497–538. DOI logoGoogle Scholar
Roca de Larios, J., Murphy, L., & Manchón, R.
(1999) The use of restructuring strategies in EFL writing: A study of Spanish learners of English as a foreign language. Journal of Second Language Writing, 8(1), 13–44. DOI logoGoogle Scholar
Roca de Larios, J., Nicolás-Conesa, F., & Coyle, Y.
(2016) Focus on writers: Processes and strategies. In R. M. Manchón & P. K. Matsuda (Eds.), Handbook of second and foreign language writing (pp. 267–287). De Gruyter. DOI logoGoogle Scholar
Roorda, M.
(2021, May 22). Writing processes in different languages [Conference presentation]. L2WR Seminar 2021. Murcia, Spain. Retrieved on 27 April 2023 from [URL]
Sasaki, M.
(2000) Toward an empirical model of EFL writing processes: An exploratory study. Journal of Second Language Writing, 9(3), 259–291. DOI logoGoogle Scholar
Shaw, S. D., & Weir, C. J.
(2007) Examining writing: Research and practice in assessing second language writing. Cambridge University Press.Google Scholar
Spelman Miller, K., Lindgren, E., & Sullivan, K. P. H.
(2008) The psycholinguistic dimension in second language writing: Opportunities for research and pedagogy using computer keystroke logging. TESOL Quarterly, 42(3), 433–454. DOI logoGoogle Scholar
Tobii Technology AB
(2012) Tobii Studio Pro (3.4.5.). Tobii Technology AB.Google Scholar
Van Waes, L., & Leijten, M.
(2019) Inputlog: Help documentation (in progress). Retrieved on 27 April 2023 from [URL]Google Scholar
Van Waes, L., Leijten, M., Pauwaert, T., & Van Horenbeeck, E.
(2019) A multilingual copy task: Measuring typing and motor skills in writing with Inputlog. Journal of Open Research Software, 7(1), 30. DOI logoGoogle Scholar
Weir, C. J.
(2005) Language testing and validation: An evidence-based approach. Palgrave Macmillan. DOI logoGoogle Scholar
Wengelin, Å., Torrance, M., Holmqvist, K., Simpson, S., Galbraith, D., Johansson, V., & Johansson, R.
(2009) Combined eyetracking and keystroke-logging methods for studying cognitive processes in text production. Behavior Research Methods, 41(2), 337–351. DOI logoGoogle Scholar
Xu, C., & Xia, J.
(2019) Scaffolding process knowledge in L2 writing development: Insights from computer keystroke log and process graph. Computer Assisted Language Learning, 34(4), 583–608. DOI logoGoogle Scholar
Yu, G., He, L., & Isaacs, T.
(2017) The cognitive processes of taking IELTS Academic Writing Task 1: An eye-tracking study. In IELTS, British Council, idp IELTS Australia & Cambridge English (Eds.), IELTS Research Reports Online Series, 2017/2. Retrieved on 27 April 2023 from [URL]