Cover not available
Article published In:
Applied Pragmatics
Vol. 5:1 (2023) ► pp.41–63

Validating a test of L2 routine formulae to detect pragmatics learning in stay abroad

ORCID logo | The University of Melbourne
 | Sophia University
ORCID logo | Waseda University
 | Kansai University
 | Keio University
There has long been interest in the effect of stay abroad on pragmatic development in second language (L2) pragmatics research. However, few testing instruments exist to assess pragmatics learning in stay abroad and provide information about learners’ development to help institutions evaluate the success of stay abroad programs. In this study, we describe the adaptation and validation of Roever’s (2005) test of second language routine formulae to be used large-scale with Japanese stay-abroad learners. We follow Kane’s (2006) argument-based approach to validation, evaluating the first four inferences: Domain description, Evaluation, Generalization, and Explanation. We also investigate the test’s practicality within the institutional setting of intended use. We find evidence supporting the use of the test for providing information on pragmatics learning during stay abroad. The test can differentiate between low- and high-ability learners, shows adequate reliability, and reflects the roles of proficiency and exposure in pragmatics learning in stay abroad contexts. The test is also practical and requires few resources. Further steps, such as the investigation of the Extrapolation and Decision inference, are discussed.
Keywords: stay abroad, study abroad, L2 pragmatics, routine formulae, conventional expressions, argument-based validation
Article outline
  • 1.Introduction
  • 2.Background
    • 2.1Pragmatic development in study abroad: The case of routine formulae
    • 2.2Past tests of routine formula recognition and the argument-based approach
  • 3.This study
  • 4.Method
    • 4.1Participants
    • 4.2Instruments
    • 4.3Procedures
    • 4.4Data analysis
  • 5.Results
    • 5.1RQ1 Practicality: What is the cost of creating, administering, and scoring the test?
    • 5.2RQ2 Domain description: Do items sample the domain of routine formula knowledge sufficiently based on expert judgment?
    • 5.3RQ3 evaluation: Can item scoring be argued to follow clear procedures, is computation of total scores defensible, and does item difficulty spread from easy to difficult?
    • 5.4RQ4 generalization: Is the sample of items representative of the universe of generalization based on acceptable Cronbach’s alpha reliability?
    • 5.5Explanation
      • 5.5.1RQ5a Criterion: Does the test correlate moderately to highly with a general proficiency measure like TOEFL?
      • 5.5.2RQ5b group differences 1: Does recognition of routine formulae increase with length of stay abroad?
      • 5.5.3RQ5c explanation: Group differences 2: Do test-takers with stay abroad outperform a proficiency-matched group without stay abroad?
  • 6.Discussion
  • 7.Conclusion
  • Acknowledgements
  • References
Published online: 4 March 2022
DOI logo
https://doi.org/10.1075/ap.20015.roe
References (54)
References
Bachman, L. F. (2004). Statistical analyses for language assessment. Cambridge University Press. DOI logoGoogle Scholar
Bachman, L. F., & Palmer, A. S. (2010). Language assessment in practice: Developing language tests and justifying their use in the real world. Oxford University Press.Google Scholar
Baker-Smemoe, W., Dewey, D. P., Bown, J., & Martinsen, R. A. (2014). Variables affecting L2 gains during study abroad. Foreign Language Annals, 47 (3), 464–486. DOI logoGoogle Scholar
Bardovi-Harlig, K. (2009). Conventional expressions as a pragmalinguistic resource: Recognition and production of conventional expressions in L2 pragmatics. Language Learning, 59 (4), 755–795. DOI logoGoogle Scholar
Bardovi-Harlig, K., & Bastos, M. T. (2011). Proficiency, length of stay, and intensity of interaction and the acquisition of conventional expressions in L2 pragmatics. Intercultural Pragmatics, 8 1, 347–384. DOI logoGoogle Scholar
Bardovi-Harlig, K., & Dörnyei, Z. (1998). Do language learners recognize pragmatic violations? Pragmatic vs. grammatical awareness in instructed L2 learning. TESOL Quarterly, 32 1, 233–259. DOI logoGoogle Scholar
Bardovi-Harlig, K., Mossman, S., & Vellenga, H. E. (2015). The effect of instruction on pragmatic routines in academic discussion. Language Teaching Research, 19 (3), 324–350. DOI logoGoogle Scholar
Barron, A. (2019). Pragmatic development and stay abroad. Journal of Pragmatics, 146 1, 43–53. DOI logoGoogle Scholar
Brecht, R., Davidson, D., & Ginsberg, R. (1995). Predictors of foreign language gain during study abroad. In B. F. Freed (Ed.), Second language acquisition in a study abroad context (pp. 37–66). John Benjamins. DOI logoGoogle Scholar
Chapelle, C. A. (2021). Argument-based validation in testing and assessment. Sage. DOI logoGoogle Scholar
Chapelle, C. A., Enright, M. K., & Jamieson, J. (Eds.) (2008). Building a validity argument for the Test of English as a Foreign Language. Routledge.Google Scholar
Coulmas, F. (1979). On the sociolinguistic relevance of routine formulae. Journal of Pragmatics, 3 1, 239–266. DOI logoGoogle Scholar
Davidson, D. E. (2010). Study abroad: When, how long, and with what results? New data from the Russian front. Foreign Language Annals, 43 (1), 6–26. DOI logoGoogle Scholar
Di Silvio, F., Donovan, A., & Malone, M. E. (2014). The effect of study abroad homestay placements: Participant perspectives and oral proficiency gains. Foreign Language Annals, 47 (1), 168–188. DOI logoGoogle Scholar
Ebel, R. L. (1964). The social consequences of educational testing. In Proceedings of the 1963 invitational conference on testing problems (pp. 130–143). Educational Testing Service.Google Scholar
ETS. (2010). Linking TOEFL iBT scores to IELTS scores – A research report. ETS.Google Scholar
Geeslin, K. L., & Schmidt, L. B. (2018). Study abroad and L2 learner attitudes. In C. Sanz & A. Morales-Font (Eds.), The Routledge handbook of study abroad research and practice (pp. 385–405). Routledge. DOI logoGoogle Scholar
Hulstijn, J. H. (2011). Language proficiency in native and nonnative speakers: An agenda for research and suggestions for second-language assessment. Language Assessment Quarterly, 8 (3), 229–249. DOI logoGoogle Scholar
(2015). Language proficiency in native and non-native speakers: Theory and research (Vol. 411). John Benjamins Publishing Company. DOI logoGoogle Scholar
Ikeda, N. (2017). Measuring L2 oral pragmatic abilities for use in social contexts: Development and validation of an assessment instrument for L2 pragmatics performance in university settings (Unpublished doctoral dissertation). The University of Melbourne, Australia.
Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed.) (pp. 17–64). American Council on Education/Praeger Publishers.Google Scholar
Matsumura, S. (2001). Learning the rules for offering advice: A quantitative approach to second language socialization. Language Learning, 51 (4), 635–679. DOI logoGoogle Scholar
(2003). Modelling the relationships among interlanguage pragmatic development, L2 proficiency, and exposure to L2. Applied Linguistics, 24 (4), 465–491. DOI logoGoogle Scholar
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational Measurement (3rd ed.) (pp. 13–103). American Council on Education & Macmillan.Google Scholar
Myles, F., & Cordier, C. (2017). Formulaic sequence (FS) cannot be an umbrella term in SLA: Focusing on psycholinguistic FSs and their identification. Studies in Second Language Acquisition, 39 (1), 3–28. DOI logoGoogle Scholar
Pawley, A., & Syder, F. H. (1983). Two puzzles for linguistic theory: Nativelike selection and nativelike fluency. In J. C. Richards & R. W. Schmidt (Eds.), Language and communication (pp. 191–226). Longman.Google Scholar
Powers, D. E., Roever, C., Huff, K., & Trapani, C. (2003). Validating LanguEdge Courseware Scores Against Faculty Ratings and Student Self-assessments. (ETS Research Report RR-03-11). Educational Testing Service. DOI logoGoogle Scholar
(2015). L2 pragmatic development in study abroad contexts. Peter Lang. DOI logoGoogle Scholar
(2019). Pragmatic development of Chinese during study abroad: A cross-sectional study of learner requests. Journal of Pragmatics, 146 1, 137–149. DOI logoGoogle Scholar
Roever, C. (2005). Testing ESL pragmatics. Peter Lang. DOI logoGoogle Scholar
(2012). What learners get for free (and when): Learning of routine formulae in ESL and EFL environments. ELT Journal, 66 (1), 10–21. DOI logoGoogle Scholar
Roever, C., Fraser, C., & Elder, C. (2014). Testing ESL sociopragmatics. Peter Lang. DOI logoGoogle Scholar
Roever, C., & Phakiti, A. (2018). Quantitative methods in language assessment and acquisition research. Routledge. DOI logoGoogle Scholar
Roever, C., Wang, S., & Brophy, S. (2014). Learner background factors and learning of second language pragmatics. International Review of Applied Linguistics, 52 (4), 377–401. DOI logoGoogle Scholar
Sánchez-Hernández, A., & Alcón-Soler, E. (2019). Pragmatic gains in the study abroad context: Learners’ experiences and recognition of pragmatic routines. Journal of Pragmatics, 146 1, 54–71. DOI logoGoogle Scholar
Schauer, G. A. (2006). Pragmatic awareness in ESL and EFL contexts: Contrast and development. Language Learning, 56 (2), 269–318. DOI logoGoogle Scholar
Schauer, G. (2009). Interlanguage pragmatic development: The study abroad context. Bloomsbury Publishing.Google Scholar
Schenker, T. (2018). Making short-term study abroad count – Effects on German language skills. Foreign Language Annals, 51 (2), 411–429. DOI logoGoogle Scholar
Sell, F., Renkwitz, K., Sickinger, P., & Schneider, K. P. (2019). Measuring pragmatic competence on the functional and lexical level: The development of German high-school students’ requests during a stay abroad in Canada. Journal of Pragmatics, 146 1, 106–120. DOI logoGoogle Scholar
Shively, R. L. (2013). Learning to be funny in Spanish during study abroad: L2 humor development. Modern Language Journal, 97 1, 930–946. DOI logoGoogle Scholar
(2015). Developing interactional competence during study abroad: Listener responses in L2 Spanish. System, 48 1, 86–98. DOI logoGoogle Scholar
Sireci, S., & Faulkner-Bond, M. (2014). Validity evidence based on test content. Psicothema, 26 (1), 100–107. DOI logoGoogle Scholar
Sinclair, J. (1991). Corpus, concordance and collocation. Oxford University Press.Google Scholar
Taguchi, N. (2014). Development of interactional competence in Japanese as a second language: Use of incomplete sentences as interactional resources. Modern Language Journal, 98 (2), 518–535. DOI logoGoogle Scholar
Taguchi, N., Li, S., & Xiao, F. (2013). Production of formulaic expressions in L2 Chinese: A developmental investigation in a study abroad context. Chinese as a Second Language Research, 2 (1), 23–58. DOI logoGoogle Scholar
Taguchi, N., Xiao, F., & Li, S. (2016). Effects of intercultural competence and social contact on speech act production in a Chinese study abroad context. Modern Language Journal, 100 1, 775–796. DOI logoGoogle Scholar
Tannenbaum, R. J., & Baron, P. A. (2011). Mapping TOEFL® ITP scores onto the Common European Framework of Reference. ETS.Google Scholar
Tare, M., Golonka, E., Lancaster, A. K., Bonilla, C., Doughty, C. J., Belnap, R. K., & Jackson, S. R. (2018). The role of cognitive aptitudes in a study abroad language-learning environment. In C. Sanz & A. Morales-Font (Eds.), The Routledge handbook of study abroad research and practice (pp. 406–420). Routledge. DOI logoGoogle Scholar
Tullock, B., & Ortega, L. (2017). Fluency and multilingualism in study abroad: Lessons from a scoping review. System, 71 1, 7–21. DOI logoGoogle Scholar
Xiao, F. (2015). Adult second language learners’ pragmatic development in the study-abroad context: A review. Frontiers: The Interdisciplinary Journal of Study Abroad, 25 1, 132–149. DOI logoGoogle Scholar
Xu, Y. (2019). Changes in interlanguage complexity during study abroad: A meta-analysis. System, 80 1, 199–211. DOI logoGoogle Scholar
Yang, J. (2016). CFL learners’ recognition and production of pragmatic routine formulae. Chinese as a Second Language. The Journal of the Chinese Language Teachers Association, USA, 51 (1), 29–61. DOI logoGoogle Scholar
Youn, S. J. (2015). Validity argument for assessing L2 pragmatics in interaction using mixed methods. Language Testing, 32 (2), 199–225. DOI logoGoogle Scholar
Cited by (1)

Cited by one other publication

Higuchi, Yuki, Makiko Nakamuro, Carsten Roever, Miyuki Sasaki & Tomoko Yashima
2023. Impact of studying abroad on language skill development: Regression discontinuity evidence from Japanese university students. Journal of the Japanese and International Economies 70 ► pp. 101284 ff. DOI logo

This list is based on CrossRef data as of 4 august 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.