Chapter 9
Task design and validity evidence for assessment of L2 pragmatics in interaction
This study examines how clear communicative goals and authentic interaction are ensured in task-based pragmatic assessment practices, particularly in designing role-play assessment tasks and developing task-appropriate rating criteria. Employing a concurrent mixed methods design, conversation analysis (CA) and multi-faceted Rasch measurement were used to investigate whether task-independent interactional features are elicited from role-play interactions and how interaction-specific rating categories function quantitatively. The CA findings indicated that various interactional organizations emerged from role-play task interactions as examinees oriented to the different degree of imposition and contextual variables embedded in the role-plays. Further, the interactional features included in the rating criteria created a substantial amount of variance in distinguishing varying degrees of pragmatic performance. The findings are discussed in terms of designing valid pragmatic assessment tasks and what features need to be included in developing rating criteria to ensure the validity evidence of task-based pragmatic assessment.
Article outline
- Introduction
- Background
- Assessing pragmatic competence in interaction and validity challenges
- Role-plays as assessment tasks and eliciting validity evidence
- The present study
- Methods
- Participants
- Examinees
- Interlocutors
- Raters
- Test instruments
- Role-plays
- Rating criteria
- Procedures
- Test administration
- Rater training
- Rating
- Data analysis
- Conversation analysis
- Multi-faceted Rasch measurement
- Results
- Conversation Analysis (CA) findings
- Recommendation letter request role-play
- Additional advising time request role-play
- Refusing a professor’s request role-play
- Deciding a meeting time
- Deciding a meeting mode
- Multi-faceted Rasch Measurement analysis
- Discussion
- Conclusion
-
References
-
Appendix
References (46)
References
Al-Gahtani, S., & Roever, C. (2012). Proficiency and sequential organization of L2 requests. Applied Linguistics, 33, 42–65. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Al-Gahtani, S., & Roever, C. (2014). Insert and post-expansion in L2 Arabic requests. System, 42, 189–206. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Atkinson, J., & Heritage, J. (Eds.) (1984). Structures of social action. Cambridge: Cambridge University Press.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Bardovi-Harlig, K., & Hartford, B. S. (Eds.) (2005). Interlanguage pragmatics: Exploring institutional talk. Mahwah, NJ: Lawrence Erlbaum Associates.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model (2nd ed.). New York, NY: Routledge.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Celce-Murcia, M. (2007). Rethinking the role of communicative competence in language teaching. In E. Alcón Soler & M. P. Safont Jordà (Eds.), Intercultural language use and language learning (pp. 41–57). Dordrecht: Springer. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Celce-Murcia, M., Dörnyei, Z., & Thurrell, S. (1995). Communicative competence: A pedagogically motivated model with content specifications. Issues in Applied Linguistics, 6, 5–35.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Chapelle, C. A. (2012). Validity argument for language assessment: The framework is simple… Language Testing, 29, 19–27. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Chapelle, C., Enright, M., & Jamieson, J. (Eds.) (2008). Building a validity argument for the Test of English as a Foreign Language. New York, NY: Routledge.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Eckes, T. (2011). Introduction to many-Facet Rasch measurement. Frankfurt: Peter Lang.![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Grabowski, K. (2013). Investigating the construct validity of a role-play test designed to measure grammatical and pragmatic knowledge at multiple proficiency levels. In S. Ross & G. Kasper (Eds.), Assessing second language pragmatics (pp. 149–171). Houndmills: Palgrave Macmillan. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Heritage, J. (1984). A change-of-state token and aspects of its sequential placement. In M. J. Atkinson & J. Heritage (Eds.), Structures of social action: Studies in conversation analysis (pp. 299–345). Cambridge: Cambridge University Press.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Hudson, T. D., Detmer, E., & Brown, J. D. (1992). A framework for testing cross-cultural pragmatics (Technical Report #2). Honolulu, HI: University of Hawai‘i, Second Language Teaching and Curriculum Center.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Hudson, T. D., Detmer, E., & Brown, J. D. (1995). Developing prototype measures of cross-cultural pragmatics (Technical Report #7). Honolulu, HI: University of Hawai‘i, Second Language Teaching and Curriculum Center.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Hughes, A. (1989). Testing for language teachers. Cambridge: Cambridge University Press.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Huth, T. (2010). Can talk be inconsequential? Social and interactional aspects of elicited second-language interaction. Modern Language Journal, 94, 537–553. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kane, M. (1992). An argument-based approach to validity. Psychological Bulletin, 112, 527–535. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kane, M. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (pp. 17–64). Westport, CT: American Council on Education.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kasper, G. (2006). Speech acts in interaction: Towards discursive pragmatics. In K. Bardovi-Harlig, C. Félix-Brasdefer & A. S. Omar (Eds.), Pragmatics and language learning (pp. 281–314). Honolulu, HI: Second Language Teaching and Curriculum Center, University of Hawai‘i.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kasper, G., & Rose, K. R. (2002). Pragmatic development in a second language. Malden, MA: Blackwell Publishing.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kasper, G. (2009). L2 pragmatic development. In W. C. Ritchie & T. K. Bhatia (Eds.), New handbook of second language acquisition (pp. 259–295). Leeds: Emerald.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kasper, G., & Ross, S. (2013). Assessing second language pragmatics: An overview and introductions. In S. J. Ross & G. Kasper (Eds.), Assessing second language pragmatics (pp. 1–40). Houndmills: Palgrave Macmillan. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kasper, G., & Youn, S. J. (2017). Transforming instruction to activity: Roleplay in language assessment. Applied Linguistics Review. Advance online publication. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Linacre, J. M. (2014). Facets Rasch measurement computer program, version 3.71.4. Chicago, IL: Winsteps.com.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Liu, J. (2007). Developing a pragmatic test for Chinese EFL learners. Language Testing, 24, 391–415. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Lunz, M. E., Wright, B. D., & Linacre, J. M. (1990). Measuring the impact of judge severity on examination scores. Applied Measurement in Education, 3, 331–345. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
McNamara, T. (1996). Measuring second language performance. New York, NY: Addison Wesley Longman.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
McNamara, T., & Knoch, U. (2012). The Rasch wars: The emergence of Rasch measurement in language testing. Language Testing, 29, 555–576. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Norris, J. M. (2009). Task-based teaching and testing. In M. H. Long & C. J. Doughty (Eds.), Handbook of language teaching (pp. 578–594). Oxford: Blackwell. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Norris, J. M. (2016). Current uses for task-based language assessment. Annual Review of Applied Linguistics, 36, 230–244.![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Okada, Y. (2010). Role-play in oral proficiency interviews: Interactive footing and interactional competencies. Journal of Pragmatics, 42, 1647–1668. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Okada, Y., & Greer, T. (2013). Pursuing a relevant response in oral proficiency interview role plays. In S. Ross & G. Kasper (Eds.), Assessing second language pragmatics (pp. 288–310). Houndmills: Palgrave Macmillan. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Pomerantz, A. (1984). Agreeing and disagreeing with assessments: Some features of preferred/dispreferred turn shapes. In J. M. Atkinson & J. Heritage (Eds.), Structures of social action: Studies in conversation analysis (pp. 57–101). Cambridge: Cambridge University Press.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Pomerantz, A., & Heritage, J. (2013). Preference. In J. Sidnell & T. Stivers (Eds.), The handbook of conversation analysis (pp. 210–228). Chichester: Wiley-Blackwell.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Purpura, J. E., Brown, J. D., & Schoonen, R. (2015). Improving the validity of quantitative measures in applied linguistics research. Language Learning, 65, 37–75. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Roever, C. (2006). Validation of a web-based test of ESL pragmalinguistics. Language Testing, 23, 229–256. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Roever, C. (2011). Testing of second language pragmatics: Past and future. Language Testing, 28, 463–481. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Ross, S. J., & Kasper, G. (Eds.) (2013). Assessing second language pragmatics. Houndmills: Palgrave Macmillan. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Sacks, H. (1992). Lectures on conversation (2 Vols.). Oxford: Blackwell.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Schegloff, E. A. (2007). Sequence organization in interaction: A primer in conversation analysis. Cambridge: Cambridge University Press. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Schumacker, R. E. (1999). Many-facet Rasch analysis with crossed, nested, and mixed designs. Journal of Outcome Measurement, 3, 323–338.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Taguchi, N., & Roever, C. (2017). Second language pragmatics. Oxford: Oxford University Press.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Teddlie, C., & Tashakkori, A. (2006). A general typology of research designs featuring mixed methods. Research in the Schools, 13(1), 12–28.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Walters, F. S. (2007). A conversation-analytic hermeneutic rating protocol to assess L2 oral pragmatic competence. Language Testing, 24, 155–183. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Youn, S. J. (2010). Task-based needs analysis of L2 pragmatics in an EAP context: Implications for teaching. Unpublished manuscript, University of Hawai‘i.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Youn, S. J. (2015). Validity argument for assessing L2 pragmatics in interaction using mixed methods. Language Testing, 32, 199–225. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Cited by (2)
Cited by two other publications
Su, Yunwen & Sun-Young Shin
2024.
Comparing two formats of data-driven rating scales for classroom assessment of pragmatic performance with roleplays.
Language Testing 41:2
► pp. 357 ff.
![DOI logo](//benjamins.com/logos/doi-logo.svg)
This list is based on CrossRef data as of 26 june 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.