Article published In:
ITL - International Journal of Applied LinguisticsVol. 166:2 (2015) ► pp.278–306
Vol. 166:2 (2015) ► pp.278–306
Assessing vocabulary size through multiple-choice formats
Issues with guessing and sampling rates
In most tests of vocabulary size, knowledge is assessed through multiple-choice formats. Despite advantages such as ease of scoring, multiple-choice tests (MCT) are accompanied with problems. One of the more central issues has to do with guessing and the presence of other construct-irrelevant strategies that can lead to overestimation of scores. A further challenge when designing vocabulary size tests is that of sampling rate. How many words constitute a representative sample of the underlying population of words that the test is intended to measure? This paper addresses these two issues through a case study based on data from a recent and increasingly used MCT of vocabulary size: the Vocabulary Size Test. Using a criterion-related validity approach, our results show that for multiple-choice items sampled from this test, there is a discrepancy between the test scores and the scores obtained from the criterion measure, and that a higher sampling rate would be needed in order to better represent knowledge of the underlying population of words. We offer two main interpretations of these results, and discuss their implications for the construction and use of vocabulary size tests.
References (44)
Alderson, J.C. (2005). Diagnosing foreign language proficiency: The interface between learning and assessment. London: Continuum.
Bachman, L.F. (1990). Fundamental considerations in language testing. Oxford: Oxford University Press.
Beglar, D. (2010). A Rasch-based validation of the Vocabulary Size Test. Language Testing, 27(1), 101–118. 
Elgort, I. (2011). Deliberate learning and vocabulary acquisition in a second language. Language Learning, 61(2), 367–413.
. (2013). Effects of L1 definitions and cognate status of test items on the Vocabulary Size Test. Language Testing, 30(2), 253–272.
Embretson, S.E. (1996). The new rules of measurement. Psychological Assessment, 8(4), 341–349.
Eyckmans, J. (2004). Measuring receptive vocabulary size: Reliability and validity of the yes/no vocabulary test for French-speaking learners of Dutch. Utrecht: LOT.
Gyllstad, H. (2012). Validating the Vocabulary Size Test. A classical test theory approach. Poster presented at The Ninth Annual Conference of EALTA, Innsbruck, Austria, 31 May – 3 June. Retrieved from: <[URL]>
Horst, M., Cobb, T., & Meara, P. (1998). Beyond A Clockwork Orange: Acquiring second language vocabulary through reading. Reading in a Foreign Language, 111, 207–223.
Huibregtse, I., Admiraal, W., & Meara, P. (2002). Scores on a yes-no vocabulary test: Correction for guessing and response style. Language Testing, 19(3), 227–245.
Kamimoto, T. (2008). Nation’s vocabulary levels test and its successors: A re-appraisal. Unpublished PhD Thesis. University of Wales, Swansea.
Kane, M.T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73.
Laufer, B., & Goldstein, Z. (2004). Testing vocabulary knowledge: Size, strength, and computer adaptiveness. Language Learning, 54(3), 399–436.
Meara, P. (1992). EFL vocabulary tests. ERIC Clearinghouse.
. (1996). The dimensions of lexical competence. In G. Brown, K. Malmkjaer, & J. Williams (Eds.), Performance and competence in second language acquisition (pp. 35–53). Cambridge: Cambridge University Press.
Meara, P., & Buxton, B. (1987). An alternative to multiple choice vocabulary tests. Language Testing, 4(2), 142–154.
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749.
Nation, I.S.P. (2006). How large a vocabulary is needed for reading and listening? Canadian Modern Language Review/La Revue Canadienne Des Langues Vivantes, 63(1), 59–82.
. (1990). Teaching and learning vocabulary. Heinle & Heinle.
Nation, I.S.P., & Beglar, D. (2007). A vocabulary size test. The Language Teacher, 31(7), 9–13.
Nation, I.S.P., & Gu, P.Y. (2007). Focus on vocabulary. Sydney: NCELTR Publications.
Nation, P. (1993). Using dictionaries to estimate vocabulary size: essential, but rarely followed, procedures. Language Testing, 10(1), 27–40.
Nation, I.S.P. (2012). Vocabulary size test instructions and description. Retrieved from <[URL]>. Last revised on October 23, 2012.
Paul, P.V., Stallman, A.C., & O’Rourke, J.P. (1990). Using three test formats to assess good and poor reader’s word knowledge. Technical Report No. 509 of the Center for the Study of Reading. University of Illinois. Retrieved from <[URL]>
Pellicer-Sánchez, A., & Schmitt, N. (2012). Scoring Yes–No vocabulary tests: Reaction time vs. nonword approaches. Language Testing, 29(4), 489–509.
Read, J. (1998). Validating a test to measure depth of vocabulary knowledge. In A.J. Kunnan (Ed.) Validation in language assessment (pp.41–61). Mahwah, NJ: Lawrence Erlbaum Associates.
. (2000). Assessing vocabulary. Cambridge: Cambridge University Press.
Rupp, A.A., Ferne, T., & Choi, H. (2006). How assessing reading comprehension with multiple-choice questions shapes the construct: A cognitive processing perspective. Language Testing, 23(4), 441–474.
Saragi, T., Nation, I.S.P., & Meister, F. (1978). Vocabulary learning and reading. System, 61, 72–78.
Schmitt, N. (2010). Researching vocabulary: A vocabulary research manual. Houndmills: Palgrave Macmillan.
Schmitt, N., & Meara, P. (1997). Researching vocabulary through a word knowledge framework: Word associations and verbal suffixes. Studies in Second Language Acquisition, 191, 17–36.
Schmitt, N., Ng, J.W.C., & Garras, J. (2011). The word associates format: Validation evidence. Language Testing, 28(1), 105–126.
Schmitt, N., & Schmitt, D. (2014). A reassessment of frequency and vocabulary size in L2 vocabulary teaching. Language Teaching, 47(4), 484–503.
Schmitt, N., Schmitt, D., & Clapham, C. (2001). Developing and exploring the behaviour of two new versions of the Vocabulary Levels Test. Language Testing, 18(1), 55–88.
Stenius Staehr, L. (2009). Vocabulary knowledge and advanced listening comprehension in English as a foreign language. Studies in Second Language Acquisition, 311, 577–607.
Stewart, J. (2012). A multiple-choice test of active vocabulary knowledge. Vocabulary Learning and Instruction, 1(1), 53–59.
. (2014). Do multiple-choice options inflate estimates of vocabulary size on the VST? Language Assessment Quarterly, 11(3), 271–282.
Stewart, J., & White, D.A. (2011). Estimating guessing effects on the vocabulary levels test for differing degrees of word knowledge. TESOL Quarterly, 45(2), 370–380.
Uden, J., Schmitt, D., & Schmitt, N. (2014). Jumping from the highest graded readers to ungraded novels: Four case studies. Reading in a Foreign Language, 26(1), 1–28.
Waring, R., & Takaki, M. (2003). At what rate do learners learn and retain new vocabulary from reading a graded reader. Reading in a Foreign Language, 15(2), 130–163.
Weir, C.J. (2005). Language testing and validation. Houndmills: Palgrave McMillan.
Wesche, M., & Paribakht, T.S. (1996). Assessing second language vocabulary knowledge: Depth versus breadth. Canadian Modern Language Review, 53(1), 13–40.
Wood, B. (1928). New York experiments with new-type modern language tests. New York, NY: Macmillan.
Zhang, X. (2013). The I don’t know option in the Vocabulary Size Test. TESOL Quarterly, 47(4), 790–811.
Cited by (77)
Cited by 77 other publications
Chen, Po-Heng, Rachael C. Hulme, Lena M. Blott & Jennifer M. Rodd
2024. The Web-based Open-access Reliable Decision on Synonyms (WORDS) English Vocabulary Test. Journal of Cognition 7:1 
Gyllstad, Henrik, Tanja Kupisch & Anika Lloyd-Smith
2024. Development and initial validation of a yes/no vocabulary test for North Sámi. ITL - International Journal of Applied Linguistics
Hamada, Akira, Yuko Hoshino, Masumi Kojima, Kazumi Aizawa, Tatsuo Iso & Yuichiro Kobayashi
2024. How many L2 word meanings can learners recall? A latent trait approach to vocabulary size estimation. Applied Linguistics
Milliner, Brett, Kriss Lange, Joshua Matthews & Riko Umeki
2024. Examining EFL learners’ comprehension of derivational forms: The role of overlap with base word knowledge, word frequency, and contextual support. Language Teaching Research
Qi, Shiwei, Mark Feng Teng & Ailan Fu
2024. LexCH: a quick and reliable receptive vocabulary size test for Chinese Learners. Applied Linguistics Review 15:2 ► pp. 643 ff.
Robles-García, Pablo, Stuart McLean, Jeffrey Stewart, Ji-young Shin & Claudia Helena Sánchez-Gutiérrez
2024. The Development and Initial Validation of O-WSVLT, a Meaning-Recall Online L2 Spanish Vocabulary Levels Test. Language Assessment Quarterly 21:2 ► pp. 181 ff.
Sevigny, Paul, Lindsay Mack, Lance Stilp & Maiko Berger
2024. High-Frequency Vocabulary: Moving From Recognition to Recall Level on Quizlet. Sage Open 14:2
Shafi, Sehrish & Sam Hellmuth
2024. Stress Placement in English Loanwords by Speakers of Mirpur Pahari in the UK. In The Phonetics and Phonology of Heritage Languages, ► pp. 107 ff.
Stewart, Jeffrey, Henrik Gyllstad, Christopher Nicklin & Stuart McLean
2024. Establishing meaning recall and meaning recognition vocabulary knowledge as distinct psychometric constructs in relation to reading proficiency. Language Testing 41:1 ► pp. 89 ff.
Stoeckel, Tim, Hung Tan Ha, Duyen Thi Bich Nguyen & Christopher Nicklin
2024. Can an extended-matching second-language vocabulary test format bridge the gap between meaning-recognition and meaning-recall?. Research Methods in Applied Linguistics 3:2 ► pp. 100109 ff.
Stoeckel, Tim, Stuart McLean, Young Ae Kim, Yukie Shinhara & Allie Spencer Patterson
2024. The role of context in the word family debate. System 126 ► pp. 103453 ff.
Uchihara, Takumi, Kazuya Saito, Satsuki Kurokawa, Kotaro Takizawa & Yui Suzukida
2024. Declarative and Automatized Phonological Vocabulary Knowledge: Recognition, Recall, Lexicosemantic Judgment, and Listening‐Focused Employability of Second Language Words. Language Learning
Vicary, Anne & Jeanine Treffers-Daller
2024. Reading against all odds: creative coping strategies of international students in HE. International Journal of Bilingual Education and Bilingualism 27:8 ► pp. 1128 ff.
Zhang, Haiwei, Peng Sun, Yaowaluk Bianglae & Winda Widiawati
2024. The development of a Chinese vocabulary proficiency test (CVPT) for learners of Chinese as a second/foreign language . Language Testing 41:2 ► pp. 412 ff.
Alzahrani, Alaa
2023. LexArabic: A receptive vocabulary size test to estimate Arabic proficiency. Behavior Research Methods 56:6 ► pp. 5529 ff.
Bardel, Camilla, Henrik Gyllstad & Jörgen Tholin
2023. Research on foreign language learning, teaching, and assessment in Sweden 2012–2021. Language Teaching 56:2 ► pp. 223 ff.
Dang, Thi Ngoc Yen, Duy Van Vu & Thi My Hang Nguyen
2023. Researching Vocabulary: Mixed Methods Research. In The Encyclopedia of Applied Linguistics, ► pp. 1 ff.
De Wilde, Vanessa
2023. The auditory picture vocabulary test for English L2: A spoken receptive meaning-recognition test intended for Dutch-speaking L2 learners of English. Language Teaching Research
Robles-García, Pablo, Glen H. Wallace & Claudia Helena Sánchez-Gutiérrez
Stoeckel, Tim, Tomoko Ishii, Young Ae Kim, Hung Tan Ha, Nam Thi Phuong Ho & Stuart McLean
2023. A comparison of contextualized and non-contextualized meaning-recall vocabulary test formats. Research Methods in Applied Linguistics 2:3 ► pp. 100075 ff.
Sun, Ting & Hyunjeong Nam
2023. Variations in Vocabulary Depth Among Second Language Learners with Comparable Vocabulary Breadths: An Empirical Investigation. Journal of Psycholinguistic Research 52:6 ► pp. 2473 ff.
Tam, Ho I (Anna) & Barry Lee Reynolds
2023. The relationship between extramural English engagement and the vocabulary size of L1 Cantonese speakers in Macau. ITL - International Journal of Applied Linguistics 174:1 ► pp. 49 ff.
Warnby, Marcus, Hans Malmström & Kajsa Yang Hansen
2023. Linking scores from two written receptive English academic vocabulary tests—The VLT-Ac and the AVT. Language Testing 40:3 ► pp. 548 ff.
Zhang, Haiwei, Peijian Paul Sun, Xueyan Zhang, Silin Fan & Yunji Zhang
2023. A Systematic Review of Chinese Character Size Tests From 1930 to 2021. Sage Open 13:3
B. Silva, Breno
2022. The Importance of General and Academic Vocabulary Learning. In Writing to Learn Academic Words [Second Language Learning and Teaching, ], ► pp. 1 ff.
B. Silva, Breno
2022. Study 1—The Assessment of Academic Vocabulary: Developing a Reliable Academic Placement Tool. In Writing to Learn Academic Words [Second Language Learning and Teaching, ], ► pp. 81 ff.
Nall, Matthew, Ari Huhta & Behzad Mansouri
2022. Close‐Ended Vocabulary Assessments. In The TESOL Encyclopedia of English Language Teaching, ► pp. 1 ff.
Urdaniz, Raquel Perez & Sophia Skoufaki
2022. Spanish L1 EFL learners’ recognition knowledge of English academic vocabulary: The role of cognateness, word frequency and length. Applied Linguistics Review 13:4 ► pp. 661 ff.
Warnby, Marcus
2022. Receptive academic vocabulary knowledge and extramural English involvement – is there a correlation?. ITL - International Journal of Applied Linguistics 173:1 ► pp. 120 ff.
Warnby, Marcus
2023. Academic vocabulary knowledge among adolescents in university preparatory programmes. Journal of English for Academic Purposes 61 ► pp. 101203 ff.
Yunjiu, Luo, Wei Wei & Ying Zheng
2022. Artificial Intelligence-Generated and Human Expert-Designed Vocabulary Tests: A Comparative Study. Sage Open 12:1
Zeng, Yuntao, Qiuxia Lu, Matthew P. Wallace, Yawei Guo, Chun-Wai Fan & Xiaofei Chen
2022. Understanding Sustainable Development of English Vocabulary Acquisition: Evidence from Chinese EFL Learners. Sustainability 14:11 ► pp. 6532 ff.
Zhang, Songshan & Xian Zhang
2022. The relationship between vocabulary knowledge and L2 reading/listening comprehension: A meta-analysis. Language Teaching Research 26:4 ► pp. 696 ff.
Busby, Nicole Louise
Busby, Nicole Louise
2024. Exploring the Effects of Early Extramural English Exposure on the Vocabulary Size of University Students. Education Sciences 14:4 ► pp. 372 ff.
Gyllstad, Henrik, Stuart McLean & Jeffrey Stewart
2021. Using confidence intervals to determine adequate item sample sizes for vocabulary tests: An essential but overlooked practice. Language Testing 38:4 ► pp. 558 ff.
Ha, Hung Tan
2021. A Rasch-based validation of the Vietnamese version of the Listening Vocabulary Levels Test. Language Testing in Asia 11:1
SALMELA, ROSA, MINNA LEHTONEN, STEFANO GARUSI & RAYMOND BERTRAM
2021. Lexize: A test to quickly assess vocabulary knowledge in Finnish. Scandinavian Journal of Psychology 62:6 ► pp. 806 ff.
Schmitt, Norbert, Karen Dunn, Barry O’Sullivan, Laurence Anthony & Benjamin Kremmel
2021. Introducing Knowledge‐based Vocabulary Lists (KVL). TESOL Journal 12:4
Stewart, Jeffrey, Tim Stoeckel, Stuart McLean, Paul Nation & Geoffrey G. Pinchbeck
2021. WHAT THE RESEARCH SHOWS ABOUT WRITTEN RECEPTIVE VOCABULARY TESTING. Studies in Second Language Acquisition 43:2 ► pp. 462 ff.
Stoeckel, Tim, Stuart McLean & Paul Nation
2021. LIMITATIONS OF SIZE AND LEVELS TESTS OF WRITTEN RECEPTIVE VOCABULARY KNOWLEDGE. Studies in Second Language Acquisition 43:1 ► pp. 181 ff.
Dong, Yang, Yi Tang, Sammy Xiao-Ying Wu, Wei-Yang Dong & Zhen Li
2020. Longitudinal Effects of Mediums of Word Explanation on L2 Vocabulary Learning Strategies Among Chinese Grade-7 Students. Frontiers in Psychology 11
Duff, Dawna & Melissa Brydon
2020. Estimates of individual differences in vocabulary size in English: how many words are needed to ‘close the vocabulary gap’?. Journal of Research in Reading 43:4 ► pp. 454 ff.
Koizumi, Rie & Yo In’nami
2020. Structural Equation Modeling of Vocabulary Size and Depth Using Conventional and Bayesian Methods. Frontiers in Psychology 11
McLean, Stuart, Jeffrey Stewart & Aaron Olaf Batty
2020. Predicting L2 reading proficiency with modalities of vocabulary knowledge: A bootstrapping approach. Language Testing 37:3 ► pp. 389 ff.
Reynolds, Barry Lee, Wei-Hua Wu & Ying-Chun Shih
2020. Which Elements Matter? Constructing Word Cards for English Vocabulary Growth. Sage Open 10:2
Robles-García, Pablo
2020. 3K-LEx. Desarrollo y validación de una prueba de amplitud léxica en español. Journal of Spanish Language Teaching 7:1 ► pp. 64 ff.
Robles‐García, Pablo
2022. Receptive vocabulary knowledge in L2 learners of Spanish: The role of high‐frequency words. Foreign Language Annals 55:4 ► pp. 963 ff.
Schmitt, Norbert, Paul Nation & Benjamin Kremmel
2020. Moving the field of vocabulary assessment forward: The need for more rigorous test development and validation. Language Teaching 53:1 ► pp. 109 ff.
Schmitt, Norbert & Diane Schmitt
2020. Vocabulary in Language Teaching,
Zhang, Pengchong & Suzanne Graham
2020. Vocabulary learning through listening: Comparing L2 explanations, teacher codeswitching, contrastive focus-on-form and incidental learning. Language Teaching Research 24:6 ► pp. 765 ff.
Zhang, Xian, Jianda Liu & Haiyang Ai
2020. Pseudowords and guessing in the Yes/No format vocabulary test. Language Testing 37:1 ► pp. 6 ff.
Aviad-Levitzky, Tami, Batia Laufer & Zahava Goldstein
2019. The New Computer Adaptive Test of Size and Strength (CATSS): Development and Validation. Language Assessment Quarterly 16:3 ► pp. 345 ff.
Masrai, Ahmed
2019. Vocabulary and Reading Comprehension Revisited: Evidence for High-, Mid-, and Low-Frequency Vocabulary Knowledge. Sage Open 9:2
Masrai, Ahmed
2022. The Development and Validation of a Lemma-Based Yes/No Vocabulary Size Test. Sage Open 12:1
Peters, Elke, Tom Velghe & Tinne Van Rompaey
Puimège, Eva & Elke Peters
2019. Learning L2 vocabulary from audiovisual input: an exploratory study into incidental learning of single words and formulaic sequences. The Language Learning Journal 47:4 ► pp. 424 ff.
Silva, Breno B. & Agnieszka Otwinowska
2019. VST as a reliable academic placement tool despite cognate inflation effects. English for Specific Purposes 54 ► pp. 35 ff.
Stoeckel, Tim, Jeffrey Stewart, Stuart McLean, Tomoko Ishii, Brandon Kramer & Yuko Matsumoto
2019. The relationship of four variants of the Vocabulary Size Test to a criterion measure of meaning recall vocabulary knowledge. System 87 ► pp. 102161 ff.
Janebi Enayat, Mostafa & Esmat Babaii
2018. Reliable predictors of reduced redundancy test performance: The interaction between lexical bonds and test takers’ depth and breadth of vocabulary knowledge. Language Testing 35:1 ► pp. 121 ff.
Masrai, Ahmed & James Milton
2018. Measuring the contribution of academic and general vocabulary knowledge to learners' academic achievement. Journal of English for Academic Purposes 31 ► pp. 44 ff.
Masrai, Ahmed & James Milton
2019. How many words do you need to speak Arabic? An Arabic vocabulary size test. The Language Learning Journal 47:5 ► pp. 519 ff.
McLean, Stuart
2018. Evidence for the Adoption of the Flemma as an Appropriate Word Counting Unit. Applied Linguistics 39:6 ► pp. 823 ff.
Peters, Elke & Stuart Webb
2018. INCIDENTAL VOCABULARY ACQUISITION THROUGH VIEWING L2 TELEVISION AND FACTORS THAT AFFECT LEARNING. Studies in Second Language Acquisition 40:3 ► pp. 551 ff.
Petrescu, Maria Claudia & Rena Helms-Park
2018. The Lexical Development of Canadian-Born Romanian L1 Bilingual Kindergarteners. Languages 3:3 ► pp. 33 ff.
Reynolds, Barry Lee, Ying-Chun Shih & Wei-Hua Wu
2018. Modeling Taiwanese adolescent learners' English vocabulary acquisition and retention: The washback effect of the College Entrance Examination Center's reference word list. English for Specific Purposes 52 ► pp. 47 ff.
Roche, Thomas & Michael Harrington
2018. Offshore and onsite placement testing for English pathway programmes. Journal of Further and Higher Education 42:3 ► pp. 415 ff.
Stoeckel, Tim & Tomomi Sukigara
2018. A Serial Multiple‐Choice Format Designed to Reduce Overestimation of Meaning‐Recall Knowledge on the Vocabulary Size Test. TESOL Quarterly 52:4 ► pp. 1050 ff.
LAUFER, BATIA & TAMI AVIAD–LEVITZKY
2017. What Type of Vocabulary Knowledge Predicts Reading Comprehension: Word Meaning Recall or Word Meaning Recognition?. The Modern Language Journal 101:4 ► pp. 729 ff.
Mainz, Nina, Zeshu Shao, Marc Brysbaert & Antje S. Meyer
2017. Vocabulary Knowledge Predicts Lexical Processing: Evidence from a Group of Participants with Diverse Educational Backgrounds. Frontiers in Psychology 8
Stewart, Jeffrey, Stuart McLean & Brandon Kramer
2017. A Response to Holster and Lake Regarding Guessing and the Rasch Model. Language Assessment Quarterly 14:1 ► pp. 69 ff.
Vilkaitė, Laura
2017. Incidental acquisition of collocations in L2. ITL - International Journal of Applied Linguistics 168:2 ► pp. 248 ff.
İnözü, Julide & Sehnaz Sahinkarakas
2016. Exploring Relations between Pre-Service English Teachers’ Psychological and Social Well-Being and their Perceptions of Positive Classroom Environment. Eurasian Journal of Applied Linguistics 2:2 ► pp. 79 ff.
Kremmel, Benjamin
2016. Word Families and Frequency Bands in Vocabulary Tests: Challenging Conventions. TESOL Quarterly 50:4 ► pp. 976 ff.
Stoeckel, Tim, Phil Bennett & Stuart Mclean
2016. Is “I Don't Know” a Viable Answer Choice on the Vocabulary Size Test?. TESOL Quarterly 50:4 ► pp. 965 ff.
Z Szabo, Csaba
2016. Exploring the Mental Lexicon of the Multilingual: Vocabulary Size, Cognate Recognition and Lexical Access in the L1, L2 and L3. Eurasian Journal of Applied Linguistics 2:2 ► pp. 1 ff.
[no author supplied]
This list is based on CrossRef data as of 8 january 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.