Texts of different lengths can be difficult to compare using quantitative methods. This is particularly true if many of the texts are extremely short, as is commonly the case with social media comments, where the median text length may be only a few dozen words. In this paper, I explore lengthwise scaling, that is, scaling applied to each text length separately, as a possible approach for getting around some of the statistical problems caused by different text lengths and short texts. I describe two implementations of this family of methods, lengthwise rarity scaling and lengthwise quantile scaling. I show in an exploratory analysis that these scaling methods support earlier results in terms of register differences between Reddit subreddits.
n.d. Reddit Comment Corpus. [URL] (27 March 2020).
Berber Sardinha, Tony
2014Comparing internet and pre-internet registers. In Multi-dimensional Analysis, 25 Years on: A Tribute to Douglas Biber [Studies in Corpus Linguistics 60], Tony Berber-Sardinha & Marcia Veirano-Pinto (eds), 81–105. Amsterdam: John Benjamins.
Biber, Douglas
1988Variation across Speech and Writing. Cambridge: CUP.
Biber, Douglas
1992The multi-dimensional approach to linguistic analyses of genre variation: An overview of methodology and findings. Computers and the Humanities 26(5–6): 331–345.
Biber, Douglas
1993Representativeness in corpus design. Literary and Linguistic Computing 8(4): 243–257.
2016Register variation on the searchable web: A multi-dimensional analysis. Journal of English Linguistics 44(2): 95–137.
Clarke, Isobelle & Grieve, Jack
2017Dimensions of abusive language on Twitter. In Proceedings of the First Workshop on Abusive Language Online, Zeerak Waseem, Wendy Hui Kyong, Dirk Hovy & Joel Tetreault (eds), 1–10. Vancouver BC: Association for Computational Linguistics.
Clarke, Isobelle & Grieve, Jack
2019Stylistic variation on the Donald Trump Twitter account: A linguistic analysis of tweets posted between 2009 and 2018. PLoS ONE 14(9).
Conrad, Susan & Biber, Douglas
2001Variation in English: Multi-dimensional Studies. Eastbourne: Pearson Education.
Covington, Michael A. & McFall, Joe D.
2010Cutting the Gordian Knot: The Moving-Average Type-Token Ratio (MATTR). Journal of Quantitative Linguistics 17(2): 94–100.
Eisenstein, Jacob
2013What to do about bad language on the internet. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), 359–369.
Francis, W. Nelson & Kučera, Henry
1964A Standard Corpus of Present-Day Edited American English, for Use with Digital Computers (Brown). Providence, RI: Brown University.
Hess, Carla W., Sefton, Karem M. & Landry, Richard G.
1986Sample size and type-token ratios for oral language of preschool children. Journal of Speech and Hearing Research 29: 129–134.
Hess, Carla W., Haug, Holly T. & Landry, Richard G.
1989The reliability of type-token ratios for the oral language of school age children. Journal of Speech and Hearing Research 32: 536–540.
Hiltunen, Turo
2014Choice of national variety in the English-language Wikipedia. In Texts and Discourses of New Media, Jukka Tyrkkö & Sirpa Leppänen (eds), n.p. Helsinki: VARIENG. [URL] (8 June 2020).
Koizumi, Rie & In’nami, Yo
2012Effects of text length on lexical diversity measures: Using short texts with less than 200 tokens. System 40(4): 554–564.
Kubát, Miroslav & Milička, Jiří
2013Vocabulary richness measure in genres. Journal of Quantitative Linguistics 20(4): 339–349.
This list is based on CrossRef data as of 15 march 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.