Cover not available
Article published In:
Dutch Journal of Applied Linguistics
Vol. 9:1/2 (2020) ► pp.79–96

The New Statistics for applied linguistics

ORCID logo | VU Amsterdam

The New Statistics is an approach to scholarly research which offers an alternative to the problematic overreliance on significance testing currently plaguing the research literature. This paper describes the problems associated with significance testing and introduces the key concepts of the data-analysis that best fits with the goals of the New Statistics: estimation of effect sizes and confidence intervals. These concepts will be applied in a reanalysis of the summary data from an article that was recently published in this journal. This makes it possible to compare the estimation approach advocated by the New Statistics to the standard significance tests and to discuss potential drawbacks of this approach as a means of gathering quantitative evidence in support of our substantive hypotheses.

Keywords: New Statistics, estimation, confidence intervals, Bayesian Credible Intervals, significance testing, p-value
Article outline
  • 1.Introduction
  • 2.Does it feel non-native?
  • 3.The New Statistics
    • 3.1An estimate of the population effect size and the confidence interval
  • 4.Interpreting confidence intervals
  • 5.Conclusion
  • Acknowledgements
  • Notes
  • References
Available under the Creative Commons Attribution (CC BY) 4.0 license.

For any use beyond this license, please contact the publisher at [email protected].

Published online: 10 September 2020
DOI logo
https://doi.org/10.1075/dujal.19019.mul
References
American Psychological Association
(2010) Publication manual of the American Psychological Association (6th ed.). Washington, DC: American Psychological Association.Google Scholar
Amrhein, V., Greenland, S., & McShane, B.
(2019) Retire statistical significance. Nature, 5671, 305–307. DOI logoGoogle Scholar
Bakan, D.
(1966) The test of significance in psychological research. Psychological Bulletin, 661, 423–437. DOI logoGoogle Scholar
Berger, O. J., & Sellke, T.
(1987) Testing a point null hypothesis: The irreconcilability of P Values and evidence. Journal of the American Statistical Association, 821, 112–122.Google Scholar
Berkson, J.
(1942) Tests of significance considered as evidence. Journal of the American Statistical Association, 371, 325–335. DOI logoGoogle Scholar
Calin-Jageman, R. J., & Cumming, G.
(2019a) The New Statistics for better science: Ask how much, how uncertain, and what else is known. The American Statistician, 701, 271–280. DOI logoGoogle Scholar
(2019b) Estimation for better inference in neuroscience. ENeuro, 61, 1–11. DOI logoGoogle Scholar
Carver, R. P.
(1978) The case against significance testing. Harvard Educational Review, 481, 378–399. DOI logoGoogle Scholar
Chambers, C.
(2018) The seven deadly sins of psychology. A manifesto for reforming the culture of scientific practice. Princeton/Oxford: Princeton University Press.Google Scholar
Cohen, J.
(1988) Statistical power analysis for the behavioral sciences (2nd edition). New York, NY: Academic Press.Google Scholar
(1994) The earth is round (p < .05). American Psychologist, 491, 997–1003. DOI logoGoogle Scholar
Cumming, G.
(2012) Understanding the New Statistics. Effect sizes, confidence intervals, and meta-analysis. New York/London: Routledge.Google Scholar
(2014) The New Statistics: Why and how. Psychological Science, 251, 7–29. DOI logoGoogle Scholar
Cumming, G. & Calin-Jageman, R. J.
(2017) Introduction to the New Statistics. Estimation, open science, & beyond. New York/London: Routledge.Google Scholar
Falk, R., & Greenbaum, C. W.
(1995) Significance tests die hard: The amazing persistence of a probabilistic misconception. Theory & Psychology, 51, 75–98. DOI logoGoogle Scholar
Field, A.
(2015) Discovering statistics using IBM SPSS Statistics (4th ed.). London: Sage.Google Scholar
Gigerenzer, G.
(2004) Mindless statistics. Journal of Socio-Economics, 331, 587–606. DOI logoGoogle Scholar
Gigerenzer, G., & Marewski, J. N.
(2015) Surrogate science: The idol of a universal method for scientific inference. Journal of Management, 411, 421–440. DOI logoGoogle Scholar
Gigerenzer, G., Swijtink, Z., Porter, T., Daston, L., Beatty, J., & Kruger, L.
(1989) The Empire of Chance. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Hacking, I.
(1965) Logic of statistical inference. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Haller, H., & Krauss, S.
(2002) Misinterpretations of significance. A problem students share with their teachers? Methods of Psychological Research Online, 71, [URL]
Hubbard, R.
(2004) Alfabet soup: Blurring the distinctions between p’s and α’s in psychological research. Theory &Psychology, 141, 295–327. DOI logoGoogle Scholar
Hubbard, R., & Lindsay, R. M.
(2008) Why P values are not a useful measure of evidence in statistical significance testing. Theory & Psychology, 181, 69–88. DOI logoGoogle Scholar
Kelley, K.
(2007) Methods for the behavioral, educational, and social sciences: An R package. Behavior Research Methods, 391, 979–384. DOI logoGoogle Scholar
Kline, R. B.
(2013) Beyond significance testing. Statistics reform in the behavioral sciences. Washington, DC: American Psychological Association. DOI logoGoogle Scholar
Kruschke, J. K.
(2015) Doing Bayesian data analysis. A tutorial with R, Jags, and Stan (2nd ed.). London: Academic Press.Google Scholar
Kruschke, J. K., & Liddell, T. M.
(2018) The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychonomic Bulletin Review, 251, 178–206. DOI logoGoogle Scholar
Lambdin, C.
(2012) Significance tests as sorcery: Science is empirical – significance tests are not. Theory & Psychology, 221, 67–90. DOI logoGoogle Scholar
Lindley, D. V.
(2000) The philosophy of statistics. The Statistician, 491, 293–337.Google Scholar
Ly, A., Raj, A., Etz, A., Marsman, M., Gronau, Q. F., & Wagenmakers, E. J.
(2018) Bayesian reanalyses from summary statistics: A guide for academic consumers. Advances in Methods and Practices in Psychological Science, 11, 367–374. DOI logoGoogle Scholar
Maxwell, S. E., Delaney, H. D., & Kelley, K.
(2017) Designing experiments and analyzing data. A model comparison perspective (3th ed.). New York, NY: Routledge. DOI logoGoogle Scholar
McShane, B. B., Gal, D., Gelman, A., Robert, C., & Tackett, J. L.
(2019) Abandon statistical significance. The American Statistician, 731, 235–245. DOI logoGoogle Scholar
Meehl, P. E.
(1978) Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 461, 806–834. DOI logoGoogle Scholar
(1997) The problem is epistemology, not statistics: Replace significance tests by confidence intervals and quantify accuracy of risky numerical predictions. In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 393–425). Mahwah, NJ: Erlbaum.Google Scholar
Morey, R. D., Hoekstra, R., Rouder, J. N., Lee, M. D., & Wagenmakers, E. J.
(2016) The fallacy of placing confidence in confidence intervals. Psychonomic Bulletin Review, 231, 103–123. DOI logoGoogle Scholar
Mulder, G.
(2016) De kwaliteit van onderzoek. Dichotoom denken versus meta-analytisch denken. Tijdschrift voor Taalbeheersing, 381, 163–173. DOI logoGoogle Scholar
(2019) Een significant probleem. Tijdschrift voor Taalbeheersing, 411, 203–213. DOI logoGoogle Scholar
Neyman, J.
(1977) Frequentist probability and frequentist statistics. Synthese, 361, 97–131. DOI logoGoogle Scholar
Nickerson, R. S.
(2000) Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 51, 241–301. DOI logoGoogle Scholar
Norouzian, R., De Miranda, M., & Plonksy, L.
(2018) The Bayesian revolution in second language research: An applied approach. Language Learning, 681, 1032–1075. DOI logoGoogle Scholar
Oakes, M.
(1986) Statistical significance. New York, NY: Wiley.Google Scholar
Perezgonzalez, J. D.
(2015) Fisher, Neyman-Pearson, or NHST? A tutorial for teaching data testing. Frontiers in Psychology, 61, 1–11. DOI logoGoogle Scholar
Polya, G.
(1954) Mathematics and plausible reasoning, V1–2. Induction and analogy in mathematics, patterns of plausible inference. Princeton, NJ: Princeton University Press. DOI logoGoogle Scholar
Roozeboom, W. W.
(1960) The fallacy of the null-hypothesis significance test. Psychological Bulletin, 571, 416–428. DOI logoGoogle Scholar
Rosenthal, R., Rosnow, R. L., & Rubin, D. B.
(2000) Contrasts and effect sizes in behavioral research. A correlational approach. Cambridge, UK: Cambridge University Press.Google Scholar
Rouder, J. N., Speckman, P. L., Dongchu, S., Morey, R. D., & Iversen, G.
(2009) Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin and Review, 161, 225–237. DOI logoGoogle Scholar
Schmidt, F. L.
(1996) Statistical significance and cumulative knowledge in psychology: Implications for the training of researchers. Psychological Methods, 11, 115–129. DOI logoGoogle Scholar
Van Hilten, M., & Van Vuuren, S.
(2017) Does it ‘feel’ non-native? Native-speaker perceptions of information-structural transfer in L1 Dutch advanced EFL writing. Dutch Journal of Applied Linguistics, 61, 197–212. DOI logoGoogle Scholar
Wasserstein, R. L., & Lazar, N.
(2016) The ASA’s statement on P-values: Context, process, and purpose. The American Statistician, 701, 129–133. DOI logoGoogle Scholar
Wasserstein, R. L., Schirm, A. L., & Lazar, N. A.
(2019) Moving to a world beyond “p < .05”. The American Statistician, 731, 1–19. DOI logoGoogle Scholar
Wiens, S., & Nilsson, M. E.
(2017) Performing contrast analysis in factorial designs: From NHST to confidence intervals and beyond. Educational and Psychological Measurement, 771, 690–715. DOI logoGoogle Scholar
Zilliak, S. T., & McClosky, D. N.
(2008) The cult of statistical significance. How the standard error costs us jobs, justice, and lives. Ann Arbor, MN: The University of Michigan Press.Google Scholar