Unquestioned Practices in Psychological Research

This presentation critically considers some practices in psychological research that are typically undertaken in the same way in any piece of research from both an historical and current perspective. Some of these practices are overt (e.g., choice of alpha, use of power, sample selection), while others are arguably more covert (e.g., insufficient consideration of plausible rival hypotheses) when undertaking research. These considerations can be historically understood by contrasting the introduction of Fisher and Neyman-and-Pearson statistical procedures in the 1940s to 1950s into psychology with Donald Campbell’s development of a taxonomy for threats to validity in research design in the late 1950s. The tension between the two aspects of research practice is arguably reflected by a regular two-decade cycle of introspection about the use of statistical procedures in psychological research since the 1950s. The most recent manifestation of this cycle being inflamed concerns about the reproducibility of published research. Contradictions arising across these cycles are reviewed. It will be argued here that the almost unavoidable use of self-selected samples in psychological research may be a more essential explanation of current concerns about reproducibility than the more-often accused culprits (i.e., lack of power and questionable research practices). Different scenarios of sample selection will be considered, based on Pearson-Lawley selection theorem, that show the inherent limitations that self-selected samples may have on many, but not, all forms of psychological research.

Speaker bio

Dr Paul Dudgeon lectures in quantitative methods in the School of Psychological Sciences at The University of Melbourne. He has published research on the use of quantitative methods in the British Journal of Mathematical and Statistical Psychology, Multivariate Behavioral Research, Psychometrika, and Structural Equation Modelling. He has been a CI on 4 NHMRC Project Grants, 3 ARC Discovery Grants, and 1 ARC Linkage Grant. He is a member of the board of statistical reviewers for Lancet Psychiatry.