The psychology of psychology: Inside MetaMelb
Led by Professor Simine Vazire in collaboration with Professor Fiona Fidler’s lab in History and Philosophy of Science, the MetaMelb lab studies what is essentially “the psychology of psychology research”. The initiative focuses on examining research practices and norms in psychology, particularly in social and personality psychology. They investigate researchers’ methodologies, their interpretation and communication of findings and scrutinise structures such as peer review and post-publication critique, all aimed at improving the field’s research practices.
The ‘Meta’ in MetaMelb refers to both ‘Metascience’ - the science of science, and ‘meta-research’ – research on research. As an interdisciplinary research group, they embrace this plurality of meaning.
The replication crisis
MetaMelb emerged in the wake of psychology’s ‘replication crisis’ and subsequent credibility revolution which led to a movement to improve research practices in the field.
“We wanted to provide some empirical evidence regarding whether and how things are improving in the field, and possible side effects or unintended consequences of reforms,” Professor Vazire explains.
These aims are exemplified by the work of Wendy Higgins, a post-doctoral research fellow at MetaMelb. During her Master of Research degree, she noticed many published studies lacked sufficient information about their measurement procedures, including studies used to support claims that autistic individuals have deficits in theory of mind and empathy. Recognising this as a substantial threat to the credibility of psychological research findings, Wendy now focuses on examining measurement practices in psychological sciences. She emphasises: “Research findings are only meaningful if they are based on valid measurements.”
Wendy Higgins, post-doctoral research fellow at MetaMelb
Likewise, PhD candidate Laura Conlon was drawn to metascience when she learned about the replication crisis and the danger questionable research practices pose to the validity and reliability of research. “I am motivated to investigate ways that we can improve scientific practices to facilitate greater trust in science,” she says.
Examining evidence synthesis and measurement practices
MetaMelb’s current research examines how the field synthesises evidence in meta-analyses and systematic reviews. They investigate the role of quality assessments and commensurability in these reviews, aiming to encourage more critical examination of primary studies when conducting reviews and meta-analyses.
“We think that systematic reviews provide an opportunity for post-publication critique and correction,” Professor Vazire explains.
Laura also emphasises the importance of this work, adding: “Meta-analysis is a form of evidence synthesis that is often used to inform policy decisions. However...the replication crisis illuminated that false positives in published papers may be more common than previously assumed.”
Her current research involves investigating the prevalence of quality assessments in published meta-analyses in social and personality psychology. She and the lab are also applying strict quality assessments to primary studies to examine potential changes in meta-analytic results.
Laura Conlon, PhD student
Simultaneously, MetaMelb is exploring the validity of measures and instruments used in psychological research, scrutinising common practices and seeking improvements. Wendy’s projects are central to this focus, including developing a new framework for evaluating measurement quality and conducting a survey of published studies to assess current measurement practices. This survey aims to answer three key questions:
- What are researchers measuring?
- How are they measuring it?
- Is there enough information to evaluate how well they are measuring it?
These efforts aim to enhance research practices and measurement validity in psychology, potentially influencing future methodological approaches.
Interdisciplinary connections
The group’s interdisciplinary connections have been fundamental to its success. As the replication crisis and credibility revolution in psychology has spurred similar meta-research movements across different fields, Professor Vazire collaborates with researchers in many other disciplines.
“It has been exciting – we frequently connect with scholars in those fields to exchange ideas and learn from each other’s attempts to improve research rigour and integrity.”
The interdisciplinary nature of meta-research has also been a highlight for Laura. “I enjoy discussing how open science practices differ across disciplines with other meta-researchers,” she shares.
Wendy agrees: “Measurement is hard, and working collaboratively to generate ideas on how to do it better is both challenging and energising.”
MetaMelb has connected with researchers from physics, economics, medicine, ecology and evolutionary biology, sports science, law, criminology and political science. In particular, their collaboration with the University’s History and Philosophy of Science program has proved essential “to better understand what makes a discipline scientific, and how to bolster the credibility of a scientific discipline”.
Broader impact
Professor Vazire’s influence extends beyond MetaMelb. She cofounded the Society for the Improvement of Psychological Science in 2016 and serves as Editor in Chief of the journal Psychological Science. “In [the Editor in Chief] role, I have the opportunity to try to put some of the improvements we study into practice and try to improve the incentive structure to encourage good research practices.”
MetaMelb, under Professor Vazire’s leadership, continues to play a crucial role in improving research practices in psychology and beyond. Their interdisciplinary collaborations and critical examination of research methodologies have put them at the forefront of the credibility revolution.
As Wendy sums up: “It is rewarding to work on projects that have the potential to lead to positive changes in research practices, which can increase the rigour of psychological research.”