Investigating Variation in Replicability: A “Many Labs” Replication Project

Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of thirteen classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, ten effects replicated consistently. One effect – imagined contact reducing prejudice – showed weak support for replicability. And two effects – flag priming influencing conservatism and currency priming influencing system justification – did not replicate. We compared whether the conditions such as lab versus online or U.S. versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect.

• Publication year

  2022

• Venue

  Social Psychology

• Publication date

  2022-02-07

• Fields of study

  Psychology

• Identifiers
  DOI 10.1027/1864-9335/A000178
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Priming, Replication, and the Hardest Science

  2014cited by this paper
• Theory building through replication response to commentaries on the "Many labs" replication project

  2014cited by this paper
• Support for the replicability of imagined contact effects

  2014cited by this paper
• The Reproducibility Project: a model of large-scale collaboration for empirical research on reproducibility

  2014cited by this paper
• The Limits of Direct Replications and the Virtues of Stimulus Sampling

  2014influential reference
• Commentary on the Attempt to Replicate the Effect of the American Flag on Increased Republican Attitudes

  2014cited by this paper
• The Replication Recipe: What Makes for a Convincing Replication?

  2014cited by this paper
• Does merely going through the same moves make for a “direct” replication? Concepts, contexts, and operationalizations.

  2014influential reference
• A new etiquette for replication.

  2014influential reference
• An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science

  2012cited by this paper
• Journal of Experimental Psychology : General Mere Exposure to Money Increases Endorsement of Free-Market Systems and Social Inequality

  2012cited by this paper
• Open Science Collaboration

  2012cited by this paper
• Implicit Social Cognitions Predict Sex Differences in Math Engagement and Achievement

  2011cited by this paper
• A Single Exposure to the American Flag Shifts Support Toward Republicanism up to 8 Months Later

  2011cited by this paper
• Most people are not WEIRD

  2010cited by this paper
• Running experiments on Amazon Mechanical Turk

  2010cited by this paper
• Elaboration enhances the imagined contact effect

  2010cited by this paper
• Instructional Manipulation Checks: Detecting Satisficing to Increase Statistical Power

  2009cited by this paper
• Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences

  2009cited by this paper
• The retrospective gambler’s fallacy: Unlikely events, constructing the past, and multiple universes

  2009cited by this paper
• Imagining Intergroup Contact Can Improve Intergroup Attitudes

  2007cited by this paper
• Why Most Published Research Findings Are False

  2005cited by this paper
• Social Cognition: Understanding Self and Others

  2004cited by this paper
• Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires.

  2004cited by this paper
• A meta-analytic review of prospective memory and aging.

  2004cited by this paper
• Complementary justice: effects of "poor but happy" and "poor but honest" stereotype exemplars on system justification and implicit activation of the justice motive.

  2003cited by this paper
• Math = male, me = female, therefore math ≠ me.

  2002cited by this paper
• Math Male , Me Female , Therefore Math Me

  2002cited by this paper
• Measures of Anchoring in Estimation Tasks

  1995cited by this paper
• The Return of the Repressed: Dissonance Theory Makes a Comeback

  1992cited by this paper
• Stereotypes as Judgmental Heuristics: Evidence of Circadian Variations in Discrimination

  1990cited by this paper
• Under what conditions does theory obstruct research progress?

  1986cited by this paper
• Response Scales: Effects of Category Range on Reported Behavior and Comparative Judgments

  1985cited by this paper
• Mental Accounting and Consumer Choice

  1985cited by this paper
• The framing of decisions and the psychology of choice.

  1981influential reference
• The TEA Set: Tacit Knowledge and Scientific Networks

  1974cited by this paper
• Postdecision dissonance at post time.

  1968cited by this paper
• EXPERIMENTS IN WORDING QUESTIONS: II

  1941cited by this paper
• Prestige, suggestion, and attitudes.

  1936cited by this paper

• Rethinking where and how we publish in health sciences

  2026cites this paper
• Using prediction markets and forecasting surveys to predict 28 replication outcomes of classic articles in social psychology and judgement and decision making

  2026cites this paper
• Psychological counsellors awareness of cognitive biases: a qualitative study based on vignettes

  2026cites this paper
• Exploring the effects of positive imagined contact with gay men and lesbians on sexual prejudice among young heterosexuals

  2026cites this paper
• Social Psychology’s Empty-Self Metaphor and the Replication Crisis

  2026cites this paper
• LLM-Assisted Replication for Quantitative Social Science

  2026cites this paper
• Identifying Methodological Language in Psychology Abstracts: A Machine Learning Approach Using NLP and Embedding-Based Clustering

  2025cites this paper
• The power and peril of first offers in negotiations: a conceptual, meta-analytic, and experimental synthesis

  2025cites this paper
• Replicability: Terminology, Measuring Success, and Strategy

  2025cites this paper
• The Sequential Nature of Science: Quantifying Learning from a Sequence of Studies

  2025cites this paper
• A Tale of Open Science: Emergence of a New Normal

  2025cites this paper
• Investigating the effects of single-dose intranasal testosterone on economic preferences in a large randomized trial of men

  2025cites this paper
• Single-Dataset Meta-Analysis For Many-Analysts And Multiverse Studies

  2025cites this paper
• Neuroimaging genetics and developmental psychopathology: A systematic review.

  2025cites this paper
• The Design of Current Replication Studies: A Systematic Literature Review on the Variation of Study Characteristics

  2025influential citation
• The Prevalence of Replication Studies in Human Factors (2020-2024)

  2025cites this paper
• Seven Challenges for the Science of Animal Minds

  2025cites this paper
• Open minds, tied hands: Awareness, behavior, and reasoning on open science and irresponsible research behavior

  2025cites this paper
• Feather keratin-based hybrid adsorbents: Reproducibility and efficacy in metal removal from synthetic wastewater

  2025cites this paper
• Life-or-Death Framing of Public-Health Policy in a Pandemic

  2025cites this paper
• Concerns About Theorizing, Relevance, Generalizability, and Methodology Across Two Crises in Social Psychology

  2025cites this paper
• Under my umbrella: Rating scales obscure statistical power and effect size heterogeneity

  2025cites this paper
• Reflections on Conducting a Large Replication Project in Sports and Exercise Science

  2025cites this paper
• Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects

  2025cites this paper
• In Search of the Lost Interaction: A Theoretical and Methodological Framework for Researching Interactions.

  2025cites this paper
• Pseudo Effects: How Method Biases Can Produce Spurious Findings About Close Relationships

  2025cites this paper
• A large-scale study across the avian clade identifies ecological drivers of neophobia

  2025cites this paper
• How to Improve Theory Specification in Psychology

  2025cites this paper
• Contextual Changes, Credible Conclusions? A Direct and Conceptual Replication of Shen et al.'s (2019) Study on Online Image Credibility

  2025cites this paper
• Sequential Dynamic Bayesian Modeling for Single-Case Experimental Designs: A Novel Approach to Many Labs Research

  2025cites this paper
• “Black-and-White” thinking: Does visual contrast polarize moral judgment? Independent replications and extension of Study 1

  2025cites this paper
• Application of nanomaterials in surface-enhanced Raman spectroscopy for pesticide detection

  2025cites this paper
• Overconsumption and tourist ideology: how and why the Protestant Work Ethic moderates the effect of spending on consumption quantity under flat-rate pricing

  2025cites this paper
• Sign-Congruence, External Validity, and Replication

  2025cites this paper
• Cross-validation and predictive metrics in psychological research: Do not leave out the leave-one-out

  2025influential citation
• Spatial Frequency Maps in Human Visual Cortex: A Replication and Extension

  2025cites this paper
• Estimation of factorial expressions and its improvement through calibration: A replication and extension of Tversky and Kahneman (1973)

  2025cites this paper
• A Systematic Review on the Evolution of Power Analysis Practices in Psychological Research

  2025cites this paper
• On the Statistical Analysis of Studies With Attention Checks

  2025influential citation
• International initiatives to enhance awareness and uptake of open research in psychology: a systematic mapping review

  2025cites this paper
• The Controversy behind the Controversies: Scientific Discourse in the Twenty-First Century

  2025cites this paper
• How Do “Ugly Science” and “Dirty Science” Affect the Practice of Clinical Science?

  2025cites this paper
• A primer on open science for cognitive psychologists

  2025cites this paper
• Open-Science Training in American Psychological Association–Accredited Clinical-Psychology Programs: A Registered Report

  2025cites this paper
• Concerns About Replicability Across Two Crises in Social Psychology

  2025cites this paper
• What is empathy for laypeople? - A replication study of Hall, Schwartz, and Duong (2021).

  2025cites this paper
• Participant and Musical Diversity in Music Psychology Research

  2025cites this paper
• Optimizing Counteroffers: How Timing and Magnitude Shape Sale Prices and Impasses in 26 Million Asynchronous Online Negotiations

  2025cites this paper
• Four heuristics for theory selection in evolutionary cognitive archaeology

  2025cites this paper
• Fashion-AlterEval: A Dataset for Improved Evaluation of Conversational Recommendation Systems with Alternative Relevant Items

  2025cites this paper
• Some comments on robustness in comparative grammar research: commentary on “Replication and methodological robustness in quantitative typology” by Becker and Guzmán Naranjo

  2025cites this paper
• The quantitative paradigm and the nature of the human mind. The replication crisis as an epistemological crisis of quantitative psychology in view of the ontic nature of the psyche

  2024cites this paper
• The Replication Database: Documenting the Replicability of Psychological Science

  2024cites this paper
• A Framework for Evaluating Reproducibility and Replicability in Economics

  2024cites this paper
• Evaluating the Construct Validity of Instructional Manipulation Checks as Measures of Careless Responding to Surveys

  2024cites this paper
• The Language of (Non)Replicable Social Science

  2024cites this paper
• Open science practices in the false memory literature

  2024cites this paper
• Quality, Replicability, and Transparency in Research in Social Psychology

  2024cites this paper
• Heterogeneity in effect size estimates

  2024cites this paper
• Challenging the N-Heuristic: Effect size, not sample size, predicts the replicability of psychological science

  2024cites this paper
• Does racial and gender bias shape perception: examining how other’s pro-environmental behaviors are viewed

  2024cites this paper
• Open Access, Scholarly Communication, and Open Science in Psychology: An Overview for Researchers

  2024cites this paper
• Risky-Choice Framing Effects Result Partly From Mismatched Option Descriptions in Gains and Losses

  2024influential citation
• Evidence for Infant-directed Speech Preference Is Consistent Across Large-scale, Multi-site Replication and Meta-analysis

  2024cites this paper
• Exploring the multiverse of analysis options for the alcohol Stroop

  2024cites this paper
• Experimenter bias: exploring the interaction between participant’s and investigator’s gender/sex in VR

  2024cites this paper
• Meta-Dominance Analysis – A Tool for the Assessment of the Quality of Digital Behavioural Data

  2024cites this paper
• Estimating the Replicability of Psychology Experiments After an Initial Failure to Replicate

  2024cites this paper
• Publication bias in psychology: A closer look at the correlation between sample size and effect size

  2024cites this paper
• Imagine Being Humble: Integrating Imagined Intergroup Contact and Cultural Humility to Foster Inclusive Intergroup Relations

  2024cites this paper
• Tackling Challenges in Data Pooling: Missing Data Handling in Latent Variable Models with Continuous and Categorical Indicators

  2024cites this paper
• Advancing OSCM scientific knowledge by replicating empirical findings: Step-by-step procedure and illustration for transformative replication endeavors

  2024cites this paper
• Finding the right power balance: Better study design and collaboration can reduce dependence on statistical power

  2024cites this paper
• Assessment of Replicability and Efforts to Refine an Operant Conditioning Procedure for Larval Zebrafish

  2024cites this paper
• Bayes factors functions based on test statistics and non-local moment prior densities

  2024cites this paper
• Exploring the intra-individual reliability of tDCS: A registered report.

  2024cites this paper
• Quantifying replicability of multiple studies in a meta-analysis

  2024cites this paper
• A Multilab Replication of the Induced-Compliance Paradigm of Cognitive Dissonance

  2024cites this paper
• Predicting the replicability of social and behavioural science claims in COVID-19 preprints

  2024influential citation
• Remembering Reasons for Reform: A More Replicable and Reproducible Communication Literature Without the Rancor

  2024cites this paper
• Sounds Good to Me: How Communication Mode and Priming Affect Auditor Performance

  2024cites this paper
• An evaluation of the replicability of analyses using synthetic health data

  2024cites this paper
• Nonverbal behaviors perceived as most empathic in a simulated medical context

  2024cites this paper
• Scientific Advice and the COVID-19 Pandemic: Revisiting the Asian Disease Problem

  2024cites this paper
• Replication of null results: Absence of evidence or evidence of absence?

  2024cites this paper
• Standardisierter Datenmanagementplan für die Bildungsforschung (Stamp) – Ein projektbegleitendes Tool für das Forschungsdatenmanagement

  2024cites this paper
• Evaluating meta-analysis as a replication success measure

  2024cites this paper
• Examining the replicability of online experiments selected by a decision market

  2024cites this paper
• Reduce, reuse, recycle: Introducing MetaPipeX, a framework for analyses of multi‐lab data

  2024cites this paper
• Measuring transparency in the social sciences: political science and international relations

  2024cites this paper
• Beyond reweighting: On the predictive role of covariate shift in effect generalization

  2024cites this paper
• Cinquante ans de travaux sur l’effet d’ancrage. Quoi de neuf depuis Tversky et Kahneman (1974) ?

  2024cites this paper
• Does putting down your smartphone make you happier? the effects of restricting digital media on well-being

  2024cites this paper
• INVITED EDITORIAL: Undergraduate Involvement in “Many Labs” Studies

  2024cites this paper
• Why are replication rates so low?

  2024cites this paper
• Time for an Update

  2024cites this paper
• Exposing omitted moderators: Explaining why effect sizes differ in the social sciences

  2024cites this paper
• Claims about scientific rigour require rigour

  2024cites this paper
• Noise Versus Signal: What Can One Conclude When a Classic Finding Fails to Replicate?

  2024influential citation
• Religion, Brain & Behavior adopts stricter transparency standards

  2024cites this paper