Has the Credibility of the Social Sciences Been Credibly Destroyed? Reanalyzing the “Many Analysts, One Data Set” Project

In 2018, Silberzahn, Uhlmann, Nosek, and colleagues published an article in which 29 teams analyzed the same research question with the same data: Are soccer referees more likely to give red cards to players with dark skin tone than light skin tone? The results obtained by the teams differed extensively. Many concluded from this widely noted exercise that the social sciences are not rigorous enough to provide definitive answers. In this article, we investigate why results diverged so much. We argue that the main reason was an unclear research question: Teams differed in their interpretation of the research question and therefore used diverse research designs and model specifications. We show by reanalyzing the data that with a clear research question, a precise definition of the parameter of interest, and theory-guided causal reasoning, results vary only within a narrow range. The broad conclusion of our reanalysis is that social science research needs to be more precise in its “estimands” to become credible.

• Publication year

  2021

• Venue

  Socius: Sociological Research for a Dynamic World

• Publication date

  2021-01-01

• Fields of study

  Sociology, Psychology

• Identifiers
  DOI 10.1177/23780231211024421
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.

  2020cited by this paper
• The Taboo Against Explicit Causal Inference in Nonexperimental Psychology

  2020cited by this paper
• What is Your Estimand? Defining the Target Quantity Connects Statistical Evidence to Theory

  2020influential reference
• The influence of hidden researcher decisions in applied microeconomics

  2020cited by this paper
• The causal interpretation of estimated associations in regression models

  2019cited by this paper
• Scientific Utopia III: Crowdsourcing Science

  2019cited by this paper
• Takeaways from the Special Issue on The Practice of Replication

  2019cited by this paper
• Transparency and Replication in Cross-national Survey Research: Identification of Problems and Possible Solutions

  2019cited by this paper
• Replicator degrees of freedom allow publication of misleading failures to replicate

  2019cited by this paper
• Transparent and Reproducible Social Science Research

  2019cited by this paper
• Model Uncertainty and the Crisis in Science

  2018cited by this paper
• Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

  2018cited by this paper
• We Ran 9 Billion Regressions: Eliminating False Positives through Computational Model Robustness

  2018cited by this paper
• Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results

  2018influential reference
• Transparency, Reproducibility, and the Credibility of Economics Research

  2018cited by this paper
• Model Uncertainty and Robustness

  2017influential reference
• Caveats and pitfalls in crowdsourcing research: the case of soccer referee bias

  2017cited by this paper
• Replication in Social Science

  2017cited by this paper
• Increasing Transparency Through a Multiverse Analysis

  2016cited by this paper
• Estimating the reproducibility of psychological science

  2015cited by this paper
• Sensitivity Analysis Without Assumptions

  2015cited by this paper
• Crowdsourced research: Many hands make tight work

  2015cited by this paper
• Data Visualization in Sociology.

  2014cited by this paper
• The New Statistics

  2014cited by this paper
• Generalized Sensitivity Analysis and Application to Quasi-experiments ∗

  2014cited by this paper
• The statistical crisis in science

  2014cited by this paper
• Endogenous Selection Bias: The Problem of Conditioning on a Collider Variable.

  2014influential reference
• Handbook of Causal Analysis for Social Research

  2013cited by this paper
• Graphical Causal Models

  2013cited by this paper
• Drug development: Raise standards for preclinical cancer research

  2012cited by this paper
• Logistic Regression: Why We Cannot Do What We Think We Can Do, and What We Can Do About It

  2010cited by this paper
• Racial Discrimination Among NBA Referees

  2010cited by this paper
• The Sociology of Discrimination: Racial Discrimination in Employment, Housing, Credit, and Consumer Markets.

  2008cited by this paper
• Kausale Perspektiven in der vergleichenden Politikwissenschaft: X-zentrierte versus Y-zentrierte Forschungsdesigns

  2005cited by this paper
• Sensitivity to Exogeneity Assumptions in Program Evaluation

  2003cited by this paper
• Regression standard errors in clustered samples

  1994cited by this paper
• The Logic of Scientific Discovery

  1988cited by this paper
• The Sociology of Science: Theoretical and Empirical Investigations

  1975cited by this paper

• Examining fifty years of replication studies in management and economics: a retrospective Scientometric-Bayesian multilevel analysis

  2026cites this paper
• Let’s talk about language—and its role for replicability

  2025cites this paper
• The reliability of replications: a study in computational reproductions

  2025cites this paper
• Statistic deviation mode balancer (SDMB): A novel sampling algorithm for imbalanced data

  2025cites this paper
• Accelerating social science knowledge production with the coordinated open-source model

  2025cites this paper
• Flexibility in variable operationalization in social disorganization theory: A pilot study

  2025cites this paper
• Towards a Clearer Understanding of Causal Estimands: The Importance of Joint Effects in Longitudinal Designs With Time-Varying Treatments

  2025cites this paper
• The insights from the crowd: Drawing inferences from many approaches to key empirical questions in international business

  2025influential citation
• What can be learned when multiple analysts arrive at different estimates

  2025cites this paper
• Marital status and risk of cardiovascular disease – a multi-analyst study in epidemiology

  2025cites this paper
• The necessity of construct and external validity for deductive causal inference

  2025cites this paper
• Causal inference on human behaviour

  2024cites this paper
• Toward a more credible assessment of the credibility of science by many-analyst studies

  2024cites this paper
• A Habitat Selection Multiverse Reveals Largely Consistent Results Despite a Multitude of Analysis Options

  2024cites this paper
• Quantification and the homogeneous representation of urban students in school choice research and politics

  2024cites this paper
• Causal inference for psychologists who think that causal inference is not for them

  2024cites this paper
• Streamlining the self-correction process: a review of the use of replication research by organizational scholars

  2024cites this paper
• Replication Studies Using Secondary or Nonexperimental Datasets

  2024cites this paper
• Meta-Dominance Analysis – A Tool for the Assessment of the Quality of Digital Behavioural Data

  2024cites this paper
• Social classes, school performance, subjective expected utilities, and educational decisions

  2024cites this paper
• Heterogeneity in effect size estimates

  2024cites this paper
• Strategic ambiguity in the social sciences

  2023cites this paper
• A universe of uncertainty hiding in plain sight

  2023cites this paper
• Reply to Engzell: Maybe in plain sight but out of focus

  2023cites this paper
• Gütekriterien in der Soziologie

  2023cites this paper
• Implementation of a multiverse approach in developmental research: The longitudinal relation between s elf‐esteem and peer victimization

  2023cites this paper
• Theory and methods of the multiverse: an application for panel-based models

  2023cites this paper
• Multidimensional Signals and Analytic Flexibility: Estimating Degrees of Freedom in Human-Speech Analyses

  2023cites this paper
• Care to share? Experimental evidence on code sharing behavior in the social sciences

  2023cites this paper
• The fraternal birth-order effect as a statistical artefact: convergent evidence from probability calculus, simulated data, and multiverse meta-analysis

  2023cites this paper
• How to understand a research question—a challenging first step in setting up a statistical model

  2023cites this paper
• Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

  2022cites this paper
• From multiverse analysis to multiverse operationalisations: 262,143 ways of measuring well-being

  2022cites this paper
• Les sciences cognitives face aux changements climatiques : apports et limites pour l’éducation relative à l’environnement

  2022cites this paper
• The role of analytical variability in secondary data replications: a replication of Kim et al. (2014)

  2022influential citation
• Strengthening derivation chains in cognitive neuroscience: A special issue of Cortex.

  2022cites this paper
• Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

  2022cites this paper
• Moderators of reputation effects in peer-to-peer online markets: a meta-analytic model selection approach

  2022cites this paper
• Towards a Design-Based Approach to Accounting Research∗

  2022cites this paper
• How Robust Are Social Structural Predictors of Carbon Dioxide Emissions? A Multiverse Analysis

  2022cites this paper
• Ten Years of Psychology's Replicability Crisis:

  2022cites this paper
• Moderators of reputation effects in peer-to-peer online markets: a meta-analytic model selection approach

  2022cites this paper
• Dietary and Health Correlates of Sweetened Beverage Intake: Sources of Variability in the National Health and Nutrition Examination Survey (NHANES)

  2021cites this paper
• Why most psychological research findings are not even wrong

  2021cites this paper
• These Are Not the Effects You Are Looking for: Causality and the Within-/Between-Persons Distinction in Longitudinal Data Analysis

  2021cites this paper
• Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology

  2020cites this paper
• Towards Sensitivity Analysis: A Workflow

  year unknowncites this paper