Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

Significance Will different researchers converge on similar findings when analyzing the same data? Seventy-three independent research teams used identical cross-country survey data to test a prominent social science hypothesis: that more immigration will reduce public support for government provision of social policies. Instead of convergence, teams’ results varied greatly, ranging from large negative to large positive effects of immigration on social policy support. The choices made by the research teams in designing their statistical tests explain very little of this variation; a hidden universe of uncertainty remains. Considering this variation, scientists, especially those working with the complexities of human societies and behavior, should exercise humility and strive to better account for the uncertainty in their work.

• Publication year

  2022

• Venue

  Proceedings of the National Academy of Sciences of the United States of America

• Publication date

  2022-10-28

• Fields of study

  Sociology, Medicine, Political Science

• Identifiers
  DOI 10.1073/pnas.2203150119PMID 36306328PMCID 9636921
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• A Traveler’s Guide to the Multiverse: Promises, Pitfalls, and a Framework for the Evaluation of Analytic Decisions

  2021cited by this paper
• Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

  2021cited by this paper
• Has the Credibility of the Social Sciences Been Credibly Destroyed? Reanalyzing the “Many Analysts, One Data Set” Project

  2021cited by this paper
• Non-Standard Errors

  2021cited by this paper
• A meta-psychological perspective on the decade of replication failures in social psychology.

  2020cited by this paper
• What is Your Estimand? Defining the Target Quantity Connects Statistical Evidence to Theory

  2020cited by this paper
• Immigration and preferences for redistribution in Europe1

  2019cited by this paper
• Time to get personal? The impact of researchers choices on the selection of treatment targets using the experience sampling methodology

  2019cited by this paper
• Why Trust Science?

  2019cited by this paper
• Variability in the analysis of a single neuroimaging dataset by many teams

  2019cited by this paper
• The association between adolescent well-being and digital technology use

  2019cited by this paper
• The Emergence of Statistical Objectivity: Changing Ideas of Epistemic Vice and Virtue in Science

  2018cited by this paper
• The Quality of Response Time Data Inference: A Blinded, Collaborative Assessment of the Validity of Cognitive Models

  2018cited by this paper
• Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

  2018cited by this paper
• Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results

  2018cited by this paper
• Immigration Attitudes and Support for the Welfare State in the American Mass Public

  2017cited by this paper
• Inequality, Labor Market Segmentation, and Preferences for Redistribution

  2017cited by this paper
• Immigration and the welfare state: A cross-regional analysis of European welfare attitudes

  2017cited by this paper
• Immigration and Support for Social Policy: An Experimental Comparison of Universal and Means-Tested Programs

  2017cited by this paper
• Multilevel Modelling of Country Effects: A Cautionary Tale

  2016cited by this paper
• Estimating the reproducibility of psychological science

  2015cited by this paper
• Immigration, Integration, and Support for Redistribution in Europe

  2014cited by this paper
• Trust in Science

  2014cited by this paper
• The statistical crisis in science

  2014cited by this paper
• Does Immigration Undermine Public Support for Social Policy?

  2014influential reference
• Data from Paper “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant”

  2014cited by this paper
• False-Positive Psychology

  2011cited by this paper
• Even in Sweden: The Effect of Immigration on Support for Welfare State Spending

  2010cited by this paper
• A Cautionary Tale

  2006cited by this paper
• Why are welfare states in the US and Europe so different

  2006cited by this paper
• Fighting Poverty in the US and Europe: A World of Difference

  2004cited by this paper
• Fighting Poverty in the US and Europe

  2004cited by this paper
• Publishing as Prostitution? – Choosing Between One's Own Ideas and Academic Success

  2003cited by this paper
• The Structural Basis of Social Inequality

  1996cited by this paper
• The Politics of Inequality

  1958cited by this paper

• How Robust are Robustness Checks?

  2026cites this paper
• Ideological bias in the production of research findings

  2026cites this paper
• AI agents are ‘aeroplanes for the mind’: five ways to ensure that scientists are responsible pilots

  2026cites this paper
• Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiverse

  2026influential citation
• Multiverse analyses can be used to evaluate cross-lagged panel network models: An example with psychological flexibility.

  2026cites this paper
• Improving studies of sensitive topics using prior evidence: an informative Bayesian approach for list experiments

  2026cites this paper
• Recent advances in reproducibility and replicability in cartography and geographic information science: introduction to a special issue

  2026cites this paper
• Do fiscal rules undermine public investments? A review of empirical evidence

  2026cites this paper
• Exploring the Garden of Forking Paths in Empirical Software Engineering Research: A Multiverse Analysis

  2025cites this paper
• Developing synthetic microdata through machine learning for firm-level business surveys

  2025cites this paper
• Re-examining the effects of Western sanctions on democracy and human rights in the 21 st century

  2025cites this paper
• The insights from the crowd: Drawing inferences from many approaches to key empirical questions in international business

  2025influential citation
• Distorted claims about distortions, a response to Reflections on the House of Mirrors

  2025cites this paper
• The Systematic Multiverse Analysis Registration Tool for defining multiverse analyses

  2025cites this paper
• Are our 95% CIs only worth 45% confidence?

  2025cites this paper
• Putting clinical studies into better perspective by defining accuracy when effect size is small.

  2025cites this paper
• Divided Perceptions of Risk? A New Online Tool to Study the Many Flavors of Polarization

  2025cites this paper
• Robust statistical methods and the credibility movement of psychological science

  2025cites this paper
• Algorithmic Tradeoffs, Applied NLP, and the State-of-the-Art Fallacy

  2025cites this paper
• A scoping review on metrics to quantify reproducibility: a multitude of questions leads to a multitude of metrics

  2025cites this paper
• Multiverse Analysis to Advance Theory and Practice: Illustrating and Investigating Variability in CEO Gender Pay Inequality

  2025cites this paper
• A Tale of Open Science: Emergence of a New Normal

  2025cites this paper
• An experimentalist’s perspective on replicability in typology: commentary on “Replication and methodological robustness in quantitative typology” by Becker and Guzmán Naranjo

  2025cites this paper
• Decoupling or dependency? Investigating the joint effects of natural gas consumption and economic growth on carbon dioxide emissions in gulf cooperation council countries

  2025cites this paper
• To fill or not to fill: Comparing imputation methods for improved riverine long‐term biodiversity monitoring

  2025cites this paper
• Threatening for whom? National boundary-making, immigration, and support for the welfare state

  2025cites this paper
• Big team science reveals promises and limitations of machine learning efforts to model physiological markers of affective experience

  2025cites this paper
• Inequality and Social Ties: Evidence from 15 U.S. Data Sets

  2025cites this paper
• PCS-UQ: Uncertainty Quantification via the Predictability-Computability-Stability Framework

  2025cites this paper
• Marital status and risk of cardiovascular disease – a multi-analyst study in epidemiology

  2025cites this paper
• Preventing Harmful Data Practices by using Participatory Input to Navigate the Machine Learning Multiverse

  2025cites this paper
• Introducing the Journal of Robustness Reports

  2025cites this paper
• Lost in a Large EEG Multiverse? Comparing Sampling Approaches for Representative Pipeline Selection

  2025cites this paper
• The hidden complexity of the simple world of basic experimental psychology: the principal and practical limits of gaining psychological knowledge using the experimental method

  2025cites this paper
• SciSciGPT: Advancing Human-AI Collaboration in the Science of Science

  2025cites this paper
• The reliability of replications: a study in computational reproductions

  2025cites this paper
• Concerns about immigration and demand for social protection: The implications of political regimes

  2025cites this paper
• Innovation in the Design of Clinical Trials for Infectious Diseases: Focusing on Patients Over Pathogens

  2025cites this paper
• Robustification of structural equation modelling via global sensitivity analysis

  2025influential citation
• Revealing hidden sources of uncertainty in biodiversity trend assessments

  2025cites this paper
• The Case for Preregistering Quasi-Experimental Program and Policy Evaluations

  2025cites this paper
• Introducing synchronous robustness reports

  2025cites this paper
• An Analyst-Inspector Framework for Evaluating Reproducibility of LLMs in Data Science

  2025cites this paper
• Unravelling the Nuances of Data With Quantile Regression: A Comprehensive Tutorial.

  2025cites this paper
• Assessing quantitative methods in archaeology via simulated datasets: The Archaeoriddle challenge. Concept, project and motivations

  2025cites this paper
• Integrating Open Science Principles into Quasi-Experimental Social Science Research

  2025cites this paper
• Exploration, Confirmation, and Replication in the Same Observational Study: A Two Team Cross-Screening Approach to Studying the Effect of Unwanted Pregnancy on Mothers' Later Life Outcomes

  2025cites this paper
• Immigration and Public Support for Social Policy: Accounting for the Gender Composition of Immigrant Populations

  2025cites this paper
• PREDOMÍNIO DO POSITIVISMO NA CONTABILIDADE

  2025cites this paper
• Versatility, value and limitations of using health and demographic surveillance system data for secondary analyses: guidance for researchers, using examples from existing analyses

  2025cites this paper
• Is Inequality the Problem?

  2025cites this paper
• Bad stats: A regular series exploring slip-ups, snafus and salutary lessons from the world of statistics

  2025cites this paper
• Replication code availability over time and across fields: Evidence from the German Socio‐Economic Panel

  2024cites this paper
• Behavioral Risk Profiling: Measuring Loss Aversion of Individual Investors

  2024cites this paper
• A Framework for Evaluating Reproducibility and Replicability in Economics

  2024cites this paper
• MxML (Exploring the Relationship between Measurement and Machine Learning): Current State of the Field

  2024cites this paper
• Depth-to-diameter Ratios of Fresh Craters on the Moon and Implications for Surface Age Estimates

  2024cites this paper
• Grilling the data: application of specification curve analysis to red meat and all-cause mortality

  2024influential citation
• Promoting Reproducibility and Replicability in Political Science

  2024cites this paper
• “Head Up CPR” Is Not Ready for Widespread Adoption

  2024cites this paper
• Artificial intelligence and illusions of understanding in scientific research

  2024cites this paper
• Challenges in the Use of AI-Driven Non-Destructive Spectroscopic Tools for Rapid Food Analysis

  2024cites this paper
• On the Efficacy of Accuracy Prompts Across Partisan Lines: An Adversarial Collaboration

  2024cites this paper
• The devil is in the method details. Comment on 'Visual mental imagery: Evidence for a heterarchical neural architecture' by Spagna et al.

  2024cites this paper
• Same model, same data, but different outcomes: Evaluating the impact of method choices in structural equation modeling

  2024cites this paper
• Mass Reproducibility and Replicability: A New Hope

  2024cites this paper
• Is economics self‐correcting? Replications in the American Economic Review

  2024cites this paper
• Streamlining the self-correction process: a review of the use of replication research by organizational scholars

  2024cites this paper
• We urgently need a culture of multi-operationalization in psychological research

  2024cites this paper
• Civil Wars; Escalation and De-escalation

  2024cites this paper
• Unraveling the genetics of arsenic toxicity with cellular morphology QTL

  2024cites this paper
• Replication Studies Using Secondary or Nonexperimental Datasets

  2024cites this paper
• Revisiting the evidence on thermostatic response to democratic change: degrees of democratic support or researcher degrees of freedom?

  2024cites this paper
• Mathematical models: a state of exception

  2024cites this paper
• Meta-Dominance Analysis – A Tool for the Assessment of the Quality of Digital Behavioural Data

  2024cites this paper
• Specification uncertainty: what the disruption index tells us about the (hidden) multiverse of bibliometric indicators

  2024cites this paper
• Automating incidence and prevalence analysis in open cohorts

  2024cites this paper
• Research design, statistics and soil carbon: Responses to von Haden et al. and Spertus

  2024cites this paper
• The multiverse of universes: A tutorial to plan, execute and interpret multiverses analyses using the R package multiverse.

  2024cites this paper
• Enhancing Variable Importance in Random Forests: A Novel Application of Global Sensitivity Analysis

  2024cites this paper
• Subjective evidence evaluation survey for many-analysts studies

  2024cites this paper
• Heterogeneity in effect size estimates

  2024cites this paper
• Subjective logic as a complementary tool to meta-analysis to explicitly address second-order uncertainty in research findings: A case from infant studies.

  2024cites this paper
• One Question, Many Results—Why Epidemiological Studies Yield Heterogeneous Findings.

  2024cites this paper
• Toward a more credible assessment of the credibility of science by many-analyst studies

  2024cites this paper
• Theories and hypotheses: The forgotten plane of the multiverse.

  2024cites this paper
• The quantitative paradigm and the nature of the human mind. The replication crisis as an epistemological crisis of quantitative psychology in view of the ontic nature of the psyche

  2024cites this paper
• Overcoming the pitfalls of categorizing continuous variables in ecology, evolution and behaviour

  2024cites this paper
• New Data Sources for Demographic Research

  2024cites this paper
• Objective causal predictions from observational data

  2024cites this paper
• Subtle methodological variations substantially impact correlation test results in ecological time series

  2024cites this paper
• Reproducibility and interpretability in radiomics: a critical assessment

  2024cites this paper
• Characterisation and calibration of multiversal methods

  2024cites this paper
• The effect of seeing scientists as intellectually humble on trust in scientists and their research

  2024cites this paper
• Examining the replicability of online experiments selected by a decision market

  2024cites this paper
• Statistical analysis measurements in equity research: Utilizing language preference and pediatric surgical outcomes as an example.

  2024cites this paper
• The potential of benchmark challenges in the social sciences

  2024cites this paper
• Ideological Bias in Estimates of the Impact of Immigration

  2024cites this paper
• The hidden problem in Big Data: even infinite information does not guarantee consistent measurement

  2024cites this paper
• Misinformation and Disinformation in Statistical Methodology for Social Sciences: causes, consequences, and remedies

  2023cites this paper