Model Uncertainty and the Crisis in Science

The “crisis in science” today is rooted in genuine problems of model uncertainty and lack of transparency. Researchers estimate a large number of models in the course of their research but only publish a small number of preferred results. Authors have much influence on the results of an empirical study through their choices about model specification. I advance methods to quantify the influence of the author—or at least demonstrate the scope an author has to choose a preferred result. Multimodel analysis, combined with modern computational power, allows authors to present their preferred estimate alongside a distribution of estimates from many other plausible models. I demonstrate the method using new software and applied empirical examples. When evaluating research results, accounting for model uncertainty and model robustness is at least as important as statistical significance.

• Publication year

  2018

• Venue

  Socius

• Publication date

  2018-05-10

• Fields of study

  Mathematics, Economics

• Identifiers
  DOI 10.1177/2378023117737206
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Model Uncertainty and Robustness

  2017cited by this paper
• Millionaire Migration and Taxation of the Elite

  2016cited by this paper
• Multicollinearity and Model Misspecification

  2016cited by this paper
• Endogenous Selection Bias: The Problem of Conditioning on a Collider Variable.

  2014cited by this paper
• The statistical crisis in science

  2014cited by this paper
• Are All Economic Facts Greatly Exaggerated? Theory Competition and Selectivity

  2013cited by this paper
• Assumptions Matter: Model Uncertainty and the Deterrent Effect of Capital Punishment

  2012cited by this paper
• Model Uncertainty in Sociological Research: An Application to Religion and Economic Growth

  2009cited by this paper
• Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference

  2007cited by this paper
• 1. The Scientific Model of Causality

  2005cited by this paper
• Why Most Published Research Findings Are False

  2005cited by this paper
• Do Frequency Representations Eliminate Conjunction Effects? An Exercise in Adversarial Collaboration

  2001cited by this paper
• Let's Take the Con Out of Econometrics

  1983cited by this paper
• Aix-Marseille University,

  year unknowncited by this paper

• Is College Really “the” Equalizer? New Evidence Addressing Unobserved Selection

  2026influential citation
• Gendered beauty inequalities? A multiverse analysis of physical attractiveness, occupational gender-typicality and earnings in the German labour market

  2025cites this paper
• The reliability of replications: a study in computational reproductions

  2025cites this paper
• Reproducibility, replicability, robustness, and generalizability in corpus linguistics

  2025cites this paper
• Problems With Large Language Models for Learner Modelling: Why LLMs Alone Fall Short for Responsible Tutoring in K-12 Education

  2025cites this paper
• Specification uncertainty: what the disruption index tells us about the (hidden) multiverse of bibliometric indicators

  2024cites this paper
• Consolation in the aftermath of violent public assaults: An interaction ethological study

  2024cites this paper
• Incorporating Machine Learning into Sociological Model-Building

  2024cites this paper
• REFORMS: Consensus-based Recommendations for Machine-learning-based Science

  2024cites this paper
• Corporate Demography and Wage Inequality: Revisited

  2024influential citation
• The quantitative paradigm and the nature of the human mind. The replication crisis as an epistemological crisis of quantitative psychology in view of the ontic nature of the psyche

  2024cites this paper
• The Life Cycle of Large Language Models: A Review of Biases in Education

  2024cites this paper
• The life cycle of large language models in education: A framework for understanding sources of bias

  2024cites this paper
• Reducing bias, increasing transparency and calibrating confidence with preregistration

  2023cites this paper
• REFORMS: Reporting Standards for Machine Learning Based Science

  2023cites this paper
• Understanding Patterns and Trends in Income Mobility through Multiverse Analysis

  2023cites this paper
• Preregistration and Registered Reports in Sociology: Strengths, Weaknesses, and Other Considerations

  2023cites this paper
• Reply to Engzell: Maybe in plain sight but out of focus

  2023cites this paper
• Exploring the Multiverse of Analytical Decisions in Scaling Educational Large-Scale Assessment Data: A Specification Curve Analysis for PISA 2018 Mathematics Data

  2022cites this paper
• A Survey of Tasks and Visualizations in Multiverse Analysis Reports

  2022cites this paper
• A Pragmatist’s Guide to Using Prediction in the Social Sciences

  2022cites this paper
• On the Choice of the Item Response Model for Scaling PISA Data: Model Selection Based on Information Criteria and Quantifying Model Uncertainty

  2022cites this paper
• How Could Venture Capitalists Improve Their Performance at Fund Management Level via Better Allocating Their Financial and Human Capitals-A Cross-Fund Approach

  2022influential citation
• How Robust Are Social Structural Predictors of Carbon Dioxide Emissions? A Multiverse Analysis

  2022cites this paper
• Quantifying the robustness of causal inferences: Sensitivity analysis for pragmatic social science.

  2022cites this paper
• Machine Learning Methods for Evaluating Public Crisis: Meta-Analysis

  2022cites this paper
• Face‐touching behaviour as a possible correlate of mask‐wearing: A video observational study of public place incidents during the COVID‐19 pandemic

  2021cites this paper
• Hyperparameter Optimization Is Deceiving Us, and How to Stop It

  2021cites this paper
• The Rise of the Nones across the United States, 1973 to 2018: State-Level Trends of Religious Affiliation and Participation in the General Social Survey

  2021cites this paper
• Has the Credibility of the Social Sciences Been Credibly Destroyed? Reanalyzing the “Many Analysts, One Data Set” Project

  2021cites this paper
• Advancements in Multiverse Analysis

  2021cites this paper
• Testing for calibration discrepancy of reported likelihood ratios in forensic science

  2021cites this paper
• 1 Uncertainty-based decision-making in fire safety : analyzing the alternatives

  2020cites this paper
• What is Your Estimand? Defining the Target Quantity Connects Statistical Evidence to Theory

  2020cites this paper
• Advancements for Functional Form Robustness

  2020cites this paper
• Cross-National CCTV Footage Shows Low Victimization Risk for Bystander Interveners in Public Conflicts

  2020cites this paper
• Methodological Uncertainty and Multi-Strategy Analysis: Case Study of the Long-Term Effects of Government Sponsored Youth Training on Occupational Mobility

  2020cites this paper
• Comparing the vibration of effects due to model, data pre-processing and sampling uncertainty on a large data set in personality psychology

  2020cites this paper
• No generalizable effect of income inequality on public support for governmental redistribution among rich democracies 1987-2010.

  2019cites this paper
• Recommendations for Increasing the Transparency of Analysis of Preexisting Data Sets

  2018cites this paper
• Comment: Some Challenges When Estimating the Impact of Model Uncertainty on Coefficient Instability

  2018cites this paper
• Rejoinder: Can We Weight Models by Their Probability of Being True?

  2018cites this paper
• We Ran 9 Billion Regressions: Eliminating False Positives through Computational Model Robustness

  2018cites this paper
• The Difference Between Causal Analysis and Predictive Models: Response to “Comment on Young and Holsteen (2017)”

  2018cites this paper
• Which Data to Meta-Analyze, and How?: A Specification-Curve and Multiverse-Analysis Approach to Meta-Analysis

  2018cites this paper