Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N > 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = -0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim. (PsycINFO Database Record (c) 2020 APA, all rights reserved).

• Publication year

  2020

• Venue

  Psychological bulletin

• Publication date

  2020-01-16

• Fields of study

  Medicine, Psychology

• Identifiers
  DOI 10.1037/bul0000220PMID 31944796
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• Investigating Variation in Replicability: A “Many Labs” Replication Project

  2022influential reference
• Modernization and Postmodernization

  2020cited by this paper
• Pathways of persuasion

  2020cited by this paper
• Scientific Utopia III: Crowdsourcing Science

  2019cited by this paper
• A creative destruction approach to replication

  2019cited by this paper
• Generalizability is not optional: insights from a cross-cultural study of social discounting

  2019cited by this paper
• Abilene Christian University

  2019cited by this paper
• Metastudies for robust tests of theory

  2018influential reference
• A Unified Framework to Quantify the Credibility of Scientific Findings

  2018cited by this paper
• UvA-DARE ( Digital Academic Repository ) Bayesian inference for psychology . Part

  2018cited by this paper
• Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

  2018cited by this paper
• Max Planck Institute for Human Cognitive and Brain Sciences

  2018cited by this paper
• What Motivates Effort?: Evidence and Expert Forecasts

  2018influential reference
• The preregistration revolution

  2018cited by this paper
• Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results

  2018cited by this paper
• Predicting Experimental Results: Who Knows What?

  2018influential reference
• Predicting replication outcomes in the Many Labs 2 study

  2018cited by this paper
• Facing One’s Implicit Biases: From Awareness to Acknowledgment

  2018cited by this paper
• Abandon Statistical Significance

  2017cited by this paper
• Our obsession with eminence warps research

  2017cited by this paper
• Registered Replication Report: Rand, Greene, and Nowak (2012)

  2017cited by this paper
• Bayesian inference for psychology. Part II: Example applications with JASP

  2017cited by this paper
• Invited Commentary: The Need for Cognitive Science in Methodology

  2017cited by this paper
• A tutorial on bridge sampling

  2017cited by this paper
• Judgment under Uncertainty

  2017cited by this paper
• Show Me the Money: A Systematic Exploration of Manipulations, Moderators, and Mechanisms of Priming Effects

  2017cited by this paper
• Fixed or Random? A Resolution Through Model Averaging: Reply to Carlsson, Schimmack, Williams, and Bürkner (2017)

  2017cited by this paper
• Estimates of Between-Study Heterogeneity for 705 Meta-Analyses Reported in Psychological Bulletin From 1990-2013

  2017cited by this paper
• When your anchor sinks your boat: Information asymmetry in distributive negotiations and the disadvantage of making the first offer

  2017cited by this paper
• A Bayesian model-averaged meta-analysis of the power pose effect with informed and default priors: the case of felt power

  2017cited by this paper
• Does expertise matter in replication? An examination of the reproducibility project: Psychology ☆

  2017cited by this paper
• Replicability and Other Features of a High-Quality Science: Toward a Balanced and Empirical Approach

  2017cited by this paper
• Informed Bayesian t-Tests

  2017cited by this paper
• bridgesampling: An R Package for Estimating Normalizing Constants

  2017cited by this paper
• UvA-DARE ( Digital Academic Repository ) Estimates of Between-Study Heterogeneity for 705 Meta-Analyses Reported in Psychological Bulletin From 1990 – 2013

  2017cited by this paper
• What’s Wrong With Using Steroids? Exploring Whether and Why People Oppose the Use of Performance Enhancing Drugs

  2017influential reference
• Large-Scale Replication Projects in Contemporary Psychological Research

  2017cited by this paper
• True Happiness: The Role of Morality in the Folk Concept of Happiness

  2017influential reference
• Charting the future of social psychology on stormy seas: Winners, losers, and recommendations

  2016cited by this paper
• The information-anchoring model of first offers: When moving first helps versus hurts negotiators.

  2016cited by this paper
• On the scientific superiority of conceptual replications for scientific progress

  2016cited by this paper
• The impact of soft skills training on female youth employment: evidence from a randomized experiment in Jordan

  2016cited by this paper
• Increasing Transparency Through a Multiverse Analysis

  2016cited by this paper
• Evaluating replicability of laboratory experiments in economics

  2016cited by this paper
• “Am I Famous Yet?” Judging Scholarly Merit in Psychological Science

  2016cited by this paper
• Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking

  2016cited by this paper
• Contextual sensitivity in scientific reproducibility

  2016influential reference
• of Experimental Psychology : General Inference of Trustworthiness From Intuitive Moral Judgments

  2016cited by this paper
• The wisdom of crowds.

  2016cited by this paper
• Third-party punishment as a costly signal of trustworthiness

  2016cited by this paper
• Crowdsourcing Hypothesis Tests

  2016cited by this paper
• The pipeline project Pre-publication independent replications of a single laboratory ' s research pipeline

  2016cited by this paper
• Bayesian inference for psychology

  2016cited by this paper
• Many Labs 3: Evaluating participant pool quality across the academic semester via replication

  2016cited by this paper
• Moral Tribes: Emotion, Reason, and the Gap between Us and Them

  2016influential reference
• Estimating the reproducibility of psychological science

  2015cited by this paper
• The Chicago face database: A free stimulus set of faces and norming data

  2015cited by this paper
• The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers

  2015cited by this paper
• Replicating Studies in Which Samples of Participants Respond to Samples of Stimuli

  2015cited by this paper
• Crowdsourced research: Many hands make tight work

  2015cited by this paper
• Sidetracked by trolleys: Why sacrificial moral dilemmas tell us little (or nothing) about utilitarian judgment

  2015cited by this paper
• Using prediction markets to estimate the reproducibility of scientific research

  2015cited by this paper
• A Person-Centered Approach to Moral Judgment

  2015cited by this paper
• A tragedy of the (academic) commons: interpreting the replication crisis in psychology as a social dilemma for early-career researchers

  2015cited by this paper
• Tandem anchoring: informational and politeness effects of range offers in social exchange.

  2015cited by this paper
• Best research practices in psychology: Illustrating epistemological and pragmatic considerations with the case of relationship science.

  2015cited by this paper
• UvA-DARE ( Digital Academic Repository ) Bayesian tests to quantify the result of a replication attempt

  2014cited by this paper
• R: A language and environment for statistical computing.

  2014cited by this paper
• The Value of Direct Replication

  2014influential reference
• P-curve: a key to the file-drawer.

  2014cited by this paper
• Registered Replication Report

  2014cited by this paper
• "Bayesian tests to quantify the result of a replication attempt": Correction to Verhagen and Wagenmakers (2014).

  2014cited by this paper
• The Replication Recipe: What Makes for a Convincing Replication?

  2014cited by this paper
• ON THE OTHER SIDE OF THE MIRROR: PRIMING IN COGNITIVE AND SOCIAL PSYCHOLOGY

  2014influential reference
• Being Tough or Being Nice? A Meta-Analysis on the Impact of Hard- and Softline Strategies in Distributive Negotiations

  2014cited by this paper
• The Alleged Crisis and the Illusion of Exact Replication

  2014cited by this paper
• The Increasing Dominance of Teams in Production of Knowledge

  2014cited by this paper
• The Limits of Direct Replications and the Virtues of Stimulus Sampling

  2014cited by this paper
• Welcome Back Theory!

  2014cited by this paper
• The Implicit Legacy of American Protestantism

  2014cited by this paper
• Public Attitudes Toward Cognitive Enhancement

  2014cited by this paper
• Multivariate meta-analysis using individual participant data

  2014cited by this paper
• Social: Why Our Brains Are Wired to Connect

  2013cited by this paper
• The Folk Probably do Think What you Think They Think

  2013cited by this paper
• Jeremy Bentham: An Introduction to the Principles of Morals and Legislation

  2013cited by this paper
• Journal of Experimental Psychology : General Awareness of Implicit Attitudes

  2013cited by this paper
• Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers

  2013cited by this paper
• Impact of Contextual Factors and Substance Characteristics on Perspectives toward Cognitive Enhancement

  2013cited by this paper
• The Remarkable Robustness of the First-Offer Effect

  2013cited by this paper
• Research Data Management with DATORIUM. Filling a Gap by Developing a Data Sharing Repository at GESIS-Leibniz Institute for the Social Sciences

  2013cited by this paper
• Risks and Rewards of Crowdsourcing Marketplaces

  2013cited by this paper
• BootES: An R package for bootstrap confidence intervals on effect sizes

  2013cited by this paper
• Starting high and ending with nothing: The role of anchors and power in negotiations

  2012cited by this paper
• Is the Replicability Crisis Overblown? Three Arguments Examined

  2012cited by this paper
• An Agenda for Purely Confirmatory Research

  2012cited by this paper
• Treating stimuli as a random factor in social psychology: a new and comprehensive solution to a pervasive but largely ignored problem.

  2012cited by this paper
• The ironic effect of significant results on the credibility of multiple-study articles.

  2012cited by this paper
• Science or Art? How Aesthetic Standards Grease the Way Through the Publication Bottleneck but Undermine Science

  2012cited by this paper
• Replications in Psychology Research

  2012cited by this paper
• The Ethics of “Smart Drugs”: Moral Judgments About Healthy People's Use of Cognitive-Enhancing Drugs

  2012cited by this paper

• Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiverse

  2026cites this paper
• Multiverse analyses can be used to evaluate cross-lagged panel network models: An example with psychological flexibility.

  2026cites this paper
• The effects of mixed-mode ventilation on energy saving and employee job satisfaction, work engagement, and job performance.

  2026cites this paper
• Comparing expert assessments of research quality between the Global North and East Africa.

  2025cites this paper
• Hypothetical nudges provide directional but noisy estimates of real behavior change

  2025cites this paper
• The reliability of replications: a study in computational reproductions

  2025cites this paper
• Agential realism as an alternative philosophy of science perspective for quantitative psychology

  2024cites this paper
• Can large language models help predict results from a complex behavioural science study?

  2024cites this paper
• Metformin in the Diabetes Prevention Program 3‐year trial: The cost‐effectiveness that never was

  2024cites this paper
• Heterogeneity in effect size estimates

  2024cites this paper
• The multiverse of universes: A tutorial to plan, execute and interpret multiverses analyses using the R package multiverse.

  2024cites this paper
• Grilling the data: application of specification curve analysis to red meat and all-cause mortality

  2024cites this paper
• An Wavelet analysis of connectedness between volatility of the energy and other markets during the COVID-19 crisis

  2023cites this paper
• One data set, many analysts: Implications for practicing scientists

  2023cites this paper
• A manifesto for applying behavioural science

  2023cites this paper
• Research-Problem Validity in Primary Research: Precision and Transparency in Characterizing Past Knowledge

  2023cites this paper
• When expert predictions fail.

  2023cites this paper
• The two-week systematic review (2weekSR) method was successfully blind-replicated by another team: a case study.

  2023cites this paper
• Meta-Analytic Associations Between the Student-Teacher Relationship Scale and Students’ Social Competence With Peers

  2023cites this paper
• The trustworthiness of the cumulative knowledge in industrial/organizational psychology: The current state of affairs and a path forward.

  2023cites this paper
• How to build up big team science: a practical guide for large-scale collaborations

  2023cites this paper
• Competition and moral behavior: A meta-analysis of forty-five crowd-sourced experimental designs

  2023influential citation
• Constructive replication, reproducibility, and generalizability: Getting theory testing for JOMSR right

  2023cites this paper
• We are all less risky and more skillful than our fellow drivers: Successful replication and extension of Svenson (1981)‎

  2023cites this paper
• (Why) Are Open Research Practices the Future for the Study of Language Learning?

  2023cites this paper
• An advanced method to streamline p-hacking

  2023cites this paper
• One finding is no finding: Toward a replication culture in family business research

  2022cites this paper
• The spatial grounding of politics

  2022cites this paper
• The Benefits, Barriers, and Risks of Big-Team Science

  2022cites this paper
• Politicizing mask-wearing: predicting the success of behavioral interventions among republicans and democrats in the U.S.

  2022cites this paper
• Feeling and Thinking about It Are Two Different Things: How to Capture Momentary Emotions of Extreme Sports in the Field

  2022cites this paper
• Interpolated testing and content pretesting as interventions to reduce task-unrelated thoughts during a video lecture

  2022cites this paper
• Ten Years of Psychology's Replicability Crisis:

  2022cites this paper
• It Takes a Village: Using a Crowdsourced Approach to Investigate Organic Matter Composition in Global Rivers Through the Lens of Ecological Theory

  2022cites this paper
• An exploration of sources of volatility in the energy market: An application of a TVP-VAR extended joint connected approach

  2022cites this paper
• Optimizing Character Animations using Online Crowdsourcing

  2022cites this paper
• Accelerating Psychological Science With Metastudies: A Demonstration Using the Risky-Choice Framing Effect

  2022cites this paper
• Examining the generalizability of research findings from archival data

  2022cites this paper
• Forecasting the publication and citation outcomes of COVID-19 preprints

  2022cites this paper
• Diagnosis and therapy of psychology by means of systematicity theory

  2022cites this paper
• The replication crisis and philosophy

  2022cites this paper
• Dynamic interlinkages between the crude oil and gold and stock during Russia-Ukraine War: evidence from an extended TVP-VAR analysis

  2022cites this paper
• Reproducibility in Neuroimaging Analysis: Challenges and Solutions.

  2022cites this paper
• The Potential of Factor Analysis for Replication, Generalization, and Integration

  2022cites this paper
• A brighter vision of the potential of open science for benefiting practice: A ManyOrgs proposal

  2022cites this paper
• Has the Credibility of the Social Sciences Been Credibly Destroyed? Reanalyzing the “Many Analysts, One Data Set” Project

  2021cites this paper
• Editorial: Special Issue on Replicability in Cognitive Science

  2021cites this paper
• Improving the reproducibility of findings by updating research methodology

  2021cites this paper
• Impacts of COVID-19 pandemic on the global energy system and the shift progress to renewable energy: Opportunities, challenges, and policy implications

  2021cites this paper
• Quantifying sources of uncertainty in drug discovery predictions with probabilistic models

  2021cites this paper
• Novel predictions arise from contradictions

  2021cites this paper
• Authentic leadership and employee health: a conditional process model

  2021cites this paper
• Editorial: Replicability in Cognitive Science

  2021cites this paper
• How open is innovation? A retrospective and ideas forward

  2021influential citation
• COVID-19 and the Global Shift Progress to Clean Energy

  2021cites this paper
• Two Challenges to “Embodied Cognition” Research And How to Overcome Them

  2021cites this paper
• Using prediction markets to predict the outcomes in the Defense Advanced Research Projects Agency's next-generation social science programme

  2021cites this paper
• Predicting replicability—Analysis of survey and prediction market data from large-scale forecasting projects

  2021cites this paper
• Priming exploration across domains: does search in a spatial environment influence search in a cognitive environment?

  2021cites this paper
• Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

  2021influential citation
• Who, What, Where: Tracking the development of COVID-19 related PsyArXiv preprints

  2021cites this paper
• Bayesian model‐averaged meta‐analysis in medicine

  2021cites this paper
• Percent framing attenuates the magnitude effect in a preference-matching task of intertemporal choice

  2021cites this paper
• The open innovation journey along heterogeneous modes of knowledge-intensive marketing collaborations: a cross-sectional study of innovative firms in Europe

  2021cites this paper
• Why most psychological research findings are not even wrong

  2021cites this paper
• Science-utility and science-trust associations and how they relate to knowledge about how science works

  2021cites this paper
• Improving the design of climate insurance: combining empirical approaches and modelling

  2021cites this paper
• Subjective interdependence and prosocial behaviour

  2020cites this paper
• Assessing heterogeneity and power in replications of psychological experiments.

  2020cites this paper
• Putting your money where your self is: Connecting dimensions of closeness and theories of personal identity

  2020cites this paper
• A Falsificationist Treatment of Auxiliary Hypotheses in Social and Behavioral Sciences: Systematic Replications Framework

  2020cites this paper
• Replicability, Robustness, and Reproducibility in Psychological Science.

  2020cites this paper
• Ostracized and observed: The presence of an audience affects the experience of being excluded

  2020cites this paper
• Credibility Beyond Replicability: Improving the Four Validities in Psychological Science

  2020cites this paper
• Creative destruction in science

  2020cites this paper
• Does preregistration improve the credibility of research findings?

  2020cites this paper
• A Primer on Bayesian Model-Averaged Meta-Analysis

  2020cites this paper
• “How do you like them now?” Expected reactions upon discovering that a friend is a political out-group member

  2020cites this paper
• Perceived efficacy of COVID-19 restrictions, reactions and their impact on mental health during the early phase of the outbreak in six countries

  2020cites this paper
• Are replication rates the same across academic fields? Community forecasts from the DARPA SCORE programme

  2020cites this paper
• Validity of energy social research during and after COVID-19: challenges, considerations, and responses

  2020cites this paper
• The influence of hidden researcher decisions in applied microeconomics

  2020cites this paper
• DESTRUCTION THROUGH REPLICATION 1 Creative Destruction in Science

  2020cites this paper
• Political conservatism and preference for (a)symmetric brand logos

  2020cites this paper
• Republicans and Democrats differ in why they support renewable energy

  2020cites this paper
• Using mouse cursor tracking to investigate online cognition: Preserving methodological ingenuity while moving toward reproducible science

  2019cites this paper
• Why diverse perspectives matter for Memory & Cognition

  2019cites this paper
• Stability of Experimental Results: Forecasts and Evidence

  2019cites this paper
• Generalizability is not optional: insights from a cross-cultural study of social discounting

  2019cites this paper
• How open science norms improve scientific practices

  2018cites this paper