The Failure of Null Hypothesis Significance Testing When Studying Incremental Changes, and What to Do About It

No abstract is available for this paper.

• Publication year

  2018

• Venue

  Personality and Social Psychology Bulletin

• Publication date

  2018-01-01

• Fields of study

  Medicine, Economics, Psychology

• Identifiers
  DOI 10.1177/0146167217729162PMID 28914154
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• Statistical Significance and the Dichotomization of Evidence

  2017cited by this paper
• Ethics and Statistics: Honesty and Transparency Are Not Enough

  2017cited by this paper
• Less Is More: Psychologists Can Learn More by Studying Fewer People

  2016cited by this paper
• Increasing Transparency Through a Multiverse Analysis

  2016cited by this paper
• Caught red-minded: Evidence-induced denial of mental transgressions.

  2016influential reference
• Estimating the reproducibility of psychological science

  2015cited by this paper
• College football, elections, and false-positive results in observational research

  2015cited by this paper
• Beyond Power Calculations

  2014cited by this paper
• Labor market returns to an early childhood stimulation intervention in Jamaica

  2014cited by this paper
• Deep Interactions with MRP: Election Turnout and Voting Patterns Among Small Electoral Subgroups

  2013cited by this paper
• A 21 Word Solution

  2012cited by this paper
• False-Positive Psychology

  2011cited by this paper
• Why We (Usually) Don't Have to Worry About Multiple Comparisons

  2009cited by this paper
• Of Beauty, Sex and Power

  2009cited by this paper
• Of beauty, sex, and power: Statistical challenges in estimating small eects

  2008cited by this paper
• The Null Hypothesis Testing Controversy in Psychology

  1999cited by this paper
• Automaticity of social behavior: direct effects of trait construct and stereotype-activation on action.

  1996cited by this paper
• Theory-Testing in Psychology and Physics: A Methodological Paradox

  1967cited by this paper
• Researchers overturn landmark study on the replicability of psychological science

  year unknowncited by this paper

• Are our 95% CIs only worth 45% confidence?

  2025cites this paper
• Beyond statistical significance: Quantifying uncertainty and statistical variability in multilingual and multitask NLP evaluation

  2025influential citation
• Surprises, Conflicting Findings, or Questionable Research Practices? A Methodology for Evaluating Cumulative Empirical Analyses and Replication Studies

  2025cites this paper
• Compatibility and S values versus traditional reporting in epidemiological studies

  2025cites this paper
• Parent-Child Interaction Therapy’s Influence on Parental Behavior and Child Compliance in a Child-Welfare Involved Randomized Clinical Trial

  2025cites this paper
• Pro-environmental behavior and life satisfaction: How strong is our evidence?

  2025cites this paper
• Generalising Social Behaviour and Theory of Social Behaviour: When Is It Statistical and When Not?

  2025cites this paper
• How can we improve statistical training in archaeological science?

  2025cites this paper
• PHÂN TÍCH BAYES VỀ TÁC ĐỘNG CỦA VỐN TÂM LÝ ĐẾN SỰ GẮN KẾT VỚI CÔNG VIỆC CỦA CÁC BÁC SĨ

  2025cites this paper
• A registered report of preregistration practices in studies of electroencephalogram (EEG) and event-related potentials (ERPs): A first look at accessibility, adherence, transparency, and selection bias.

  2025cites this paper
• Flexible Empirical Bayesian Approaches to Pharmacovigilance for Simultaneous Signal Detection and Signal Strength Estimation in Spontaneous Reporting Systems Data

  2025cites this paper
• The potential effect of d-cycloserine on the development of anxiety during exposure sessions in patients with agoraphobia.

  2024cites this paper
• P>0.05 Is Good: The NORD-h Protocol for Several Hypothesis Analysis Based on Known Risks, Costs, and Benefits

  2024influential citation
• Threats to personal control fuel similarity attraction

  2024cites this paper
• Have attitudes toward democracy polarized in the U.S.?

  2024cites this paper
• Compatibility ranges as a practical alternative to the “significant/non-significant” statistical dichotomy

  2024cites this paper
• For a proper use of frequentist inferential statistics in public health

  2024cites this paper
• How Statistical Challenges and Misreadings of the Literature Combine to Produce Unreplicable Science: An Example From Psychology

  2024cites this paper
• Conflicting Results and Statistical Malleability: Embracing Pluralism of Empirical Results

  2024influential citation
• Choosing not to get anchored: A choice mindset reduces the anchoring bias

  2024cites this paper
• Changes in methodological study characteristics in psychology between 2010-2021

  2023cites this paper
• Neurophysiological measures and correlates of cognitive load in attention‐deficit/hyperactivity disorder (ADHD), autism spectrum disorder (ASD) and dyslexia: A scoping review and research recommendations

  2023cites this paper
• Accounting Research as Bayesian Inference to the Best Explanation

  2023cites this paper
• A New Look at P Values for Randomized Clinical Trials.

  2023cites this paper
• A Chain as Strong as Its Strongest Link? Understanding the Causes and Consequences of Biases Arising from Selective Analysis and Reporting of Research Results

  2023cites this paper
• The winner's curse and related perils of low statistical power − spelled out and illustrated

  2023cites this paper
• Human presence and infrastructure impact wildlife nocturnality differently across an assemblage of mammalian species

  2023cites this paper
• veRification: an R Shiny application for laboratory method verification and validation

  2023cites this paper
• Causal Quartets: Different Ways to Attain the Same Average Treatment Effect

  2023cites this paper
• Interdependencies between family and friends in daily life: Personality differences and associations with affective well-being across the lifespan

  2022cites this paper
• Criticism as asynchronous collaboration: An example from social science research

  2022cites this paper
• Causal quartets: Different ways to attain the same average treatment effect ∗

  2022cites this paper
• A New Way to Think About Internal and External Validity

  2022cites this paper
• Predicted as observed? How to identify empirically adequate theoretical constructs

  2022cites this paper
• Beyond Chronology, Using Bayesian Inference to Evaluate Hypotheses in Archaeology

  2022cites this paper
• Can the Replication Rate Tell Us About Publication Bias?

  2022cites this paper
• 50 shades of Bayesian testing of hypotheses

  2022cites this paper
• Implementing planned missingness in stimulus sampling designs: Strategies for optimizing statistical power and precision while limiting participant burden

  2022cites this paper
• Personality Traits of Special Forces Operators

  2022cites this paper
• Selecting on statistical significance and practical importance is wrong

  2022cites this paper
• Personality traits of special forces operators: Comparing commandos, candidates, and controls.

  2022cites this paper
• Personality Traits of Special Forces Operators

  2022cites this paper
• Commentary on “On Loss Functions and Bias–Variance Tradeoffs in Causal Estimation and Decisions”

  2022cites this paper
• The Worst of Both Worlds: A Comparative Analysis of Errors in Learning from Data in Psychology and Machine Learning

  2022influential citation
• Equivalent statistics for a one-sample t-test

  2022cites this paper
• Are most published research findings false in a continuous universe?

  2022cites this paper
• LET's CONNECT Community Mentorship Program for Adolescents with Peer Social Problems: A Randomized Intervention Trial.

  2021cites this paper
• Comparing cross-classified mixed effects and Bayesian structural equations modeling for stimulus sampling designs: A simulation study

  2021cites this paper
• Ecophylogenetics redux.

  2021cites this paper
• Estimation Statistics, One Year Later

  2021cites this paper
• Is Supported Employment Effective for Young Adults with Disability Pension?

  2021cites this paper
• Meaningful associations in the adolescent brain cognitive development study

  2021cites this paper
• Rumination Derails Reinforcement Learning With Possible Implications for Ineffective Behavior

  2021cites this paper
• Needed: More Reliable Bioeffects Studies at “High Band” 5G Frequencies

  2021cites this paper
• Perceived threat and officer-involved shootings: characteristics leading to fatal vs non-fatal instances

  2021cites this paper
• Effect Sizes Reported in Highly Cited Emotion Research Compared With Larger Studies and Meta-Analyses Addressing the Same Questions

  2021cites this paper
• Digital Pathology During the COVID-19 Outbreak in Italy: Survey Study

  2020cites this paper
• Evaluating the Use of Covariance‐Based Structural Equation Modelling with Reflective Measurement in Organizational and Management Research: A Review and Recommendations for Best Practice

  2020cites this paper
• Thinking about effect sizes: From the replication crisis to a cumulative psychological science.

  2020cites this paper
• Measuring affective states from technical debt

  2020cites this paper
• Statistical Nonsignificance in Empirical Economics

  2020cites this paper
• Meaningful Effects in the Adolescent Brain Cognitive Development Study

  2020cites this paper
• Contemporary Empirical Methods in Software Engineering

  2020cites this paper
• Time investments in rituals are associated with social bonding, affect and subjective health: a longitudinal study of Diwali in two Indian communities

  2020cites this paper
• Subjective status and perceived legitimacy across countries

  2020cites this paper
• Improving the Rigor and Replicability of Applied Psychophysiology Research: Sample Size, Standardization, Transparency, and Preregistration

  2020cites this paper
• What Experts Appreciate in Patterns: Art Expertise Modulates Preference for Asymmetric and Face-Like Patterns

  2020cites this paper
• Embracing Robustness and Reliability in the Science of Organizations

  2020cites this paper
• Assessment of demand for methamphetamine and cigarettes among individuals with methamphetamine use disorder.

  2020cites this paper
• Replications in Comparative Cognition: What Should We Expect and How Can We Improve?

  2020cites this paper
• From Absolution to Action: Examining Americans’ Reactions to High‐Profile Corporate Scandals 

  2020cites this paper
• Effect size reporting and interpretation in social personality research

  2020cites this paper
• Designing Studies and Evaluating Research Results: Type M and Type S Errors for Pearson Correlation Coefficient

  2020cites this paper
• Comparing multiple comparisons: practical guidance for choosing the best multiple comparisons test

  2020cites this paper
• Promoting and supporting credibility in neuroscience

  2019cites this paper
• Objective Randomised Blinded Investigation With Optimal Medical Therapy of Angioplasty in Stable Angina (ORBITA) and coronary stents: A case study in the analysis and reporting of clinical trials.

  2019cites this paper
• Using automation to combat the replication crisis: A case study from controlled-rearing studies of newborn chicks.

  2019cites this paper
• Enhancing Rigor in Quantitative Entrepreneurship Research

  2019cites this paper
• Bayesian Data Analysis in Empirical Software Engineering: The Case of Missing Data

  2019cites this paper
• Beyond Bonferroni revisited: concerns over inflated false positive research findings in the fields of conservation genetics, biology, and medicine

  2019influential citation
• Using Simulations to Explore Sampling Distributions: An Antidote to Hasty and Extravagant Inferences

  2019cites this paper
• Measuring individual differences in cognitive abilities in the lab and on the web

  2019cites this paper
• Indices of Effect Existence and Significance in the Bayesian Framework

  2019cites this paper
• The Practicalities and Perils of Ambulatory Assessment's Promise: Introduction to a Special Section.

  2019cites this paper
• Enhancing Statistical Inference in Psychological Research via Prospective and Retrospective Design Analysis

  2019cites this paper
• Understanding the Average Impact of Microcredit Expansions: A Bayesian Hierarchical Analysis of Seven Randomized Experiments

  2019influential citation
• Integrating Pre-Registered Re-examinations (p-rex) Proposals into Research Programs and Doctoral Seminars

  2019cites this paper
• Volume-wise analysis of fMRI time series allows accurate prediction of language lateralization

  2019cites this paper
• A best-case scenario of the Semantic Priming Stroop task; can specific thoughts be found?

  2019influential citation
• On de-bunking “Fake News” in the post-truth era: How to reduce statistical error in research

  2019cites this paper
• Methodological reporting behavior, sample sizes, and statistical power in studies of event-related potentials: Barriers to reproducibility and replicability.

  2019cites this paper
• Non-significant p-values? Strategies to understand and better determine the importance of effects and interactions in logistic regression

  2018cites this paper
• The statistical significance filter leads to overoptimistic expectations of replicability

  2018cites this paper
• P values in display items are ubiquitous and almost invariably significant: A survey of top science journals

  2018cites this paper
• Benefits and limitations of randomized controlled trials: A commentary on Deaton and Cartwright.

  2018cites this paper
• Behavioural Insights and (Un)healthy Dietary Choices: a Review of Current Evidence

  2018cites this paper
• Behavioural Insights and (Un)healthy Dietary Choices: a Review of Current Evidence

  2018cites this paper
• Response to Leslie Hayduk’s Review of Principles and Practice of Structural Equation Modeling, 4th Edition

  2018cites this paper
• How much does teacher quality vary across teacher preparation programs? Reanalyses from six states.

  2018influential citation
• THREE INVESTIGATIONS INTO THE DYNAMICS AND IMPLICATIONS OF IDENTITY-PROTECTIVE COGNITION FOR PUBLIC RESPONSES TO ENVIRONMENTAL PROBLEMS

  2018cites this paper