On the Limitations of Manipulation Checks: An Obstacle Toward Cumulative Science

Manipulation checks do not allow ruling out or accepting alternative explanations of causal effects (Sigall & Mills, 1998). In order to gauge the influence of this argument on current research practices, we surveyed the views of researchers on manipulation checks. Results confirmed that a manipulation check still stands as a totem of experimental rigor. Except in rare circumstances, such as when pilot testing, manipulation checks do not provide information relevant to construct validity. While it seems cost free to include seemingly informative manipulation checks, we claim it is actually costly because it wrongly enhances subjective confidence in the validity of research findings. We conclude that manipulation checks may hinder efforts to adopt a cumulative culture and practice of hypothesis testing.

• Publication year

  2017

• Venue

  Revue Internationale De Psychologie Sociale-international Review of Social Psychology

• Publication date

  2017-06-20

• Fields of study

  Engineering, Education, Psychology

• Identifiers
  DOI 10.5334/IRSP.102
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Construct Validation in Social and Personality Research

  2017cited by this paper
• Reconceptualizing replication as a sequence of different studies: A replication typology

  2016cited by this paper
• A Closer Look at Social Psychologists’ Silver Bullet

  2016cited by this paper
• Best research practices in psychology: Illustrating epistemological and pragmatic considerations with the case of relationship science.

  2015cited by this paper
• The Replication Recipe: What Makes for a Convincing Replication?

  2014cited by this paper
• The Alleged Crisis and the Illusion of Exact Replication

  2014influential reference
• Hypotheses, Operationalizations, and Manipulation Checks

  2014cited by this paper
• Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling

  2012cited by this paper
• What mediation analysis can (not) do

  2011cited by this paper
• Interactions do not only tell us when, but can also tell us how: Testing process hypotheses by interaction

  2011cited by this paper
• Manipulation Checks as Manipulation: Another Look at the Ease-of-Retrieval Heuristic

  2010cited by this paper
• The Art of Laboratory Experimentation

  2010influential reference
• Strong Inference

  2007cited by this paper
• Handbook of Industrial and Organizational Psychology

  2005cited by this paper
• STANDARDS OF VALIDITY AND THE VALIDITY OF STANDARDS IN PERFORMANCE ASSESSMENT

  2005cited by this paper
• The persistence of underpowered studies in psychological research: causes, consequences, and remedies.

  2004cited by this paper
• Experimental design and causality in social psychology research

  2004cited by this paper
• Message Properties, Mediating States, and Manipulation Checks: Claims, Evidence, and Data Analysis in Experimental Persuasive Message Effects Research

  2003cited by this paper
• Things I Have Learned ( So Far )

  2002cited by this paper
• Experimental and Quasi-Experimental Designs for Generalized Causal Inference

  2001cited by this paper
• Research Design and Issues of Validity

  2000influential reference
• Measures of Independent Variables and Mediators Are Useful in Social Psychology Experiments: But Are They Necessary?

  1998influential reference
• Heuristic versus systematic information processing and the use of source versus message cues in persuasion.

  1980cited by this paper
• Theories in Social Psychology.

  1966cited by this paper
• Strong Inference: Certain systematic methods of scientific thinking may produce much more rapid progress than others.

  1964cited by this paper

• (Mis)representing the Opposition and Rhetorical Success: Experimental Evidence on Faithful and Inaccurate Reformulations

  2026cites this paper
• Good practices in experimental research

  2026cites this paper
• Inside the experimental causal black box: A heuristic for generating (alternative) explanations in experiments

  2026cites this paper
• Integrating Civic and Artificial Intelligence in Policymaking: Experimental Insights on Public Perceptions of Policy Proposals

  2025cites this paper
• How and When Paradoxical Leadership Nurture Innovation: The Role of Self-Leadership and Leader-Member Exchange

  2025cites this paper
• Exploring entomophagy behavior in restaurants and the influence of social norms

  2025cites this paper
• Bias in manipulation checks of students’ race

  2025cites this paper
• Does reflection increase accuracy rather than bias in the assessments of political fake news?

  2025cites this paper
• How Stereotype Threat Influences Cognitive Performance: It All Depends on How You Feel!

  2025cites this paper
• Beyond Income Disparities: Perceived Health and Education Inequities Drive Actions to Reduce Economic Inequality

  2025cites this paper
• Unboxing Manipulation Checks for Voice UX

  2025cites this paper
• Manipulation Checks in Focus of Attention Research: A Methodological Systematic Review

  2025cites this paper
• Perfectionism, excellencism, and impostor feelings: a meta-analysis and two studies on the moderating role of research productivity in graduate students

  2025cites this paper
• Strengthening the heart by means of a gratitude intervention?

  2025cites this paper
• Implicit Juror Bias: Does Racial Priming Impact the Race-Crime Congruency Effect?

  2024cites this paper
• Manipulation in Organizational Research: On Executing and Interpreting Designs from Treatments to Primes

  2024cites this paper
• The Detection of Invalid Responses Using the Dissociative Experiences Scale-V (DES-V)

  2024cites this paper
• Societal impact of Brand Public Relations on gender equality: evidence for a mediation-type mechanism

  2024cites this paper
• Intimidation and Information Repetition: How Client Behaviors Influence Auditor Judgments

  2024cites this paper
• Assessing the moderating effects of involvement on tourist attitudes and intentions through virtual reality applications

  2024cites this paper
• Message-sidedness in performance information disclosure and citizens’ perceived accountability: an experimental study

  2024cites this paper
• Does metaverse fidelity matter? Testing the impact of fidelity on consumer responses in virtual retail stores

  2024cites this paper
• No politics in funding pitches: An expectancy violations theory perspective of entrepreneurs' political expressions in crowdfunding

  2024cites this paper
• The role of sociopolitical workplace networks in involuntary employee turnover

  2024cites this paper
• Being seen… by human or machine? Acknowledgment effects on customer responses differ between human and robotic service workers

  2023cites this paper
• Men’s Gender Norms and Gender-Hierarchy-Legitimizing Ideologies: The Effect of Priming Traditional Masculinity Versus a Feminization of Men’s Norms

  2023cites this paper
• Entitlement Versus Obligation: The Role of Attributed Motives in Subordinate Reactions to Leader Leniency

  2023cites this paper
• Absolved from the neoliberal burden of responsibility: The effect of populist victim rhetoric on leader support

  2023cites this paper
• Using Toulmin's Argumentation Model to Enhance Trust in Analytics-Based Advice Giving Systems

  2023cites this paper
• I’ll have what they’re having: a descriptive social norm increases choice for vegetables in students

  2023cites this paper
• How workplace identities and team management practices affect distributed team auditors' willingness to speak up

  2023cites this paper
• LMXSC Elicits Hubristic Pride and Social Undermining in Individuals with High Trait Dominance

  2023cites this paper
• Female-to-Male Sexual Assault: The Role of the Perpetrator's Attractiveness and Attributed Emotional States on Victim Blame.

  2023cites this paper
• Common methodological mistakes

  2023cites this paper
• A teacher strategic mindset predicts efficacious and effective teaching

  2023cites this paper
• « Le » Covid-19 est-il plus grave que « la » Covid-19 ? Impact du genre grammatical attribué à la maladie sur sa perception et sur les mesures de protection prises pour s’en prémunir

  2022cites this paper
• Crisis response strategy manipulation: A systematic review and a test of nuances

  2022cites this paper
• How better client service performance affects auditors' willingness to challenge management's preferred accounting

  2022cites this paper
• Attachment Security Priming Delayed Negative Information-Related Attentional Disengagement Among Anxiously Attached Individuals: Evidence From Behavioral and Functional MRI Experiments

  2022cites this paper
• The influence of positive affect on sensitivity to important omissions

  2022cites this paper
• Letting the Dogs Out: The Impact of Team Service Dogs on Brand Image, Perceived Altruism, and Behavioral Intention

  2022cites this paper
• Mitigating the Default? The Influence of Ingroup Diversity on Outgroup Trust

  2021cites this paper
• Protest Under Uncertainty: Evidence from a Survey Experiment

  2021cites this paper
• Task choice immunizes against incidental affective influences in volition.

  2021cites this paper
• Advancing the psychology of social class with large-scale replications in four countries

  2021cites this paper
• AP-MOTJ210007 229..241

  2021cites this paper
• The problem of demand effects in power studies: Moving beyond power priming

  2021cites this paper
• Self-Affirmation and Identity-Driven Political Behavior

  2021cites this paper
• “People help the people”: Der Einfluss von Musik mit prosozialem Text in sozialen Werbespots

  2020cites this paper
• Power reduces the goal gradient effect

  2020cites this paper
• To be or not to be in the sample? On using manipulation checks in experimental accounting research

  2020cites this paper
• Jury simulation studies: To exclude or not to exclude participants based on a lack of comprehension of the case?

  2020cites this paper
• Induced Mate Abundance Increases Women’s Expectations for Engagement Ring Size and Cost

  2020cites this paper
• Making inferential leaps: Manipulation checks and the road towards strong inference

  2020cites this paper
• The effect of specific emotions on conformity

  2020cites this paper
• When Victims Help Their Abusive Supervisors: The Role of LMX, Self-Blame, and Guilt

  2020cites this paper
• A conceptual replication of ambidextrous leadership theory: An experimental approach

  2020cites this paper
• Always play against par? The effect of inter-team coopetition on individual team productivity

  2020cites this paper
• Is Ego Depletion Real? An Analysis of Arguments

  2019cites this paper
• Construct Validation of Experimental Manipulations in Social Psychology: Current Practices and Recommendations for the Future

  2019cites this paper
• Who should be prioritized? : the impact of meritocracy norm on medical decisions toward a low status group

  2019cites this paper
• “Rarely safe to assume”: Evaluating the use and interpretation of manipulation checks in experimental social psychology

  2019cites this paper
• Do You Like What You See? How Nonprofit Campaigns With Output, Outcome, and Impact Effectiveness Indicators Influence Charitable Behavior

  2019cites this paper
• Primes and Consequences: A Systematic Review of Meritocracy in Intergroup Relations

  2019cites this paper
• Does voluntary disclosure matter when nonprofit organizations violate stakeholder trust ?

  2019cites this paper
• Underlying processes of SCCT: Mediating roles of preventability, blame, and trust

  2019cites this paper
• What Doesn’t Kill You . . .: Means for Avoidance Goal Pursuit Are Less Enjoyable Than Means for Approach Goal Pursuit

  2019cites this paper
• When stereotype threat makes me more or less intelligent: The informative role of emotions in effort mobilization and task performance.

  2019cites this paper
• All click, no action? Online action, efficacy perceptions, and prior experience combine to affect future collective action

  2019cites this paper
• Testing Trait-State Isomorphism in a New Domain: An Exploratory Manipulation of Openness to Experience

  2018cites this paper
• Are Manipulation Checks Necessary?

  2018cites this paper
• Perceptions of suspect statements: a comparison of exposed lies and confessions

  2018cites this paper
• “I Know Things They Don’t Know!”

  2017cites this paper
• “I Know Things They Don’t Know!”: The Role of Need for Uniqueness in Belief in Conspiracy Theories

  2017cites this paper
• I Think Therefore I Am (Influenced): Perceptions of Social Influence on Self and Others

  2017cites this paper
• The meaning of a claim is its reproducibility

  2017cites this paper