Participant Nonnaiveté and the reproducibility of cognitive psychology

Many argue that there is a reproducibility crisis in psychology. We investigated nine well-known effects from the cognitive psychology literature—three each from the domains of perception/action, memory, and language, respectively—and found that they are highly reproducible. Not only can they be reproduced in online environments, but they also can be reproduced with nonnaïve participants with no reduction of effect size. Apparently, some cognitive tasks are so constraining that they encapsulate behavior from external influences, such as testing situation and prior recent experience with the experiment to yield highly robust effects.

• Publication year

  2017

• Venue

  Psychonomic Bulletin & Review

• Publication date

  2017-07-25

• Fields of study

  Medicine, Psychology

• Identifiers
  DOI 10.3758/s13423-017-1348-yPMID 28744765
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• Creating false memories: Remembering words not presented in lists.

  2018cited by this paper
• Registered Replication Report: Rand, Greene, and Nowak (2012)

  2017cited by this paper
• Registered Replication Report

  2016cited by this paper
• PSYCHOLOGY. Mechanical Turk upends social sciences.

  2016cited by this paper
• Registered Replication Report

  2016cited by this paper
• A Multilab Preregistered Replication of the Ego-Depletion Effect

  2016cited by this paper
• Replications with MTurkers who are naïve versus experienced with academic studies: A comment on Connors, Khamitov, Moroz, Campbell, and Henderson (2015)

  2016cited by this paper
• The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers

  2015cited by this paper
• Estimating the reproducibility of psychological science

  2015cited by this paper
• Using Nonnaive Participants Can Reduce Effect Sizes

  2015influential reference
• Registered Replication Report

  2014cited by this paper
• Social heuristics shape intuitive cooperation

  2014cited by this paper
• If It’s Difficult to Pronounce, It Might Not Be Risky: The Effect of Fluency on Judgment of Risk Does Not Generalize to New Stimuli

  2014cited by this paper
• Inside the Turk

  2014cited by this paper
• Controlled stripes of ultrafine ferroelectric domains

  2014cited by this paper
• Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research

  2013cited by this paper
• Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers

  2013cited by this paper
• Revisiting Mental Simulation in Language Comprehension: Six Replication Attempts

  2012cited by this paper
• Amazon's Mechanical Turk

  2011cited by this paper
• Language Comprehenders Mentally Represent the Shapes of Objects

  2002cited by this paper
• Word naming and memory load: Still searching for an individual differences explanation.

  1998cited by this paper
• Spacing effects in memory: evidence for a two-process account

  1989cited by this paper
• REPETITION PRIMING AND FREQUENCY ATTENUATION IN LEXICAL ACCESS

  1984cited by this paper
• Effects of noise letters upon the identification of a target letter in a nonsearch task

  1974cited by this paper
• Facilitation in recognizing pairs of words: evidence of a dependence between retrieval operations.

  1971cited by this paper
• Processing symbolic information from a visual display: interference from an irrelevant directional cue.

  1970cited by this paper
• Statistical significance in psychological research.

  1968cited by this paper
• The serial position effect of free recall

  1962cited by this paper

• Investigating object orientation effects across 18 languages

  2025cites this paper
• Psychology's Questionable Research Fundamentals (QRFs): Key problems in quantitative psychology and psychological measurement beyond Questionable Research Practices (QRPs)

  2025cites this paper
• Four (and a Half) Preregistered Failures to Replicate the Weapon Focus Effect in Online Samples.

  2025cites this paper
• Effects of multisensory study on long-term memory for pictures and sounds.

  2025cites this paper
• The Language of (Non)Replicable Social Science

  2024cites this paper
• Automatic mental simulation in native and non-native speakers

  2024cites this paper
• Putting the Spotlight Back Onto the Flanker Task in Autism: Autistic Adults Show Increased Interference from Foils Compared with Non-autistic Adults

  2024cites this paper
• Ergonomics & Human factors: fade of a discipline

  2024cites this paper
• Two replications of Wiechert et al.’s (2023) online Think/No-Think study in undergraduate students

  2024cites this paper
• Non-native language comprehenders encode implied shapes of objects in memory.

  2024influential citation
• Suppression-induced forgetting: a pre-registered replication of the think/no-think paradigm

  2023cites this paper
• Mutual interference between memory encoding and motor skills: the influence of motor expertise

  2023cites this paper
• We don't know what you did last summer. On the importance of transparent reporting of reaction time data pre-processing.

  2023cites this paper
• Brain Signatures of Embodied Semantics and Language: A Consensus Paper

  2023cites this paper
• A pilot randomized controlled trial comparing the effectiveness of different spaced learning models used during school examination revision: the SMART Spaces 24/10 model

  2023cites this paper
• The replication crisis in science and its overlap into kinantropology: an introduction

  2023cites this paper
• Metacognitive monitoring in schizotypy: Systematic literature review and new empirical data.

  2023cites this paper
• Effects of caffeine intake and performance pressure on working memory

  2023influential citation
• Realistic guidelines on expert witness work concerning memory

  2023cites this paper
• ERP Evidences of Rapid Semantic Learning in Foreign Language Word Comprehension

  2022cites this paper
• Beyond prosociality: Recalling many types of moral behavior produces positive emotion

  2022cites this paper
• Older Adults Show Intraindividual Variation in the Use of Predictive Processing

  2022cites this paper
• God in body and space: Investigating the sensorimotor grounding of abstract concepts

  2022cites this paper
• What can expert witnesses reliably say about memory in the courtroom?

  2022cites this paper
• A Novel Deep Framework for English Communication Based on Educational Psychology Perspective

  2022cites this paper
• Table tennis expertise influences dual-task costs in timed and self-initiated tasks.

  2022cites this paper
• Replication: Revisiting Tversky and Shafir’s (1992) Disjunction Effect with an extension comparing between and within subject designs

  2021cites this paper
• Misprediction of affective outcomes due to different evaluation modes: Replication and extension of two distinction bias experiments by Hsee and Zhang (2004)

  2021cites this paper
• The Talent Training Mode of International Service Design Using a Human–Computer Interaction Intelligent Service Robot From the Perspective of Cognitive Psychology

  2021cites this paper
• (Non-) impact of task experience on behavioral economic decision-making.

  2021cites this paper
• Crowdsourcing in Cognitive and Systems Neuroscience

  2021cites this paper
• Manipulations of List Type in the DRM Paradigm: A Review of How Structural and Conceptual Similarity Affect False Memory

  2021cites this paper
• Collective memory for American leaders: Measuring recognition for the names and faces of the US presidents

  2021cites this paper
• The impact of pandemic-related worry on cognitive functioning and risk-taking

  2021cites this paper
• Reducing the number of non-naïve participants in Mechanical Turk samples

  2020cites this paper
• University of Southern Denmark Misprediction of affective outcomes due to different evaluation modes: Replication and extension of two distinction bias experiments by Hsee and Zhang (2004) Anvari,

  2020cites this paper
• Grounding language in the neglected senses of touch, taste, and smell

  2020cites this paper
• Processing symbolic magnitude information conveyed by number words and by scalar adjectives

  2020cites this paper
• Les pratiques de recherche ouvertes en psychologie

  2020cites this paper
• Crowdsourcing the Implicit Association Test: Limitations and Best Practices

  2020cites this paper
• Approaches from cognitive neuroscience and comparative cognition

  2020cites this paper
• Efficiency model of micro-course study based on cognitive psychology in the college

  2020cites this paper
• Understanding Replication in a Way That Is True to Science

  2020cites this paper
• Replications in Comparative Cognition: What Should We Expect and How Can We Improve?

  2020cites this paper
• The feature boost in false memory: the roles of monitoring and critical item identifiability

  2020cites this paper
• Analyze the Attentive & Bypass Bias: Mock Vignette Checks in Survey Experiments

  2020cites this paper
• A Broader Application of the Detection Response Task to Cognitive Tasks and Online Environments

  2019cites this paper
• Crowdsourcing visual perception experiments: a case of contrast threshold

  2019influential citation
• C OGNITIVE P SYCHOLOGY A NALYSIS ON THE C AUSES , M ECHANISM AND C OUNTERMEASURES OF THE S PREAD OF I NTERNET R UMORS

  2019cites this paper
• Modulating brain activity and behaviour with tDCS: Rumours of its death have been greatly exaggerated.

  2019cites this paper
• REGISTERED REPORTS AS A METHOD TO INCREASE CREDIBILITY OF SCIENCE – EXPERIMENTAL STUDY AMONG PSYCHOLOGICAL STUDENTS

  2019cites this paper
• Online panels in social science research: Expanding sampling methods beyond Mechanical Turk

  2019cites this paper
• How Many Participants Do We Have to Include in Properly Powered Experiments? A Tutorial of Power Analysis with Reference Tables

  2019cites this paper
• Test-retest reliability for common tasks in vision science

  2019cites this paper
• What makes euthanasia justifiable? The role of symptoms’ characteristics and interindividual differences

  2019cites this paper
• A Novel Framework to Measure Executive Function in Down Syndrome With Applications for Early Clinical Diagnosis of Dementia.

  2019cites this paper
• Understanding of researcher behavior is required to improve data reliability

  2019cites this paper
• Conducting Web-Based Experiments for Numerical Cognition Research

  2019cites this paper
• Body Odor Disgust Sensitivity Predicts Moral Harshness Toward Moral Violations of Purity

  2019cites this paper
• Dis)continuing the continued influence effect of misinformation

  2018cites this paper
• Communicating the validity of structured job interviews with graphical visual aids

  2018cites this paper
• Comparing the use of open and closed questions for Web-based measures of the continued-influence effect

  2018cites this paper
• Writing Empirical Articles: Transparency, Reproducibility, Clarity, and Memorability

  2018cites this paper
• “Keep on Turkin’”?

  2018cites this paper
• Low and Variable Correlation Between Reaction Time Costs and Accuracy Costs Explained by Accumulation Models: Meta-Analysis and Simulations

  2018cites this paper
• A PLACE AMONG THE STARS? THE INFLUENCE OF RELIGION AND CREATIONISM ON ATTITUDES TOWARDS SPACE EXPLORATION AND BELIEFS IN EXTRATERRESTRIAL LIFE

  2018cites this paper
• Crowdsourcing Samples in Cognitive Science.

  2017cites this paper
• The meaning of a claim is its reproducibility

  2017cites this paper
• Manipulating the Alpha Level Cannot Cure Significance Testing

  2017cites this paper
• Behavioral and Brain Sciences (forthcoming)

  2017cites this paper
• Visual Motion Prediction and Verbal False Memory Performance in Autistic Children

  2017cites this paper
• Pervasiveness of effects in sample-based experimental psychology: A Re-examination of replication data from nine famous psychology experiments

  year unknowncites this paper