Crowdsourcing Samples in Cognitive Science.

Crowdsourcing data collection from research participants recruited from online labor markets is now common in cognitive science. We review who is in the crowd and who can be reached by the average laboratory. We discuss reproducibility and review some recent methodological innovations for online experiments. We consider the design of research studies and arising ethical issues. We review how to code experiments for the web, what is known about video and audio presentation, and the measurement of reaction times. We close with comments about the high levels of experience of many participants and an emerging tragedy of the commons.

• Publication year

  2017

• Venue

  Trends in Cognitive Sciences

• Publication date

  2017-10-01

• Fields of study

  Medicine, Computer Science, Psychology

• Identifiers
  DOI 10.1016/j.tics.2017.06.007PMID 28803699
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• Investigating Variation in Replicability: A “Many Labs” Replication Project

  2022cited by this paper
• Generalizing from Survey Experiments Conducted on Mechanical Turk: A Replication Approach

  2018cited by this paper
• A manifesto for reproducible science

  2017cited by this paper
• Morningness/eveningness and the synchrony effect for spatial attention.

  2017cited by this paper
• The Opportunities and Limitations of Using Mechanical Turk (MTURK) in Public Administration and Management Scholarship

  2017cited by this paper
• Intertemporal Differences Among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection

  2017cited by this paper
• Crowdsourcing participants for psychological research in Australia: A test of Microworkers

  2017cited by this paper
• An Analysis of Data Quality: Professional Panels, Student Subject Pools, and Amazon's Mechanical Turk

  2017cited by this paper
• Beyond the Turk: Alternative Platforms for Crowdsourcing Behavioral Research

  2017influential reference
• Crowdsourcing Consumer Research

  2017cited by this paper
• Lie for a Dime

  2017cited by this paper
• nodeGame: Real-time, synchronous, online experiments in the browser

  2017cited by this paper
• Lookit (Part 1): A New Online Platform for Developmental Research

  2017cited by this paper
• A Community Rather Than A Union: Understanding Self-Organization Phenomenon on MTurk and How It Impacts Turkers and Requesters

  2017cited by this paper
• Conducting Online Behavioral Research Using Crowdsourcing Services in Japan

  2017cited by this paper
• Online recruitment and testing of infants with Mechanical Turk.

  2017cited by this paper
• Turking overtime: how participant characteristics and behavior vary over time and day on Amazon Mechanical Turk

  2017cited by this paper
• Conducting interactive experiments online

  2017cited by this paper
• Participant Nonnaiveté and the reproducibility of cognitive psychology

  2017cited by this paper
• Cognitive Reflection and Decision Making

  2016cited by this paper
• Mechanical Novel: Crowdsourcing Complex Work through Reflection and Revision

  2016cited by this paper
• The Communication Network Within the Crowd

  2016cited by this paper
• Online psychophysics: reaction time effects in cognitive experiments

  2016cited by this paper
• Crowd-sourced Text Analysis: Reproducible and Agile Production of Political Data

  2016cited by this paper
• Auditory presentation and synchronization in Adobe Flash and HTML5/JavaScript Web experiments

  2016influential reference
• Conducting Clinical Research Using Crowdsourced Convenience Samples.

  2016influential reference
• The importance of assessing clinical phenomena in Mechanical Turk research.

  2016cited by this paper
• TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences

  2016cited by this paper
• Using Crowdsourcing Websites for Sociological Research: The Case of Amazon Mechanical Turk

  2016cited by this paper
• Income Security in the On-Demand Economy: Findings and Policy Lessons from a Survey of Crowdworkers

  2016cited by this paper
• Investigating an alternate form of the cognitive reflection test

  2016cited by this paper
• The Validity of Crowdsourcing Data in Studying Anger and Aggressive Behavior

  2016cited by this paper
• The Demographic and Political Composition of Mechanical Turk Samples

  2016cited by this paper
• Replications with MTurkers who are naïve versus experienced with academic studies: A comment on Connors, Khamitov, Moroz, Campbell, and Henderson (2015)

  2016cited by this paper
• Reaction time effects in lab- versus Web-based research: Experimental evidence

  2015cited by this paper
• Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants

  2015cited by this paper
• Are samples drawn from Mechanical Turk valid for research on political ideology?

  2015cited by this paper
• Incentivizing High Quality Crowdwork

  2015cited by this paper
• The Generalizability of Survey Experiments*

  2015cited by this paper
• Conducting spoken word recognition research online: Validation and a new timing method

  2015cited by this paper
• Using Nonnaive Participants Can Reduce Effect Sizes

  2015cited by this paper
• The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers

  2015cited by this paper
• Internet research in psychology.

  2015cited by this paper
• psiTurk: An open-source framework for conducting replicable behavioral experiments online

  2015cited by this paper
• Estimating the reproducibility of psychological science

  2015cited by this paper
• Psychophysics in a Web browser? Comparing response times collected with JavaScript and Psychophysics Toolbox in a visual search task

  2015cited by this paper
• Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments

  2014influential reference
• The Alleged Crisis and the Illusion of Exact Replication

  2014cited by this paper
• Social heuristics shape intuitive cooperation

  2014cited by this paper
• Small Telescopes

  2014cited by this paper
• The New Statistics

  2014cited by this paper
• Cross-Sample Comparisons and External Validity

  2014cited by this paper
• Does merely going through the same moves make for a “direct” replication? Concepts, contexts, and operationalizations.

  2014cited by this paper
• Negative affect intensity influences drinking to cope through facets of emotion dysregulation

  2014cited by this paper
• Crowd-powered positive psychological interventions

  2014cited by this paper
• The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk

  2014cited by this paper
• Long-run learning in games of cooperation

  2014cited by this paper
• Posterior-Hacking: Selective Reporting Invalidates Bayesian Results Also

  2014cited by this paper
• jsPsych: A JavaScript library for creating behavioral experiments in a Web browser

  2014cited by this paper
• Data from Paper “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant”

  2014cited by this paper
• An online daily diary study of alcohol use using Amazon's Mechanical Turk.

  2014cited by this paper
• Crowd-based Semantic Event Detection and Video Annotation for Sports Videos

  2014cited by this paper
• Mechanical Turk is Not Anonymous

  2013cited by this paper
• Identifying and Training Adaptive Cross-Cultural Management Skills: The Crucial Role of Cultural Metacognition

  2013cited by this paper
• ScriptingRT: A Software Library for Collecting Response Latencies in Online Studies of Cognition

  2013cited by this paper
• Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research

  2013influential reference
• Power failure: why small sample size undermines the reliability of neuroscience

  2013cited by this paper
• Separate but equal? A comparison of participants and data gathered via Amazon's MTurk, social media, and face-to-face behavioral testing

  2013cited by this paper
• Reputation as a sufficient condition for data quality on Amazon Mechanical Turk

  2013cited by this paper
• Data collection in a flat world: the strengths and weaknesses of mechanical turk samples

  2013cited by this paper
• Collecting response times using Amazon Mechanical Turk and Adobe Flash

  2013cited by this paper
• Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers

  2013influential reference
• Using Mechanical Turk to Study Clinical Populations

  2013cited by this paper
• Assessing the Effects of Technical Variance on the Statistical Outcomes of Web Experiments Measuring Response Times

  2012cited by this paper
• Assessing Crowdsourcing Quality through Objective Tasks

  2012cited by this paper
• Towards an integrated crowdsourcing definition

  2012cited by this paper
• An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science

  2012cited by this paper
• Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk

  2012cited by this paper
• Emailing Workers Using Python

  2012cited by this paper
• Response time accuracy in Apple Macintosh computers

  2011cited by this paper
• False-Positive Psychology

  2011cited by this paper
• The viability of crowdsourcing for survey research

  2011influential reference
• Usability of residential thermostats: Preliminary investigations

  2011cited by this paper
• Amazon's Mechanical Turk

  2011cited by this paper
• Cooks or cobblers?: crowd creativity through combination

  2011cited by this paper
• Doing Bayesian Data Analysis: A Tutorial with R and BUGS

  2010cited by this paper
• VizWiz: nearly real-time answers to visual questions

  2010cited by this paper
• PsyToolkit: A software package for programming psychological experiments using Linux

  2010cited by this paper
• Most people are not WEIRD

  2010cited by this paper
• Do Mechanical Turks dream of square pie charts?

  2010cited by this paper
• Running experiments on Amazon Mechanical Turk

  2010cited by this paper
• Financial incentives and the "performance of crowds"

  2010cited by this paper
• The online laboratory: conducting experiments in a real labor market

  2010cited by this paper
• Developing a tool for measuring the decision-making competence of older adults.

  2010cited by this paper
• Crowdsourcing user studies with Mechanical Turk

  2008cited by this paper
• Experimental practices in economics: A methodological challenge for psychologists?

  2001cited by this paper
• Experimental practices in economics: A challenge for psychologists?

  2000cited by this paper
• A better stopping rule for conventional statistical tests

  1998cited by this paper
• Statistical Power Analysis for the Behavioral Sciences (2nd ed.)

  1989cited by this paper
• Time resolution of clocks: Effects on reaction time measurement—Good news for bad clocks

  1989cited by this paper

• Sensory profile of the highly sensitive personality

  2026cites this paper
• Purpose and profit: Understanding consumer reactions to social enterprises in retail and services

  2026cites this paper
• Brand origin and country-of-manufacture origin changes: do losses loom larger than gains?

  2026cites this paper
• Testing the testing effect on prolific: when retrieval practice fails to boost learning

  2026cites this paper
• On Benchmarking Human-Like Intelligence in Machines

  2025cites this paper
• Addressing low statistical power in computational modelling studies in psychology and neuroscience

  2025cites this paper
• Variability and stability of autistic traits in the general population: A systematic comparison between online and in-lab samples

  2025cites this paper
• A body detection inversion effect revealed by a large-scale inattentional blindness experiment

  2025cites this paper
• When AI explains in natural language: Unveiling the impact of generative AI explanations on educators’ grading and feedback practices

  2025cites this paper
• From lab to web: Replicating cross-language translation priming asymmetry in an online environment

  2025cites this paper
• Second Language Phonology

  2025cites this paper
• The L-maze task and web-based data collection in second language sentence processing research

  2025cites this paper
• Assessing the reliability of an online measure of the temporal binding window of audiovisual integration

  2025cites this paper
• Shedding light on public perceptions of scientists who engage in wrongness admission amidst a failed replication.

  2025cites this paper
• A ruler-based technique to rigorously control the size of visual stimuli for online psychological experiments

  2025cites this paper
• Reliable Online Auditory Cognitive Testing: An observational study

  2024cites this paper
• Moving experimental psychology online: How to obtain high quality data when we can’t see our participants

  2024cites this paper
• The effect of cognitive load, ego depletion, induction and time restriction on moral judgments about sacrificial dilemmas: a meta-analysis

  2024cites this paper
• Comparing online versus laboratory measures of speech perception in older children and adolescents

  2024cites this paper
• Amazon Mechanical Turk: The Human Sciences’ Labor Problem

  2024cites this paper
• The emotional impact of baseless discrediting of knowledge: An empirical investigation of epistemic injustice.

  2024cites this paper
• Reliable Web-Based Auditory Cognitive Testing: Observational Study

  2024cites this paper
• Audiomotor temporal recalibration modulates feeling of control: Exploration through an online experiment and Bayesian modeling.

  2024cites this paper
• Revisiting the role of computational neuroimaging in the era of integrative neuroscience

  2024cites this paper
• Analysis of Motivational Theories in Crowdsourcing Using Long Tail Theory: A Systematic Literature Review

  2024cites this paper
• High in the Cloud: Alcohol-, Cannabis-, and Co-Use Before and During Remote Research Participation

  2024cites this paper
• Perceptual representations mediate effects of stimulus properties on liking for music

  2024cites this paper
• Learning from academic video with subtitles: When foreign language proficiency matters

  2024cites this paper
• Informing Machine Perception With Psychophysics

  2024cites this paper
• Leveraging Online Treatment to Re-examine the Association Between Alcohol Use and Disinhibition.

  2024cites this paper
• Studying trait-characteristics and neural correlates of the emotional ego- and altercentric bias using an audiovisual paradigm

  2023cites this paper
• Undue Influence or Exploitation — A Qualitative Inquiry into an Ethical Dilemma Around Payment in Crowd Work-Based Research in the U.S.

  2023cites this paper
• From Lab-Testing to Web-Testing in Cognitive Research: Who You Test is More Important than how You Test

  2023cites this paper
• The ethical ambiguity of AI data enrichment: Measuring gaps in research ethics norms and practices

  2023cites this paper
• Subjective signal strength distinguishes reality from imagination

  2023cites this paper
• PACT: Developing and Evaluating a Digital Sexual Consent Program for Youth

  2023cites this paper
• Large and small financial incentives may motivate COVID-19 vaccination: A randomized, controlled survey experiment

  2023cites this paper
• Topical Preference Trumps Other Features in News Recommendation: A Conjoint Analysis on a Representative Sample from Norway

  2023cites this paper
• L2 self-paced reading data collection across three contexts: In-person, online, and crowdsourcing

  2023cites this paper
• Development and validation of the male postcoital affect scale for heterosexual men.

  2023cites this paper
• Methodological considerations for behavioral studies relying on response time outcomes through online crowdsourcing platforms

  2023cites this paper
• Probabilistic prediction and context tree identification in the Goalkeeper game

  2023cites this paper
• [Developing Knowledge Together: Participatory Methods in Psychological and Neuroscientific Research with Children and Adolescents].

  2023cites this paper
• The Effects of Interaction Conflicts, Levels of Automation, and Frequency of Automation on Human Automation Trust and Acceptance

  2023cites this paper
• Gay vs. straight? Implications of intergroup perceptions on minority stress and the mental health of lesbian and gay people

  2023cites this paper
• Using curiosity to counter health information avoidance.

  2023cites this paper
• Benchmark Evaluation for Tasks with Highly Subjective Crowdsourced Annotations: Case study in Argument Mining of Political Debates

  2023cites this paper
• Effects of pay rate and instructions on attrition in crowdsourcing research

  2023cites this paper
• Forms of Overconfidence: Reconciling Divergent Levels with Consistent Individual Differences

  2023cites this paper
• Tragedy of the Commons in Crowd Work-Based Research

  2023cites this paper
• “Buy local” campaigns in times of crisis: insights from reactance theory

  2023cites this paper
• The Dark Side of Recruitment in Crowdsourcing: Ethics and Transparency in Micro-Task Marketplaces

  2023cites this paper
• Exploring synchrony effects in performance on tasks involving cognitive inhibition: An online study of young adults

  2023cites this paper
• Comparing data quality from an online and in-person lab sample on dynamic theory of mind tasks

  2023cites this paper
• A Case Study of the Validity of Web-based Visuomotor Rotation Experiments

  2023cites this paper
• Inattentive responding can induce spurious associations between task behaviour and symptom measures

  2023cites this paper
• Assessing the reliability of web-based measurements of visual function

  2023cites this paper
• Online data collection to address language sampling bias: lessons from the COVID-19 pandemic

  2022cites this paper
• How to Maintain Data Quality When You Can't See Your Participants

  2022cites this paper
• Using Human Perception to Regularize Transfer Learning

  2022cites this paper
• Understanding crowdsourcing in science

  2022cites this paper
• Recruiting Sexual and Gender Minority United States Military Personnel for Research Via Peer Networks: Addressing the Challenges

  2022cites this paper
• Comparing random dot motion in MATLAB vs. Inquisit Millisecond

  2022cites this paper
• Developing skin cancer education materials for darker skin populations: crowdsourced design, message targeting, and acral lentiginous melanoma

  2022cites this paper
• Frustration and ennui among Amazon MTurk workers

  2022cites this paper
• Drivers of Data Quality in Advertising Research: Differences across MTurk and Professional Panel Samples

  2022cites this paper
• Guiding Machine Perception with Psychophysics

  2022cites this paper
• A method to induce stress in human subjects in online research environments

  2022cites this paper
• Towards a typology development of crowdsourcing in science

  2022cites this paper
• Crowdsourced EEG experiments: A proof of concept for remote EEG acquisition using EmotivPRO Builder and EmotivLABS

  2022cites this paper
• Crowdsourcing Creative Work

  2022cites this paper
• Media Exposure and Risk Perception as Predictors of Engagement in COVID-19 Preventive Behaviors: Extending the Theory of Planned Behavior Across Two Cultures

  2022cites this paper
• Subjective Socioeconomic Status, Cognitive Abilities, and Personal Control: Associations With Health Behaviours

  2022cites this paper
• Replacing vertical actions by mouse movements: a web-suited paradigm for investigating vertical spatial associations

  2022cites this paper
• Measuring event segmentation: An investigation into the stability of event boundary agreement across groups

  2022cites this paper
• A Note on Increases in Inattentive Online Survey-Takers Since 2020

  2022cites this paper
• Comparative effectiveness of mandates and financial policies targeting COVID-19 vaccine hesitancy: A randomized, controlled survey experiment

  2022cites this paper
• Blend in or stand out: social anxiety levels shape information-sharing strategies

  2022cites this paper
• Online data collection to address language sampling bias: lessons from the COVID-19 pandemic

  2022cites this paper
• Cognitive logic in management

  2021cites this paper
• Online data collection to address language sampling bias: lessons from the COVID-19 pandemic

  2021cites this paper
• Isolated Sandbox Environment Architecture for Running Cognitive Psychological Experiments in Web Platforms

  2021cites this paper
• False beliefs about sleep and their associations with sleep-related behavior.

  2021cites this paper
• UvA-DARE (Digital Academic Repository) Lured into listening Engaging games as an alternative to reward-based crowdsourcing in music research Lured Into Listening Engaging Games as an Alternative to Reward-Based Crowdsourcing in Music Research

  2021cites this paper
• Distinct but correlated latent factors support the regulation of learned conflict-control and task-switching.

  2021cites this paper
• Beyond online participant crowdsourcing: The benefits and opportunities of big team addiction science.

  2021cites this paper
• Agricultural Data Collection to Minimize Measurement Error and Maximize Coverage

  2021influential citation
• Normal colour perception in developmental prosopagnosia

  2021cites this paper
• Are any samples representative or unbiased? reply to Pickering and Blaszczynski

  2021cites this paper
• Relation Classification with Cognitive Attention Supervision

  2021cites this paper
• Attitudes and habits regarding brain training applications and games among Japanese consumers: a cross-sectional study

  2021cites this paper
• Paid online convenience samples in gambling studies: questionable data quality

  2021cites this paper
• Do data from mechanical Turk subjects replicate accuracy, response time, and diffusion modeling results?

  2021cites this paper
• Compassionate goals, prosocial emotions, and prosocial behaviours during the COVID‐19 pandemic

  2021cites this paper
• A THREE-LEVEL APPROACH TO UNDERSTAND CULTURAL VARIABILITY AND THE EVOLUTION OF HUMAN ATTENTION

  2021cites this paper
• Reward-Related Attentional Capture Moderates the Association between Fear-Driven Motives and Heavy Drinking

  2021cites this paper
• Implicit Communication Through Social Distancing: Can Social Navigation Communicate Social Norms?

  2021cites this paper
• Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

  2021cites this paper
• Reliability of web-based affective auditory stimulus presentation

  2021cites this paper
• Smartphones and the Neuroscience of Mental Health.

  2021cites this paper