Tell Us What You Really Think: A Think Aloud Protocol Analysis of the Verbal Cognitive Reflection Test

The standard interpretation of cognitive reflection tests assumes that correct responses are reflective and lured responses are unreflective. However, prior process-tracing of mathematical reflection tests has cast doubt on this interpretation. In two studies (N = 201), we deployed a validated think-aloud protocol in-person and online to test how this assumption is satisfied by the new, validated, less familiar, and non-mathematical verbal Cognitive Reflection Test (vCRT). Verbalized thoughts in both studies revealed that most (but not all) correct responses involved reflection and that most (but not all) lured responses lacked reflection. The think-aloud protocols seemed to reflect business-as-usual performance: thinking aloud did not disrupt test performance compared to a control group. These data suggest that the vCRT usually satisfies the standard interpretation of the reflection tests (albeit not without exceptions) and that the vCRT can be a good measure of the construct theorized by the two-factor explication of ‘reflection’ (as deliberate and conscious).

• Publication year

  2023

• Venue

  Journal of Intelligence

• Publication date

  2023-04-01

• Fields of study

  Medicine, Psychology

• Identifiers
  DOI 10.3390/jintelligence11040076PMID 37103261PMCID 10146599
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• A Two-Factor Explication of “Reflection”

  2025cited by this paper
• The factor structure of cognitive reflection, numeracy, and fluid intelligence: The evidence from the Polish adaptation of the Verbal CRT

  2022cited by this paper
• Great Minds do not Think Alike: Philosophers’ Views Predicted by Reflection, Education, Personality, and Other Demographic Differences

  2022cited by this paper
• Bounded reflectivism and epistemic identity

  2021cited by this paper
• A mistaken confidence in data

  2021cited by this paper
• Why do people fail to see simple solutions? Using think-aloud protocols to uncover the mechanism behind the Einstellung (mental set) effect

  2020cited by this paper
• A reflection on cognitive reflection – testing convergent validity of two versions of the Cognitive Reflection Test

  2020cited by this paper
• Activating reflective thinking with decision justification and debiasing training

  2020cited by this paper
• Domain-specific experience and dual-process thinking

  2020cited by this paper
• A reflection on cognitive reflection – testing convergent/divergent validity of two measures of cognitive reflection

  2020cited by this paper
• Reasoning strategies predict use of very fast logical reasoning

  2020cited by this paper
• The false allure of fast lures

  2020cited by this paper
• Individual differences in logical intuitions on reasoning problems presented under two-response paradigm

  2019cited by this paper
• The Smart System 1: evidence for the intuitive nature of correct responding on the bat-and-ball problem

  2019cited by this paper
• What we can (and can’t) infer about implicit bias from debiasing experiments

  2019cited by this paper
• The cognitive reflection test is robust to multiple exposures

  2018cited by this paper
• Performance on the Cognitive Reflection Test is stable across time

  2018cited by this paper
• Miserliness in human cognition: the interaction of detection, override and mindware

  2018cited by this paper
• Measuring cognitive reflection without maths: Development and validation of the verbal cognitive reflection test

  2018cited by this paper
• The cognitive reflection test revisited: exploring the ways individuals solve the test

  2017cited by this paper
• Prolific.ac—A subject pool for online experiments

  2017cited by this paper
• Beyond the Turk: Alternative Platforms for Crowdsourcing Behavioral Research

  2017cited by this paper
• Slower is not always better: Response-time evidence clarifies the limited role of miserly information processing in the Cognitive Reflection Test

  2017cited by this paper
• A limitation of the Cognitive Reflection Test: familiarity

  2016cited by this paper
• Belief bias, base rates and moral judgment: re-evaluating the default interventionist dual process account

  2016cited by this paper
• Dual-process theories and consciousness: the case for ‘Type Zero’ cognition

  2016cited by this paper
• The Thinking Mind : A Festschrift for Ken Manktelow

  2016cited by this paper
• Is the cognitive reflection test a measure of both reflection and intuition?

  2016cited by this paper
• What makes us think? A three-stage dual-process model of analytic engagement.

  2015cited by this paper
• Conflict, metacognition, and analytic thinking

  2014cited by this paper
• Assessing miserly information processing: An expansion of the Cognitive Reflection Test

  2014cited by this paper
• A gap in Nisbett and Wilson's findings? A first-person access to our cognitive processes.

  2013cited by this paper
• Life after P-Hacking

  2013cited by this paper
• Matching bias on the selection task: It's fast and feels good

  2013cited by this paper
• Protocol Analysis

  2012cited by this paper
• Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods.

  2011cited by this paper
• Intuition, reason, and metacognition.

  2011cited by this paper
• Dual-Process and Dual-System Theories of Reasoning

  2010cited by this paper
• Evidence for a Verbally-Based Analytic Component to Insight Problem Solving

  2009cited by this paper
• On the resolution of conflict in dual process theories of reasoning

  2007cited by this paper
• Measuring Deliberation’s Content: A Coding Scheme

  2007cited by this paper
• Cognitive Reflection and Decision Making

  2005cited by this paper
• Valid and Non-Reactive Verbalization of Thoughts During Performance of Tasks Towards a Solution to the Central Problems of Introspection as a Source of Scientific Data

  2003cited by this paper
• Representativeness revisited: Attribute substitution in intuitive judgment.

  2002cited by this paper
• Thoughts beyond words : When language overshadows insight

  1993cited by this paper
• Protocol Analysis: Verbal Reports as Data

  1984cited by this paper
• On the conflict between logic and belief in syllogistic reasoning

  1983cited by this paper
• Verbal reports as data.

  1980cited by this paper
• The accuracy of verbal reports about the effects of stimuli on evaluations and behavior

  1978cited by this paper
• The measurement of observer agreement for categorical data.

  1977cited by this paper
• Science Perspectives on Psychological

  year unknowncited by this paper

• Exploring multimodal, non-invasive stress assessment through audio-visual and textual cues integrated with psychometric survey data

  2026cites this paper
• Ambient Intelligence Highlights Familiar Obstacles to Pre-Deployment Testing of Healthcare Innovations

  2026cites this paper
• Metareasoning: Theoretical and Methodological Developments

  2025cites this paper
• Analytic atheism and analytic apostasy across cultures

  2025cites this paper
• Standing Up for Philosophy

  2025cites this paper
• Does performance on the cognitive reflection test indicate cognitive miserliness? Evidence from response times, working memory capacity, and warnings

  2025influential citation
• Strategic Reflectivism In Intelligent Systems

  2025cites this paper
• System 2 and cognitive transparency: deliberation helps to justify sound intuitions during reasoning

  2025cites this paper
• WIP: Motivating Calculus for Computer Science Majors

  2025cites this paper
• Online monitoring and evaluation for conceptual design of product morphological structure driven by the retrospective analysis of cognitive behaviour

  2024cites this paper
• An Analysis and Reflection Pedagogical Toolkit Used in Induction Mentoring

  2024cites this paper
• Using text mining and machine learning to predict reasoning activities from think-aloud transcripts in computer assisted learning

  2024cites this paper
• Zermelo Game: All or None?

  2024cites this paper
• Increased reliance on heuristic thinking in mild cognitive impairment

  2024cites this paper
• Great Minds do not Think Alike: Philosophers’ Views Predicted by Reflection, Education, Personality, and Other Demographic Differences

  2022influential citation
• Who detects and why: how do individual differences in cognitive characteristics underpin different types of responses to reasoning tasks?

  2022influential citation
• Bounded reflectivism and epistemic identity

  2021cites this paper
• The psychology of philosophy: Associating philosophical views with psychological traits in professional philosophers

  2021cites this paper
• Reflective reasoning & philosophy

  2021cites this paper
• Your health vs. my liberty: Philosophical beliefs dominated reflection and identifiable victim effects when predicting public health recommendation compliance during the COVID-19 pandemic

  2020cites this paper