Show Your Title! A Scoping Review on Verbalization in Software Engineering with LLM-Assisted Screening

Understanding how software developers think, make decisions, and behave remains a key challenge in software engineering (SE). Verbalization techniques (methods that capture spoken or written thought processes) offer a lightweight and accessible way to study these cognitive aspects. This paper presents a scoping review of research at the intersection of SE and psychology (PSY), focusing on the use of verbal data. To make large-scale interdisciplinary reviews feasible, we employed a large language model (LLM)-assisted screening pipeline using GPT to assess the relevance of over 9,000 papers based solely on titles. We addressed two questions: what themes emerge from verbalization-related work in SE, and how effective are LLMs in supporting interdisciplinary review processes? We validated GPT's outputs against human reviewers and found high consistency, with a 13\% disagreement rate. Prominent themes mainly were tied to the craft of SE, while more human-centered topics were underrepresented. The data also suggests that SE frequently draws on PSY methods, whereas the reverse is rare.

• Publication year

  2025

• Venue

  arXiv.org

• Publication date

  2025-10-14

• Fields of study

  Computer Science, Psychology

• Identifiers
  DOI 10.48550/arXiv.2510.12294arXiv 2510.12294
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Exploring Debugging Challenges and Strategies Using Structural Topic Model: A Comparative Analysis of High and Low-Performing Students

  2024cited by this paper
• Threats to Validity in Software Engineering – hypocritical paper section or essential analysis?

  2024cited by this paper
• A Flipped Systematic Debugging Approach to Enhance Elementary Students’ Program Debugging Performance and Optimize Cognitive Load

  2023cited by this paper
• A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT

  2023cited by this paper
• SEGRESS: Software Engineering Guidelines for REporting Secondary Studies

  2023cited by this paper
• Comparing Methods for Large-Scale Agile Software Development: A Systematic Literature Review

  2022cited by this paper
• Using contextual knowledge in interactive fault localization

  2022cited by this paper
• Gamification in Software Engineering: A literature Review

  2021cited by this paper
• Programmatically Identifying Cognitive Biases Present in Software Development

  2021cited by this paper
• Identifying, categorizing and mitigating threats to validity in software engineering secondary studies

  2019cited by this paper
• Exploring principles of user-centered agile software development: A literature review

  2015cited by this paper
• Do feelings matter? On the correlation of affects and the self‐assessed productivity in software engineering

  2014cited by this paper
• Expertise-dependent visual attention strategies develop over time during debugging with multiple code representations

  2012influential reference
• Systematic literature reviews in software engineering - A systematic literature review

  2009cited by this paper
• Empirical studies of agile software development: A systematic review

  2008cited by this paper
• Lessons from applying the systematic literature review process within the software engineering domain

  2007cited by this paper
• Scoping studies: towards a methodological framework

  2005cited by this paper

• No citing papers are available for this paper.