A Critical Reflection on the State of Data Analysis in Empirical Software Engineering

Context. Empirical Software Engineering drives innovation in SE through qualitative and quantitative studies. Since the 2006 Dagstuhl seminar, concerns about methodological rigor persist. Recent studies have highlighted misconceptions in statistical practices in ESE, yet their impact on the field’s progress and verifiability remains uninvestigated. Aim. To analyze three decades of SE research to uncover flaws in statistical methods used for data analysis in empirical studies. Next, to observe the capability of current empirical software engineering experts to identify and address these issues. Method. We conducted a large-scale literature survey, collecting over 27,000 empirical SE papers. Using a Large Language Model (LLM), we categorized studies into methodologically adequate and not adequate categories, and selected 28 primary studies (14 from each category) for expert evaluation via a focus group-based workshop. Results. Our findings reveal widespread misuse of statistical methods in empirical SE studies. Additionally, experts often struggle to detect these flaws and provide proper corrections, raising concerns about methodological rigor in the field. Conclusions. This study highlights the risks of perpetuating statistical misconceptions and advocates for a critical reform in the approach to data analysis in ESE. We advocate for developing frameworks that foster methodological awareness and rigor.

• Publication year

  2026

• Venue

  ACM Transactions on Software Engineering and Methodology

• Publication date

  2026-02-28

• Fields of study

  Not labeled

• Identifiers
  DOI 10.1145/3799715
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Network Centrality as a New Perspective on Microservice Architecture

  2025influential reference
• Handbook on Teaching Empirical Software Engineering

  2024cited by this paper
• On Large Language Models in Mission-Critical IT Governance: Are We Ready Yet?

  2024influential reference
• All You Need is Attention: Lightweight Attention-based Data Augmentation for Text Classification

  2024cited by this paper
• Experimentation in Software Engineering

  2024cited by this paper
• Statistical Inference

  2024cited by this paper
• The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews

  2024cited by this paper
• Leveraging Large Language Models for Preliminary Security Risk Analysis: A Mission-Critical Case Study

  2024cited by this paper
• VALIDATE: A deep dive into vulnerability prediction datasets

  2024influential reference
• An Extensive Comparison of Static Application Security Testing Tools

  2024cited by this paper
• A Survey of Text Classification With Transformers: How Wide? How Large? How Long? How Accurate? How Expensive? How Safe?

  2024cited by this paper
• A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT

  2023cited by this paper
• Pitfalls in Experiments with DNN4SE: An Analysis of the State of the Practice

  2023cited by this paper
• Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models

  2023cited by this paper
• Enhancing the defectiveness prediction of methods and classes via JIT

  2023cited by this paper
• Prompted LLMs as Chatbot Modules for Long Open-domain Conversation

  2023cited by this paper
• Towards a Robust Approach to Analyze Time-Dependent Data in Software Engineering

  2022cited by this paper
• On effort-aware metrics for defect prediction

  2022cited by this paper
• History of Software Engineering

  2021cited by this paper
• Natural Language Processing Recipes: Unlocking Text Data with Machine Learning and Deep Learning Using Python

  2021cited by this paper
• Language Models are Few-Shot Learners

  2020cited by this paper
• Cohort Studies in Software Engineering: A Vision of the Future

  2020cited by this paper
• Deep Learning for NLP and Speech Recognition

  2020cited by this paper
• RETRACTED ARTICLE: The correlation between social transformation economic risk and internet public opinion

  2020cited by this paper
• Natural Language Processing Recipes

  2019cited by this paper
• The Evolution of Empirical Methods in Software Engineering

  2019cited by this paper
• Problems with Statistical Practice in Human-Centric Software Engineering Experiments

  2019cited by this paper
• Text Classification Algorithms: A Survey

  2019cited by this paper
• Performance Metrics (Error Measures) in Machine Learning Regression, Forecasting and Prognostics: Properties and Typology

  2018cited by this paper
• Statistical Errors in Software Engineering Experiments: A Preliminary Literature Review

  2018influential reference
• Prioritizing Corrective Maintenance Activities for Android Applications: An Industrial Case Study on Android Crash Reports

  2018cited by this paper
• The kappa paradox

  2018cited by this paper
• Business Process Model and Notation - BPMN

  2015cited by this paper
• The Science of Computing: Shaping a Discipline

  2014cited by this paper
• The Empirical Paradigm

  2014cited by this paper
• A practical guide for using statistical tests to assess randomized algorithms in software engineering

  2011cited by this paper
• Statistical Data Analysis: A Practical Guide

  2011cited by this paper
• Steven Bird, Ewan Klein and Edward Loper: Natural Language Processing with Python, Analyzing Text with the Natural Language Toolkit

  2010cited by this paper
• A Practical Guide to Scientific Data Analysis

  2009influential reference
• Guidelines for conducting and reporting case study research in software engineering

  2009cited by this paper
• The Science of Computing

  2009cited by this paper
• Empirical Software Engineering Issues. Critical Assessment and Future Directions, International Workshop, Dagstuhl Castle, Germany, June 26-30, 2006. Revised Papers

  2007cited by this paper
• New Opportunities for Empirical Research

  2006cited by this paper
• Empirical Evaluation in Software Engineering: Role, Strategy, and Limitations

  2006cited by this paper
• Knowledge Acquisition in Software Engineering Requires Sharing of Data and Artifacts

  2006cited by this paper
• Measurement and Model Building Discussion and Summary

  2006cited by this paper
• Evidence-based software engineering

  2004cited by this paper
• Goal Question Metric (GQM) Approach

  2002cited by this paper
• Bias and causal associations in observational research.

  2002cited by this paper
• Inter-Rater Reliability: Dependency on Trait Prevalence and Marginal Homogeneity

  2002cited by this paper
• Christopher D. Manning and Hinrich Schutze. Foundations of Statistical Natural Language Processing. MIT Press, 2000. ISBN 0-262-13360-1. 620 pp. $64.95/£44.95 (cloth).

  2002cited by this paper
• Statistical Software Engineering

  1998cited by this paper
• Sample size in qualitative research.

  1995cited by this paper
• The roots of software engineering

  1990cited by this paper
• An Analysis of Variance Test for Normality (Complete Samples)

  1965cited by this paper
• Asymptotic Theory of Certain "Goodness of Fit" Criteria Based on Stochastic Processes

  1952cited by this paper
• Individual Comparisons by Ranking Methods

  1945cited by this paper
• Convergence of probability measures

  year unknowncited by this paper
• The Thermal Efficiency of Steam Engines

  year unknowninfluential reference

• No citing papers are available for this paper.