"Bad smells" in software analytics papers

Abstract Context There has been a rapid growth in the use of data analytics to underpin evidence-based software engineering. However the combination of complex techniques, diverse reporting standards and poorly understood underlying phenomena are causing some concern as to the reliability of studies. Objective Our goal is to provide guidance for producers and consumers of software analytics studies (computational experiments and correlation studies). Method We propose using “bad smells”, i.e., surface indications of deeper problems and popular in the agile software community and consider how they may be manifest in software analytics studies. Results We list 12 “bad smells” in software analytics papers (and show their impact by examples). Conclusions We believe the metaphor of bad smell is a useful device. Therefore we encourage more debate on what contributes to the validity of software analytics studies (so we expect our list will mature over time).

• Publication year

  2018

• Venue

  Information and Software Technology

• Publication date

  2018-03-14

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1016/J.INFSOF.2019.04.005arXiv 1803.05518
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Randomization

  2020cited by this paper
• Why Most Published Research Findings Are False

  2019cited by this paper
• Data-Driven Search-Based Software Engineering

  2018cited by this paper
• Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results

  2018cited by this paper
• The Impact of Automated Parameter Optimization on Defect Prediction Models

  2018cited by this paper
• A manifesto for reproducible science

  2017cited by this paper
• Simpler Transfer Learning (Using "Bellwethers")

  2017cited by this paper
• Is "Better Data" Better Than "Better Data Miners"?

  2017cited by this paper
• We Don't Need Another Hero?: The Impact of "Heroes" on Software Development

  2017cited by this paper
• Easy over hard: a case study on deep learning

  2017cited by this paper
• Revisiting unsupervised learning for defect prediction

  2017cited by this paper
• 2. Introduction

  2017cited by this paper
• Automatic Database Management System Tuning Through Large-scale Machine Learning

  2017cited by this paper
• Supervised vs Unsupervised Models: A Holistic Look at Effort-Aware Just-in-Time Defect Prediction

  2017cited by this paper
• Writing Good Software Engineering Research Papers: Revisited

  2017cited by this paper
• Robust Statistical Methods for Empirical Software Engineering

  2017cited by this paper
• What is the Connection Between Issues, Bugs, and Enhancements?

  2017cited by this paper
• Equivalence Tests

  2017cited by this paper
• Power, false discovery rate and Winner’s Curse in eQTL studies

  2017cited by this paper
• "Better Data" is Better than "Better Data Miners" (Benefits of Tuning SMOTE for Defect Prediction)

  2017influential reference
• Measurement error and the replication crisis

  2017cited by this paper
• Refactoring improving the design of existing code

  2017cited by this paper
• Incorrect results in software engineering experiments

  2016cited by this paper
• Belief & Evidence in Empirical Software Engineering

  2016cited by this paper
• What is wrong with topic modeling? And how to fix it using search-based software engineering

  2016cited by this paper
• The natural selection of bad science

  2016cited by this paper
• Prediction Interval: What to Expect When You’re Expecting … A Replication

  2016cited by this paper
• What is Wrong with Topic Modeling? (and How to Fix it Using Search-based SE)

  2016cited by this paper
• Why is Differential Evolution Better than Grid Search for Tuning Defect Predictors?

  2016cited by this paper
• Would wider adoption of reproducible research be beneficial for empirical software engineering research?

  2016cited by this paper
• What does research reproducibility mean?

  2016cited by this paper
• Incorrect results in software engineering experiments: How to improve research practices

  2016influential reference
• The 'BigSE' Project: Lessons Learned from Validating Industrial Text Mining

  2016cited by this paper
• Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature

  2016cited by this paper
• Tuning for Software Analytics: is it Really Necessary?

  2016influential reference
• Data Sets and Data Quality in Software Engineering: Eight Years On

  2016cited by this paper
• Grounded Theory in Software Engineering Research: A Critical Review and Guidelines

  2016cited by this paper
• Automated Parameter Optimization of Classification Techniques for Defect Prediction Models

  2016cited by this paper
• “Sampling” as a Baseline Optimizer for Search-Based Software Engineering

  2016cited by this paper
• Replication

  2015cited by this paper
• Cost-Efficient Sampling for Performance Prediction of Configurable Systems (T)

  2015cited by this paper
• Evidence-Based Software Engineering and Systematic Reviews

  2015cited by this paper
• How practitioners perceive the relevance of software engineering research

  2015cited by this paper
• Revisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models

  2015influential reference
• Replication, falsification, and the crisis of confidence in social psychology

  2015cited by this paper
• Estimating the reproducibility of psychological science

  2015cited by this paper
• Equivalence Tests

  2015cited by this paper
• Guidelines for conducting systematic mapping studies in software engineering: An update

  2015cited by this paper
• A Baseline Model for Software Effort Estimation

  2015cited by this paper
• Deep Learning

  2015cited by this paper
• An Evolutionary Many-Objective Optimization Algorithm Using Reference-Point-Based Nondominated Sorting Approach, Part I: Solving Problems With Box Constraints

  2014cited by this paper
• Researcher Bias: The Use of Machine Learning in Software Defect Prediction

  2014cited by this paper
• Sailing From the Seas of Chaos Into the Corridor of Stability

  2014cited by this paper
• Ranking and Clustering Software Cost Estimation Models through a Multiple Comparisons Algorithm

  2013cited by this paper
• Data Quality: Some Comments on the NASA Software Defect Datasets

  2013cited by this paper
• Uncovering theories in software engineering

  2013cited by this paper
• Pareto-optimal search-based software engineering (POSBSE): A literature survey

  2013cited by this paper
• Power failure: why small sample size undermines the reliability of neuroscience

  2013cited by this paper
• Data quality in empirical software engineering: a targeted review

  2013cited by this paper
• Sampling

  2013cited by this paper
• Can We Trust Our Results? A Mapping Study on Data Quality

  2013cited by this paper
• A practical guide to controlled experiments of software engineering tools with human participants

  2013cited by this paper
• Analyze this! 145 questions for data scientists in software engineering

  2013cited by this paper
• Special issue on repeatable results in software engineering prediction

  2012cited by this paper
• Experimentation in Software Engineering

  2012cited by this paper
• Random Search for Hyper-Parameter Optimization

  2012influential reference
• Where's the Theory for Software Engineering?

  2012cited by this paper
• Practical Bayesian Optimization of Machine Learning Algorithms

  2012cited by this paper
• A method for evaluating rigor and industrial relevance of technology evaluations

  2011cited by this paper
• A practical guide for using statistical tests to assess randomized algorithms in software engineering

  2011cited by this paper
• Research synthesis in software engineering: A tertiary study

  2011cited by this paper
• Reporting computing projects through structured abstracts: a quasi-experiment

  2011cited by this paper
• Starfish: A Self-tuning System for Big Data Analytics

  2011cited by this paper
• False-Positive Psychology

  2011cited by this paper
• The essential guide to effect sizes : statistical power, meta-analysis, and the interpretation of research results

  2010cited by this paper
• Making Sense of Data II: A Practical Guide to Data Visualization, Advanced Data Mining Methods, and Applications

  2010cited by this paper
• On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation

  2010cited by this paper
• Guidelines for conducting and reporting case study research in software engineering

  2009cited by this paper
• Introduction to Meta‐Analysis

  2009cited by this paper
• Making Sense of Data II

  2009cited by this paper
• Sample size planning for statistical power and accuracy in parameter estimation.

  2008cited by this paper
• Strength of evidence in systematic reviews in software engineering

  2008cited by this paper
• Length and readability of structured software engineering abstracts

  2008cited by this paper
• Benchmarking Classification Models for Software Defect Prediction: A Proposed Framework and Novel Findings

  2008cited by this paper
• Modern robust statistical methods: an easy way to maximize the accuracy and power of your research.

  2008influential reference
• Building Theories in Software Engineering

  2008cited by this paper
• Data sets and data quality in software engineering

  2008cited by this paper
• Data Mining Static Code Attributes to Learn Defect Predictors

  2007cited by this paper
• An Introduction to the Bootstrap

  2007cited by this paper
• A systematic review of effect size in software engineering experiments

  2007cited by this paper
• MOEA/D: A Multiobjective Evolutionary Algorithm Based on Decomposition

  2007cited by this paper
• Guide to Advanced Empirical Software Engineering

  2007cited by this paper
• A systematic review of statistical power in software engineering experiments

  2006cited by this paper
• How to lie with bad data

  2005cited by this paper
• Finding the right data for software cost modeling

  2005cited by this paper
• Why Most Published Research Findings Are False

  2005cited by this paper
• The Problem of Overfitting

  2004cited by this paper
• Mining version histories to guide software changes

  2004cited by this paper
• Evidence-based software engineering

  2004cited by this paper
• Benchmarking Attribute Selection Techniques for Discrete Class Data Mining

  2003cited by this paper

• Evidence-Based Software Engineering Guidelines Revisited

  2025cites this paper
• A Generalized Method for Filtering Noise in Open-Source Project Selection

  2025cites this paper
• On the Energy Footprint of Using a Small Language Model for Unit Test Generation

  2025cites this paper
• Exploring the Garden of Forking Paths in Empirical Software Engineering Research: A Multiverse Analysis

  2025cites this paper
• Crossover Designs in Software Engineering Experiments: Review of the State of Analysis

  2024cites this paper
• On the effectiveness of developer features in code smell prioritization: A replication study

  2024cites this paper
• A Second Look at the Impact of Passive Voice Requirements on Domain Modeling: Bayesian Reanalysis of an Experiment

  2024cites this paper
• SEGRESS: Software Engineering Guidelines for REporting Secondary Studies

  2023cites this paper
• Towards Causal Analysis of Empirical Software Engineering Data: The Impact of Programming Languages on Coding Competitions

  2023influential citation
• Empirical research in software architecture - Perceptions of the community

  2023cites this paper
• Evaluating classifiers in SE research: the ECSER pipeline and two replication studies

  2022cites this paper
• Semi-Automatic Mapping Technique Using Snowballing to Support Massive Literature Searches in Software Engineering

  2022cites this paper
• Ground-Truth Deficiencies in Software Engineering: When Codifying the Past Can Be Counterproductive

  2022cites this paper
• Use and Misuse of the Term “Experiment” in Mining Software Repositories Research

  2021influential citation
• Applying Bayesian Analysis Guidelines to Empirical Software Engineering Data: The Case of Programming Languages and Code Quality

  2021cites this paper
• Predicting Design Impactful Changes in Modern Code Review: A Large-Scale Empirical Study

  2021cites this paper
• Conclusion stability for natural language based mining of design discussions

  2021cites this paper
• Unveiling multiple facets of design degradation in modern code review

  2021cites this paper
• Ground-Truth Deficiencies in Software Engineering: When Codifying the Past Can Be Counterproductive

  2021cites this paper
• Data‐driven benchmarking in software development effort estimation: The few define the bulk

  2020cites this paper
• Cross-Dataset Design Discussion Mining

  2020cites this paper
• Publish or perish, but do not forget your software artifacts

  2020cites this paper
• An empirical study of Linespots: A novel past‐fault algorithm

  2020cites this paper
• Contemporary Empirical Methods in Software Engineering

  2020cites this paper
• Predicting Defective Lines Using a Model-Agnostic Technique

  2020cites this paper
• Educating Future Software Architects in the Art and Science of Analysing Software Data

  2020cites this paper
• Improved Recognition of Security Bugs via Dual Hyperparameter Optimization

  2019cites this paper
• Bayesian Data Analysis in Empirical Software Engineering: The Case of Missing Data

  2019cites this paper
• Mining Text in Incident Repositories: Experiences and Perspectives on Adopting Machine Learning Solutions in Practice

  2019cites this paper
• Data of low quality is better than no data

  2019cites this paper
• How to Better Distinguish Security Bug Reports (Using Dual Hyperparameter Optimization)

  2019cites this paper
• Better Security Bug Report Classification via Hyperparameter Optimization

  2019cites this paper
• Integrating runtime data with development data to monitor external quality: challenges from practice

  2019cites this paper
• Processing Large Datasets of Fined Grained Source Code Changes

  2019cites this paper
• Pitfalls Analyzer: Quality Control for Model-Driven Data Science Pipelines

  2019influential citation
• A Method to Assess and Argue for Practical Significance in Software Engineering

  2018cites this paper
• Better software analytics via “DUO”: Data mining algorithms using/used-by optimizers

  2018cites this paper
• Arguing Practical Significance in Empirical Software Engineering

  2018cites this paper
• Bayesian Data Analysis in Empirical Software Engineering Research

  2018cites this paper