Quality Assessment of ChatGPT Generated Code and their Use by Developers

The release of large language models (LLMs) like ChatGPT has revolutionized software development. Prior works explored ChatGPT’s generated response quality, the effectiveness of different prompting techniques, its performance in programming contests, etc. However, there is limited information regarding the practical usage of ChatGPT by software developers. This data mining challenge focuses on DevGPT, a curated dataset of developer-ChatGPT conversations encompassing prompts with ChatGPT’s responses, including code snippets. Our paper leverages this dataset to investigate (RQ1) whether ChatGPT generates Python & Java code with quality issues; (RQ2) whether ChatGPT-generated code is merged into a repository, and, if it does, to what extent developers change them; and (RQ3) what are the main use cases for ChatGPT besides code generation. We found that ChatGPT-generated code suffers from using undefined/unused variables and improper documentation. They also have security issues related to improper resources and exception management. Our results show that ChatGPT-generated codes are hardly merged, and they are significantly modified before merging. Based on an analysis of developers’ discussions and the developer-ChatGPT chats, we found that developers use ChatGPT for every stage of software development and leverage it to learn about new frameworks and development kits.CCS CONCEPTS• Software and its engineering → Software performance; Software usability; Empirical software validation.

• Publication year

  2024

• Venue

  IEEE Working Conference on Mining Software Repositories

• Publication date

  2024-04-15

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1145/3643991.3645071
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Sallm: Security Assessment of Generated Code

  2023cited by this paper
• Do Users Write More Insecure Code with AI Assistants?

  2022cited by this paper
• SecurityEval dataset: mining vulnerability examples to evaluate machine learning-based code generation techniques

  2022cited by this paper
• An Empirical Study of Code Smells in Transformer-based Code Generation Techniques

  2022cited by this paper
• Productivity assessment of neural code completion

  2022cited by this paper
• Is GitHub’s Copilot as bad as humans at introducing vulnerabilities in code?

  2022cited by this paper
• M2TS: Multi-Scale Multi-Modal Approach Based on Transformer for Source Code Summarization

  2022cited by this paper
• CodeFill: Multi-token Code Completion by Jointly learning from Structure and Naming Sequences

  2022cited by this paper
• Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models

  2022cited by this paper
• The Prevalence of Code Smells in Machine Learning projects

  2021cited by this paper
• Evaluating Large Language Models Trained on Code

  2021cited by this paper
• CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation

  2021cited by this paper
• Language Models are Few-Shot Learners

  2020cited by this paper
• CodeBERT: A Pre-Trained Model for Programming and Natural Languages

  2020cited by this paper
• Code Prediction by Feeding Trees to Transformers

  2020cited by this paper
• Machine Learning Techniques for Code Smells Detection: A Systematic Mapping Study

  2019cited by this paper
• Share, But be Aware: Security Smells in Python Gists

  2019cited by this paper
• The Seven Sins: Security Smells in Infrastructure as Code Scripts

  2019cited by this paper
• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network

  2018cited by this paper
• Refactoring improving the design of existing code

  2017cited by this paper
• On the evaluation of code smells and detection tools

  2017cited by this paper
• Security Smells in Android

  2017cited by this paper
• Attention is All you Need

  2017cited by this paper
• Detecting Code Smells in Python Programs

  2016cited by this paper
• A Cooperative Parallel Search-Based Software Engineering Approach for Code-Smells Detection

  2014cited by this paper
• Sequence to Sequence Learning with Neural Networks

  2014cited by this paper

• AnCoder: Anchored Code Generation via Discrete Diffusion Models

  2026cites this paper
• Mining Type Constructs Using Patterns in AI-Generated Code

  2026cites this paper
• Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions

  2026cites this paper
• AI builds, We Analyze: An Empirical Study of AI-Generated Build Code Quality

  2026cites this paper
• Generative AI as an infrastructure copilot: automating Infrastructure-As-Code across the DevSecOps lifecycle

  2026cites this paper
• Parameter-Efficient Multi-Task Fine-Tuning in Code-Related Tasks

  2026cites this paper
• Securing LLM Code Generation: Leveraging Prompt Engineering to Mitigate Vulnerabilities Across Models and Languages

  2026cites this paper
• Do LLMs consider security? an empirical study on responses to programming questions

  2025influential citation
• AutoPAC: Exploring LLMs for Automating Policy to Code Conversion in Business Organizations

  2025cites this paper
• Studying How Configurations Impact Code Generation in LLMs: The Case of ChatGPT

  2025cites this paper
• LLMs' Reshaping of People, Processes, Products, and Society in Software Development: A Comprehensive Exploration with Early Adopters

  2025cites this paper
• A Taxonomy of Inefficiencies in LLM-Generated Python Code

  2025cites this paper
• Enhancing High-Quality Code Generation in Large Language Models with Comparative Prefix-Tuning

  2025influential citation
• Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarios

  2025cites this paper
• SandboxEval: Towards Securing Test Environment for Untrusted Code

  2025cites this paper
• So much more than test cases - An industrial study on testing of software units and components

  2025cites this paper
• Unveiling the Role of ChatGPT in Software Development: Insights from Developer-ChatGPT Interactions on GitHub

  2025cites this paper
• PatchTrack: A Comprehensive Analysis of ChatGPT's Influence on Pull Request Outcomes

  2025cites this paper
• LLM-Driven Personalized Answer Generation and Evaluation

  2025cites this paper
• Automated Feedback Generation for Programming Assignments Through Diversification

  2025cites this paper
• Can LLMs Generate Higher Quality Code Than Humans? An Empirical Study

  2025cites this paper
• ChatGPT Choreography: Discovering Developer Dialogues and Potential Software Development Lifecycle Applications

  2025cites this paper
• How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench

  2025influential citation
• ReCatcher: Towards LLMs Regression Testing for Code Generation

  2025cites this paper
• Self-Admitted GenAI Usage in Open-Source Software

  2025cites this paper
• Is Quantization a Deal-Breaker? Empirical Insights From Large Code Models

  2025cites this paper
• Is LLM-Generated Code More Maintainable & Reliable Than Human-Written Code?

  2025cites this paper
• SPENCER: Self-Adaptive Model Distillation for Efficient Code Retrieval

  2025cites this paper
• EyeMulator: Improving Code Language Models by Mimicking Human Visual Attention

  2025cites this paper
• How do Humans and LLMs Process Confusing Code?

  2025cites this paper
• Evaluation of the Code Generated By Large Language Models: The State of the Art

  2025cites this paper
• Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity

  2025cites this paper
• Developer-LLM Conversations: An Empirical Study of Interactions and Generated Code Quality

  2025cites this paper
• HFuzzer: Testing Large Language Models for Package Hallucinations via Phrase-based Fuzzing

  2025cites this paper
• Refactoring Loops in the Era of LLMs: A Comprehensive Study

  2025cites this paper
• Discrete prompt optimization using genetic algorithm for secure Python code generation

  2025cites this paper
• Bugs in AI-Generated Code - Understanding Bug Patterns and Possible Fix Strategies

  2025cites this paper
• Retrieve, Refine, or Both? Using Task-Specific Guidelines for Secure Python Code Generation

  2025cites this paper
• An Empirical Study of Java Code Improvements Based on Stack Overflow Answer Edits

  2025cites this paper
• A Causal Perspective on Measuring, Explaining and Mitigating Smells in LLM-Generated Code

  2025cites this paper
• Bridging AI and Human Knowledge: Towards a Deeper Understanding of Stack Overflow and ChatGPT

  2025cites this paper
• State of the Art of the Security of Code Generated by LLMs: A Multivocal Literature Review

  2025cites this paper
• Advancing Secure and Standard Source Code Generation Techniques

  2025cites this paper
• Exploring Code Comprehension in Scientific Programming: Preliminary Insights from Research Scientists

  2025cites this paper
• Automated Refactoring of Non-Idiomatic Python Code: A Differentiated Replication with LLMS

  2025cites this paper
• IDE Native, Foundation Model Based Agents for Software Refactoring

  2025cites this paper
• Analysis of ChatGPT-Generated Codes Across Multiple Programming Languages

  2025cites this paper
• Generative AI & Changing Work: Systematic Review of Practitioner-led Work Transformations through the Lens of Job Crafting

  2025cites this paper
• The Impact of Generative Artificial Intelligence on Software Development Teams During the Construction Phase: Systematic Mapping Study

  2025cites this paper
• KNSE-Code: A Knowledge-Aware, Safety-Enhanced, Rule-Driven Framework for Aviation Safety-Critical Software Code Generation

  2025cites this paper
• Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarios

  2024cites this paper
• Assessing ChatGPT’s Code Generation Capabilities with Short vs Long Context Programming Problems

  2024cites this paper
• Large Language Models as Robust Data Generators in Software Analytics: Are We There Yet?

  2024cites this paper
• Codellm-Devkit: A Framework for Contextualizing Code LLMs with Program Analysis Insights

  2024cites this paper
• Are Humans and LLMs Confused by the Same Code? An Empirical Study on Fixation-Related Potentials and LLM Perplexity

  year unknowncites this paper
• A Systematic Survey on Large Language Models for Code Generation

  year unknowninfluential citation