Standing on the Shoulders of Giants: Bug-Aware Automated GUI Testing via Retrieval Augmentation

In software development, similar apps often encounter similar bugs due to shared functionalities and implementation methods. However, current automated GUI testing methods mainly focus on generating test scripts to cover more pages by analyzing the internal structure of the app, without targeted exploration of paths that may trigger bugs, resulting in low efficiency in bug discovery. Considering that a large number of bug reports on open source platforms can provide external knowledge for testing, this paper proposes BugHunter, a novel bug-aware automated GUI testing approach that generates exploration paths guided by bug reports from similar apps, utilizing a combination of Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG). Instead of focusing solely on coverage, BugHunter dynamically adapts the testing process to target bug paths, thereby increasing bug detection efficiency. BugHunter first builds a high-quality bug knowledge base from historical bug reports. Then it retrieves relevant reports from this large bug knowledge base using a two-stage retrieval process, and generates test paths based on similar apps’ bug reports. BugHunter also introduces a local and global path-planning mechanism to handle differences in functionality and UI design across apps, and the ambiguous behavior or missing steps in the online bug reports. We evaluate BugHunter on 121 bugs across 71 apps and compare its performance against 16 state-of-the-art baselines. BugHunter achieves 60% improvement in bug detection over the best baseline, with comparable or higher coverage against the baselines. Furthermore, BugHunter successfully detects 49 new crash bugs in real-world apps from Google Play, with 33 bugs fixed, 9 confirmed, and 7 pending feedback.

• Publication year

  2025

• Venue

  Proc. ACM Softw. Eng.

• Publication date

  2025-06-19

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1145/3715755
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Enhancing Exploratory Testing by Large Language Model and Knowledge Graph

  2024cited by this paper
• Explainable automated debugging via large language model-driven scientific debugging

  2023cited by this paper
• Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing

  2023cited by this paper
• An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation

  2023cited by this paper
• Learning-based Widget Matching for Migrating GUI Test Cases

  2023cited by this paper
• ChatUniTest: A Framework for LLM-Based Test Generation

  2023cited by this paper
• Prompting Is All You Need: Automated Android Bug Replay with Large Language Models

  2023cited by this paper
• Constructing a System Knowledge Graph of User Tasks and Failures from Bug Reports to Support Soap Opera Testing

  2022cited by this paper
• Guided Bug Crush: Assist Manual GUI Testing of Android Apps via Hint Moves

  2022cited by this paper
• Vet: identifying and avoiding UI exploration tarpits

  2021cited by this paper
• Deep Reinforcement Learning for Black-box Testing of Android Apps

  2021cited by this paper
• An infrastructure approach to improving effectiveness of Android UI testing tools

  2021cited by this paper
• Benchmarking automated GUI testing for Android against real-world bugs

  2021cited by this paper
• Generating Accurate Assert Statements for Unit Test Cases using Pretrained Transformers

  2020cited by this paper
• ComboDroid: Generating High-Quality Test Inputs for Android Apps via Use Case Combinations

  2020cited by this paper
• Time-travel Testing of Android Apps

  2020cited by this paper
• Reinforcement learning based curiosity-driven testing of Android applications

  2020cited by this paper
• Owl Eyes: Spotting UI Display Issues via Visual Understanding

  2020cited by this paper
• Fastbot: A Multi-Agent Model-Based Test Generation System : Beijing Bytedance Network Technology Co., Ltd.

  2020cited by this paper
• Mobolic: An automated approach to exercising mobile application GUIs using symbiosis of online testing technique and customated input generation

  2018cited by this paper
• AppFlow: using machine learning to synthesize robust, reusable UI tests

  2018cited by this paper
• Automatically translating bug reports into test cases for mobile apps

  2018cited by this paper
• Guided, stochastic model-based GUI testing of Android apps

  2017cited by this paper
• Sapienz: multi-objective automated testing for Android applications

  2016cited by this paper
• Automated test input generation for Android: are we really there yet in an industrial case?

  2016cited by this paper
• Reducing Combinatorics in GUI Testing of Android Applications

  2016cited by this paper
• Dynodroid: an input generation system for Android apps

  2013cited by this paper
• Designing and comparing automated test oracles for GUI-based software applications

  2007cited by this paper
• The Complete Essays

  2004cited by this paper

• AI Agents for Web Testing: A Case Study in the Wild

  2025cites this paper
• Aligning Academia with Industry: An Empirical Study of Industrial Needs and Academic Capabilities in AI-Driven Software Engineering

  2025influential citation
• Beyond Static GUI Agent: Evolving LLM-based GUI Testing via Dynamic Memory

  2025cites this paper
• RAG4Test: Retrieving GUI States for Multilingual Bug Report and Test Case Generation via LLMs

  2025cites this paper