Finding Critical Scenarios for Automated Driving Systems: A Systematic Mapping Study

Scenario-based approaches have been receiving a huge amount of attention in research and engineering of automated driving systems. Due to the complexity and uncertainty of the driving environment, and the complexity of the driving task itself, the number of possible driving scenarios that an Automated Driving System or Advanced Driving-Assistance System may encounter is virtually infinite. Therefore it is essential to be able to reason about the identification of scenarios and in particular critical ones that may impose unacceptable risk if not considered. Critical scenarios are particularly important to support design, verification and validation efforts, and as a basis for a safety case. In this paper, we present the results of a systematic mapping study in the context of autonomous driving. The main contributions are: (i) introducing a comprehensive taxonomy for critical scenario identification methods; (ii) giving an overview of the state-of-the-art research based on the taxonomy encompassing 86 papers between 2017 and 2020; and (iii) identifying open issues and directions for further research. The provided taxonomy comprises three main perspectives encompassing the problem definition (the why), the solution (the methods to derive scenarios), and the assessment of the established scenarios. In addition, we discuss open research issues considering the perspectives of coverage, practicability, and scenario space explosion.

• Publication year

  2023

• Venue

  IEEE Transactions on Software Engineering

• Publication date

  2023-03-01

• Fields of study

  Computer Science, Engineering

• Identifiers
  DOI 10.1109/TSE.2022.3170122
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Finding Critical Scenarios for Automated Driving Systems: The Data Extraction Form

  2022influential reference
• Test Scenario Generation and Optimization Technology for Intelligent Driving Systems

  2022cited by this paper
• Criticality Analysis for the Verification and Validation of Automated Vehicles

  2021cited by this paper
• A Theoretical Foundation of Intelligence Testing and Its Application for Intelligent Vehicles

  2021cited by this paper
• Cyber-physical systems have far-reaching implications

  2021cited by this paper
• Safety assessment of highly automated driving systems in test tracks: A new framework.

  2020cited by this paper
• Safety Challenges for Autonomous Vehicles in the Absence of Connectivity

  2020cited by this paper
• Survey on Scenario-Based Safety Assessment of Automated Vehicles

  2020cited by this paper
• Fundamental Considerations around Scenario-Based Testing for Automated Driving

  2020influential reference
• Testing Scenario Library Generation for Connected and Automated Vehicles: An Adaptive Framework

  2020cited by this paper
• A Survey of Algorithms for Black-Box Safety Validation

  2020influential reference
• Combining Reachability Analysis and Importance Sampling for Accelerated Evaluation of Highway Automated Vehicles at Pedestrian Crossing

  2020cited by this paper
• Towards an Operational Design Domain That Supports the Safety Argumentation of an Automated Driving System

  2020cited by this paper
• Training Adversarial Agents to Exploit Weaknesses in Deep Control Policies

  2020cited by this paper
• Ontology-based test generation for automated and autonomous driving functions

  2020cited by this paper
• Is neuron coverage a meaningful measure for testing deep neural networks?

  2020cited by this paper
• Identifying the Operational Design Domain for an Automated Driving System through Assessed Risk

  2020cited by this paper
• Identification and Quantification of Hazardous Scenarios for Automated Driving

  2020cited by this paper
• Mining and comparative analysis of typical pre-crash scenarios from IGLAD.

  2020influential reference
• Low-Cost Urban Test Scenario Generation Using Microscopic Traffic Simulation

  2020influential reference
• A Trajectory-Based Method for Scenario Analysis and Test Effort Reduction for Highly Automated Vehicle

  2019cited by this paper
• Generating Adversarial Driving Scenarios in High-Fidelity Simulators

  2019cited by this paper
• Towards Corner Case Detection for Autonomous Driving

  2019influential reference
• Liability, Ethics, and Culture-Aware Behavior Specification using Rulebooks

  2019cited by this paper
• How Many Operational Design Domains, Objects, and Events?

  2019cited by this paper
• Credible Autonomy Safety Argumentation

  2019cited by this paper
• Algorithms for Optimization

  2019cited by this paper
• Fitness Functions for Testing Automated and Autonomous Driving Systems

  2019influential reference
• Application of Collision Probability Estimation to Calibration of Advanced Driver Assistance Systems

  2019cited by this paper
• Recent Progress on Generative Adversarial Networks (GANs): A Survey

  2019cited by this paper
• Parallel testing of vehicle intelligence via virtual-real interaction

  2019cited by this paper
• Generating Traffic Safety Test Scenarios for Automated Vehicles using a Big Data Technique

  2019cited by this paper
• A Test Scenario Automatic Generation Strategy for Intelligent Driving Systems

  2019cited by this paper
• Driver Behavior Classification under Cut-In Scenarios Using Support Vector Machine Based on Naturalistic Driving Data

  2019cited by this paper
• Testing Scenario Library Generation for Connected and Automated Vehicles, Part I: Methodology

  2019influential reference
• Testing Scenario Library Generation for Connected and Automated Vehicles, Part II: Case Studies

  2019influential reference
• Automatically Reconstructing Car Crashes from Police Reports for Testing Self-Driving Cars

  2019cited by this paper
• Towards Certification of Autonomous Driving: Systematic Test Case Generation for a Comprehensive but Economically-Feasible Assessment of Lane Keeping Assist Algorithms

  2019cited by this paper
• On the Industrial Application of Combinatorial Testing for Autonomous Driving Functions

  2019cited by this paper
• ML-Based Fault Injection for Autonomous Vehicles: A Case for Bayesian Fault Injection

  2019influential reference
• Performance Boundary Identification for the Evaluation of Automated Vehicles using Gaussian Process Classification

  2019influential reference
• Automatically testing self-driving cars with search-based procedural content generation

  2019cited by this paper
• Efficient Autonomy Validation in Simulation with Adaptive Stress Testing

  2019influential reference
• Ponn 1 AN OPTIMIZATION-BASED METHOD TO IDENTIFY RELEVANT SCENARIOS FOR TYPE APPROVAL OF AUTOMATED VEHICLES

  2019cited by this paper
• Requirements-Driven Test Generation for Autonomous Vehicles With Machine Learning Components

  2019influential reference
• Adaptive Stress Testing with Reward Augmentation for Autonomous Vehicle Validatio

  2019influential reference
• Quantifying Vision Zero: Crash avoidance in rural and motorway accident scenarios by combination of ACC, AEB, and LKS projected to German accident occurrence

  2019influential reference
• A framework for definition of logical scenarios for safety assurance of automated driving

  2019influential reference
• Functional decomposition—A contribution to overcome the parameter space explosion during validation of highly automated driving

  2019influential reference
• Generating effective test cases for self-driving cars from police reports

  2019cited by this paper
• Computationally Efficient Safety Falsification of Adaptive Cruise Control Systems

  2019influential reference
• A clustering approach to developing car-to-two-wheeler test scenarios for the assessment of Automated Emergency Braking in China using in-depth Chinese crash data.

  2019influential reference
• Beyond-Design-Basis Evaluation of Advanced Driver Assistance Systems

  2019cited by this paper
• Generating Critical Test Scenarios for Automated Vehicles with Evolutionary Algorithms

  2019influential reference
• Formalisation and algorithmic approach to the automated driving validation problem

  2019cited by this paper
• Genetic Algorithm-Based Test Parameter Optimization for ADAS System Testing

  2019cited by this paper
• Performance Comparison of Two Search-Based Testing Strategies for ADAS System Validation

  2019cited by this paper
• Rapidly-exploring Random Trees for Testing Automated Vehicles

  2019influential reference
• Automatic Testing and Falsification with Dynamically Constrained Reinforcement Learning

  2019cited by this paper
• Towards Vision Zero: Addressing White Spots by Accident Data based ADAS Design and Evaluation

  2019influential reference
• A Risk-index based Sampling Method to Generate Scenarios for the Evaluation of Automated Driving Vehicle Safety*

  2019influential reference
• Identification of Test Cases for Automated Driving Systems Using Bayesian Optimization

  2019cited by this paper
• An Agency-Directed Approach to Test Generation for Simulation-based Autonomous Vehicle Verification

  2019cited by this paper
• Experience Paper: Search-Based Testing in Automated Driving Control Applications

  2019cited by this paper
• Optimizing coverage of simulated driving scenarios for the autonomous vehicle

  2019cited by this paper
• An Automated Learning-Based Procedure for Large-scale Vehicle Dynamics Modeling on Baidu Apollo Platform

  2019cited by this paper
• Finding Diverse Failure Scenarios in Autonomous Systems Using Adaptive Stress Testing

  2019cited by this paper
• Virtual Assessment of Automated Driving: Methodology, Challenges, and Lessons Learned

  2019cited by this paper
• Microscopic and Macroscopic Risk Metrics for the Safety Validation of Automated Driving

  2019cited by this paper
• Differential Adaptive Stress Testing of Airborne Collision Avoidance Systems

  2018cited by this paper
• Automatic Generation of Safety-Critical Test Scenarios for Collision Avoidance of Road Vehicles

  2018influential reference
• Using Time-to-React based on Naturalistic Traffic Object Behavior for Scenario-Based Risk Assessment of Automated Driving

  2018cited by this paper
• Rule‐based searching for collision test cases of autonomous vehicles simulation

  2018cited by this paper
• Artificial intelligence test: a case study of intelligent vehicles

  2018cited by this paper
• Test Scenario Design for Intelligent Driving System Ensuring Coverage and Effectiveness

  2018cited by this paper
• PEGASUS—First Steps for the Safe Introduction of Automated Driving

  2018cited by this paper
• Adaptive Stress Testing for Autonomous Vehicles

  2018influential reference
• A Framework for Identifying and Simulating Worst-Case Animal-Vehicle Interactions

  2018cited by this paper
• Architecting Safety Supervisors for High Levels of Automated Driving

  2018cited by this paper
• DeepRoad: GAN-Based Metamorphic Testing and Input Validation Framework for Autonomous Driving Systems

  2018influential reference
• Safety Verification Of ADAS By Collision-free Boundary Searching Of A Parameterized Catalog

  2018cited by this paper
• Using Control Synthesis to Generate Corner Cases: A Case Study on Autonomous Driving

  2018influential reference
• Intelligent corner synthesis via cycle-consistent generative adversarial networks for efficient validation of autonomous driving systems

  2018influential reference
• A Tutorial on Bayesian Optimization

  2018cited by this paper
• Explainable artificial intelligence: A survey

  2018cited by this paper
• Controller Synthesis Made Real: Reach-Avoid Specifications and Linear Dynamics

  2018cited by this paper
• Autoware on Board: Enabling Autonomous Vehicles with Embedded Systems

  2018cited by this paper
• Scenarios for Development, Test and Validation of Automated Vehicles

  2018influential reference
• A Graded Offline Evaluation Framework for Intelligent Vehicle’s Cognitive Ability

  2018cited by this paper
• Application Oriented Testcase Generation for Validation of Environment Perception Sensor in Automated Driving Systems

  2018cited by this paper
• Traffic Sensory Data Classification by Quantifying Scenario Complexity

  2018influential reference
• Testing Vision-Based Control Systems Using Learnable Evolutionary Algorithms

  2018cited by this paper
• Gaining Knowledge on Automated Driving’s Safety—The Risk-Free VAAFO Tool

  2018influential reference
• Adaptive generation of challenging scenarios for testing and evaluation of autonomous vehicles

  2018influential reference
• Criticality Metric for the Safety Validation of Automated Driving using Model Predictive Trajectory Optimization

  2018cited by this paper
• Accelerated Evaluation of Autonomous Vehicles in the Lane Change Scenario Based on Subset Simulation Technique

  2018cited by this paper
• Study on the Controlled Field Test Scenarios of Automated Vehicles

  2018cited by this paper
• Towards Criticality Characterization of Situational Space

  2018influential reference
• Accident Scenario Generation with Recurrent Neural Networks

  2018influential reference
• Using Ontologies for Test Suites Generation for Automated and Autonomous Driving Functions

  2018cited by this paper

• Resilience-by-Design: Extracting Resilience Requirements Using the Resilience Graph in the Automotive Concept Phase

  2026influential citation
• Trustworthy Compiler Design for Generating Concrete Scenarios Toward Certifying Autonomous Driving Safety

  2025cites this paper
• AMACollision: An advanced framework for testing autonomous vehicles based on adversarial multi-agent

  2025cites this paper
• An Evolving Scenario Generation Method based on Dual-modal Driver Model Trained by Multi-Agent Reinforcement Learning

  2025cites this paper
• Optimizing Scenario-Based Testing in ADS: Parameter Selection and Sensitivity Analysis Based Parameter Elimination

  2025cites this paper
• Bayesian Optimization applied for accelerated Virtual Validation of the Autonomous Driving Function

  2025cites this paper
• Search-to-Crash: Generating safety-critical scenarios from in-depth crash data for testing autonomous vehicles

  2025cites this paper
• High-Risk Trajectories Generation for Safety Testing of Autonomous Vehicles Based on In-Depth Crash Data

  2025cites this paper
• Fuzzy Knowledge-Driven Human-in-the-Loop Learning for Autonomous Driving

  2025cites this paper
• Driving Risk Assessment for Intelligent Vehicles Based on Entropy-Informed Graph Neural Networks and Gaussian Distributions

  2025cites this paper
• Challenges and Recommendations Using Surrogates for Safety Impact Assessment: Correcting Misestimations

  2025cites this paper
• Evaluation of Different Generative Models to Support the Validation of Advanced Driver Assistance Systems

  2025cites this paper
• A Survey on the Application of Large Language Models in Scenario-Based Testing of Automated Driving Systems

  2025cites this paper
• Challenges and Advances in Scenario-Based Validation for Automated Driving Systems

  2025cites this paper
• Validation of a Critical Driving Scenario Identification Algorithm for Automated Driving Functions Using Real and Synthetic Object Lists

  2025cites this paper
• Navigating the Landscape of Operational Design Domains: A Comprehensive Mapping Study

  2025cites this paper
• A Generative Digital Twin for Continually Enhancing the Intended Functional Safety of Cyber–Physical Systems

  2025cites this paper
• Multidimensional Evaluation of Autonomous Driving Test Scenarios Based on AHP-EWN-TOPSIS Models

  2025cites this paper
• A Tool for Benchmarking Large Language Models' Robustness in Assessing the Realism of Driving Scenarios

  2025cites this paper
• Complexity learning-driven route planning accelerating automated electric truck deployments in semi-open port logistics

  2025cites this paper
• Impact of Adverse Weather on Road Safety: A Survey of Test Methods for Enhancing Safety of Automated Vehicles and Sensor Robustness in Challenging Environmental Conditions

  2025cites this paper
• Unifying expert knowledge and field data toward an enhanced scenario description for CAV certification: a comprehensive scenario-based approach

  2025cites this paper
• Multimodal Large Language Model Framework for Safe and Interpretable Grid-Integrated EVs

  2025cites this paper
• Safety behavior abstraction and model evolution in autonomous driving

  2025cites this paper
• Increasing Safety of Vulnerable Road Users in Scenarios With Occlusion: A Collaborative Approach for Smart Infrastructures and Automated Vehicles

  2025cites this paper
• Software testing for extended reality applications: a systematic mapping study

  2025cites this paper
• Testing the Fault-Tolerance of Multi-sensor Fusion Perception in Autonomous Driving Systems

  2025cites this paper
• Specific Scenario Generation Method for Trustworthiness Testing of Autonomous Vehicles Based on Interaction Coding

  2025cites this paper
• Make Full Use of Testing Information: An Integrated Accelerated Testing and Evaluation Method for Autonomous Driving Systems

  2025cites this paper
• The Road to Safe Automated Driving Systems: A Review of Methods Providing Safety Evidence

  2025cites this paper
• INSIGHT: Enhancing Autonomous Driving Safety through Vision-Language Models on Context-Aware Hazard Detection and Edge Case Evaluation

  2025cites this paper
• Combinatorial-Testing-Based Multi-Ship Encounter Scenario Generation for Collision Avoidance Algorithm Evaluation

  2025cites this paper
• A Novel Methodology for Planning Urban Road Safety Interventions

  2025cites this paper
• Multi-Objective Reinforcement Learning for Critical Scenario Generation of Autonomous Vehicles

  2025cites this paper
• Navigating the Edge with the State-of-the-Art Insights into Corner Case Identification and Generation for Enhanced Autonomous Vehicle Safety

  2025cites this paper
• A Systematic Literature Review on Safety of the Intended Functionality for Automated Driving Systems

  2025cites this paper
• A review of scenario cases for autonomous transportation system: Insights from CAV safety testing and scenario generation.

  2025cites this paper
• Trajectory Planning for Autonomous Cars in Low-Structured and Unstructured Environments: A Systematic Review

  2025cites this paper
• Risk Assessment Method for Autonomous Vehicles Violating Safety Common Sense Based on Driving Behavior

  2025cites this paper
• ATOS: An Open-Source Platform for Testing AI-Based Automated Vehicle Systems in Integrated Simulated and Physical Scenarios

  2025cites this paper
• Synthetic versus real: an analysis of critical scenarios for autonomous vehicle testing

  2025cites this paper
• Structuring Automotive Data for Systems Engineering: A Taxonomy-Based Approach

  2025cites this paper
• State-of-the-Art of Ontology-based Testing

  2025cites this paper
• Ontology-based Testing Revisited

  2025cites this paper
• Advancing autonomous driving system testing: Demands, challenges, and future directions

  2025cites this paper
• Lightweight Attention-based Temporal Modeling of Key Facial Features for Driver Fatigue Detection in Intelligent Driver Monitoring Systems

  2025cites this paper
• Reducing the Number of Scenarios in Automated Driving Systems: Methods and Strategies

  2024cites this paper
• Cone Slalom With Automated Sports Car–Trajectory Planning Algorithm

  2024cites this paper
• Causal Models to Support Scenario-Based Testing of ADAS

  2024cites this paper
• Test Scenario Fusion: How to Fuse Scenarios From Accident and Traffic Observation Data

  2024cites this paper
• Task-Driven Controllable Scenario Generation Framework Based on AOG

  2024cites this paper
• Few-Shot Scenario Testing for Autonomous Vehicles Based on Neighborhood Coverage and Similarity

  2024cites this paper
• A novel framework for adaptive stress testing of autonomous vehicles in highways

  2024cites this paper
• Development and classification of autonomous vehicle's ambiguous driving scenario.

  2024cites this paper
• Reality Bites: Assessing the Realism of Driving Scenarios with Large Language Models

  2024cites this paper
• Categorizing Data-Driven Methods for Test Scenario Generation to Assess Automated Driving Systems

  2024influential citation
• Evaluation method with digital expert on the criticality of car-following scenarios for autonomous vehicles testing

  2024cites this paper
• A Concept for Semi-Automatic Configuration of Sufficiently Valid Simulation Setups for Automated Driving Systems

  2024cites this paper
• LLMScenario: Large Language Model Driven Scenario Generation

  2024cites this paper
• Efficient Generation of Safety-Critical Scenarios Combining Dynamic and Static Scenario Parameters

  2024cites this paper
• Development and validation of automated driving behavior questionnaire (ADBQ)

  2024cites this paper
• Digital Twin Test Method With LTE-V2X for Autonomous Vehicle Safety Test

  2024cites this paper
• IAE-KM3D A 3D Object Detection Method Based on an Improved KM3D Network

  2024cites this paper
• Search-Based DNN Testing and Retraining With GAN-Enhanced Simulations

  2024cites this paper
• Model-Based Generation of Representative Rear-End Crash Scenarios Across the Full Severity Range Using Pre-Crash Data

  2024cites this paper
• Behavior Forests: Real-Time Discovery of Dynamic Behavior for Data Selection

  2024cites this paper
• Assessing Unknown Hazards for SOTIF Based on Twin Scenarios Empowered Autonomous Driving

  2024cites this paper
• Augmenting Scenario Description Languages for Intelligence Testing of Automated Driving Systems

  2024cites this paper
• Generation of Ego-Liable Hazardous-Test-Cases for Validating Automated Driving Systems in Junction-Scenes

  2024cites this paper
• Formulating a dissimilarity metric for comparison of driving scenarios for Automated Driving Systems

  2024cites this paper
• Importance Sampling-Guided Meta-Training for Intelligent Agents in Highly Interactive Environments

  2024cites this paper
• CommonUppRoad: A Framework of Formal Modelling, Verifying, Learning, and Visualisation of Autonomous Vehicles

  2024cites this paper
• A Testing and Evaluation Method for the Car-Following Models of Automated Vehicles Based on Driving Simulator

  2024cites this paper
• MACS: Multi-Agent Adversarial Reinforcement Learning for Finding Diverse Critical Driving Scenarios

  2024cites this paper
• SEAL: Towards Safe Autonomous Driving via Skill-Enabled Adversary Learning for Closed-Loop Scenario Generation

  2024cites this paper
• Summary of “An empirical comparison of combinatorial testing and search-based testing in the context of automated and autonomous driving systems”

  2024cites this paper
• Testing cyber-physical systems with explicit output coverage

  2024cites this paper
• A novel framework for adaptive stress testing of autonomous vehicles in multi-lane roads

  2024cites this paper
• Few-Shot Testing of Autonomous Vehicles With Scenario Similarity Learning

  2024cites this paper
• Utilizing Genetic Algorithms for Generating Critical Scenarios for Testing Autonomous Driving Functions

  2024cites this paper
• The effect of dual training on the hazard response and attention allocation of novice drivers when driving with advanced driver assistance system.

  2024cites this paper
• A Systematic Review of Edge Case Detection in Automated Driving: Methods, Challenges and Future Directions

  2024cites this paper
• A survey of autonomous driving frameworks and simulators

  2024cites this paper
• Study of Contactless Computer Vision-Based Road Condition Estimation Methods Within the Framework of an Operational Design Domain Monitoring System

  2024cites this paper
• Foundation Models for Rapid Autonomy Validation

  2024cites this paper
• Model Checking for Reinforcement Learning in Autonomous Driving: One Can Do More Than You Think!

  2024cites this paper
• Detection and analysis of corner case scenarios at a signalized urban intersection.

  2024cites this paper
• Comparison of System Theoretic Process Analysis and Cause Tree Analysis Applied on an Autonomous Parking System from Safety of the Intended Functionality Perspective

  2024cites this paper
• Model vs system level testing of autonomous driving systems: a replication and extension study

  2023cites this paper
• SAB: Stacking Action Blocks for Efficiently Generating Diverse Multimodal Critical Driving Scenario

  2023cites this paper
• Specification-Guided Critical Scenario Identification for Automated Driving

  2023influential citation
• Overview of the Operational Design Domain Monitoring for Safe Intelligent Vehicle Navigation

  2023cites this paper
• Oriented Bounding Box Detection Robust to Vehicle Shape on Road Under Real-Time Constraints

  2023cites this paper
• Continual Reinforcement Learning for Autonomous Driving with Application on Velocity Control under Various Environment

  2023cites this paper
• Using Data Abstraction for Clustering in the Context of Test Case Generation

  2023cites this paper
• Vectorized Scenario Description and Motion Prediction for Scenario-Based Testing

  2023cites this paper
• EpiTESTER: Testing Autonomous Vehicles With Epigenetic Algorithm and Attention Mechanism

  2023cites this paper
• TARGET: Traffic Rule-Based Test Generation for Autonomous Driving via Validated LLM-Guided Knowledge Extraction

  2023cites this paper
• Pretrain, Prompt, and Transfer: Evolving Digital Twins for Time-to-Event Analysis in Cyber-Physical Systems

  2023cites this paper
• A Systematic Approach Towards the Definition of the Terms Edge Case and Corner Case for Automated Driving

  2023cites this paper