How Simulation Helps Autonomous Driving: A Survey of Sim2real, Digital Twins, and Parallel Intelligence

Developing autonomous driving technologies necessitates addressing safety and cost concerns. Both academic research and commercial applications of autonomous driving vehicles require extensive simulation and real-world testing. The challenge lies in effectively transferring driving knowledge from the virtual simulation world to the reality world, known as the reality gap (RG). This gap arises due to differences in lighting, textures, vehicle dynamics, and agents' behaviors between the two environments. To address this issue, researchers have explored three main approaches: sim2real, digital twins (DTs), and parallel intelligence (PI) technologies. This article reviews these solutions and examines their applications and innovations in autonomous driving. Furthermore, we delve into the state-of-the-art algorithms, models, and involved simulators, and discuss the developmental process from sim2real to DTs and PI. The presentation also sheds light on the challenges and future perspectives in the development of sim2real, DTs, and PI in the field of autonomous driving.

• Publication year

  2023

• Venue

  IEEE Transactions on Intelligent Vehicles

• Publication date

  2023-05-02

• Fields of study

  Computer Science, Engineering

• Identifiers
  DOI 10.1109/TIV.2023.3312777arXiv 2305.01263
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• A time-evolving digital twin tool for engineering dynamics applications

  2023cited by this paper
• Motion Planning for Autonomous Driving: The State of the Art and Future Perspectives

  2023cited by this paper
• ParallelEye Pipeline: An Effective Method to Synthesize Images for Improving the Visual Intelligence of Intelligent Vehicles

  2023cited by this paper
• DAO to HANOI via DeSci: AI Paradigm Shifts from AlphaGo to ChatGPT

  2023cited by this paper
• Hierarchical Interpretable Imitation Learning for End-to-End Autonomous Driving

  2023cited by this paper
• Robust Lane Change Decision Making for Autonomous Vehicles: An Observation Adversarial Reinforcement Learning Approach

  2023cited by this paper
• Robust Decision Making for Autonomous Vehicles at Highway On-Ramps: A Constrained Adversarial Reinforcement Learning Approach

  2023cited by this paper
• A Novel Scenarios Engineering Methodology for Foundation Models in Metaverse

  2023cited by this paper
• Milestones in Autonomous Driving and Intelligent Vehicles—Part I: Control, Computing System Design, Communication, HD Map, Testing, and Human Behaviors

  2023cited by this paper
• Parallel Learning: Overview and Perspective for Computational Learning Across Syn2Real and Sim2Real

  2023cited by this paper
• Milestones in Autonomous Driving and Intelligent Vehicles—Part II: Perception and Planning

  2023cited by this paper
• Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveys

  2023cited by this paper
• What Does ChatGPT Say: The DAO from Algorithmic Intelligence to Linguistic Intelligence

  2023cited by this paper
• Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training

  2022cited by this paper
• An integrated mixed reality system for safety-aware human-robot collaboration using deep learning and digital twin generation

  2022cited by this paper
• Study on UAV Parallel Planning System for Transmission Line Project Acceptance Under the Background of Industry 5.0

  2022cited by this paper
• The Role of Digital Twins in Connected and Automated Vehicles

  2022cited by this paper
• Reinforcement Learning from Simulation to Real World Autonomous Driving using Digital Twin

  2022cited by this paper
• Parallel Driving OS: A Ubiquitous Operating System for Autonomous Driving in CPSS

  2022cited by this paper
• Enabling Digital Twin in Vehicular Edge Computing: A Multi-Agent Multi-Objective Deep Reinforcement Learning Solution

  2022cited by this paper
• Self-Distillation for Robust LiDAR Semantic Segmentation in Autonomous Driving

  2022cited by this paper
• From Features Engineering to Scenarios Engineering for Trustworthy AI: I&I, C&C, and V&V

  2022cited by this paper
• Autonomous Vehicles Digital Twin: A Practical Paradigm for Autonomous Driving System Development

  2022cited by this paper
• Review and Perspectives on Driver Digital Twin and Its Enabling Technologies for Intelligent Vehicles

  2022cited by this paper
• Digital twin modeling

  2022cited by this paper
• Parallel Manufacturing for Industrial Metaverses: A New Paradigm in Smart Manufacturing

  2022cited by this paper
• Effects of Augmented-Reality-Based Assisting Interfaces on Drivers’ Object-wise Situational Awareness in Highly Autonomous Vehicles

  2022cited by this paper
• An End-to-End Curriculum Learning Approach for Autonomous Driving Scenarios

  2022cited by this paper
• Design and Implementation of Digital Twin-Assisted Simulation Method for Autonomous Vehicle in Car-Following Scenario

  2022cited by this paper
• PointDistiller: Structured Knowledge Distillation Towards Efficient and Compact 3D Detection

  2022cited by this paper
• Event-Triggered Near-Optimal Control for Unknown Discrete-Time Nonlinear Systems Using Parallel Control

  2022cited by this paper
• Parallel Intelligence in Metaverses: Welcome to Hanoi!

  2022cited by this paper
• Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Data

  2022cited by this paper
• A Mixed Reality Interface for a Digital Twin Based Crane

  2021cited by this paper
• A robust lateral tracking control strategy for autonomous driving vehicles

  2021cited by this paper
• Digital Twin-enabled Reinforcement Learning for End-to-end Autonomous Driving

  2021cited by this paper
• Automatic traffic modelling for creating digital twins to facilitate autonomous vehicle development

  2021cited by this paper
• Autonomous Driving Test Method Based on Digital Twin: A Survey

  2021cited by this paper
• Autonomous Overtaking in Gran Turismo Sport Using Curriculum Reinforcement Learning

  2021cited by this paper
• A Real-Time Distance Measurement System for a Digital Twin Using Mixed Reality Goggles

  2021cited by this paper
• Towards the future of smart electric vehicles: Digital twin technology

  2021cited by this paper
• Digital Twin Analysis to Promote Safety and Security in Autonomous Vehicles

  2021cited by this paper
• Learning a Deep Cascaded Neural Network for Multiple Motion Commands Prediction in Autonomous Driving

  2021cited by this paper
• Curriculum Learning for Vehicle Lateral Stability Estimations

  2021cited by this paper
• Deep Neural Network Based Vehicle and Pedestrian Detection for Autonomous Driving: A Survey

  2021cited by this paper
• Crossing the Reality Gap: a Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning

  2021cited by this paper
• Digital Twin: Origin to Future

  2021cited by this paper
• MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning

  2021cited by this paper
• High-Speed Collision Avoidance using Deep Reinforcement Learning and Domain Randomization for Autonomous Vehicles

  2020cited by this paper
• Learning Driving Models From Parallel End-to-End Driving Data Set

  2020cited by this paper
• A novel background subtraction algorithm based on parallel vision and Bayesian GANs

  2020cited by this paper
• A parallel vision approach to scene-specific pedestrian detection

  2020cited by this paper
• Augmented Reality-Based Advanced Driver-Assistance System for Connected Vehicles

  2020cited by this paper
• Enhanced Transfer Learning for Autonomous Driving with Systematic Accident Simulation

  2020cited by this paper
• Sensor Fusion of Camera and Cloud Digital Twin Information for Intelligent Vehicles

  2020cited by this paper
• Mixed reality test environment for autonomous cars using Unity 3D and SUMO

  2020cited by this paper
• LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving

  2020cited by this paper
• Augmented reality assisted calibration of digital twins of mobile robots

  2020cited by this paper
• System Design for a Driverless Autonomous Racing Vehicle

  2020cited by this paper
• Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey

  2020cited by this paper
• Principles of application of virtual and physical simulation technology in production of digital twin of active vehicle safety systems

  2020cited by this paper
• Positioning Accuracy Improvement of Automated Guided Vehicles Based on a Novel Magnetic Tracking Approach

  2020cited by this paper
• Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World

  2020cited by this paper
• Finite-time optimized robust control with adaptive state estimation algorithm for autonomous heavy vehicle

  2020cited by this paper
• Autonomous Vehicle Control based on HoloLens Technology and Raspberry Pi Platform: an Educational Perspective

  2019cited by this paper
• Parallel planning: a new motion planning framework for autonomous driving

  2019cited by this paper
• Parallel Motion Planning: Learning a Deep Planning Model against Emergencies

  2019cited by this paper
• Generalization through Simulation: Integrating Simulated and Real Data into Deep Reinforcement Learning for Vision-Based Autonomous Flight

  2019cited by this paper
• Parallel testing of vehicle intelligence via virtual-real interaction

  2019cited by this paper
• Risk Averse Robust Adversarial Reinforcement Learning

  2019cited by this paper
• Meta-Sim: Learning to Generate Synthetic Datasets

  2019cited by this paper
• Deep Integration: A Multi-Label Architecture for Road Scene Recognition

  2019cited by this paper
• Semantic Segmentation with Transfer Learning for Off-Road Autonomous Driving

  2019cited by this paper
• Real-World Robotic Perception and Control Using Synthetic Data

  2019cited by this paper
• ROADS: Randomization for Obstacle Avoidance and Driving in Simulation

  2019cited by this paper
• Distilling Knowledge From a Deep Pose Regressor Network

  2019cited by this paper
• 車両シミュレーションと実車検証を統合する デジタルツイン環境の構築 Digital Twin Environment to Integrate Vehicle Simulation and Physical Verification

  2019cited by this paper
• Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization Without Accessing Target Domain Data

  2019cited by this paper
• Digital Twins for Dependability Improvement of Autonomous Driving

  2019cited by this paper
• SUMMIT: A Simulator for Urban Driving in Massive Mixed Traffic

  2019cited by this paper
• Curriculum Self-Paced Learning for Cross-Domain Object Detection

  2019cited by this paper
• Multi-Vehicle Mixed-Reality Reinforcement Learning for Autonomous Multi-Lane Driving

  2019cited by this paper
• Digital twin for propulsion drive of autonomous electric vehicle

  2019cited by this paper
• Context-Aware Autonomous Driving Using Meta-Reinforcement Learning

  2019cited by this paper
• Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance?

  2019cited by this paper
• SAM: Squeeze-and-Mimic Networks for Conditional Visual Driving Policy Learning

  2019cited by this paper
• Dynamic path planning for autonomous driving on various roads with avoidance of static and moving obstacles

  2018cited by this paper
• Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning

  2018cited by this paper
• Vehicle-In-the-Loop Test Environment for Autonomous Driving with Microscopic Traffic Simulation

  2018cited by this paper
• Automatically Generated Curriculum based Reinforcement Learning for Autonomous Vehicles in Urban Environment

  2018cited by this paper
• Digital Behavioral Twins for Safe Connected Cars

  2018cited by this paper
• The ApolloScape Dataset for Autonomous Driving

  2018cited by this paper
• The sleepwalker framework: Verification and validation of autonomous vehicles by mixed reality LiDAR stimulation

  2018cited by this paper
• Regularized Evolution for Image Classifier Architecture Search

  2018cited by this paper
• Autoware on Board: Enabling Autonomous Vehicles with Embedded Systems

  2018cited by this paper
• Meta Reinforcement Learning with Latent Variable Gaussian Processes

  2018cited by this paper
• On modeling

  2018cited by this paper
• An Explanatory Windshield Display Interface with Augmented Reality Elements for Urban Autonomous Driving

  2018cited by this paper
• C2PS: A Digital Twin Architecture Reference Model for the Cloud-Based Cyber-Physical Systems

  2017cited by this paper
• Domain randomization for transferring deep neural networks from simulation to the real world

  2017cited by this paper

• Digital Twin Technology for Intelligent Vehicles and Transportation Systems: A Survey on Applications, Challenges and Future Directions

  2026cites this paper
• Learning to Tune Pure Pursuit in Autonomous Racing: Joint Lookahead and Steering-Gain Control With PPO

  2026cites this paper
• Developing AI Agents with Simulated Data: Why, what, and how?

  2026cites this paper
• Advancing Digital Twin Generation Through a Novel Simulation Framework and Quantitative Benchmarking

  2026cites this paper
• Motion Capture is Not the Target Domain: Scaling Synthetic Data for Learning Motion Representations

  2026cites this paper
• DrivIng: A Large-Scale Multimodal Driving Dataset with Full Digital Twin Integration

  2026cites this paper
• Long- and Short-Term Constraint-Driven Safe Reinforcement Learning for Autonomous Driving

  2026cites this paper
• Advancing intelligent transportation through digital twin: Challenges, models, and future prospects

  2026cites this paper
• CARLoS: Customizable Autonomy Research Low-Fidelity Simulator for Safety Testing of Agents

  2025cites this paper
• FalconGym: A Photorealistic Simulation Framework for Zero-Shot Sim-to-Real Vision-Based Quadrotor Navigation

  2025cites this paper
• A Performance Evaluation of Open Source Autonomous Driving Frameworks: Case Studies of Apollo and Autoware

  2025cites this paper
• A review of scenario cases for autonomous transportation system: Insights from CAV safety testing and scenario generation.

  2025cites this paper
• Hydra-NeXt: Robust Closed-Loop Driving with Open-Loop Training

  2025cites this paper
• Industrial-Grade Sensor Simulation via Gaussian Splatting: A Modular Framework for Scalable Editing and Full-Stack Validation

  2025cites this paper
• SimWorld: A Unified Benchmark for Simulator-Conditioned Scene Generation via World Model

  2025cites this paper
• Autonomous Vehicle for Industry 5.0: Digital Twin for System Safety Validation

  2025cites this paper
• TKSF-KAN: Transformer-enhanced oat yield modeling and transferability across major oat-producing regions in China using UAV multisource data

  2025cites this paper
• Digital Twin Technologies for Vehicular Prototyping: A Survey

  2025cites this paper
• Eclipse SUMO and Unity 3D integration for emission studies based on driving behaviour in virtual reality environment

  2025cites this paper
• On the Need for a Statistical Foundation in Scenario-Based Testing of Autonomous Vehicles

  2025cites this paper
• RAFT -- A Domain Adaptation Framework for RGB&LiDAR Semantic Segmentation

  2025cites this paper
• Predicting emergency decision-making in conditionally automated driving from personality traits and heart rate

  2025cites this paper
• AI Simulation by Digital Twins: Systematic Survey, Reference Framework, and Mapping to a Standardized Architecture

  2025influential citation
• $\mathcal{R}-\mathbf{CARLA}$: High-Fidelity Sensor Simulations with Interchangeable Dynamics for Autonomous Racing

  2025cites this paper
• Mixed Motivation Driven Social Multi-Agent Reinforcement Learning for Autonomous Driving

  2025cites this paper
• AI-driven visual monitoring of industrial assembly tasks

  2025cites this paper
• A Vehicle-in-the-Loop Simulator with AI-Powered Digital Twins for Testing Automated Driving Controllers

  2025cites this paper
• Research on Autonomous Vehicle Lane-Keeping and Navigation System Based on Deep Reinforcement Learning: From Simulation to Real-World Application

  2025cites this paper
• A Physics-Based Continuum Model for Versatile, Scalable, and Fast Terramechanics Simulation

  2025cites this paper
• A Digital Twin Approach to Autonomous Emergency Braking: HiL Testing with Raspberry Pi and Standardized Scenarios

  2025cites this paper
• Generative policy models for autonomous governance in edge AI

  2025cites this paper
• Bridging Simulation and Usability: A User-Friendly Framework for Scenario Generation in CARLA

  2025cites this paper
• The public perception and adaptability of laws and regulations of autonomous driving vehicles

  2025cites this paper
• A Digital Twin Approach for Perception Development and Validation in Autonomous Vehicles*

  2025cites this paper
• Engineering Automotive Digital Twins on Standardized Architectures: A Case Study

  2025cites this paper
• Real-Time Model Checking for Closed-Loop Robot Reactive Planning

  2025cites this paper
• RealMirror: A Comprehensive, Open-Source Vision-Language-Action Platform for Embodied AI

  2025cites this paper
• A Multi-Modality Evaluation of the Reality Gap in Autonomous Driving Systems

  2025cites this paper
• Seeing What Matters: Attentional (MIS-) Alignment between Humans and AI in VR-Simulated Prediction of Driving Accidents

  2025cites this paper
• World model-based reinforcement learning for autonomous ship safe collision avoidance

  2025cites this paper
• Enriched Pedestrian Crossing Prediction Using Carla Synthetic Data

  2025cites this paper
• Shaping the future of nuclear reactors with digital twins: Current developments and perspectives

  2025cites this paper
• VLN-ChEnv: Vision-language Navigation in Changeable Environments

  2025cites this paper
• Toward Intelligent AIoT: A Comprehensive Survey on Digital Twin and Multimodal Generative AI Integration

  2025cites this paper
• A Framework for Generating Synthetic Urban Mobility Datasets With Customizable Anomalous Scenarios

  2025cites this paper
• A review of reinforcement learning approaches for pursuit-evasion games

  2025cites this paper
• LiDAR-GS++:Improving LiDAR Gaussian Reconstruction via Diffusion Priors

  2025cites this paper
• Zero-Shot Sim-to-Real Visual Quadrotor Control with Hard Constraints

  2025cites this paper
• HABIT: Human Action Benchmark for Interactive Traffic in CARLA

  2025cites this paper
• Poster: Digital Twin-Assisted Closed Loop Reinforcement Learning for Autonomous Vehicle Lane Changing

  2025cites this paper
• Energy Conservation for Autonomous Agents Using Reinforcement Learning

  2025cites this paper
• Digital Twin-based Architecture and Design for Co-simulation of Omnidirectional Mobile Robots*

  2025cites this paper
• vEDGAR - Can CARLA Do HiL?

  2025cites this paper
• DTCCL: Disengagement-Triggered Contrastive Continual Learning for Autonomous Bus Planners

  2025cites this paper
• Enhancing Real-Time Decision-Making and Resilience in Cyber-Physical Systems with Digital Twins for Smart Manufacturing

  2025cites this paper
• BiguaSim: A Hybrid Multi-Domain Simulator for Robotics High-Fidelity Simulation and Synthetic Dataset Generation

  2025cites this paper
• Quantifying and Bridging the Fidelity Gap: A Decisive-Feature Approach to Comparing Synthetic and Real Imagery

  2025cites this paper
• Digital Twin Modeling Method Based on the Dynamic Characteristics of Mobile Robots

  2025cites this paper
• Safedrive dreamer: Navigating safety–critical scenarios in autonomous driving with world models

  2025cites this paper
• Toward Real-Time and Efficient Perception Workflows in Software-Defined Vehicles

  2025cites this paper
• Adaptive Digital Twin Migration in Vehicular Edge Computing and Networks

  2025cites this paper
• RALAD: Bridging the Real-to-Sim Domain Gap in Autonomous Driving with Retrieval-Augmented Learning

  2025cites this paper
• A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models

  2025cites this paper
• Synthetic Data for Enhancing Perception and Cooperative Localization in Autonomous Vehicles

  2025cites this paper
• Cutting-edge 3D reconstruction solutions for underwater coral reef images: A review and comparison

  2025influential citation
• Quantifying the Sim2Real Gap: Model-Based Verification and Validation in Autonomous Ground Systems

  2025cites this paper
• Navigating the Edge with the State-of-the-Art Insights into Corner Case Identification and Generation for Enhanced Autonomous Vehicle Safety

  2025cites this paper
• Learning-Based NMPC Adaptation for Autonomous Driving Using Parallelized Digital Twin

  2024cites this paper
• Are NeRFs ready for autonomous driving? Towards closing the real-to-simulation gap

  2024cites this paper
• Addressing Data Annotation Challenges in Multiple Sensors: A Solution for Scania Collected Datasets

  2024cites this paper
• Long and Short-Term Constraints Driven Safe Reinforcement Learning for Autonomous Driving

  2024cites this paper
• Wcdt: World-Centric Diffusion Transformer for Traffic Scene Generation

  2024cites this paper
• Exploring Generative AI for Sim2Real in Driving Data Synthesis

  2024cites this paper
• Planning the path with Reinforcement Learning: Optimal Robot Motion Planning in RoboCup Small Size League Environments

  2024cites this paper
• Recent advances in reinforcement learning-based autonomous driving behavior planning: A survey

  2024cites this paper
• Low Fidelity Digital Twin for Automated Driving Systems: Use Cases and Automatic Generation

  2024cites this paper
• OpenConvoy: Universal Platform for Real-World Testing of Cooperative Driving Systems

  2024cites this paper
• REvolve: Reward Evolution with Large Language Models for Autonomous Driving

  2024cites this paper
• DiffPoGAN: Diffusion Policies with Generative Adversarial Networks for Offline Reinforcement Learning

  2024cites this paper
• Behavioural Gap Assessment of Human-Vehicle Interaction in Real and Virtual Reality-Based Scenarios in Autonomous Driving

  2024cites this paper
• Learning Realistic and Reactive Traffic Agents

  2024cites this paper
• Decision Making for Autonomous Driving Stack: Shortening the Gap from Simulation to Real-World Implementations

  2024cites this paper
• Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey

  2024cites this paper
• GLANCE: Graph-based Learnable Digital Twin for Communication Networks

  2024cites this paper
• A Survey on Recent Advancements in Autonomous Driving Using Deep Reinforcement Learning: Applications, Challenges, and Solutions

  2024cites this paper
• TinyLidarNet: 2D LiDAR-based End-to-End Deep Learning Model for F1TENTH Autonomous Racing

  2024cites this paper
• Efficient Domain Augmentation for Autonomous Driving Testing Using Diffusion Models

  2024cites this paper
• Autonomous Driving at Unsignalized Intersections: A Review of Decision-Making Challenges and Reinforcement Learning-Based Solutions

  2024cites this paper
• A Sim-to-Real Vision-based Lane Keeping System for a 1:10-scale Autonomous Vehicle

  2024cites this paper
• A Systematic Review of Edge Case Detection in Automated Driving: Methods, Challenges and Future Directions

  2024cites this paper
• Augmented Intelligence in Smart Intersections: Local Digital Twins-Assisted Hybrid Autonomous Driving

  2024cites this paper
• Foundation Intelligence for Smart Infrastructure Services in Transportation 5.0

  2024cites this paper
• REvolve: Reward Evolution with Large Language Models using Human Feedback

  2024cites this paper
• AI Simulation by Digital Twins: Systematic Survey of the State of the Art and a Reference Framework

  2024cites this paper
• MetaSSC: Enhancing 3D semantic scene completion for autonomous driving through meta-learning and long-sequence modeling

  2024cites this paper
• Metaverse-Enabled Intelligence for Open-Terrain Field Vehicle Fleets: Leveraging Parallel Intelligence and Edge Computing

  2024cites this paper
• Accelerated Safety Testing for Highly Automated Vehicles: Application and Capability Comparison of Surrogate Models

  2024cites this paper
• Smart Mobility Digital Twin Based Automated Vehicle Navigation System: A Proof of Concept

  2024cites this paper
• Deep Reinforcement Learning-Based Off-Road Path Planning via Low-Dimensional Simulation

  2024cites this paper
• Quantifying the Sim2real Gap for GPS and IMU Sensors

  2024cites this paper