Challenges in Designing Datasets and Validation for Autonomous Driving

Autonomous driving is getting a lot of attention in the last decade and will be the hot topic at least until the first successful certification of a car with Level 5 autonomy. There are many public datasets in the academic community. However, they are far away from what a robust industrial production system needs. There is a large gap between academic and industrial setting and a substantial way from a research prototype, built on public datasets, to a deployable solution which is a challenging task. In this paper, we focus on bad practices that often happen in the autonomous driving from an industrial deployment perspective. Data design deserves at least the same amount of attention as the model design. There is very little attention paid to these issues in the scientific community, and we hope this paper encourages better formalization of dataset design. More specifically, we focus on the datasets design and validation scheme for autonomous driving, where we would like to highlight the common problems, wrong assumptions, and steps towards avoiding them, as well as some open problems.

• Publication year

  2019

• Venue

  VISIGRAPP

• Publication date

  2019-01-26

• Fields of study

  Mathematics, Computer Science, Engineering

• Identifiers
  DOI 10.5220/0007690706530659arXiv 1901.09270
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Undoing the Damage

  2019cited by this paper
• The ApolloScape Dataset for Autonomous Driving

  2018cited by this paper
• Everybody Dance Now

  2018cited by this paper
• Driving Policy Transfer via Modularity and Abstraction

  2018cited by this paper
• CARLA: An Open Urban Driving Simulator

  2017cited by this paper
• DeepXplore: Automated Whitebox Testing of Deep Learning Systems

  2017cited by this paper
• The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes

  2017cited by this paper
• CyCADA: Cycle-Consistent Adversarial Domain Adaptation

  2017cited by this paper
• Negative results in computer vision: A perspective

  2017cited by this paper
• DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars

  2017cited by this paper
• Understanding deep learning requires rethinking generalization

  2016cited by this paper
• VirtualWorlds as Proxy for Multi-object Tracking Analysis

  2016influential reference
• The Cityscapes Dataset for Semantic Urban Scene Understanding

  2016cited by this paper
• The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes

  2016influential reference
• The Cityscapes Dataset

  2015cited by this paper
• ImageNet Large Scale Visual Recognition Challenge

  2014cited by this paper
• ImageNet classification with deep convolutional neural networks

  2012influential reference
• Undoing the Damage of Dataset Bias

  2012cited by this paper
• Object Detection with Discriminatively Trained Part Based Models

  2010influential reference
• Segmentation and Recognition Using Structure from Motion Point Clouds

  2008cited by this paper

• Let’s Go Bananas: Beyond Bounding Box Representations for Fisheye Camera-Based Object Detection in Autonomous Driving

  2025cites this paper
• SimBEV: A Synthetic Multi-Task Multi-Sensor Driving Data Generation Tool and Dataset

  2025cites this paper
• Dataset Safety in Autonomous Driving: Requirements, Risks, and Assurance

  2025cites this paper
• DaF-BEVSeg: Distortion-aware Fisheye Camera based Bird's Eye View Segmentation with Occlusion Reasoning

  2024cites this paper
• An empirical investigation of challenges of specifying training data and runtime monitors for critical software with machine learning and their relation to architectural decisions

  2024cites this paper
• Automated and Connected Driving: State-of-the-Art and Implications for Future Scenario Analysis

  2024cites this paper
• FisheyeDetNet: 360{\deg} Surround view Fisheye Camera based Object Detection System for Autonomous Driving

  2024cites this paper
• Modified RRT* for Path Planning in Autonomous Driving

  2024cites this paper
• Neural Rendering based Urban Scene Reconstruction for Autonomous Driving

  2024cites this paper
• Navya3DSeg - Navya 3D Semantic Segmentation Dataset Design & Split Generation for Autonomous Vehicles

  2023cites this paper
• WoodScape Motion Segmentation for Autonomous Driving - CVPR 2023 OmniCV Workshop Challenge

  2023cites this paper
• Multi-camera Bird's Eye View Perception for Autonomous Driving

  2023cites this paper
• LiDAR-BEVMTN: Real-Time LiDAR Bird’s-Eye View Multi-Task Perception Network for Autonomous Driving

  2023cites this paper
• An investigation of challenges encountered when specifying training data and runtime monitors for safety critical ML applications

  2023cites this paper
• Evaluating the effect of data augmentation and BALD heuristics on distillation of Semantic-KITTI dataset

  2023cites this paper
• Monocular three-dimensional object detection using data augmentation and self-supervised learning in autonomous driving

  2022cites this paper
• Efficient Characterization Method for Big Automotive Datasets used for Perception System Development and Verification

  2022cites this paper
• Evaluation Methodology for Object Detection and Tracking in Bounding Box Based Perception Modules

  2022cites this paper
• Surround-View Fisheye Camera Perception for Automated Driving: Overview, Survey & Challenges

  2022cites this paper
• Woodscape Fisheye Object Detection for Autonomous Driving - CVPR 2022 OmniCV Workshop Challenge

  2022cites this paper
• Let’s Get Dirty: GAN Based Data Augmentation for Camera Lens Soiling Detection in Autonomous Driving

  2021cites this paper
• Phenomenological Modelling of Camera Performance for Road Marking Detection

  2021cites this paper
• Near-Field Perception for Low-Speed Vehicle Automation Using Surround-View Fisheye Cameras

  2021cites this paper
• Ensemble-Based Semi-Supervised Learning to Improve Noisy Soiling Annotations in Autonomous Driving

  2021cites this paper
• Pseudo-Label Ensemble-based Semi-supervised Learning for Handling Noisy Soiling Segmentation Annotations

  2021cites this paper
• Weather and Light Level Classification for Autonomous Driving: Dataset, Baseline and Active Learning

  2021cites this paper
• Near-field Sensing Architecture for Low-Speed Vehicle Automation using a Surround-view Fisheye Camera System

  2021cites this paper
• TiledSoilingNet: Tile-level Soiling Detection on Automotive Surround-view Cameras Using Coverage Metric

  2020cites this paper
• Securing Connected & Autonomous Vehicles: Challenges Posed by Adversarial Machine Learning and the Way Forward

  2019cites this paper
• Let's Get Dirty: GAN Based Data Augmentation for Soiling and Adverse Weather Classification in Autonomous Driving

  2019cites this paper
• DeepTrailerAssist: Deep Learning Based Trailer Detection, Tracking and Articulation Angle Estimation on Automotive Rear-View Camera

  2019cites this paper
• WoodScape: A Multi-Task, Multi-Camera Fisheye Dataset for Autonomous Driving

  2019cites this paper
• Integration IMVIP 2019 : Irish Machine Vision and Image Processing 2019 FisheyeMultiNet : Real-time Multitask Learning Architecture for Surround-view Automated Parking System

  2019cites this paper
• FisheyeMultiNet: Real-time Multi-task Learning Architecture for Surround-view Automated Parking System

  2019cites this paper
• FisheyeMODNet: Moving Object detection on Surround-view Cameras for Autonomous Driving

  2019cites this paper
• FisheyeDetNet: Object Detection on Fisheye Surround View Camera Systems for Automated Driving

  year unknowncites this paper