Universal Semantic Segmentation for Fisheye Urban Driving Images

Semantic segmentation is a critical method in the field of autonomous driving. When performing semantic image segmentation, a wider field of view (FoV) helps to obtain more information about the surrounding environment, making automatic driving safer and more reliable, which could be offered by fisheye cameras. However, large public fisheye datasets are not available, and the fisheye images captured by the fisheye camera with large FoV comes with large distortion, so commonly-used semantic segmentation model cannot be directly utilized. In this paper, a seven degrees of freedom (DoF) augmentation method is proposed to transform rectilinear image to fisheye image in a more comprehensive way. In the training process, rectilinear images are transformed into fisheye images in seven DoF, which simulates the fisheye images taken by cameras of different positions, orientations and focal lengths. The result shows that training with the seven-DoF augmentation can improve the model’s accuracy and robustness against different distorted fisheye data. This seven-DoF augmentation provides a universal semantic segmentation solution for fisheye cameras in different autonomous driving applications. Also, we provide specific parameter settings of the augmentation for autonomous driving. At last, we tested our universal semantic segmentation model on real fisheye images and obtained satisfactory results. The code and configurations are released at https://github.com/Yaozhuwa/FisheyeSeg.

• Publication year

  2020

• Venue

  IEEE International Conference on Systems, Man and Cybernetics

• Publication date

  2020-01-31

• Fields of study

  Mathematics, Computer Science, Engineering

• Identifiers
  DOI 10.1109/SMC42975.2020.9283099arXiv 2002.03736
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• The OmniScape Dataset

  2020cited by this paper
• Oriented Spatial Transformer Network for Pedestrian Detection Using Fish-Eye Camera

  2020cited by this paper
• PASS: Panoramic Annular Semantic Segmentation

  2020cited by this paper
• DS-PASS: Detail-Sensitive Panoramic Annular Semantic Segmentation through SwaftNet for Surrounding Sensing

  2019cited by this paper
• Real-Time Semantic Segmentation for Fisheye Urban Driving Images Based on ERFNet

  2019cited by this paper
• In Defense of Pre-Trained ImageNet Architectures for Real-Time Semantic Segmentation of Road-Driving Images

  2019cited by this paper
• WoodScape: A Multi-Task, Multi-Camera Fisheye Dataset for Autonomous Driving

  2019cited by this paper
• Can we PASS beyond the Field of View? Panoramic Annular Semantic Segmentation for Real-World Surrounding Perception

  2019cited by this paper
• RGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Driving

  2019cited by this paper
• Semantic Segmentation of Fisheye Images

  2018cited by this paper
• Fusion of millimeter wave radar and RGB-depth sensors for assisted navigation of the visually impaired

  2018cited by this paper
• The ApolloScape Dataset for Autonomous Driving

  2018cited by this paper
• VH-HFCN based Parking Slot and Lane Markings Segmentation on Panoramic Surround View

  2018cited by this paper
• Scene Understanding Networks for Autonomous Driving Based on Around View Monitoring System

  2018influential reference
• Restricted Deformable Convolution-Based Road Scene Semantic Segmentation Using Surround View Cameras

  2018cited by this paper
• Unifying terrain awareness through real-time semantic segmentation

  2018cited by this paper
• Understanding 3D Semantic Structure around the Vehicle with Monocular Cameras

  2018cited by this paper
• CNN-based Fisheye Image Real-Time Semantic Segmentation

  2018cited by this paper
• An Environmental Perception and Navigational Assistance System for Visually Impaired Persons Based on Semantic Stixels and Sound Interaction

  2018cited by this paper
• CNN based semantic segmentation for urban traffic scenes using fisheye camera

  2017cited by this paper
• Focal Loss for Dense Object Detection

  2017cited by this paper
• Deformable Convolutional Networks

  2017cited by this paper
• The Cityscapes Dataset for Semantic Urban Scene Understanding

  2016influential reference
• U-Net: Convolutional Networks for Biomedical Image Segmentation

  2015cited by this paper
• Fully convolutional networks for semantic segmentation

  2014cited by this paper
• International Conference on Robotics and Automation (ICRA)

  2008cited by this paper

• Unlocking Constraints: Source-Free Occlusion-Aware Seamless Segmentation

  2025cites this paper
• A High-Speed Omnidirectional Video-Shooting System for Automatic Multiobject Tracking

  2025cites this paper
• Mamba4PASS: Vision Mamba for PAnoramic Semantic Segmentation

  2025cites this paper
• A Comparison of Spherical Neural Networks for Surround-View Fisheye Image Semantic Segmentation

  2025influential citation
• Depth Any Camera: Zero-Shot Metric Depth Estimation from Any Camera

  2025cites this paper
• Data Augmentation is All You Need for Robust Fisheye Object Detection

  2025cites this paper
• Prompt-Based and Transformer-Based Models Evaluation for Semantic Segmentation of Crowdsourced Urban Imagery Under Projection and Geometric Symmetry Variations

  2025influential citation
• Exploring Surround-View Fisheye Camera 3D Object Detection

  2025cites this paper
• DarSwin-Unet: Distortion Aware Architecture

  2024cites this paper
• Deformable Convolution Based Road Scene Semantic Segmentation of Fisheye Images in Autonomous Driving

  2024cites this paper
• Occlusion-Aware Seamless Segmentation

  2024cites this paper
• Transparency Distortion Robustness for SOTA Image Segmentation Tasks

  2024cites this paper
• Adapting CNNs for Fisheye Cameras without Retraining

  2024cites this paper
• Location-Guided Head Pose Estimation for Fisheye Image

  2024cites this paper
• DarSwin-Unet: Distortion Aware Encoder-Decoder Architecture

  2024cites this paper
• AirEyeSeg: Teacher-Student Insights into Robust Fisheye UAV Detection

  2024cites this paper
• OARPD: occlusion-aware rotated people detection in overhead fisheye images

  2024cites this paper
• Overhead fisheye cameras for indoor monitoring: challenges and recent progress

  2024cites this paper
• FishSegSSL: A Semi-Supervised Semantic Segmentation Framework for Fish-Eye Images

  2024cites this paper
• SlaBins: Fisheye Depth Estimation using Slanted Bins on Road Environments

  2023cites this paper
• Full Semantic Constructed Network for Urban Use Classification From Very High-Resolution Optical Remote Sensing Imagery

  2023cites this paper
• F2BEV: Bird's Eye View Generation from Surround-View Fisheye Camera Images for Automated Driving

  2023cites this paper
• FishDreamer: Towards Fisheye Semantic Completion via Unified Image Outpainting and Segmentation

  2023influential citation
• Sector Patch Embedding: An Embedding Module Conforming to The Distortion Pattern of Fisheye Image

  2023cites this paper
• DarSwin: Distortion Aware Radial Swin Transformer

  2023cites this paper
• Surround-View Fisheye Camera Viewpoint Augmentation for Image Semantic Segmentation

  2023cites this paper
• Evaluation of the Human Body Mask Effects on GNSS Wearable Devices for Outdoor Pedestrian Navigation Using Fisheye Sky Views

  2023cites this paper
• ADD: An Automatic Desensitization Fisheye Dataset for Autonomous Driving

  2023cites this paper
• LineMarkNet: Line Landmark Detection for Valet Parking

  2023cites this paper
• Equirectangular image construction method for standard CNNs for Semantic Segmentation

  2023cites this paper
• Automatic segmentation of inconstant fractured fragments for tibia/fibula from CT images using deep learning

  2023cites this paper
• Efficient Edge Computing Device for Traffic Monitoring Using Deep Learning Detectors

  2023cites this paper
• Monocular Depth Estimation from a Fisheye Camera Based on Knowledge Distillation

  2023cites this paper
• CANet: cross attention network for food image segmentation

  2023cites this paper
• Research on Semantic Segmentation of Fish-Eye Images for Autonomous Driving

  2023cites this paper
• Semantic Segmentation Using Transfer Learning on Fisheye Images

  2023cites this paper
• Bending Reality: Distortion-aware Transformers for Adapting to Panoramic Semantic Segmentation

  2022cites this paper
• Fisheye object detection based on standard image datasets with 24-points regression strategy

  2022cites this paper
• Survey on Fish-Eye Cameras and Their Applications in Intelligent Vehicles

  2022cites this paper
• Unsupervised Domain Adaptation for Semantic Segmentation of Urban Street Scenes Reflected by Convex Mirrors

  2022cites this paper
• The Robust Semantic SLAM System for Texture-Less Underground Parking Lot

  2022cites this paper
• Surround-View Fisheye Camera Perception for Automated Driving: Overview, Survey & Challenges

  2022influential citation
• Review on Panoramic Imaging and Its Applications in Scene Understanding

  2022cites this paper
• FisheyeHDK: Hyperbolic Deformable Kernel Learning for Ultra-Wide Field-of-View Image Recognition

  2022cites this paper
• Stitching images from a conventional camera and a fisheye camera based on nonrigid warping

  2022cites this paper
• Adaptable Deformable Convolutions for Semantic Segmentation of Fisheye Images in Autonomous Driving Systems

  2021cites this paper
• Transfer Beyond the Field of View: Dense Panoramic Semantic Segmentation via Unsupervised Domain Adaptation

  2021cites this paper
• A Proposal to Integrate ORB-Slam Fisheye and Convolutional Neural Networks for Outdoor Terrestrial Mobile Mapping

  2021cites this paper
• DensePASS: Dense Panoramic Semantic Segmentation via Unsupervised Domain Adaptation with Attention-Augmented Context Exchange

  2021cites this paper
• Aerial-PASS: Panoramic Annular Scene Segmentation in Drone Videos

  2021cites this paper
• Capturing Omni-Range Context for Omnidirectional Segmentation

  2021cites this paper
• Omnisupervised Omnidirectional Semantic Segmentation

  2020cites this paper
• Lifting the Structural Morphing for Wide-Angle Images Rectification: Unified Content and Boundary Modeling

  year unknowncites this paper