Real-time Multi-view Omnidirectional Depth Estimation for Real Scenarios based on Teacher-Student Learning with Unlabeled Data

Omnidirectional depth estimation enables efficient 3D perception over a full 360-degree range. However, in real-world applications such as autonomous driving and robotics, achieving real-time performance and robust cross-scene generalization remains a significant challenge for existing algorithms. In this paper, we propose a real-time omnidirectional depth estimation method for edge computing platforms named Rt-OmniMVS, which introduces the Combined Spherical Sweeping method and implements the lightweight network structure to achieve real-time performance on edge computing platforms. To achieve high accuracy, robustness, and generalization in real-world environments, we introduce a teacher-student learning strategy. We leverage the high-precision stereo matching method as the teacher model to predict pseudo labels for unlabeled real-world data, and utilize data and model augmentation techniques for training to enhance performance of the student model Rt-OmniMVS. We also propose HexaMODE, an omnidirectional depth sensing system based on multi-view fisheye cameras and edge computation device. A large-scale hybrid dataset contains both unlabeled real-world data and synthetic data is collected for model training. Experiments on public datasets demonstrate that proposed method achieves results comparable to state-of-the-art approaches while consuming significantly less resource. The proposed system and algorithm also demonstrate high accuracy in various complex real-world scenarios, both indoors and outdoors, achieving an inference speed of 15 frames per second on edge computing platforms.

• Publication year

  2024

• Venue

  Unknown venue

• Publication date

  2024-09-12

• Fields of study

  Computer Science, Engineering

• Identifiers
  arXiv 2409.07843
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• OmniStereo: Real-Time Omnidireactional Depth Estimation with Multiview Fisheye Cameras

  2025cited by this paper
• Deep Stereo Network With MRF-Based Cost Aggregation

  2024cited by this paper
• RomniStereo: Recurrent Omnidirectional Stereo Matching

  2024influential reference
• CasOmniMVS: Cascade Omnidirectional Depth Estimation with Dynamic Spherical Sweeping

  2024cited by this paper
• OmniVidar: Omnidirectional Depth Estimation from Multi-Fisheye Images

  2023cited by this paper
• S-OmniMVS: Incorporating Sphere Geometry into Omnidirectional Stereo Matching

  2023cited by this paper
• Omnidirectional Depth Estimation With Hierarchical Deep Network for Multi-Fisheye Navigation Systems

  2023cited by this paper
• FastOmniMVS: Real-time Omnidirectional Depth Estimation from Multiview Fisheye Images

  2023cited by this paper
• Unsupervised OmniMVS: Efficient Omnidirectional Depth Inference via Establishing Pseudo-Stereo Supervision

  2023cited by this paper
• 360MVSNet: Deep Multi-view Stereo Network with 360° Images for Indoor Scene Reconstruction

  2023cited by this paper
• MODE: Multi-view Omnidirectional Depth Estimation with 360$^\circ $ Cameras

  2022cited by this paper
• OmniFusion: 360 Monocular Depth Estimation via Geometry-Aware Fusion

  2022cited by this paper
• BiFuse++: Self-Supervised and Efficient Bi-Projection Fusion for 360° Depth Estimation

  2022cited by this paper
• 360 Depth Estimation in the Wild - the Depth360 Dataset and the SegFuse Network

  2022cited by this paper
• Semi-Supervised 360° Depth Estimation from Multiple Fisheye Cameras with Pixel-Level Selective Loss

  2022cited by this paper
• Practical Stereo Matching via Cascaded Recurrent Network with Adaptive Correlation

  2022influential reference
• Attention Concatenation Volume for Accurate and Efficient Stereo Matching

  2022cited by this paper
• RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching

  2021cited by this paper
• UniFuse: Unidirectional Fusion for 360° Panorama Depth Estimation

  2021cited by this paper
• CFNet: Cascade and Fused Cost Volume for Robust Stereo Matching

  2021cited by this paper
• Real-Time Sphere Sweeping Stereo from Multiview Fisheye Images

  2021cited by this paper
• Omnidirectional stereo depth estimation based on spherical deep network

  2021cited by this paper
• 360° Depth Estimation from Multiple Fisheye Images with Origami Crown Representation of Icosahedron

  2020cited by this paper
• BiFuse: Monocular 360 Depth Estimation via Bi-Projection Fusion

  2020cited by this paper
• End-to-End Learning for Omnidirectional Stereo Matching With Uncertainty Prior

  2020influential reference
• AANet: Adaptive Aggregation Network for Efficient Stereo Matching

  2020cited by this paper
• OmniMVS: End-to-End Learning for Omnidirectional Stereo Matching

  2019influential reference
• Spherical View Synthesis for Self-Supervised 360° Depth Estimation

  2019cited by this paper
• 360SD-Net: 360° Stereo Depth Estimation with Learnable Cost Volume

  2019cited by this paper
• SweepNet: Wide-baseline Omnidirectional Depth Estimation

  2019influential reference
• EdgeStereo: An Effective Multi-task Learning Network for Stereo Matching and Edge Detection

  2019cited by this paper
• Pyramid Stereo Matching Network

  2018cited by this paper
• Self-Supervised Learning of Depth and Camera Motion from 360° Videos

  2018cited by this paper
• Fast R-CNN

  2015cited by this paper
• Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches

  2015cited by this paper
• A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation

  2015cited by this paper
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network

  2014cited by this paper
• Pulling Things out of Perspective

  2014cited by this paper

• No citing papers are available for this paper.