Veritatem Dies Aperit - Temporally Consistent Depth Prediction Enabled by a Multi-Task Geometric and Semantic Scene Understanding Approach

Robust geometric and semantic scene understanding is ever more important in many real-world applications such as autonomous driving and robotic navigation. In this paper, we propose a multi-task learning-based approach capable of jointly performing geometric and semantic scene understanding, namely depth prediction (monocular depth estimation and depth completion) and semantic scene segmentation. Within a single temporally constrained recurrent network, our approach uniquely takes advantage of a complex series of skip connections, adversarial training and the temporal constraint of sequential frame recurrence to produce consistent depth and semantic class labels simultaneously. Extensive experimental evaluation demonstrates the efficacy of our approach compared to other contemporary state-of-the-art techniques.

• Publication year

  2019

• Venue

  Computer Vision and Pattern Recognition

• Publication date

  2019-03-26

• Fields of study

  Computer Science, Engineering

• Identifiers
  DOI 10.1109/CVPR.2019.00349arXiv 1903.10764
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Unsupervised Learning of Depth and Ego-Motion from Continuous Monocular Images

  2021influential reference
• Generative adversarial framework for depth filling via Wasserstein metric, cosine transform and domain transfer

  2019cited by this paper
• Learning a Discriminative Feature Network for Semantic Segmentation

  2018cited by this paper
• Perceptually aware image inpainting

  2018cited by this paper
• Extended Patch Prioritization for Depth Filling Within Constrained Exemplar-Based RGB-D Image Completion

  2018cited by this paper
• GENERATIVE ADVERSARIAL NETS

  2018cited by this paper
• Generative Image Inpainting with Contextual Attention

  2018influential reference
• Deep Depth Completion of a Single RGB-D Image

  2018cited by this paper
• Low-Latency Video Semantic Segmentation

  2018cited by this paper
• Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction

  2018influential reference
• Dynamic Video Segmentation Network

  2018cited by this paper
• A comparative review of plausible hole filling strategies in the context of scene depth image completion

  2018cited by this paper
• Real-Time Monocular Depth Estimation Using Synthetic Data with Domain Adaptation via Image Style Transfer

  2018influential reference
• Skip Connections Eliminate Singularities

  2017cited by this paper
• Globally and locally consistent image completion

  2017influential reference
• Semi-Supervised Deep Learning for Monocular Depth Map Prediction

  2017influential reference
• Augmented Reality Meets Computer Vision: Efficient Data Generation for Urban Driving Scenes

  2017cited by this paper
• DepthComp: Real-time Depth Image Completion Based on Prior Semantic Scene Segmentation

  2017cited by this paper
• Object scene flow

  2017influential reference
• Semantic Video CNNs Through Representation Warping

  2017cited by this paper
• Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics

  2017cited by this paper
• Image Super-Resolution Using Dense Skip Connections

  2017cited by this paper
• Automatic differentiation in PyTorch

  2017cited by this paper
• CyCADA: Cycle-Consistent Adversarial Domain Adaptation

  2017cited by this paper
• Flow-Guided Feature Aggregation for Video Object Detection

  2017cited by this paper
• Fast and Accurate Image Super Resolution by Deep CNN with Skip Connection and Network in Network

  2017cited by this paper
• Unsupervised Learning of Depth and Ego-Motion from Video

  2017influential reference
• Clockwork Convnets for Video Semantic Segmentation

  2016cited by this paper
• Deep Feature Flow for Video Recognition

  2016cited by this paper
• Optical Flow Estimation Using a Spatial Pyramid Network

  2016cited by this paper
• Context Encoders: Feature Learning by Inpainting

  2016cited by this paper
• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

  2016influential reference
• Pyramid Scene Parsing Network

  2016cited by this paper
• Deep3D: Fully Automatic 2D-to-3D Video Conversion with Deep Convolutional Neural Networks

  2016cited by this paper
• Pixel-Level Encoding and Depth Layering for Instance-Level Semantic Labeling

  2016cited by this paper
• RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation

  2016cited by this paper
• Image-to-Image Translation with Conditional Adversarial Networks

  2016cited by this paper
• Semantic Video Segmentation by Gated Recurrent Flow Propagation

  2016cited by this paper
• Generating Images with Perceptual Similarity Metrics based on Deep Networks

  2016cited by this paper
• Depth Image Inpainting: Improving Low Rank Matrix Completion With Low Gradient Regularization

  2016cited by this paper
• STFCN: Spatio-Temporal FCN for Semantic Video Segmentation

  2016cited by this paper
• Building Scene Models by Completing and Hallucinating Depth and Semantics

  2016cited by this paper
• Deeper Depth Prediction with Fully Convolutional Residual Networks

  2016cited by this paper
• The Cityscapes Dataset for Semantic Urban Scene Understanding

  2016cited by this paper
• Back to Butterworth - a Fourier basis for 3D surface relief hole filling within RGB-D imagery

  2016influential reference
• Semantic Image Inpainting with Deep Generative Models

  2016cited by this paper
• The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes

  2016influential reference
• Unsupervised Monocular Depth Estimation with Left-Right Consistency

  2016influential reference
• Spatial Transformer Networks

  2015cited by this paper
• Attention to Scale: Scale-Aware Semantic Image Segmentation

  2015cited by this paper
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

  2015cited by this paper
• Depth image enhancement using local tangent plane approximations

  2015cited by this paper
• FlowNet: Learning Optical Flow with Convolutional Networks

  2015cited by this paper
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

  2015cited by this paper
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

  2015cited by this paper
• Parallel Multi-Dimensional LSTM, With Application to Fast Biomedical Volumetric Image Segmentation

  2015cited by this paper
• Depth from shading, defocus, and correspondence using light-field angular coherence

  2015cited by this paper
• ReSeg: A Recurrent Neural Network-Based Model for Semantic Segmentation

  2015cited by this paper
• Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields

  2015influential reference
• U-Net: Convolutional Networks for Biomedical Image Segmentation

  2015influential reference
• Learning Deconvolution Network for Semantic Segmentation

  2015cited by this paper
• Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding

  2015cited by this paper
• Conditional Random Fields as Recurrent Neural Networks

  2015cited by this paper
• SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation

  2015influential reference
• Semantic Image Segmentation via Deep Parsing Network

  2015cited by this paper
• Improved 3D Sparse Maps for High-performance Structure from Motion with Low-cost Omnidirectional Robots - Evaluation Dataset

  2015cited by this paper
• Depth and surface normal estimation from monocular images using regression on deep features and hierarchical CRFs

  2015cited by this paper
• Im2depth: Scalable exemplar based depth transfer

  2014cited by this paper
• Depth-based inpainting for disocclusion filling

  2014cited by this paper
• An improved edge detection algorithm for depth map inpainting

  2014cited by this paper
• Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture

  2014cited by this paper
• Fully convolutional networks for semantic segmentation

  2014influential reference
• Adam: A Method for Stochastic Optimization

  2014cited by this paper
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network

  2014influential reference
• PM-Huber: PatchMatch with Huber Regularization for Stereo Matching

  2013cited by this paper
• Depth Map Inpainting under a Second-Order Smoothness Prior

  2013cited by this paper
• Depth inpainting by tensor voting.

  2013cited by this paper
• Vision meets robotics: The KITTI dataset

  2013influential reference
• Guided inpainting and filtering for Kinect depth maps

  2012influential reference
• A hierarchical extension to 3D non-parametric surface relief completion

  2012cited by this paper
• A Naturalistic Open Source Movie for Optical Flow Evaluation

  2012cited by this paper
• Heliometric Stereo: Shape from Sun Position

  2012cited by this paper
• A Variational Framework for Exemplar-Based Image Inpainting

  2011cited by this paper
• Semantic object classes in video: A high-definition ground truth database

  2009cited by this paper
• An Image Inpainting Technique Based on the Fast Marching Method

  2004cited by this paper
• Poisson image editing

  2003influential reference
• Navier-Stokes, Fluid Dynamics, and Image and Video Inpainting

  2001cited by this paper
• Navier-stokes, fluid dynamics, and image and video inpainting

  2001cited by this paper
• A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms

  2001cited by this paper
• Video

  1999cited by this paper
• Ieee Transactions on Pattern Analysis and Machine Intelligence 1 Stereo Processing by Semi-global Matching and Mutual Information

  year unknowncited by this paper

• Hadamard Product in Deep Learning: Introduction, Advances and Challenges

  2025cites this paper
• Unsupervised Domain Adaptation Depth Estimation Based on Self-attention Mechanism and Edge Consistency Constraints

  2024cites this paper
• Less is More: Reducing Task and Model Complexity for 3D Point Cloud Semantic Segmentation

  2023cites this paper
• A Threefold Review on Deep Semantic Segmentation: Efficiency-oriented, Temporal and Depth-aware design

  2023cites this paper
• Real-Time Dense Reconstruction with Binocular Endoscopy Based on StereoNet and ORB-SLAM

  2023cites this paper
• SPNet: Structure preserving network for depth completion

  2023cites this paper
• Joint Optimization of Depth and Ego-Motion for Intelligent Autonomous Vehicles

  2023cites this paper
• Sparse depth densification for monocular depth estimation

  2023cites this paper
• Mining Regional Relation from Pixel-wise Annotation for Scene Parsing

  2022cites this paper
• PMOD-Net: point-cloud-map-based metric scale obstacle detection by using a monocular camera

  2022cites this paper
• Robust Multimodal Depth Estimation using Transformer based Generative Adversarial Networks

  2022cites this paper
• Image Segmentation Techniques: Statistical, Comprehensive, Semi-Automated Analysis and an Application Perspective Analysis of Mathematical Expressions

  2022cites this paper
• Semisupervised learning-based depth estimation with semantic inference guidance

  2022cites this paper
• Deep learning for monocular depth estimation: A review

  2021cites this paper
• DPSNet: Multitask Learning Using Geometry Reasoning for Scene Depth and Semantics

  2021cites this paper
• Semi-supervised Multi-task Learning for Semantics and Depth

  2021cites this paper
• Indoor scene understanding based on manhattan and non-manhattan projection of spatial right-angles

  2021cites this paper
• Resolving the cybersecurity Data Sharing Paradox to scale up cybersecurity via a co-production approach towards data sharing

  2020cites this paper
• Avoiding non-Manhattan obstacles based on projection of spatial corners in indoor environment

  2020cites this paper
• Context-Aware Multi-Task Learning for Traffic Scene Recognition in Autonomous Vehicles

  2020cites this paper
• Understanding of wheelchair ramp scenes for disabled people with visual impairments

  2020cites this paper
• MD-ST: Monocular Depth Estimation Based on Spatio-Temporal Correlation Features

  2020cites this paper
• When Self-Supervised Learning Meets Scene Classification: Remote Sensing Scene Classification Based on a Multitask Learning Framework

  2020cites this paper
• Competitive Simplicity for Multi-Task Learning for Real-Time Foggy Scene Understanding via Domain Adaptation

  2020influential citation
• Domain Adaptation via Image Style Transfer

  2020cites this paper
• DESC: Domain Adaptation for Depth Estimation via Semantic Consistency

  2020cites this paper
• Rank over Class: The Untapped Potential of Ranking in Natural Language Processing

  2020cites this paper
• Adaptive Context-Aware Multi-Modal Network for Depth Completion

  2020cites this paper
• Semi-Supervised Monocular Depth Estimation Based on Semantic Supervision

  2020cites this paper
• From Depth What Can You See? Depth Completion via Auxiliary Image Reconstruction

  2020cites this paper
• Temporally Coherent General Dynamic Scene Reconstruction

  2019cites this paper
• To Complete or to Estimate, That is the Question: A Multi-Task Approach to Depth Completion and Monocular Depth Estimation

  2019cites this paper
• Multi-Task Regression-Based Learning for Autonomous Unmanned Aerial Vehicle Flight Control Within Unstructured Outdoor Environments

  2019cites this paper
• A King’s Ransom for Encryption: Ransomware Classification using Augmented One-Shot Learning and Bayesian Approximation

  2019cites this paper
• Monocular Segment-Wise Depth: Monocular Depth Estimation Based on a Semantic Segmentation Prior

  2019cites this paper
• for Monocular Depth Estimation: A Review.

  year unknowncites this paper