MoDAR: Using Motion Forecasting for 3D Object Detection in Point Cloud Sequences

Occluded and long-range objects are ubiquitous and challenging for 3D object detection. Point cloud sequence data provide unique opportunities to improve such cases, as an occluded or distant object can be observed from different viewpoints or gets better visibility over time. However, the efficiency and effectiveness in encoding longterm sequence data can still be improved. In this work, we propose MoDAR, using motion forecasting outputs as a type of virtual modality, to augment LiDAR point clouds. The MoDAR modality propagates object information from temporal contexts to a target frame, represented as a set of virtual points, one for each object from a waypoint on a forecasted trajectory. A fused point cloud of both raw sensor points and the virtual points can then be fed to any off-the-shelf point-cloud based 3D object detector. Evaluated on the Waymo Open Dataset, our method significantly improves prior art detectors by using motion forecasting from extra-long sequences (e.g. 18 seconds), achieving new state of the arts, while not adding much computation overhead.

• Publication year

  2023

• Venue

  Computer Vision and Pattern Recognition

• Publication date

  2023-06-01

• Fields of study

  Computer Science, Engineering, Environmental Science

• Identifiers
  DOI 10.1109/CVPR52729.2023.00900arXiv 2306.03206
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Wayformer: Motion Forecasting via Simple & Efficient Attention Networks

  2022cited by this paper
• MPPNet: Multi-Frame Feature Intertwining with Proxy Points for 3D Temporal Object Detection

  2022cited by this paper
• Forecasting from LiDAR via Future Object Detection

  2022cited by this paper
• Sparse Fuse Dense: Towards High Quality 3D Detection with Depth Completion

  2022cited by this paper
• DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection

  2022cited by this paper
• DCMS: Motion Forecasting with Dual Consistency and Multi-Pseudo-Target Supervision

  2022cited by this paper
• Lidar Augment: Searching for Scalable 3D LiDAR Data Augmentations

  2022cited by this paper
• CramNet: Camera-Radar Fusion with Ray-Constrained Cross-Attention for Robust 3D Object Detection

  2022cited by this paper
• SWFormer: Sparse Window Transformer for 3D Object Detection in Point Clouds

  2022influential reference
• LidarNAS: Unifying and Searching Neural Architectures for 3D Point Clouds

  2022cited by this paper
• 3D-MAN: 3D Multi-frame Attention Network for Object Detection

  2021cited by this paper
• Multimodal Motion Prediction with Stacked Transformers

  2021cited by this paper
• Offboard 3D Object Detection from Point Cloud Sequences

  2021influential reference
• Weighted boxes fusion: Ensembling boxes from different object detection models

  2021influential reference
• LaneRCNN: Distributed Representations for Graph-Centric Motion Forecasting

  2021cited by this paper
• Auto4D: Learning to Label 4D Objects from Sequential Point Clouds

  2021cited by this paper
• MP3: A Unified Model to Map, Perceive, Predict and Plan

  2021cited by this paper
• Embracing Single Stride 3D Object Detector with Sparse Transformer

  2021cited by this paper
• MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction

  2021cited by this paper
• Multimodal Virtual Point 3D Detection

  2021cited by this paper
• Scene Transformer: A unified architecture for predicting multiple agent trajectories

  2021cited by this paper
• 4D-Net for Learned Multi-Modal Alignment

  2021cited by this paper
• DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets

  2021cited by this paper
• PointAugmenting: Cross-Modal Augmentation for 3D Object Detection

  2021cited by this paper
• RSN: Range Sparse Net for Efficient, Accurate LiDAR 3D Object Detection

  2021cited by this paper
• Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset

  2021influential reference
• STINet: Spatio-Temporal-Interactive Network for Pedestrian Detection and Trajectory Prediction

  2020cited by this paper
• 3DSSD: Point-Based 3D Single Stage Object Detector

  2020cited by this paper
• HVNet: Hybrid Voxel Network for LiDAR Based 3D Object Detection

  2020cited by this paper
• Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud

  2020cited by this paper
• MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird’s Eye View Maps

  2020cited by this paper
• Range Conditioned Dilated Convolutions for Scale Invariant 3D Object Detection

  2020cited by this paper
• PnPNet: End-to-End Perception and Prediction With Tracking in the Loop

  2020cited by this paper
• Center-based 3D Object Detection and Tracking

  2020influential reference
• AFDet: Anchor Free One Stage 3D Object Detection

  2020cited by this paper
• Pillar-based Object Detection for Autonomous Driving

  2020cited by this paper
• LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Driving

  2019cited by this paper
• PointPainting: Sequential Fusion for 3D Object Detection

  2019cited by this paper
• Scalability in Perception for Autonomous Driving: Waymo Open Dataset

  2019influential reference
• StarNet: Targeted Computation for Object Detection in Point Clouds

  2019cited by this paper
• A Baseline for 3D Multi-Object Tracking

  2019cited by this paper
• Deep Hough Voting for 3D Object Detection in Point Clouds

  2019cited by this paper
• Objects as Points

  2019cited by this paper
• nuScenes: A Multimodal Dataset for Autonomous Driving

  2019cited by this paper
• PIXOR: Real-time 3D Object Detection from Point Clouds

  2018cited by this paper
• SECOND: Sparsely Embedded Convolutional Detection

  2018cited by this paper
• Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net

  2018influential reference
• IPOD: Intensive Point-based Object Detector for Point Cloud

  2018cited by this paper
• PointPillars: Fast Encoders for Object Detection From Point Clouds

  2018cited by this paper
• PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud

  2018cited by this paper
• ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst

  2018cited by this paper
• Complex-YOLO: An Euler-Region-Proposal for Real-Time 3D Object Detection on Point Clouds

  2018cited by this paper
• IntentNet: Learning to Predict Intention from Raw Sensor Data

  2018cited by this paper
• Joint 3D Proposal Generation and Object Detection from View Aggregation

  2017cited by this paper
• VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection

  2017cited by this paper
• Frustum PointNets for 3D Object Detection from RGB-D Data

  2017cited by this paper
• Vehicle Detection from 3D Lidar Using Fully Convolutional Network

  2016cited by this paper
• 3D fully convolutional network for vehicle detection in point cloud

  2016cited by this paper
• Vote3Deep: Fast object detection in 3D point clouds using efficient convolutional neural networks

  2016cited by this paper
• Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images

  2015cited by this paper
• Voting for Voting in Online Point Cloud Object Detection

  2015cited by this paper
• Multimodal decision-level fusion for person authentication

  1999cited by this paper
• Multisensor image fusion in remote sensing: Concepts, methods and applications

  1998cited by this paper

• A Frustum-Aware Fusion Network With Cross-Attention for Multi-Modal 3D Detection

  2025cites this paper
• ForeSight: Multi-View Streaming Joint Object Detection and Trajectory Forecasting

  2025cites this paper
• A Survey on End-to-end Perception and Prediction for Autonomous Driving

  2025cites this paper
• Is Intermediate Fusion All You Need for UAV-based Collaborative Perception?

  2025cites this paper
• MAD: Memory-Augmented Detection of 3D Objects

  2025influential citation
• SOR: Semi-Automatic Object Box and Road Element Annotation Pipeline

  2025cites this paper
• Leveraging Temporal Cues for Semi-Supervised Multi-View 3D Object Detection

  2025cites this paper
• Towards Flexible 3D Perception: Object-Centric Occupancy Completion Augments 3D Object Detection

  2024influential citation
• FuDensityNet: Fusion-Based Density-Enhanced Network for Occlusion Handling

  2024cites this paper
• T4P: Test-Time Training of Trajectory Prediction via Masked Autoencoder and Actor-Specific Token Memory

  2024cites this paper
• Future Does Matter: Boosting 3D Object Detection with Temporal Motion Estimation in Point Cloud Sequences

  2024cites this paper
• TrajSSL: Trajectory-Enhanced Semi-Supervised 3D Object Detection

  2024cites this paper
• DG-BEV: Depth-Guided BEV 3D Object Detection with Sparse LiDAR Data

  2024cites this paper
• Frame Fusion with Vehicle Motion Prediction for 3D Object Detection

  2023cites this paper
• PTT: Point-Trajectory Transformer for Efficient Temporal 3D Object Detection

  2023cites this paper
• Practical Collaborative Perception: A Framework for Asynchronous and Multi-Agent 3D Object Detection

  2023cites this paper
• A survey on deep learning approaches for data integration in autonomous driving system

  2023cites this paper