Excavator 3D pose estimation from point cloud with self‐supervised deep learning

Pose estimation of excavators is a fundamental yet challenging task with significant implications for intelligent construction. Traditional methods based on cameras or sensors are often limited by their ability to perceive spatial structures. To address this, 3D light detection and ranging has emerged as a promising paradigm for excavator pose estimation. However, these methods face significant challenges: (1) accurate 3D pose annotations are labor‐intensive and costly, and (2) excavators exhibit complex kinematics and geometric structures, further complicating pose estimation. In this study, a novel framework is proposed for full‐body excavator pose estimation directly from 3D point clouds, without relying on manual 3D annotations. The excavator pose is parameterized using pose parameters of geometric primitives under kinematic constraints. A unified deep network is designed to predict pose parameters from point clouds. The network is initially pre‐trained on synthetic data to provide parameter initialization and then fine‐tuned using real‐world data. To facilitate label‐free training, the self‐supervised loss functions are designed by exploiting the geometric and kinematic consistency between point clouds and excavators. Experimental results on real‐world construction sites demonstrate the effectiveness and robustness of the proposed method, achieving an average pose estimation accuracy of 0.26 m. The method also exhibits promising performance across various excavator operational scenarios, highlighting its potential for real‐world applications.

• Publication year

  2025

• Venue

  Comput. Aided Civ. Infrastructure Eng.

• Publication date

  2025-05-03

• Fields of study

  Computer Science, Engineering, Environmental Science

• Identifiers
  DOI 10.1111/mice.13500
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• On Boundary Discontinuity in Angle Regression Based Arbitrary Oriented Object Detection

  2024cited by this paper
• Robust Shape Fitting for 3D Scene Abstraction

  2024cited by this paper
• Deep learning applications for point clouds in the construction industry

  2024cited by this paper
• Deep learning framework with Local Sparse Transformer for construction worker detection in 3D with LiDAR

  2024influential reference
• Multisensory and BIM-Integrated Digital Twin to Improve Urban Excavation Safety

  2023cited by this paper
• 3D pose estimation and localization of construction equipment from single camera images by virtual model integration

  2023cited by this paper
• Excavator 3D pose estimation using deep learning and hybrid datasets

  2023influential reference
• 3D Human Keypoints Estimation from Point Clouds in the Wild without Human Labels

  2023cited by this paper
• Monocular 3D object detection for construction scene analysis

  2023cited by this paper
• Intention‐aware robot motion planning for safe worker–robot collaboration

  2023cited by this paper
• Real-Time Bucket Pose Estimation Based on Deep Neural Network and Registration Using Onboard 3D Sensor

  2023cited by this paper
• Memory-Augmented Point Cloud Registration Network for Bucket Pose Estimation of the Intelligent Mining Excavator

  2022cited by this paper
• HUM3DIL: Semi-supervised Multi-modal 3D Human Pose Estimation for Autonomous Driving

  2022cited by this paper
• Point Transformer V2: Grouped Vector Attention and Partition-based Pooling

  2022cited by this paper
• Self-Supervised Learning for Electroencephalography

  2022cited by this paper
• Sensing, perception, decision, planning and action of autonomous excavators

  2022cited by this paper
• Implementation experiments on convolutional neural network training using synthetic images for 3D pose estimation of an excavator on real images

  2022cited by this paper
• PriFit: Learning to Fit Primitives Improves Few Shot Point Cloud Segmentation

  2021cited by this paper
• Synthesizing Pose Sequences from 3D Assets for Vision-Based Activity Analysis

  2021cited by this paper
• CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds

  2021cited by this paper
• Proactive proximity monitoring with instance segmentation and unmanned aerial vehicle‐acquired video‐frame prediction

  2021cited by this paper
• HandFoldingNet: A 3D Hand Pose Estimation Network Using Multiscale-Feature Guided Folding of a 2D Hand Skeleton

  2021cited by this paper
• Deep learning‐based object identification with instance segmentation and pseudo‐LiDAR point cloud for work zone safety

  2021cited by this paper
• Point Transformer

  2020cited by this paper
• SUSTech POINTS: A Portable 3D Point Cloud Interactive Annotation Platform System

  2020cited by this paper
• Lidar for Autonomous Driving: The Principles, Challenges, and Trends for Automotive Lidar and Perception Systems

  2020cited by this paper
• SAPIEN: A SimulAted Part-Based Interactive ENvironment

  2020cited by this paper
• A vision-based marker-less pose estimation system for articulated construction robots

  2019cited by this paper
• Category-Level Articulated Object Pose Estimation

  2019cited by this paper
• Scalability in Perception for Autonomous Driving: Waymo Open Dataset

  2019cited by this paper
• PyTorch: An Imperative Style, High-Performance Deep Learning Library

  2019cited by this paper
• Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation

  2019cited by this paper
• Point-to-Point Regression PointNet for 3D Hand Pose Estimation

  2018cited by this paper
• Super-convergence: very fast training of neural networks using large learning rates

  2018influential reference
• An evidence‐based approach to object pose estimation from LiDAR measurements in challenging environments

  2018cited by this paper
• Framework for Location Data Fusion and Pose Estimation of Excavators Using Stereo Vision

  2018cited by this paper
• Detection of Outliers Using Interquartile Range Technique from Intrusion Dataset

  2018cited by this paper
• Self-corrective knowledge-based hybrid tracking system using BIM and multimodal sensors

  2017cited by this paper
• Integral Human Pose Regression

  2017cited by this paper
• Optimization-based excavator pose estimation using real-time location systems

  2015cited by this paper
• Projection-Recognition-Projection Method for Automatic Object Recognition and Registration for Dynamic Heavy Equipment Operations

  2014cited by this paper
• Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments

  2014cited by this paper
• Laser Scanner-based End-effector Tracking and Joint Variable Extraction for Heavy Machinery

  2010cited by this paper
• A New Approach for Health Monitoring of Structures: Terrestrial Laser Scanning

  2007cited by this paper

• Detecting worker loss of balance events from point cloud sequence using unsupervised motion-pose learning

  2025cites this paper
• A point cloud dataset and deep learning method for semantic segmentation of underground garages

  2025cites this paper
• Three‐dimensional reconstruction of loose defects in semi‐rigid base layers using enhanced deep learning and point cloud from GPR images

  2025cites this paper
• Visual-IMU fusion for 3D pose measurement of articulated construction equipment

  2025cites this paper