D-REX: Differentiable Real-to-Sim-to-Real Engine for Learning Dexterous Grasping

Simulation provides a cost-effective and flexible platform for data generation and policy learning to develop robotic systems. However, bridging the gap between simulation and real-world dynamics remains a significant challenge, especially in physical parameter identification. In this work, we introduce a real-to-sim-to-real engine that leverages the Gaussian Splat representations to build a differentiable engine, enabling object mass identification from real-world visual observations and robot control signals, while enabling grasping policy learning simultaneously. Through optimizing the mass of the manipulated object, our method automatically builds high-fidelity and physically plausible digital twins. Additionally, we propose a novel approach to train force-aware grasping policies from limited data by transferring feasible human demonstrations into simulated robot demonstrations. Through comprehensive experiments, we demonstrate that our engine achieves accurate and robust performance in mass identification across various object geometries and mass values. Those optimized mass values facilitate force-aware policy learning, achieving superior and high performance in object grasping, effectively reducing the sim-to-real gap.

• Publication year

  2026

• Venue

  Unknown venue

• Publication date

  2026-03-01

• Fields of study

  Computer Science, Engineering

• Identifiers
  arXiv 2603.01151
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Novel Demonstration Generation with Gaussian Splatting Enables Robust One-Shot Manipulation

  2025cited by this paper
• Crossing the Human-Robot Embodiment Gap with Sim-to-Real RL using One Human Demonstration

  2025influential reference
• Scalable Real2Sim: Physics-Aware Asset Generation Via Robotic Pick-and-Place Setups

  2025cited by this paper
• Web2Grasp: Learning Functional Grasps from Web Images of Hand-Object Interactions

  2025cited by this paper
• PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videos

  2025cited by this paper
• Learning 3D-Gaussian Simulators from RGB Videos

  2025cited by this paper
• Real-is-Sim: Bridging the Sim-to-Real Gap with a Dynamic Digital Twin for Real-World Robot Policy Evaluation

  2025cited by this paper
• RobustDexGrasp: Robust Dexterous Grasping of General Objects

  2025cited by this paper
• Bridging the Human to Robot Dexterity Gap Through Object-Oriented Rewards

  2024cited by this paper
• GPT-4o System Card

  2024cited by this paper
• Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modeling

  2024cited by this paper
• Learning Object Properties Using Robot Proprioception via Differentiable Robot-Object Interaction

  2024cited by this paper
• Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction

  2024cited by this paper
• Hand-Object Interaction Pretraining from Videos

  2024cited by this paper
• gsplat: An Open-Source Library for Gaussian Splatting

  2024cited by this paper
• Robo-GS: A Physics Consistent Spatial-Temporal Model for Robotic Arm with Hybrid Representation

  2024influential reference
• Flow as the Cross-Domain Manipulation Interface

  2024cited by this paper
• Controlling diverse robots by inferring Jacobian fields with deep networks

  2024cited by this paper
• TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approach

  2024cited by this paper
• StableNormal: Reducing Diffusion Variance for Stable and Sharp Normal

  2024cited by this paper
• Physically Embodied Gaussian Splatting: A Realtime Correctable World Model for Robotics

  2024cited by this paper
• Real2Code: Reconstruct Articulated Objects via Code Generation

  2024cited by this paper
• URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Images

  2024cited by this paper
• PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation

  2024cited by this paper
• Reconstructing Hand-Held Objects in 3D from Images and Videos

  2024influential reference
• GauStudio: A Modular Framework for 3D Gaussian Splatting and Beyond

  2024cited by this paper
• 2D Gaussian Splatting for Geometrically Accurate Radiance Fields

  2024cited by this paper
• Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation

  2024cited by this paper
• Agile But Safe: Learning Collision-Free High-Speed Legged Locomotion

  2024cited by this paper
• One-Shot Real-to-Sim via End-to-End Differentiable Simulation and Rendering

  2024cited by this paper
• DexGraspNet 2.0: Learning Generative Dexterous Grasping in Large-scale Synthetic Cluttered Scenes

  2024influential reference
• Cross-Embodiment Dexterous Grasping with Reinforcement Learning

  2024cited by this paper
• D(R, O) Grasp: A Unified Representation of Robot and Object Interaction for Cross-Embodiment Dexterous Grasping

  2024cited by this paper
• 3D Gaussian Splatting for Real-Time Radiance Field Rendering

  2023cited by this paper
• LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learning

  2023cited by this paper
• AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System

  2023cited by this paper
• Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments

  2023cited by this paper
• Mastering Diverse Domains through World Models

  2023cited by this paper
• DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics

  2023cited by this paper
• FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects

  2023influential reference
• Reconstructing Hands in 3D with Transformers

  2023cited by this paper
• PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics

  2023cited by this paper
• CuRobo: Parallelized Collision-Free Minimum-Jerk Robot Motion Generation

  2023cited by this paper
• Eureka: Human-Level Reward Design via Coding Large Language Models

  2023cited by this paper
• DiffClothAI: Differentiable Cloth Simulation with Intersection-free Frictional Contact and Differentiable Two-Way Coupling with Articulated Rigid Bodies

  2023cited by this paper
• 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering

  2023cited by this paper
• Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping

  2023cited by this paper
• Affordances from Human Videos as a Versatile Representation for Robotics

  2023cited by this paper
• PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification

  2023influential reference
• FluidLab: A Differentiable Environment for Benchmarking Complex Fluid Manipulation

  2023cited by this paper
• AdaptSim: Task-Driven Simulation Adaptation for Sim-to-Real Transfer

  2023cited by this paper
• NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesis

  2023cited by this paper
• Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarios

  2022cited by this paper
• Evolution Gym: A Large-Scale Benchmark for Evolving Soft Robots

  2022cited by this paper
• Ditto: Building Digital Twins of Articulated Objects from Interaction

  2022cited by this paper
• R3M: A Universal Visual Representation for Robot Manipulation

  2022cited by this paper
• DiffCloud: Real-to-Sim from Point Clouds with Differentiable Simulation and Rendering of Deformable Objects

  2022cited by this paper
• Accelerated Policy Learning with Parallel Differentiable Simulation

  2022cited by this paper
• VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training

  2022cited by this paper
• Real-World Robot Learning with Masked Visual Pre-training

  2022cited by this paper
• Legged Locomotion in Challenging Terrains using Egocentric Vision

  2022cited by this paper
• A Real2Sim2Real Method for Robust Object Grasping with Neural Surface Reconstruction

  2022cited by this paper
• NeRF2Real: Sim2real Transfer of Vision-guided Bipedal Motion Skills using Neural Radiance Fields

  2022cited by this paper
• Hierarchical Model-Based Imitation Learning for Planning in Autonomous Driving

  2022cited by this paper
• Learning to Imitate Object Interactions from Internet Videos

  2022cited by this paper
• VideoDex: Learning Dexterity from Internet Videos

  2022cited by this paper
• Evo-NeRF: Evolving NeRF for Sequential Robot Grasping of Transparent Objects

  2022cited by this paper
• Visual Dexterity: In-hand Dexterous Manipulation from Depth

  2022cited by this paper
• The Role of Physics-Based Simulators in Robotics

  2021cited by this paper
• An End-to-End Differentiable Framework for Contact-Aware Robot Design

  2021cited by this paper
• Brax - A Differentiable Physics Engine for Large Scale Rigid Body Simulation

  2021influential reference
• PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics

  2021cited by this paper
• Reward boosts reinforcement-based motor learning

  2021cited by this paper
• HyperDynamics: Meta-Learning Object and Agent Dynamics with Hypernetworks

  2021cited by this paper
• A System for General In-Hand Object Re-Orientation

  2021cited by this paper
• gradSim: Differentiable simulation for system identification and visuomotor control

  2021influential reference
• Learning Mesh-Based Simulation with Graph Networks

  2020cited by this paper
• RetinaGAN: An Object-aware Approach to Sim-to-Real Transfer

  2020cited by this paper
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains

  2020cited by this paper
• DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactions

  2019cited by this paper
• Vid2Param: Modeling of Dynamics Parameters From Video

  2019cited by this paper
• The frontier of simulation-based inference

  2019cited by this paper
• Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research

  2019cited by this paper
• Learning agile and dynamic motor skills for legged robots

  2019cited by this paper
• Solving Rubik's Cube with a Robot Hand

  2019cited by this paper
• DiffTaichi: Differentiable Programming for Physical Simulation

  2019cited by this paper
• BayesSim: adaptive domain randomization via probabilistic inference for robotics simulators

  2019cited by this paper
• End-to-End Differentiable Physics for Learning and Control

  2018cited by this paper
• Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids

  2018cited by this paper
• Neural Ordinary Differential Equations

  2018cited by this paper
• Sim-to-Real: Learning Agile Locomotion For Quadruped Robots

  2018cited by this paper
• Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping

  2018cited by this paper
• image2mass: Estimating the Mass of an Object from Its Image

  2017cited by this paper
• A Survey on Position Based Dynamics

  2017cited by this paper
• Neural Scene De-rendering

  2017cited by this paper
• Sim-to-Real Transfer of Robotic Control with Dynamics Randomization

  2017cited by this paper
• Domain randomization for transferring deep neural networks from simulation to the real world

  2017cited by this paper
• XPBD: position-based simulation of compliant constrained dynamics

  2016cited by this paper
• Structure-from-Motion Revisited

  2016cited by this paper
• (CAD)$^2$RL: Real Single-Image Flight without a Single Real Image

  2016cited by this paper

• No citing papers are available for this paper.