TossingBot: Learning to Throw Arbitrary Objects With Residual Physics

We investigate whether a robot arm can learn to pick and throw arbitrary rigid objects into selected boxes quickly and accurately. Throwing has the potential to increase the physical reachability and picking speed of a robot arm. However, precisely throwing arbitrary objects in unstructured settings presents many challenges: from acquiring objects in grasps suitable for reliable throwing, to handling varying object-centric properties (e.g., mass distribution, friction, shape) and complex aerodynamics. In this work, we propose an end-to-end formulation that jointly learns to infer control parameters for grasping and throwing motion primitives from visual observations (RGB-D images of arbitrary objects in a bin) through trial and error. Within this formulation, we investigate the synergies between grasping and throwing (i.e., learning grasps that enable more accurate throws) and between simulation and deep learning (i.e., using deep networks to predict residuals on top of control parameters predicted by a physics simulator). The resulting system, TossingBot, is able to grasp and successfully throw arbitrary objects into boxes located outside its maximum reach range at 500+ mean picks per hour (600+ grasps per hour with 85% throwing accuracy); and generalizes to new objects and target locations.

• Publication year

  2019

• Venue

  IEEE Transactions on robotics

• Publication date

  2019-03-27

• Fields of study

  Mathematics, Physics, Computer Science, Engineering

• Identifiers
  DOI 10.1109/TRO.2020.2988642arXiv 1903.11239
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Learning to See before Learning to Act: Visual Pre-training for Manipulation

  2020influential reference
• Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly

  2019cited by this paper
• Learning Visual Affordances for Robotic Manipulation

  2019cited by this paper
• Learning ambidextrous robot grasping policies

  2019cited by this paper
• DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactions

  2019cited by this paper
• Clear Grasp: 3D Shape Estimation of Transparent Objects for Manipulation

  2019cited by this paper
• Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning

  2018cited by this paper
• Residual Policy Learning

  2018cited by this paper
• Data-Augmented Contact Model for Rigid Body Simulation

  2018cited by this paper
• Residual Reinforcement Learning for Robot Control

  2018cited by this paper
• Augmenting Physical Simulators with Stochastic Neural Networks: Case Study of Planar Pushing and Bouncing

  2018cited by this paper
• Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching

  2017cited by this paper
• Adapting learned robotics behaviours through policy adjustment

  2017cited by this paper
• Deep predictive policy training using reinforcement learning

  2017cited by this paper
• Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics

  2017cited by this paper
• Learning Data-Efficient Rigid-Body Contact Models: Case Study of Planar Impact

  2017cited by this paper
• Combining learned and analytical models for predicting action effects from sensory data

  2017cited by this paper
• Optimal shape and motion planning for dynamic planar manipulation

  2017cited by this paper
• Cartman: The Low-Cost Cartesian Manipulator that Won the Amazon Robotics Challenge

  2017influential reference
• 3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions

  2016cited by this paper
• Fast R-CNN

  2015cited by this paper
• Prioritized Experience Replay

  2015cited by this paper
• Deep Residual Learning for Image Recognition

  2015cited by this paper
• SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation

  2015cited by this paper
• Fully convolutional networks for semantic segmentation

  2014cited by this paper
• Extrinsic dexterity: In-hand manipulation with external forces

  2014cited by this paper
• Learning task error models for manipulation

  2013cited by this paper
• Motion Control of a Ball Throwing Robot with a Flexible Robotic Arm

  2013cited by this paper
• KinectFusion: Real-time dense surface mapping and tracking

  2011cited by this paper
• A ball-throwing robot with visual feedback

  2010cited by this paper
• Understanding the difficulty of training deep feedforward neural networks

  2010cited by this paper
• Automated Construction of Robotic Manipulation Programs

  2010cited by this paper
• 1-DOF dynamic pitching robot that independently controls velocity, Angular velocity, and direction of a ball: Contact models and motion planning

  2009cited by this paper
• High-speed throwing motion based on kinetic chain approach

  2008cited by this paper
• Using inaccurate models in reinforcement learning

  2006cited by this paper
• Dynamic Nonprehensile Manipulation: Controllability, Planning, and Experiments

  1999cited by this paper
• A volumetric method for building complex models from range images

  1996cited by this paper
• Dynamic manipulation

  1993influential reference
• Task-level robot learning

  1988cited by this paper
• Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence Reinforcement Learning to Adjust Robot Movements to New Situations

  year unknowncited by this paper

• Physics-Guided Language Model Via Low-Rank Adaptation for Path Loss Prediction

  2026cites this paper
• HEAPGrasp: Hand-Eye Active Perception to Grasp Objects With Diverse Optical Properties

  2026cites this paper
• RFS: Reinforcement Learning with Residual Flow Steering for Dexterous Manipulation

  2026cites this paper
• Hybrid symbolic regression framework for digital twin modeling of gas internal combustion engines

  2026cites this paper
• A Survey on Deep Generative Models for Robot Learning From Multimodal Demonstrations

  2026cites this paper
• DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulation

  2026cites this paper
• Static Is Not Enough: A Comparative Study of VR and SpaceMouse in Static and Dynamic Teleoperation Tasks

  2026cites this paper
• Adaptive robust control of multi-DOF hydraulic manipulator with precise online load estimation

  2026cites this paper
• Scheduling Adaptive Imitation Learning for Long-Horizon Dexterous Robot Micromanipulation of Deformable Cell

  2026cites this paper
• On Transient Release Dynamics in Robot Throwing: A Sliding Pivot Model

  2026influential citation
• Learning to Throw-Flip

  2025cites this paper
• Learning to Double Guess: An Active Perception Approach for Estimating the Center of Mass of Arbitrary Objects

  2025cites this paper
• Throwing Planning Diffusion: A Solution to Learning and Planning of Robotic Throwing

  2025cites this paper
• Residual Reinforcement Learning to Generate a Closed-Loop Policy from a Keyframe Kick Motion

  2025cites this paper
• A Modular Residual Learning Framework to Enhance Model-Based Approach for Robust Locomotion

  2025cites this paper
• Machine Learning-Based Ball Throwing Machine: An Intelligent Approach to Precision and Adaptability

  2025cites this paper
• Low Impedance Rendering Toward Safe Human-Robot Interaction

  2025cites this paper
• SpikePingpong: Spike Vision-based Fast-Slow Pingpong Robot System

  2025cites this paper
• Bridging the Sim-to-Real Gap for Athletic Loco-Manipulation

  2025cites this paper
• Fishing for Data: Modeling, Optimal Planning, and Iterative Learning Control for Flexible Link Robots

  2025influential citation
• 多尺度区域注意力堆叠物体抓取检测网络

  2025cites this paper
• Learning Accurate Whole-body Throwing with High-frequency Residual Policy and Pullback Tube Acceleration

  2025cites this paper
• MoRE: Mixture of Residual Experts for Humanoid Lifelike Gaits Learning on Complex Terrains

  2025cites this paper
• DartBot: Overhand Throwing of Deformable Objects With Tactile Sensing and Reinforcement Learning

  2025cites this paper
• A low-cost and lightweight 6 DoF bimanual arm for dynamic and contact-rich manipulation

  2025cites this paper
• Dual-Arm Fabric Manipulation Learning With Grasp Pose Constraints

  2025cites this paper
• Neural Robot Dynamics

  2025cites this paper
• Self-Supervised Learning of Grasping Arbitrary Objects On-the-Move

  2025cites this paper
• In-Context Iterative Policy Improvement for Dynamic Manipulation

  2025cites this paper
• Dexterous Ungrasping in Three-Dimensional Space: Stability and Planning

  2025cites this paper
• SEM: Enhancing Spatial Understanding for Robust Robot Manipulation

  2025cites this paper
• Latent Adaptive Planner for Dynamic Manipulation

  2025cites this paper
• From Overfitting to Reliability: Introducing the Hierarchical Approximate Bayesian Neural Network

  2025cites this paper
• Comparative analysis of deep Q-learning algorithms for object throwing using a robot manipulator

  2025cites this paper
• Soft Robotic Dynamic in-Hand Pen Spinning

  2025cites this paper
• Towards Damage‐Less Robotic Fragile Fruit Grasping: A Systematic Review on System Design, End Effector, and Visual and Tactile Feedback

  2025cites this paper
• Mobile Manipulator For Robotic Lacrosse: Learning to Pass the Ball

  2025influential citation
• Design and Implementation of a UAV-Based Fire Suppression System with Robotic Arm for High-Rise Building

  2025cites this paper
• A Dexterous and Compliant (DexCo) Hand Based on Soft Hydraulic Actuation for Human-Inspired Fine In-Hand Manipulation

  2025cites this paper
• Experimental Validation of Sensitivity-Aware Trajectory Planning for a Redundant Robotic Manipulator Under Payload Uncertainty

  2025cites this paper
• Implicit Physics-aware Policy for Dynamic Manipulation of Rigid Objects via Soft Body Tools

  2025cites this paper
• DEFT: Differentiable Branched Discrete Elastic Rods for Modeling Furcated DLOs in Real-Time

  2025cites this paper
• HDVIO2.0: Wind and Disturbance Estimation With Hybrid Dynamics VIO

  2025cites this paper
• Dual-Arm Robotic Fabric Manipulation With Quasi-Static and Dynamic Primitives for Rapid Garment Flattening

  2025cites this paper
• Dynamics-Compliant Trajectory Diffusion for Super-Nominal Payload Manipulation

  2025cites this paper
• ActivePusher: Active Learning and Planning with Residual Physics for Nonprehensile Manipulation

  2025cites this paper
• MATTERIX: toward a digital twin for robotics-assisted chemistry laboratory automation

  2025cites this paper
• DA-MMP: Learning Coordinated and Accurate Throwing with Dynamics-Aware Motion Manifold Primitives

  2025cites this paper
• Hearing the Slide: Acoustic-Guided Constraint Learning for Fast Non-Prehensile Transport

  2025cites this paper
• A Novel Formulation for Adaptive Computed Torque Control Enabling Low Feedback Gains in Highly Dynamical Tasks

  2025cites this paper
• IMA-catcher: An IMpact-aware nonprehensile catching framework based on combined optimization and learning

  2025cites this paper
• Central Limit Theorems for Asynchronous Averaged Q-Learning

  2025cites this paper
• Leveraging Neural Descriptor Fields for Learning Contact-Aware Dynamic Recovery

  2025cites this paper
• Robotic throwing based on deep reinforcement learning with predictive reward system

  2025cites this paper
• Prior Reinforce: Mastering Agile Tasks with Limited Trials

  2025cites this paper
• Train-Once Plan-Anywhere Kinodynamic Motion Planning via Diffusion Trees

  2025cites this paper
• Dexterous Ungrasping Manipulation in Three Dimensions

  2025cites this paper
• Cross-embodied Co-design for Dexterous Hands

  2025cites this paper
• Real-Time Trajectory Adaptation for Tossing Robots Using Soft Switching Multiple Model Predictive Control

  2025cites this paper
• Dynamic Self-Righting of Planar-Based Objects on Dual Supports and its Implications for Robotic Object Placement

  2025cites this paper
• RoboTidy : A 3D Gaussian Splatting Household Tidying Benchmark for Embodied Navigation and Action

  2025cites this paper
• AeroThrow: An Autonomous Aerial Throwing System for Precise Payload Delivery

  2025cites this paper
• Object Grasping Assist System with Reading Human Intention and Predicting Object Position

  2025cites this paper
• RM-Planner: Integrating Reinforcement Learning with Whole-Body Model Predictive Control for Mobile Manipulation

  2025cites this paper
• Speedtuning: Speeding Up Policy Execution with Lightweight Reinforcement Learning

  2025cites this paper
• Energy-Optimal Motion Planning for Manipulator Throwing

  2025cites this paper
• DyDexHandover: Human-like Bimanual Dynamic Dexterous Handover using RGB-only Perception

  2025cites this paper
• Near-Optimal Sample Complexity Bounds for Constrained Average-Reward MDPs

  2025cites this paper
• Dynamic Throwing with Robotic Material Handling Machines

  2024cites this paper
• Autonomous Algorithm for Training Autonomous Vehicles with Minimal Human Intervention

  2024cites this paper
• PhyPlan: Generalizable and Rapid Physical Task Planning with Physics Informed Skill Networks for Robot Manipulators

  2024cites this paper
• Differentiable Discrete Elastic Rods for Real-Time Modeling of Deformable Linear Objects

  2024cites this paper
• Physics-Informed Model and Hybrid Planning for Efficient Dyna-Style Reinforcement Learning

  2024cites this paper
• Active Signal Emitter Placement in Complex Environments

  2024cites this paper
• Physics-integrated generative modeling using attentive planar normalizing flow based variational autoencoder

  2024cites this paper
• Self-Supervised Learning of Dynamic Planar Manipulation of Free-End Cables

  2024cites this paper
• TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction

  2024cites this paper
• Tube Acceleration: Robust Dexterous Throwing Against Release Uncertainty

  2024influential citation
• TossNet: Learning to Accurately Measure and Predict Robot Throwing of Arbitrary Objects in Real Time With Proprioceptive Sensing

  2024influential citation
• Self-Supervised Multi-Modal Learning for Collaborative Robotic Grasp-Throw

  2024influential citation
• DexDribbler: Learning Dexterous Soccer Manipulation via Dynamic Supervision

  2024cites this paper
• Explore until Confident: Efficient Exploration for Embodied Question Answering

  2024cites this paper
• Unsupervised Learning of Hybrid Latent Dynamics: A Learn-to-Identify Framework

  2024cites this paper
• Vision Transformer and Residual Network-Based Autoencoder for RGBD Data Processing in Robotic Grasping of Noodle-Like Objects

  2024cites this paper
• Impact-Aware Bimanual Catching of Large-Momentum Objects

  2024cites this paper
• Solving Multi-Entity Robotic Problems Using Permutation Invariant Neural Networks

  2024cites this paper
• Enabling Versatility and Dexterity of the Dual-Arm Manipulators: A General Framework Toward Universal Cooperative Manipulation

  2024cites this paper
• Goal-Oriented Semantic Communication for Robot Arm Reconstruction in Digital Twin: Feature and Temporal Selections

  2024cites this paper
• One Fling to Goal: Environment-aware Dynamics for Goal-conditioned Fabric Flinging

  2024cites this paper
• LeTac-MPC: Learning Model Predictive Control for Tactile-Reactive Grasping

  2024cites this paper
• Harnessing the Synergy between Pushing, Grasping, and Throwing to Enhance Object Manipulation in Cluttered Scenarios

  2024cites this paper
• QuasiSim: Parameterized Quasi-Physical Simulators for Dexterous Manipulations Transfer

  2024cites this paper
• Application of physics-informed machine learning for excavator working resistance modeling

  2024cites this paper
• Survey of learning-based approaches for robotic in-hand manipulation

  2024cites this paper
• Planning Impact-Driven Logistic Tasks

  2024cites this paper
• RESPRECT: Speeding-up Multi-Fingered Grasping With Residual Reinforcement Learning

  2024cites this paper
• Sim-to-Real of Soft Robots With Learned Residual Physics

  2024cites this paper
• A Systematic Literature Review of Digital Twin Research for Healthcare Systems: Research Trends, Gaps, and Realization Challenges

  2024cites this paper
• Learning Object Compliance via Young's Modulus from Single Grasps with Camera-Based Tactile Sensors

  2024cites this paper
• Task-Oriented Language Grounding for Robot via Learning Object Mask

  2024cites this paper