Visual closed-loop control for pouring liquids

Pouring a specific amount of liquid is a challenging task. In this paper we develop methods for robots to use visual feedback to perform closed-loop control for pouring liquids. We propose both a model-based and a model-free method utilizing deep learning for estimating the volume of liquid in a container. Our results show that the model-free method is better able to estimate the volume. We combine this with a simple PID controller to pour specific amounts of liquid, and show that the robot is able to achieve an average 38ml deviation from the target amount. To our knowledge, this is the first use of raw visual feedback to pour liquids in robotics.

• Publication year

  2016

• Venue

  IEEE International Conference on Robotics and Automation

• Publication date

  2016-10-09

• Fields of study

  Computer Science, Engineering

• Identifiers
  DOI 10.1109/ICRA.2017.7989307arXiv 1610.02610
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Envisioning the qualitative effects of robot manipulation actions using simulation-based projections

  2017cited by this paper
• Guided Policy Search with Delayed Senor Measurements

  2016cited by this paper
• Towards Learning to Perceive and Reason About Liquids

  2016influential reference
• Multi-sensor surface analysis for robotic ironing

  2016cited by this paper
• Stereo vision of liquid and particle flow for robot pouring

  2016cited by this paper
• Differential dynamic programming with temporally decomposed dynamics

  2015cited by this paper
• LSTM: A Search Space Odyssey

  2015cited by this paper
• Half Full or Empty: Cues That Lead Wine Drinkers to Unintentionally Overpour

  2014cited by this paper
• Enhanced robotic cleaning with a low-cost tool attachment

  2014cited by this paper
• Fully convolutional networks for semantic segmentation

  2014cited by this paper
• Adam: A Method for Stochastic Optimization

  2014influential reference
• Incorporating Failure-to-Success Transitions in Imitation Learning for a Dynamic Pouring Task

  2014cited by this paper
• Caffe: Convolutional Architecture for Fast Feature Embedding

  2014cited by this paper
• Force-based robot learning of pouring skills using parametric hidden Markov models

  2013influential reference
• On Cross-Spectral Stereo Matching using Dense Gradient Features

  2012cited by this paper
• Object Categorization in the Sink : Learning Behavior – Grounded Object Categories with Water

  2012cited by this paper
• Learning to place new objects in a scene

  2012cited by this paper
• Designing robot learners that ask good questions

  2012cited by this paper
• Daytime water detection based on sky reflections

  2011cited by this paper
• Learning to pour with a robot arm combining goal and shape learning for dynamic movement primitives

  2011cited by this paper
• 3D is here: Point Cloud Library (PCL)

  2011cited by this paper
• On behalf of: Multimedia Archives

  2011cited by this paper
• Daytime water detection based on color variation

  2010cited by this paper
• Vision based behavior verification system of humanoid robot for daily environment tasks

  2006cited by this paper
• Efficient feature extraction for 2D/3D objects in mesh representation

  2001cited by this paper
• Long Short-Term Memory

  1997cited by this paper
• Hidden Markov models.

  1996cited by this paper

• ViscNet: Vision-Based In-line Viscometry for Fluid Mixing Process

  2025cites this paper
• DexPour: Effective and Efficient High-DoF Robotic Hand Liquid Pouring via Hierarchical Reward with Approximated Proxy Abstraction

  2025cites this paper
• SCOOP'D: Learning Mixed-Liquid-Solid Scooping via Sim2Real Generative Policy

  2025cites this paper
• Uncertainty-aware Planning with Inaccurate Models for Robotized Liquid Handling

  2025cites this paper
• The MOTIF Hand: A Robotic Hand for Multimodal Observations with Thermal, Inertial, and Force Sensors

  2025cites this paper
• SimLiquid: A Simulation‐Based Liquid Perception Pipeline for Robot Liquid Manipulation

  2025cites this paper
• A Modular Framework for Knowledge-Based Servoing: Plugging Symbolic Theories into Robotic Controllers

  2025cites this paper
• Liquids Identification and Manipulation via Digitally Fabricated Impedance Sensors

  2024cites this paper
• An Intelligent Robotic System for Perceptive Pancake Batter Stirring and Precise Pouring

  2024cites this paper
• A Physics-Informed Demonstration-Guided Learning Framework for Granular Material Manipulation.

  2024cites this paper
• Efficient Preference-based Reinforcement Learning via Aligned Experience Estimation

  2024cites this paper
• Privileged Sensing Scaffolds Reinforcement Learning

  2024cites this paper
• RoboCAP: Robotic Classification and Precision Pouring of Diverse Liquids and Granular Media with Capacitive Sensing

  2024influential citation
• Vision-based robot manipulation of transparent liquid containers in a laboratory setting

  2024cites this paper
• Trajectory-Oriented Policy Optimization with Sparse Rewards

  2024cites this paper
• Task-Oriented Active Learning of Model Preconditions for Inaccurate Dynamics Models

  2024cites this paper
• A survey on the visual perception of humanoid robot

  2024cites this paper
• The Sound of Water: Inferring Physical Properties from Pouring Liquids

  2024cites this paper
• GMM Enabled by Multimodal Information Fusion Network for Detection and Motion Planning of Robotic Liquid Pouring

  2024influential citation
• Continuously Improving Mobile Manipulation with Autonomous Real-World RL

  2024cites this paper
• Closed-Loop Visuomotor Control with Generative Expectation for Robotic Manipulation

  2024cites this paper
• A multi-purpose robot perception system enabling closed-loop control for zero defect manufacturing in gluing processes of large parts

  2024cites this paper
• PourIt!: Weakly-supervised Liquid Perception from a Single Image for Visual Closed-Loop Robotic Pouring

  2023cites this paper
• DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physics

  2023cites this paper
• Offline Feature-Based Reinforcement Learning with Preprocessed Image Inputs for Liquid Pouring Control

  2023cites this paper
• Kitchen Artist: Precise Control of Liquid Dispensing for Gourmet Plating

  2023cites this paper
• Pour me a drink: Robotic Precision Pouring Carbonated Beverages into Transparent Containers

  2023cites this paper
• Dynamic liquid volume estimation using optical tactile sensors and spiking neural network

  2023influential citation
• Fluid viscosity prediction leveraging computer vision and robot interaction

  2023cites this paper
• Liquid Content Detection In Transparent Containers: A Benchmark

  2023cites this paper
• DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics

  2023cites this paper
• Inverse Preference Learning: Preference-based RL without a Reward Function

  2023cites this paper
• Preference Transformer: Modeling Human Preferences using Transformers for RL

  2023cites this paper
• Self-supervised Transparent Liquid Segmentation for Robotic Pouring

  2022cites this paper
• PourNet: Robust Robotic Pouring Through Curriculum and Curiosity-based Reinforcement Learning

  2022cites this paper
• Training Robots to Evaluate Robots: Example-Based Interactive Reward Functions for Policy Learning

  2022cites this paper
• ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds

  2022cites this paper
• AutoBag: Learning to Open Plastic Bags and Insert Objects

  2022cites this paper
• Planning with Spatial and Temporal Abstraction from Point Clouds for Deformable Object Manipulation

  2022cites this paper
• Robot gaining robust pouring skills through fusing vision and audio.

  2022cites this paper
• Research on Intelligent Detection Technology of Transparent Liquid based on Style Transfer

  2022cites this paper
• Versatile Control of Fluid-directed Solid Objects Using Multi-task Reinforcement Learning

  2022cites this paper
• A Geometric Approach for Partial Liquids' Pouring from a Regular Container by a Robotic Manipulator

  2022cites this paper
• Robot Learning Physical Object Properties from Human Visual Cues: A novel approach to infer the fullness level in containers

  2022cites this paper
• Pouring by Feel: An Analysis of Tactile and Proprioceptive Sensing for Accurate Pouring

  2022cites this paper
• Visual-tactile Sensing for Real-time Liquid Volume Estimation in Grasping

  2022cites this paper
• One-Shot Domain-Adaptive Imitation Learning via Progressive Learning Applied to Robotic Pouring

  2022cites this paper
• Data augmentation with mixtures of max-entropy transformations for filling-level classification

  2022cites this paper
• Transparent Liquid Segmentation Without Annotations

  2021cites this paper
• Improving Filling Level Classification with Adversarial Training

  2021cites this paper
• Explainable Hierarchical Imitation Learning for Robotic Drink Pouring

  2021cites this paper
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training

  2021cites this paper
• Neural Task Success Classifiers for Robotic Manipulation from Few Real Demonstrations

  2021cites this paper
• Automated System for Pouring and Filtration Tasks in Laboratory Applications

  2021cites this paper
• A Workflow for Offline Model-Free Robotic Reinforcement Learning

  2021cites this paper
• Experimental Verification of Real-Time Flow-Rate Estimations in a Tilting-Ladle-Type Automatic Pouring Machine

  2021cites this paper
• Manipulation of Granular Materials by Learning Particle Interactions

  2021cites this paper
• Active Vapor-Based Robotic Wiper

  2021cites this paper
• Image Based Reconstruction of Liquids from 2D Surface Detections

  2021cites this paper
• Using Graph Neural Networks to Simulate Manipulated Granular Material

  2021cites this paper
• Transparent Liquid Segmentation for Robotic Pouring

  2021cites this paper
• Can I Pour Into It? Robot Imagining Open Containability Affordance of Previously Unseen Objects via Physical Simulations

  2020influential citation
• The Ingredients of Real-World Robotic Reinforcement Learning

  2020cites this paper
• Robust Robotic Pouring using Audition and Haptics

  2020cites this paper
• Generalizing Learned Manipulation Skills in Practice

  2020cites this paper
• Learning compositional models of robot skills for task and motion planning

  2020cites this paper
• Robot Gaining Accurate Pouring Skills through Self-Supervised Learning and Generalization

  2020influential citation
• COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning

  2020cites this paper
• Leveraging Active Perception for Improving Embedding-based Deep Face Recognition

  2020cites this paper
• Learning a Pile Loading Controller from Demonstrations

  2020cites this paper
• Stir to Pour: Efficient Calibration of Liquid Properties for Pouring Actions

  2020influential citation
• Review of Deep Reinforcement Learning-Based Object Grasping: Techniques, Open Challenges, and Recommendations

  2020cites this paper
• AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos

  2019cites this paper
• Water Streak Detection with Convolutional Neural Networks for Scrubber Dryers

  2019cites this paper
• Computer Vision Systems: 12th International Conference, ICVS 2019, Thessaloniki, Greece, September 23–25, 2019, Proceedings

  2019influential citation
• Contextual Imagined Goals for Self-Supervised Robotic Learning

  2019cites this paper
• Adversarial Skill Networks: Unsupervised Robot Skill Learning from Video

  2019cites this paper
• Robot Learning for Manipulation of Granular Materials Using Vision and Sound

  2019cites this paper
• Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation

  2019cites this paper
• Modeling and Control for Robotic Assistants: Single and Multi-robot Manipulation

  2019cites this paper
• Analyzing Liquid Pouring Sequences via Audio-Visual Neural Networks

  2019cites this paper
• Accurate Robotic Pouring for Serving Drinks

  2019cites this paper
• Implementation and Experimental Verification of Flow Rate Control Based on Differential Flatness in a Tilting-Ladle-Type Automatic Pouring Machine

  2019cites this paper
• End-to-End Robotic Reinforcement Learning without Reward Engineering

  2019cites this paper
• Pouring from Deformable Containers Using Dual-Arm Manipulation and Tactile Sensing

  2019cites this paper
• Autonomous Precision Pouring From Unknown Containers

  2019cites this paper
• Making Sense of Audio Vibration for Liquid Height Estimation in Robotic Pouring

  2019influential citation
• Visual attention for high speed driving

  2019cites this paper
• Pouring from Deformable Containers Based on Tactile Information Using a Dual-Arm Robot

  2019cites this paper
• A Kinematic Controller for Liquid Pouring between Vessels Modelled with Smoothed Particle Hydrodynamics

  2019cites this paper
• Haptic Perception of Liquids Enclosed in Containers

  2019cites this paper
• Precision Pouring into Unknown Containers by Service Robots

  2019cites this paper
• Accurate Pouring using Model Predictive Control Enabled by Recurrent Neural Network

  2019cites this paper
• Real-time Liquid Pouring Motion Generation: End-to-End Sensorimotor Coordination for Unknown Liquid Dynamics Trained with Deep Neural Networks

  2019cites this paper
• Task-Conditioned Variational Autoencoders for Learning Movement Primitives

  2019cites this paper
• Few-Shot Goal Inference for Visuomotor Learning and Planning

  2018cites this paper
• Adapting Everyday Manipulation Skills to Varied Scenarios

  2018cites this paper
• Motion Planning for High-DOF Deformable Objects using Nonlinear Dimension Reduction

  2018cites this paper
• Active Model Learning and Diverse Action Sampling for Task and Motion Planning

  2018cites this paper
• To Stir or Not to Stir: Online Estimation of Liquid Properties for Pouring Actions

  2018cites this paper