Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing

Executing contact-rich manipulation tasks necessitates the fusion of tactile and visual feedback. However, the distinct nature of these modalities poses significant challenges. In this paper, we introduce a system that leverages visual and tactile sensory inputs to enable dexterous in-hand manipulation. Specifically, we propose Robot Synesthesia, a novel point cloudbased tactile representation inspired by human tactile-visual synesthesia. This approach allows for the simultaneous and seamless integration of both sensory inputs, offering richer spatial information and facilitating better reasoning about robot actions. Comprehensive ablations are performed on how the integration of vision and touch can improve reinforcement learning and Sim2Real performance. Our project page is available at https://yingyuan0414.github.io/visuotactile/.

• Publication year

  2023

• Venue

  IEEE International Conference on Robotics and Automation

• Publication date

  2023-12-04

• Fields of study

  Computer Science, Engineering

• Identifiers
  DOI 10.1109/ICRA57147.2024.10610532arXiv 2312.01853
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects

  2023cited by this paper
• Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play

  2023cited by this paper
• Rotating without Seeing: Towards In-hand Dexterity through Touch

  2023influential reference
• Dynamic Handover: Throw and Catch with Bimanual Hands

  2023cited by this paper
• Estimator-Coupled Reinforcement Learning for Robust Purely Tactile In-Hand Manipulation

  2023cited by this paper
• See to Touch: Learning Tactile Dexterity through Visual Incentives

  2023cited by this paper
• General In-Hand Object Rotation with Vision and Touch

  2023cited by this paper
• AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System

  2023cited by this paper
• LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learning

  2023cited by this paper
• In-Hand Object Rotation via Rapid Motor Adaptation

  2022cited by this paper
• Visuo-Tactile Transformers for Manipulation

  2022cited by this paper
• Visuotactile-RL: Learning Multimodal Manipulation Policies with Deep Reinforcement Learning

  2022cited by this paper
• Learning Continuous Grasping Function With a Dexterous Hand From Human Demonstrations

  2022cited by this paper
• From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation From Single-Camera Teleoperation

  2022cited by this paper
• Learning Generalizable Dexterous Manipulation from Human Grasp Affordance

  2022cited by this paper
• Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulation

  2022cited by this paper
• Complex In-Hand Manipulation via Compliance-Enabled Finger Gaiting and Multi-Modal Planning

  2022cited by this paper
• Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube

  2022cited by this paper
• Visual Dexterity: In-hand Dexterous Manipulation from Depth

  2022cited by this paper
• In-Hand Manipulation of Unknown Objects with Tactile Sensing for Insertion

  2022cited by this paper
• HERD: Continuous Human-to-Robot Evolution for Learning from Human Demonstration

  2022cited by this paper
• Learning to Imitate Object Interactions from Internet Videos

  2022cited by this paper
• DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulation

  2022cited by this paper
• DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality

  2022cited by this paper
• Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point Clouds

  2022cited by this paper
• Holo-Dex: Teaching Dexterity with Immersive Mixed Reality

  2022cited by this paper
• DexMV: Imitation Learning for Dexterous Manipulation from Human Videos

  2021cited by this paper
• A System for General In-Hand Object Re-Orientation

  2021cited by this paper
• On the Feasibility of Learning Finger-gaiting In-hand Manipulation with Intrinsic Sensing

  2021cited by this paper
• Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning

  2021cited by this paper
• Surprisingly Robust In-Hand Manipulation: An Empirical Study

  2021cited by this paper
• DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor With Application to In-Hand Manipulation

  2020cited by this paper
• In-Hand Object Pose Tracking via Contact Feedback and GPU-Accelerated Robotic Simulation

  2020cited by this paper
• Trends and challenges in robot manipulation

  2019cited by this paper
• Deep Dynamics Models for Learning Dexterous Manipulation

  2019cited by this paper
• Learning dexterous in-hand manipulation

  2018cited by this paper
• More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch

  2018cited by this paper
• Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks

  2018cited by this paper
• 3D Shape Perception from Monocular Vision, Touch, and Shape Priors

  2018cited by this paper
• Active learning with query paths for tactile object shape exploration

  2017cited by this paper
• GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force

  2017cited by this paper
• Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations

  2017cited by this paper
• Sampling-based Planning of In-Hand Manipulation with External Pushes

  2017cited by this paper
• Proximal Policy Optimization Algorithms

  2017cited by this paper
• Cross-modal visuo-tactile object recognition using robotic active exploration

  2017cited by this paper
• PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

  2016cited by this paper
• Dynamic in-hand sliding manipulation

  2015cited by this paper
• Real-time behaviour synthesis for dynamic hand-manipulation

  2014cited by this paper
• Dexterous manipulation using both palm and fingers

  2014cited by this paper
• On rolling contact motion by robotic fingers via prescribed performance control

  2013cited by this paper
• A sensational illusion: vision-touch synaesthesia and the rubber hand paradigm.

  2013cited by this paper
• The color of touch: A case of tactile–visual synaesthesia

  2012cited by this paper
• Global Localization of Objects via Touch

  2011cited by this paper
• Towards an automatic robot regrasping movement based on human demonstration using tangle topology

  2011cited by this paper
• No-Regret Reductions for Imitation Learning and Structured Prediction

  2010cited by this paper
• Pivoting Manipulation of a Large Object: A Study of Application using Humanoid Platform

  2005cited by this paper
• International Conference on Intelligent Robots and Systems Yokohama , Japan July 2630 , 1993 PIVOTING : A NEW METHOD OF GRASPLESS MANIPULATION OF OBJECT BY ROBOT FINGERS

  2004cited by this paper
• An Overview of Dexterous Manipulation

  1999cited by this paper
• Integrating vision, touch and natural language in the control of a situation-oriented behavior-based humanoid robot

  1999cited by this paper
• Planning quasi-static fingertip manipulations for reconfiguring objects

  1999cited by this paper
• Dextrous manipulation by rolling and finger gaiting

  1998cited by this paper
• Dextrous manipulation with rolling contacts

  1997cited by this paper
• Visual and Somatosensory Information about Object Shape Control Manipulative Fingertip Forces

  1997cited by this paper
• Dexterous manipulation through rolling

  1995cited by this paper
• Dynamic control of sliding by robot hands for regrasping

  1992cited by this paper
• Regrasping

  1987cited by this paper

• Closing the Reality Gap: Zero-Shot Sim-to-Real Deployment for Dexterous Force-Based Grasping and Manipulation

  2026cites this paper
• Compliant In-hand Rolling Manipulation Using Tactile Sensing

  2026cites this paper
• One Hand to Rule Them All: Canonical Representations for Unified Dexterous Manipulation

  2026cites this paper
• Rigidity-Based Multi-Finger Coordination for Precise In-Hand Manipulation of Force-Sensitive Objects

  2026cites this paper
• Why Look at It at All?: Vision-Free Multifingered Blind Grasping Using Uniaxial Fingertip Force Sensing

  2026cites this paper
• Synesthesia of Vehicles: Tactile Data Synthesis from Visual Inputs

  2026cites this paper
• Spatially anchored Tactile Awareness for Robust Dexterous Manipulation

  2025cites this paper
• From Simple to Complex Skills: The Case of In-Hand Object Reorientation

  2025cites this paper
• GelStereo Tip: A Spherical Fingertip Visuotactile Sensor for Multi-Finger Screwing Manipulation

  2025cites this paper
• Subtask-Aware Visual Reward Learning from Segmented Demonstrations

  2025cites this paper
• Reactive Diffusion Policy: Slow-Fast Visual-Tactile Policy Learning for Contact-Rich Manipulation

  2025cites this paper
• Towards Generalization of Tactile Image Generation: Reference-Free Evaluation in a Leakage-Free Setting

  2025cites this paper
• Sequential Multi-Object Grasping with One Dexterous Hand

  2025cites this paper
• Learning Adaptive Dexterous Grasping from Single Demonstrations

  2025cites this paper
• RUKA: Rethinking the Design of Humanoid Hands with Learning

  2025cites this paper
• GeoDEx: A Unified Geometric Framework for Tactile Dexterous and Extrinsic Manipulation under Force Uncertainty

  2025cites this paper
• Zero-shot Sim2Real Transfer for Magnet-Based Tactile Sensor on Insertion Tasks

  2025cites this paper
• Composing Dextrous Grasping and In-Hand Manipulation via Scoring with a Reinforcement Learning Critic

  2025cites this paper
• Adaptive Visuo-Tactile Fusion with Predictive Force Attention for Dexterous Manipulation

  2025cites this paper
• LocoTouch: Learning Dexterous Quadrupedal Transport with Tactile Sensing

  2025cites this paper
• eFlesh: Highly customizable Magnetic Touch Sensing using Cut-Cell Microstructures

  2025cites this paper
• ViTacFormer: Learning Cross-Modal Representation for Visuo-Tactile Dexterous Manipulation

  2025cites this paper
• The MOTIF Hand: A Robotic Hand for Multimodal Observations with Thermal, Inertial, and Force Sensors

  2025cites this paper
• Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper

  2025cites this paper
• FBI: Learning Dexterous In-hand Manipulation with Dynamic Visuotactile Shortcut Policy

  2025influential citation
• LocoTouch: Learning Dynamic Quadrupedal Transport with Tactile Sensing

  2025cites this paper
• VT-Refine: Learning Bimanual Assembly with Visuo-Tactile Feedback via Simulation Fine-Tuning

  2025cites this paper
• Force-Modulated Visual Policy for Robot-Assisted Dressing with Arm Motions

  2025cites this paper
• exUMI: Extensible Robot Teaching System with Action-aware Task-agnostic Tactile Representation

  2025cites this paper
• RealMirror: A Comprehensive, Open-Source Vision-Language-Action Platform for Embodied AI

  2025cites this paper
• DexSkin: High-Coverage Conformable Robotic Skin for Learning Contact-Rich Manipulation

  2025cites this paper
• Learning-Based Collaborative Control for Bi-Manual Tactile-Reactive Grasping

  2025cites this paper
• In-Hand Manipulation of Articulated Tools with Dexterous Robot Hands with Sim-to-Real Transfer

  2025cites this paper
• Multi-Modal Manipulation via Multi-Modal Policy Consensus

  2025cites this paper
• Temporally Masked Diffusion: An Effective Behavioral Cloning Method in Robot Action Sequence Generation

  2025cites this paper
• DexNDM: Closing the Reality Gap for Dexterous In-Hand Rotation via Joint-Wise Neural Dynamics Model

  2025cites this paper
• Enhancing Tactile-based Reinforcement Learning for Robotic Control

  2025cites this paper
• JST Divergence for Better Dreamer Learning in Humanoid Robot Manipulation

  2025cites this paper
• GLINT: An Idle-Stroke Grasp-and-Lift Hand for In-Hand Manipulation

  2025cites this paper
• TARS: Tactile Affordance in Robot Synesthesia for Dexterous Manipulation

  2025cites this paper
• VTAO-BiManip: Masked Visual-Tactile-Action Pre-training with Object Understanding for Bimanual Dexterous Manipulation

  2025cites this paper
• Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding

  2025cites this paper
• Sim2Real Manipulation on Unknown Objects with Tactile-based Reinforcement Learning

  2024cites this paper
• ViViDex: Learning Vision-Based Dexterous Manipulation from Human Videos

  2024cites this paper
• AnyRotate: Gravity-Invariant In-Hand Object Rotation with Sim-to-Real Touch

  2024cites this paper
• RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packing

  2024cites this paper
• Learning In-Hand Translation Using Tactile Skin with Shear and Normal Force Sensing

  2024cites this paper
• Learning a Shape-Conditioned Agent for Purely Tactile In-Hand Manipulation of Various Objects

  2024cites this paper
• Lessons from Learning to Spin "Pens"

  2024influential citation
• In-Hand Singulation and Scooping Manipulation with a 5 DOF Tactile Gripper

  2024cites this paper
• Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrations

  2024cites this paper
• EyeSight Hand: Design of a Fully-Actuated Dexterous Robot Hand with Integrated Vision-Based Tactile Sensors and Compliant Actuation

  2024cites this paper
• Learning to Singulate Objects in Packed Environments using a Dexterous Hand

  2024cites this paper
• AnySkin: Plug-and-Play Skin Sensing for Robotic Touch

  2024cites this paper
• SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting

  2024cites this paper
• MoDex: Planning High-Dimensional Dexterous Control via Learning Neural Hand Models

  2024cites this paper
• Canonical Representation and Force-Based Pretraining of 3D Tactile for Dexterous Visuo-Tactile Policy Learning

  2024cites this paper
• On the Feasibility of A Mixed-Method Approach for Solving Long Horizon Task-Oriented Dexterous Manipulation

  2024cites this paper
• Learning Precise, Contact-Rich Manipulation through Uncalibrated Tactile Skins

  2024cites this paper
• GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policy

  2024cites this paper
• 3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing

  2024cites this paper
• Learning Time-Optimal and Speed-Adjustable Tactile In-Hand Manipulation

  2024cites this paper
• TEXterity: Tactile Extrinsic deXterity

  2024cites this paper
• BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objects

  2024cites this paper
• HandySense: A Multimodal Collection System for Human Two-Handed Dexterous Manipulation

  2024cites this paper
• Multi-Scenario Reasoning: Unlocking Cognitive Autonomy in Humanoid Robots for Multimodal Understanding

  2024cites this paper
• Twisting Lids Off with Two Hands

  2024cites this paper
• Survey of learning-based approaches for robotic in-hand manipulation

  2024cites this paper
• CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulation

  2024cites this paper
• MoDex: Planning High-Dimensional Dexterous Control via Learning Neural Internal Models

  2024cites this paper
• R ×\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\times $$\end{document} R: Rapid eXploration for Reinforcement lear

  2024cites this paper
• HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation

  2024cites this paper
• 3D Diffusion Policy

  2024cites this paper
• Purdue e-Pubs Purdue e-Pubs

  year unknowncites this paper