CAVER: Curious Audiovisual Exploring Robot

Multimodal audiovisual perception can enable new avenues for robotic manipulation, from better material classification to the imitation of demonstrations for which only audio signals are available (e.g., playing a tune by ear). However, to unlock such multimodal potential, robots need to learn the correlations between an object's visual appearance and the sound it generates when they interact with it. Such an active sensorimotor experience requires new interaction capabilities, representations, and exploration methods to guide the robot in efficiently building increasingly rich audiovisual knowledge. In this work, we present CAVER, a novel robot that builds and utilizes rich audiovisual representations of objects. CAVER includes three novel contributions: 1) a novel 3D printed end-effector, attachable to parallel grippers, that excites objects'audio responses, 2) an audiovisual representation that combines local and global appearance information with sound features, and 3) an exploration algorithm that uses and builds the audiovisual representation in a curiosity-driven manner that prioritizes interacting with high uncertainty objects to obtain good coverage of surprising audio with fewer interactions. We demonstrate that CAVER builds rich representations in different scenarios more efficiently than several exploration baselines, and that the learned audiovisual representation leads to significant improvements in material classification and the imitation of audio-only human demonstrations. https://caver-bot.github.io/

• Publication year

  2025

• Venue

  arXiv.org

• Publication date

  2025-11-10

• Fields of study

  Computer Science, Engineering

• Identifiers
  DOI 10.48550/arXiv.2511.07619arXiv 2511.07619
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Hearing Anywhere in Any Environment

  2025cited by this paper
• Differentiable Room Acoustic Rendering with Multi-View Vision Priors

  2025cited by this paper
• ForceMimic: Force-Centric Imitation Learning with Force-Motion Capture System for Contact-Rich Manipulation

  2024cited by this paper
• Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes

  2024cited by this paper
• Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulation

  2024cited by this paper
• ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data

  2024cited by this paper
• Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videos

  2024cited by this paper
• Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks

  2024cited by this paper
• DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

  2024cited by this paper
• Machine Learning Framework for Audio-Based Content Evaluation using MFCC, Chroma, Spectral Contrast, and Temporal Feature Engineering

  2024cited by this paper
• Physics-Driven Diffusion Models for Impact Sound Synthesis from Videos

  2023cited by this paper
• Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0

  2023cited by this paper
• Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models

  2023cited by this paper
• The Object Folder Benchmark : Multisensory Learning with Neural and Real Objects

  2023cited by this paper
• Conditional Generation of Audio from Video via Foley Analogies

  2023cited by this paper
• Segment Anything

  2023cited by this paper
• Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection

  2023cited by this paper
• Interactive Object Segmentation in 3D Point Clouds

  2022cited by this paper
• That Sounds Right: Auditory Self-Supervision for Dynamic Robot Manipulation

  2022cited by this paper
• Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learning

  2022cited by this paper
• ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer

  2022cited by this paper
• DiffImpact: Differentiable Rendering and Identification of Impact Sounds

  2021cited by this paper
• Taming Visually Guided Sound Generation

  2021cited by this paper
• Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learning

  2020cited by this paper
• Multimodal Sensor Fusion with Differentiable Filters

  2020cited by this paper
• Generating Visually Aligned Sound From Videos

  2020cited by this paper
• Learning to Generalize Kinematic Models to Novel Objects

  2019cited by this paper
• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms

  2019cited by this paper
• Tactile Regrasp: Grasp Adjustments via Simulated Tactile Transformations

  2018cited by this paper
• Audio-Visual Scene Analysis with Self-Supervised Multisensory Features

  2018cited by this paper
• Autonomous Sensorimotor Learning for Sound Source Localization by a Humanoid Robot

  2018cited by this paper
• Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks

  2018cited by this paper
• Interactive data collection for deep learning object detectors on humanoid robots

  2017cited by this paper
• Object Detection and Recognition for Assistive Robots: Experimentation and Implementation

  2017cited by this paper
• Cross-modal interpretation of multi-modal sensor streams in interactive perception based on coupled recursion

  2017cited by this paper
• Curiosity-Driven Exploration by Self-Supervised Prediction

  2017cited by this paper
• The Curious Robot: Learning Visual Representations via Physical Interactions

  2016cited by this paper
• Interactive Perception: Leveraging Action in Perception and Perception in Action

  2016cited by this paper
• From passive to interactive object learning and recognition through self-identification on a humanoid robot

  2015cited by this paper
• End-to-End Training of Deep Visuomotor Policies

  2015cited by this paper
• Deep Residual Learning for Image Recognition

  2015cited by this paper
• Visually Indicated Sounds

  2015cited by this paper
• Online interactive perception of articulated objects with multi-level recursive estimation based on task-specific priors

  2014cited by this paper
• Learning robot tactile sensing for object manipulation

  2014cited by this paper
• Tracking-based interactive segmentation of textureless objects

  2013cited by this paper
• OctoMap: an efficient probabilistic 3D mapping framework based on octrees

  2013cited by this paper
• Children's Scientific Curiosity: In Search of an Operational Definition of an Elusive Concept.

  2012cited by this paper
• Interactive object recognition using proprioceptive and auditory feedback

  2011cited by this paper
• Active segmentation for robotics

  2009cited by this paper
• Interactive segmentation for manipulation in unstructured environments

  2009cited by this paper
• Sensorimotor coordination in a "baby" robot: learning about objects through grasping.

  2007cited by this paper
• The Development of Embodied Cognition: Six Lessons from Babies

  2005cited by this paper
• Tapping into Touch

  2005cited by this paper
• Rapidly-Exploring Random Trees: Progress and Prospects

  2000cited by this paper
• The psychology of curiosity: A review and reinterpretation.

  1994cited by this paper
• Mel-cepstral distance measure for objective speech quality assessment

  1993cited by this paper
• THE ORIGINS OF INTELLIGENCE IN CHILDREN

  1976cited by this paper
• Recognition

  1902cited by this paper

• No citing papers are available for this paper.