BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning

In this paper, we study the problem of enabling a vision-based robotic manipulation system to generalize to novel tasks, a long-standing challenge in robot learning. We approach the challenge from an imitation learning perspective, aiming to study how scaling and broadening the data collected can facilitate such generalization. To that end, we develop an interactive and flexible imitation learning system that can learn from both demonstrations and interventions and can be conditioned on different forms of information that convey the task, including pre-trained embeddings of natural language or videos of humans performing the task. When scaling data collection on a real robot to more than 100 distinct tasks, we find that this system can perform 24 unseen manipulation tasks with an average success rate of 44%, without any robot demonstrations for those tasks.

• Publication year

  2022

• Venue

  Conference on Robot Learning

• Publication date

  2022-02-04

• Fields of study

  Computer Science, Engineering

• Identifiers
  arXiv 2202.02005
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Zero-shot Task Adaptation using Natural Language

  2021cited by this paper
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale

  2021cited by this paper
• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills

  2021cited by this paper
• Transformers for One-Shot Visual Imitation

  2020cited by this paper
• RetinaGAN: An Object-aware Approach to Sim-to-Real Transfer

  2020cited by this paper
• Grounding Language in Play

  2020cited by this paper
• Language-Conditioned Imitation Learning for Robot Manipulation Tasks

  2020cited by this paper
• Visual Imitation Made Easy

  2020cited by this paper
• Learning from Interventions: Human-robot interaction as both explicit and implicit feedback

  2020cited by this paper
• Action Image Representation: Learning Scalable Deep Grasping Policies with Zero Real World Data

  2020cited by this paper
• RoboNet: Large-Scale Multi-Robot Learning

  2019cited by this paper
• Watch, Try, Learn: Meta-Learning from Demonstrations and Reward

  2019cited by this paper
• Causal Confusion in Imitation Learning

  2019cited by this paper
• Active Domain Randomization

  2019cited by this paper
• End-to-End Learning of Visual Representations From Uncurated Instructional Videos

  2019cited by this paper
• Learning One-Shot Imitation From Humans Without Humans

  2019cited by this paper
• Multilingual Universal Sentence Encoder for Semantic Retrieval

  2019cited by this paper
• One-Shot High-Fidelity Imitation: Training Large-Scale Deep Nets with RL

  2018cited by this paper
• One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning

  2018influential reference
• VR-Goggles for Robots: Real-to-Sim Domain Adaptation for Visual Control

  2018cited by this paper
• Zero-Shot Visual Imitation

  2018cited by this paper
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation

  2018cited by this paper
• Representation Learning with Contrastive Predictive Coding

  2018cited by this paper
• Neural Task Graphs: Generalizing to Unseen Tasks From a Single Video Demonstration

  2018cited by this paper
• Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias

  2018cited by this paper
• HG-DAgger: Interactive Imitation Learning with Human Experts

  2018influential reference
• Task-Embedded Control Networks for Few-Shot Imitation Learning

  2018influential reference
• Sim-To-Real via Sim-To-Sim: Data-Efficient Robotic Grasping via Randomized-To-Canonical Adaptation Networks

  2018cited by this paper
• Sim2Real View Invariant Visual Servoing by Recurrent Control

  2017cited by this paper
• DART: Noise Injection for Robust Imitation Learning

  2017cited by this paper
• Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics

  2017cited by this paper
• Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-to-End Learning from Demonstration

  2017cited by this paper
• Domain Randomization and Generative Models for Robotic Grasping

  2017cited by this paper
• Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions

  2017cited by this paper
• Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation

  2017cited by this paper
• mixup: Beyond Empirical Risk Minimization

  2017cited by this paper
• One-Shot Imitation Learning

  2017cited by this paper
• FiLM: Visual Reasoning with a General Conditioning Layer

  2017influential reference
• One-Shot Visual Imitation Learning via Meta-Learning

  2017cited by this paper
• Deep visual foresight for planning robot motion

  2016cited by this paper
• Identity Mappings in Deep Residual Networks

  2016cited by this paper
• Supersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours

  2015cited by this paper
• ALVINN, an autonomous land vehicle in a neural network

  2015cited by this paper
• Reinforcement and Imitation Learning via Interactive No-Regret Learning

  2014cited by this paper
• Learning to select and generalize striking movements in robot table tennis

  2012cited by this paper
• A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

  2010cited by this paper
• Learning collaborative manipulation tasks by demonstration using a haptic interface

  2009cited by this paper
• Learning and generalization of motor skills by learning from demonstration

  2009cited by this paper
• Learning to Walk through Imitation

  2007cited by this paper
• Learning Movement Primitives

  2005cited by this paper
• Discovering optimal imitation strategies

  2004cited by this paper
• Robotics and autonomous systems

  1988cited by this paper
• On robust estimation of the location parameter

  1980cited by this paper
• UvA-DARE (Digital Academic Repository) Self-Supervised Video Representation Learning With Odd-One-Out Networks

  year unknowncited by this paper

• On scientific foundation models: Rigorous definitions, key applications, and a comprehensive survey.

  2026cites this paper
• Deep fine-grained clustering with model reusing.

  2026cites this paper
• LeRobot: An Open-Source Library for End-to-End Robot Learning

  2026cites this paper
• Self-Expert Imitation With Purifying Latent Feature for Generalization in Visual Reinforcement Learning

  2026cites this paper
• GraspLDP: Towards Generalizable Grasping Policy via Latent Diffusion

  2026cites this paper
• SOP: A Scalable Online Post-Training System for Vision-Language-Action Models

  2026cites this paper
• Viewpoint Matters: Dynamically Optimizing Viewpoints with Masked Autoencoder for Visual Manipulation

  2026cites this paper
• Demystifying Action Space Design for Robotic Manipulation Policies

  2026cites this paper
• ExoGS: A 4D Real-to-Sim-to-Real Framework for Scalable Manipulation Data Collection

  2026cites this paper
• Scheduling Adaptive Imitation Learning for Long-Horizon Dexterous Robot Micromanipulation of Deformable Cell

  2026cites this paper
• A survey of robotic manipulation: From bottom-up approaches to end-to-end paradigms with LLMs

  2026cites this paper
• Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learning

  2026cites this paper
• RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalization

  2026cites this paper
• A Survey on Deep Generative Models for Robot Learning From Multimodal Demonstrations

  2026cites this paper
• CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videos

  2026cites this paper
• Efficient Long-Horizon Vision-Language-Action Models via Static-Dynamic Disentanglement

  2026influential citation
• TRTP: a three-stage robust task planning framework for open worlds via visual-language models and digital twin simulation

  2026cites this paper
• Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons

  2026cites this paper
• Scaling World Model for Hierarchical Manipulation Policies

  2026cites this paper
• Sim-and-Human Co-training for Data-Efficient and Generalizable Robotic Manipulation

  2026cites this paper
• Large-scale multimodal model based embodied intelligent robots: A survey

  2026cites this paper
• MGAD-FMDM: A Modular Robotic Multitask Imitation Learning Method

  2026cites this paper
• ActiveVLA: Injecting Active Perception into Vision-Language-Action Models for Precise 3D Robotic Manipulation

  2026influential citation
• RoboPaint: From Human Demonstration to Any Robot and Any View

  2026cites this paper
• LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfer

  2026cites this paper
• Primary-Fine Decoupling for Action Generation in Robotic Imitation

  2026cites this paper
• GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learning

  2026cites this paper
• Contact-Anchored Policies: Contact Conditioning Creates Strong Robot Utility Models

  2026cites this paper
• Grounded Vision-Language Interpreter for Integrated Task and Motion Planning

  2025cites this paper
• Text-to-Decision Agent: Offline Meta-Reinforcement Learning from Natural Language Supervision

  2025influential citation
• Learning a thousand tasks in a day

  2025influential citation
• VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning

  2025cites this paper
• RH20T-P: A Primitive-Level Robotic Manipulation Dataset towards Composable Generalization Agents in Real-world Scenarios

  2025cites this paper
• OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation

  2025cites this paper
• BEAST: Efficient Tokenization of B-Splines Encoded Action Sequences for Imitation Learning

  2025influential citation
• OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoning

  2025cites this paper
• DataMIL: Selecting Data for Robot Imitation Learning with Datamodels

  2025cites this paper
• H2R: A Human-to-Robot Data Augmentation for Robot Pre-training from Videos

  2025cites this paper
• Benchmarking Vision, Language, & Action Models in Procedurally Generated, Open Ended Action Environments

  2025cites this paper
• ZAS-F: A Zero-Shot Abstract Sub-Goal Framework Empowers Robots for Long Horizontal Inventory Tasks

  2025cites this paper
• X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real

  2025cites this paper
• L2D2: Robot Learning from 2D drawings

  2025cites this paper
• BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Models

  2025influential citation
• OPAL: Encoding Causal Understanding of Physical Systems for Robot Learning

  2025cites this paper
• Slot-Level Robotic Placement via Visual Imitation from Single Human Video

  2025cites this paper
• Joint Action Language Modelling for Transparent Policy Execution

  2025cites this paper
• CIVIL: Causal and Intuitive Visual Imitation Learning

  2025cites this paper
• TesserAct: Learning 4D Embodied World Models

  2025cites this paper
• LAV-ACT: Language-Augmented Visual Action Chunking with Transformers for Bimanual Robotic Manipulation

  2025cites this paper
• UniVLA: Learning to Act Anywhere with Task-centric Latent Actions

  2025influential citation
• UniSkill: Imitating Human Videos via Cross-Embodiment Skill Representations

  2025cites this paper
• RobotMover: Learning to Move Large Objects From Human Demonstrations

  2025cites this paper
• X2C: A Dataset Featuring Nuanced Facial Expressions for Realistic Humanoid Imitation

  2025cites this paper
• ReWiND: Language-Guided Rewards Teach Robot Policies without New Demonstrations

  2025cites this paper
• APEX: Empowering LLMs with Physics-Based Task Planning for Real-time Insight

  2025cites this paper
• CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning

  2025cites this paper
• ScanBot: Towards Intelligent Surface Scanning in Embodied Robotic Systems

  2025cites this paper
• Bootstrapping Imitation Learning for Long-horizon Manipulation via Hierarchical Data Collection Space

  2025cites this paper
• EgoZero: Robot Learning from Smart Glasses

  2025cites this paper
• This&That: Language-Gesture Controlled Video Generation for Robot Planning

  2025cites this paper
• Visual Embodied Brain: Let Multimodal Large Language Models See, Think, and Control in Spaces

  2025cites this paper
• Fast-in-Slow: A Dual-System Foundation Model Unifying Fast Manipulation within Slow Reasoning

  2025influential citation
• Long-Horizon Language-Conditioned Imitation Learning for Robotic Manipulation

  2025cites this paper
• Dense Policy: Bidirectional Autoregressive Learning of Actions

  2025cites this paper
• RoboFlamingo-Plus: Fusion of Depth and RGB Perception with Vision-Language Models for Enhanced Robotic Manipulation

  2025cites this paper
• Towards Forceful Robotic Foundation Models: a Literature Survey

  2025cites this paper
• Training-Free Task Planning by Parsing Language Signals With Common Sense

  2025cites this paper
• KUDA: Keypoints to Unify Dynamics Learning and Visual Prompting for Open-Vocabulary Robotic Manipulation

  2025cites this paper
• AutoSkill: Hierarchical Open-Ended Skill Acquisition for Long-Horizon Manipulation Tasks via Language-Modulated Rewards

  2025cites this paper
• Zero-Shot Peg Insertion: Identifying Mating Holes and Estimating SE(2) Poses with Vision-Language Models

  2025cites this paper
• dARt Vinci: Egocentric Data Collection for Surgical Robot Learning at Scale

  2025cites this paper
• VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation

  2025cites this paper
• Image-Goal Navigation Using Refined Feature Guidance and Scene Graph Enhancement

  2025cites this paper
• Human2bot: learning zero-shot reward functions for robotic manipulation from human demonstrations

  2025cites this paper
• ROCKET-2: Steering Visuomotor Policy via Cross-View Goal Alignment

  2025cites this paper
• Data Augmentation for Instruction Following Policies via Trajectory Segmentation

  2025cites this paper
• A2Perf: Real-World Autonomous Agents Benchmark

  2025cites this paper
• Phantom: Training Robots Without Robots Using Only Human Videos

  2025cites this paper
• SRSA: Skill Retrieval and Adaptation for Robotic Assembly Tasks

  2025cites this paper
• VLA Model-Expert Collaboration for Bi-directional Manipulation Learning

  2025cites this paper
• Look Before You Leap: Using Serialized State Machine for Language Conditioned Robotic Manipulation

  2025cites this paper
• AgiBot World Colosseo: A Large-Scale Manipulation Platform for Scalable and Intelligent Embodied Systems

  2025cites this paper
• iManip: Skill-Incremental Learning for Robotic Manipulation

  2025cites this paper
• RoboCopilot: Human-in-the-loop Interactive Imitation Learning for Robot Manipulation

  2025cites this paper
• HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Model

  2025cites this paper
• Integrating Failures in Robot Skill Acquisition with Offline Action-Sequence Diffusion RL

  2025cites this paper
• TASTE-Rob: Advancing Video Generation of Task-Oriented Hand-Object Interaction for Generalizable Robotic Manipulation

  2025cites this paper
• SharedAssembly: A Data Collection Approach via Shared Tele-Assembly

  2025cites this paper
• ReBot: Scaling Robot Learning with Real-to-Sim-to-Real Robotic Video Synthesis

  2025cites this paper
• Robotic Paper Wrapping by Learning Force Control

  2025cites this paper
• Learning Adaptive Dexterous Grasping from Single Demonstrations

  2025cites this paper
• R900: Understanding the Cost-Effectiveness of Random Exploration from 900 Hours of Robotic Data Collection

  2025cites this paper
• Dexterous Manipulation Through Imitation Learning: A Survey

  2025cites this paper
• Tool-as-Interface: Learning Robot Policies from Human Tool Usage through Imitation Learning

  2025cites this paper
• RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins

  2025influential citation
• Text-to-Decision Agent: Learning Generalist Policies from Natural Language Supervision

  2025influential citation
• UTTG_ A Universal Teleoperation Approach via Online Trajectory Generation

  2025cites this paper
• GPA-RAM: Grasp-Pretraining Augmented Robotic Attention Mamba for Spatial Task Learning

  2025cites this paper
• Interleave-VLA: Enhancing Robot Manipulation with Interleaved Image-Text Instructions

  2025cites this paper
• A Taxonomy for Evaluating Generalist Robot Policies

  2025cites this paper