Curiosity Driven Exploration of Learned Disentangled Goal Spaces

Intrinsically motivated goal exploration processes enable agents to autonomously sample goals to explore efficiently complex environments with high-dimensional continuous actions. They have been applied successfully to real world robots to discover repertoires of policies producing a wide diversity of effects. Often these algorithms relied on engineered goal spaces but it was recently shown that one can use deep representation learning algorithms to learn an adequate goal space in simple environments. However, in the case of more complex environments containing multiple objects or distractors, an efficient exploration requires that the structure of the goal space reflects the one of the environment. In this paper we show that using a disentangled goal space leads to better exploration performances than an entangled goal space. We further show that when the representation is disentangled, one can leverage it by sampling goals that maximize learning progress in a modular manner. Finally, we show that the measure of learning progress, used to drive curiosity-driven exploration, can be used simultaneously to discover abstract independently controllable features of the environment.

• Publication year

  2018

• Venue

  Conference on Robot Learning

• Publication date

  2018-07-04

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 1807.01521
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Variation Network: Learning High-level Attributes for Controlled Input Manipulation

  2019cited by this paper
• From Babies to Robots: The Contribution of Developmental Robotics to Developmental Psychology

  2018cited by this paper
• Computational Theories of Curiosity-Driven Learning

  2018cited by this paper
• Understanding disentangling in β-VAE

  2018cited by this paper
• Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration

  2018influential reference
• Disentangling the independently controllable factors of variation by interacting with the world

  2018cited by this paper
• Zero-Shot Visual Imitation

  2018cited by this paper
• A Laplacian Framework for Option Discovery in Reinforcement Learning

  2017cited by this paper
• Hindsight Experience Replay

  2017cited by this paper
• Intrinsically motivated model learning for developing curious robots

  2017cited by this paper
• Reverse Curriculum Generation for Reinforcement Learning

  2017cited by this paper
• Curiosity-Driven Exploration by Self-Supervised Prediction

  2017cited by this paper
• Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents

  2017cited by this paper
• Independently Controllable Factors

  2017cited by this paper
• GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms

  2017cited by this paper
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets

  2016cited by this paper
• VIME: Variational Information Maximizing Exploration

  2016cited by this paper
• Early Visual Concept Learning with Unsupervised Deep Learning

  2016cited by this paper
• Unifying Count-Based Exploration and Intrinsic Motivation

  2016cited by this paper
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning

  2016cited by this paper
• Behavioral Diversity Generation in Autonomous Exploration through Reuse of Past Experience

  2016cited by this paper
• Curiosity-driven Exploration in Deep Reinforcement Learning via Bayesian Neural Networks

  2016cited by this paper
• beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework

  2016cited by this paper
• Modular active curiosity-driven discovery of tool use

  2016cited by this paper
• Adam: A Method for Stochastic Optimization

  2014cited by this paper
• Robots that can adapt like animals

  2014cited by this paper
• Active Learning of Parameterized Skills

  2014cited by this paper
• Intrinsic Motivation and Reinforcement Learning

  2013cited by this paper
• Active learning of inverse models with intrinsically motivated goal exploration in robots

  2013influential reference
• Intrinsically Motivated Learning in Natural and Artificial Systems

  2013cited by this paper
• Socially guided intrinsic motivation for robot learning of motor skills

  2013cited by this paper
• Representation Learning: A Review and New Perspectives

  2012cited by this paper
• Abandoning Objectives: Evolution Through the Search for Novelty Alone

  2011cited by this paper
• PowerPlay: Training an Increasingly General Problem Solver by Continually Searching for the Simplest Still Unsolvable Problem

  2011cited by this paper
• Goal Babbling Permits Direct Learning of Inverse Kinematics

  2010cited by this paper
• Robotics and autonomous systems

  1988cited by this paper
• Curiosity and exploration.

  1966cited by this paper
• This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION 1 Intrinsic Motivation Systems for Autonomous Mental Development

  year unknowncited by this paper

• A Behavioral Decision-Making Model of Learning and Memory for Mobile Robot Triggered by Curiosity

  2025cites this paper
• Parameter Estimation using Reinforcement Learning Causal Curiosity: Limits and Challenges

  2025cites this paper
• MAGELLAN: Metacognitive predictions of learning progress guide autotelic LLM agents in large goal spaces

  2025cites this paper
• Disentangled Representations for Causal Cognition

  2024cites this paper
• Open-Vocabulary Predictive World Models from Sensor Observations

  2024cites this paper
• Wind Estimation in Unmanned Aerial Vehicles with Causal Machine Learning

  2024cites this paper
• ACES: Generating Diverse Programming Puzzles with Autotelic Language Models and Semantic Descriptors

  2023influential citation
• ACES: Generating Diverse Programming Puzzles with with Autotelic Generative Models

  2023cites this paper
• A Review of Symbolic, Subsymbolic and Hybrid Methods for Sequential Decision Making

  2023cites this paper
• Predictive World Models from Real-World Partial Observations

  2023cites this paper
• Disentangled (Un)Controllable Features

  2022cites this paper
• Hierarchical quality-diversity for online damage recovery

  2022cites this paper
• Symmetry-Based Representations for Artificial and Biological General Intelligence

  2022cites this paper
• Online Damage Recovery for Physical Robots with Hierarchical Quality-Diversity

  2022cites this paper
• DisCo RL: Distribution-Conditioned Reinforcement Learning for General-Purpose Policies

  2021cites this paper
• Curiosity-driven exploration: Diversity of mechanisms and functions

  2021cites this paper
• Curiosity eliminates the exploration-exploitation dilemma

  2021cites this paper
• A Maze Robot Autonomous Navigation Method Based on Curiosity and Reinforcement Learning

  2021cites this paper
• Discovering and Exploiting Sparse Rewards in a Learned Behavior Space

  2021cites this paper
• Provable Rich Observation Reinforcement Learning with Combinatorial Latent States

  2021cites this paper
• DisTop: Discovering a Topological Representation to Learn Diverse and Rewarding Skills

  2021cites this paper
• Against Amnesic Robots: a Developmental Bayesian Optimization Framework enhanced with Past Experiences and Knowledge from Long-Term Memory Against Amnesic Robots: a Developmental Bayesian Optimization Framework enhanced with Past Experiences and Knowledge from Long-Term Memory

  2021cites this paper
• Towards Teachable Autonomous Agents

  2021cites this paper
• Multi-Objective Exploration for Proximal Policy Optimization

  2021cites this paper
• Rule-Based Reinforcement Learning for Efficient Robot Navigation With Space Reduction

  2021cites this paper
• Exploration With Intrinsic Motivation Using Object–Action–Outcome Latent Space

  2021cites this paper
• Intrinsically Motivated Exploration of Learned Goal Spaces

  2021cites this paper
• Vocal Imitation in Sensorimotor Learning Models: A Comparative Review

  2021cites this paper
• A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation

  2020cites this paper
• GLIB: Exploration via Goal-Literal Babbling for Lifted Operator Learning

  2020cites this paper
• Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration

  2020cites this paper
• Automatic Curriculum Learning For Deep RL: A Short Survey

  2020cites this paper
• Flexible and Efficient Long-Range Planning Through Curious Exploration

  2020cites this paper
• Trying AGAIN instead of Trying Longer: Prior Learning for Automatic Curriculum Learning

  2020cites this paper
• DREAM Architecture: a Developmental Approach to Open-Ended Learning in Robotics

  2020cites this paper
• Language-Conditioned Goal Generation: a New Approach to Language Grounding for RL

  2020cites this paper
• A Framework for Reinforcement Learning and Planning

  2020cites this paper
• Model-based Reinforcement Learning: A Survey

  2020cites this paper
• Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes

  2020cites this paper
• Hierarchically-Organized Latent Modules for Exploratory Search in Morphogenetic Systems

  2020influential citation
• Bayesian Optimization for Developmental Robotics with Meta-Learning by Parameters Bounds Reduction

  2020cites this paper
• Generalization to New Actions in Reinforcement Learning

  2020cites this paper
• Intrinsic Motivation in Object-Action-Outcome Blending Latent Space

  2020cites this paper
• GLIB: Efficient Exploration for Relational Model-Based Reinforcement Learning via Goal-Literal Babbling

  2020cites this paper
• GRIMGEP: Learning Progress for Robust Goal Sampling in Visual Deep Reinforcement Learning

  2020influential citation
• Representation Matters: Improving Perception and Exploration for Robotics

  2020cites this paper
• Leveraging Forward Model Prediction Error for Learning Control

  2020cites this paper
• Self-supervised Visual Reinforcement Learning with Object-centric Representations

  2020cites this paper
• Modular deep reinforcement learning from reward and punishment for robot navigation

  2020cites this paper
• Towards Better Understanding of Disentangled Representations via Mutual Information

  2019cites this paper
• From exploration to control: learning object manipulation skills through novelty search and local adaptation

  2019cites this paper
• Continual Learning for Robotics

  2019cites this paper
• Disentangled Cumulants Help Successor Representations Transfer to New Tasks

  2019cites this paper
• Bridging Disentanglement with Independence and Conditional Independence via Mutual Information for Representation Learning

  2019cites this paper
• Contrastive Learning of Structured World Models

  2019cites this paper
• On the Fairness of Disentangled Representations

  2019cites this paper
• Intrinsically Motivated and Interactive Reinforcement Learning: a Developmental Approach. (Apprentissage par Renforcement Intrinsèquement Motivé et Interactif: une approche développementale)

  2019cites this paper
• Unsupervised Model Selection for Variational Disentangled Representation Learning

  2019cites this paper
• COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration

  2019cites this paper
• Mega-Reward: Achieving Human-Level Play without Extrinsic Rewards

  2019cites this paper
• Disentangling Factors of Variation Using Few Labels

  2019cites this paper
• Maximum entropy searching

  2019cites this paper
• Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments

  2019cites this paper
• Curious iLQR: Resolving Uncertainty in Model-based RL

  2019cites this paper
• Learning Independently-Obtainable Reward Functions

  2019cites this paper
• Intrinsically Motivated Discovery of Diverse Patterns in Self-Organizing Systems

  2019influential citation
• Distributional Reward Decomposition for Reinforcement Learning

  2019influential citation
• A survey on intrinsic motivation in reinforcement learning

  2019cites this paper
• Intrinsically Motivated Exploration for Automated Discovery of Patterns in Morphogenetic Systems

  2019cites this paper
• Learning to Explore in Motion and Interaction Tasks

  2019cites this paper
• Embracing curiosity eliminates the exploration-exploitation dilemma

  2019cites this paper
• Curiosity-Driven Multi-Criteria Hindsight Experience Replay

  2019cites this paper
• On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset

  2019cites this paper
• Autonomous Goal Exploration using Learned Goal Spaces for Visuomotor Skill Acquisition in Robots

  2019influential citation
• On the Emergence of Biologically-Plausible Representation in Recurrent Neural Networks for Navigation

  2019cites this paper
• Micro-Data Reinforcement Learning for Adaptive Robots

  2018cites this paper
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations

  2018cites this paper
• Unsupervised Control Through Non-Parametric Discriminative Rewards

  2018cites this paper
• Adversarial Active Exploration for Inverse Dynamics Model Learning

  2018cites this paper
• Policy Search in Continuous Action Domains: an Overview

  2018cites this paper
• Learning Actionable Representations with Goal-Conditioned Policies

  2018cites this paper
• CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning

  2018cites this paper
• Policy Search in Continuous Action Domains: an Overview

  2018cites this paper
• Towards a Definition of Disentangled Representations

  2018cites this paper
• CURIOUS: Intrinsically Motivated Multi-Task, Multi-Goal Reinforcement Learning

  2018cites this paper
• Computational mechanisms of curiosity and goal-directed exploration

  2018cites this paper
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning

  2017cites this paper
• Autotelic LLM-based exploration for goal-conditioned RL

  year unknowncites this paper
• Physics of Life Reviews

  year unknowncites this paper