LTL2Action: Generalizing LTL Instructions for Multi-Task RL

We address the problem of teaching a deep reinforcement learning (RL) agent to follow instructions in multi-task environments. Instructions are expressed in a well-known formal language -- linear temporal logic (LTL) -- and can specify a diversity of complex, temporally extended behaviours, including conditionals and alternative realizations. Our proposed learning approach exploits the compositional syntax and the semantics of LTL, enabling our RL agent to learn task-conditioned policies that generalize to new instructions, not observed during training. To reduce the overhead of learning LTL semantics, we introduce an environment-agnostic LTL pretraining scheme which improves sample-efficiency in downstream environments. Experiments on discrete and continuous domains target combinatorial task sets of up to $\sim10^{39}$ unique tasks and demonstrate the strength of our approach in learning to solve (unseen) tasks, given LTL instructions.

• Publication year

  2021

• Venue

  International Conference on Machine Learning

• Publication date

  2021-02-13

• Fields of study

  Computer Science

• Identifiers
  arXiv 2102.06858
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• LTL2Action: Generalizing LTL Instructions for Multi-Task RL (Appendix)

  2021cited by this paper
• Grounded Language Learning Fast and Slow

  2020cited by this paper
• Learning a natural-language to LTL executable semantic parser for grounded robotics

  2020cited by this paper
• Learning Branching Heuristics for Propositional Model Counting

  2020cited by this paper
• Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning

  2020cited by this paper
• Temporal-Logic-Based Reward Shaping for Continuing Learning Tasks

  2020cited by this paper
• Systematic Generalisation through Task Temporal Logic and Deep Reinforcement Learning

  2020influential reference
• Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulas

  2020cited by this paper
• Temporal Logic Monitoring Rewards via Transducers

  2020cited by this paper
• Restraining Bolts for Reinforcement Learning Agents

  2020cited by this paper
• Program Guided Agent

  2020influential reference
• A Composable Specification Language for Reinforcement Learning Tasks

  2020influential reference
• Deep Reinforcement Learning with Temporal Logics

  2020cited by this paper
• Automated synthesis of decentralized controllers for robot swarms from high-level temporal logic specifications

  2019cited by this paper
• Transfer of Temporal Logic Formulas in Reinforcement Learning

  2019cited by this paper
• A Survey of Reinforcement Learning Informed by Natural Language

  2019cited by this paper
• LTL and Beyond: Formal Languages for Reward Function Specification in Reinforcement Learning

  2019cited by this paper
• Language as an Abstraction for Hierarchical Deep Reinforcement Learning

  2019cited by this paper
• Planning With Uncertain Specifications (PUnS)

  2019cited by this paper
• Benchmarking Safe Exploration in Deep Reinforcement Learning

  2019cited by this paper
• Synthesis of LTL Formulas from Natural Language Texts: State of the Art and Research Directions

  2019cited by this paper
• Compositional generalization through meta sequence-to-sequence learning

  2019cited by this paper
• Modular Deep Reinforcement Learning with Temporal Logic Specifications

  2019cited by this paper
• Learning Loop Invariants for Program Verification

  2018cited by this paper
• Interactive Grounded Language Acquisition and Generalization in a 2D World

  2018cited by this paper
• Learning a SAT Solver from Single-Bit Supervision

  2018cited by this paper
• Logically-Constrained Reinforcement Learning

  2018cited by this paper
• Teaching Multiple Tasks to an RL Agent using LTL

  2018influential reference
• Using Reward Machines for High-Level Task Specification and Decomposition in Reinforcement Learning

  2018cited by this paper
• Learning to Understand Goal Specifications by Modelling Reward

  2018cited by this paper
• Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependencies

  2018influential reference
• Bayesian Inference of Temporal Task Specifications from Demonstrations

  2018cited by this paper
• Quantifying Generalization in Reinforcement Learning

  2018cited by this paper
• A Policy Search Method For Temporal Logic Specified Reinforcement Learning Tasks

  2017cited by this paper
• Interpretable apprenticeship learning with temporal logic specifications

  2017cited by this paper
• Proximal Policy Optimization Algorithms

  2017influential reference
• Grounded Language Learning in a Simulated 3D World

  2017cited by this paper
• Environment-Independent Task Specifications via GLTL

  2017cited by this paper
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

  2017cited by this paper
• Modeling Relational Data with Graph Convolutional Networks

  2017cited by this paper
• Neural Task Programming: Learning to Generalize Across Hierarchical Tasks

  2017cited by this paper
• Decentralized control of robotic swarms from high-level temporal logic specifications

  2017cited by this paper
• Learning to Represent Programs with Graphs

  2017cited by this paper
• Gated-Attention Architectures for Task-Oriented Language Grounding

  2017cited by this paper
• Decision-Making with Non-Markovian Rewards: From LTL to automata-based reward shaping

  2017cited by this paper
• Programmable Agents

  2017cited by this paper
• Reinforcement learning with temporal logic rewards

  2016cited by this paper
• Spot 2 . 0 — a framework for LTL and ω-automata manipulation

  2016cited by this paper
• Modular Multitask Reinforcement Learning with Policy Sketches

  2016influential reference
• Q-Learning for robust satisfaction of signal temporal logic specifications

  2016cited by this paper
• Temporal logic robot mission planning for slow and fast actions

  2012cited by this paper
• A Task Specification Language for Bootstrap Learning

  2009cited by this paper
• What to do and how to do it: Translating natural language directives into temporal and dynamic logic representation for goal management and action execution

  2009cited by this paper
• The Graph Neural Network Model

  2009cited by this paper
• Principles of model checking

  2008cited by this paper
• A new model for learning in graph domains

  2005cited by this paper
• Using temporal logics to express search control knowledge for planning

  2000influential reference
• Model Checking of Safety Properties

  1999cited by this paper
• Temporal Logic of Programs

  1987cited by this paper
• Programs with common sense

  1960cited by this paper

• Specification-Guided Reinforcement Learning

  2026cites this paper
• Integrating LTL Constraints into PPO for Safe Reinforcement Learning

  2026cites this paper
• Grounding LTL Tasks in Sub-Symbolic RL Environments for Zero-Shot Generalization

  2026influential citation
• Semantically Labelled Automata for Multi-Task Reinforcement Learning with LTL Instructions

  2026influential citation
• DeepDFA: Injecting Temporal Logic in Deep Learning for Sequential Subsymbolic Applications

  2026cites this paper
• Probabilistic Performance Guarantees for Multi-Task Reinforcement Learning

  2026cites this paper
• PlatoLTL: Learning to Generalize Across Symbols in LTL Instructions for Multi-Task RL

  2026cites this paper
• Formal Methods in Robot Policy Learning and Verification: A Survey on Current Techniques and Future Directions

  2026cites this paper
• Beyond Sliding Windows: Learning to Manage Memory in Non-Markovian Environments

  2025cites this paper
• General agents need world models

  2025cites this paper
• Zero-Shot Instruction Following in RL via Structured LTL Representations

  2025influential citation
• Automating the Refinement of Reinforcement Learning Specifications

  2025cites this paper
• Beyond Fixed Tasks: Unsupervised Environment Design for Task-Level Pairs

  2025influential citation
• ETELTLf: A Recursive Embedding Approach for LTLf Satisfiability Checking in CPSs

  2025cites this paper
• Automata-Conditioned Cooperative Multi-Agent Reinforcement Learning

  2025cites this paper
• Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Data

  2025cites this paper
• Automaton Constrained Q-Learning

  2025influential citation
• TGPO: Temporal Grounded Policy Optimization for Signal Temporal Logic Tasks

  2025cites this paper
• A Novel Task-Driven Diffusion-Based Policy with Affordance Learning for Generalizable Manipulation of Articulated Objects

  2025cites this paper
• One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learning

  2025influential citation
• Scenario-Free Autonomous Driving With Multi-Task Offline-to-Online Reinforcement Learning

  2025cites this paper
• General agents contain world models

  2025cites this paper
• Learning with Expert Abstractions for Efficient Multi-Task Continuous Control

  2025influential citation
• Provably Correct Automata Embeddings for Optimal Automata-Conditioned Reinforcement Learning

  2025cites this paper
• Learning Reward Machines from Partially Observed Policies

  2025cites this paper
• Learning to Check LTL Satisfiability and to Generate Traces via Differentiable Trace Checking

  2024cites this paper
• Planning with a Learned Policy Basis to Optimally Solve Complex Tasks

  2024influential citation
• LTL-Constrained Policy Optimization with Cycle Experience Replay

  2024influential citation
• Inductive Generalization in Reinforcement Learning from Specifications

  2024cites this paper
• LTLDoG: Satisfying Temporally-Extended Symbolic Constraints for Safe Diffusion-Based Planning

  2024cites this paper
• Belief-State Query Policies for User-Aligned POMDPs

  2024cites this paper
• Reward Machines for Deep RL in Noisy and Uncertain Environments

  2024cites this paper
• Temporal Logic Guided Affordance Learning for Generalizable Dexterous Manipulation

  2024cites this paper
• Scalable Signal Temporal Logic Guided Reinforcement Learning via Value Function Space Optimization

  2024cites this paper
• Skill Transfer for Temporal Task Specification

  2024cites this paper
• Projection-Based Fast and Safe Policy Optimization for Reinforcement Learning

  2024cites this paper
• Neural Reward Machines

  2024cites this paper
• Directed Exploration in Reinforcement Learning from Linear Temporal Logic

  2024cites this paper
• Generalization of temporal logic tasks via future dependent options

  2024influential citation
• Diffusion Meets Options: Hierarchical Generative Skill Composition for Temporally-Extended Tasks

  2024influential citation
• DeepLTL: Learning to Efficiently Satisfy Complex LTL Specifications

  2024influential citation
• Generalization of Compositional Tasks with Logical Specification via Implicit Planning

  2024influential citation
• BlendRL: A Framework for Merging Symbolic and Neural Policy Learning

  2024cites this paper
• Compositional Automata Embeddings for Goal-Conditioned Reinforcement Learning

  2024influential citation
• Guiding Multiagent Multitask Reinforcement Learning by a Hierarchical Framework With Logical Reward Shaping

  2024cites this paper
• Exploiting Hybrid Policy in Reinforcement Learning for Interpretable Temporal Logic Manipulation

  2024cites this paper
• Robust Reinforcement Learning for Linear Temporal Logic Specifications with Finite Trajectory Duration

  2024cites this paper
• A Value Function Space Approach for Hierarchical Planning With Signal Temporal Logic Tasks

  2024cites this paper
• Instructing Goal-Conditioned Reinforcement Learning Agents with Temporal Logic Objectives

  2023influential citation
• Data-Driven Safe Policy Optimization for Black-Box Dynamical Systems With Temporal Logic Specifications

  2023cites this paper
• Creative Agents: Empowering Agents with Imagination for Creative Tasks

  2023cites this paper
• Simple Embodied Language Learning as a Byproduct of Meta-Reinforcement Learning

  2023cites this paper
• MER: Modular Element Randomization for robust generalizable policy in deep reinforcement learning

  2023cites this paper
• Compositional Policy Learning in Stochastic Control Systems with Formal Guarantees

  2023cites this paper
• Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learning

  2023cites this paper
• Reinforcement Learning of Action and Query Policies With LTL Instructions Under Uncertain Event Detector

  2023influential citation
• Task-Driven Reinforcement Learning With Action Primitives for Long-Horizon Manipulation Skills

  2023influential citation
• Learning Belief Representations for Partially Observable Deep RL

  2023cites this paper
• Robust Subtask Learning for Compositional Generalization

  2023cites this paper
• Contextual Pre-Planning on Reward Machine Abstractions for Enhanced Transfer in Deep Reinforcement Learning

  2023cites this paper
• Eventual Discounting Temporal Logic Counterfactual Experience Replay

  2023cites this paper
• Exploiting Contextual Structure to Generate Useful Auxiliary Tasks

  2023cites this paper
• Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Survey

  2023cites this paper
• Safety-Constrained Policy Transfer with Successor Features

  2022cites this paper
• Learning to Follow Instructions in Text-Based Games

  2022cites this paper
• Integrating Symbolic Planning and Reinforcement Learning for Following Temporal Logic Specifications

  2022cites this paper
• Exploiting Transformer in Reinforcement Learning for Interpretable Temporal Logic Motion Planning

  2022influential citation
• Noisy Symbolic Abstractions for Deep RL: A case study with Reward Machines

  2022cites this paper
• Generalizing LTL Instructions via Future Dependent Options

  2022influential citation
• Skill Machines: Temporal Logic Skill Composition in Reinforcement Learning

  2022cites this paper
• Checking LTL Satisfiability via End-to-end Learning

  2022influential citation
• Overcoming Exploration: Deep Reinforcement Learning for Continuous Control in Cluttered Environments From Temporal Logic Specifications

  2022cites this paper
• Neural Controller Synthesis for Signal Temporal Logic Specifications Using Encoder-Decoder Structured Networks

  2022cites this paper
• Categorical semantics of compositional reinforcement learning

  2022cites this paper
• Goal-Conditioned Q-Learning as Knowledge Distillation

  2022cites this paper
• Robust Option Learning for Adversarial Generalization

  2022cites this paper
• Real-Time Heuristic Search with LTLf Goals

  2022cites this paper
• Constrained Training of Neural Networks via Theorem Proving

  2022cites this paper
• Exploring Long-Horizon Reasoning with Deep RL in Combinatorially Hard Tasks

  2022cites this paper
• Policy Optimization with Linear Temporal Logic Constraints

  2022cites this paper
• LTL-Transfer: Skill Transfer for Temporal Task Specification

  2022cites this paper
• Challenges to Solving Combinatorially Hard Long-Horizon Deep RL Tasks

  2022cites this paper
• Learning Deterministic Finite Automata Decompositions from Examples and Demonstrations

  2022cites this paper
• A Framework for Following Temporal Logic Instructions with Unknown Causal Dependencies

  2022cites this paper
• Overcoming Exploration: Deep Reinforcement Learning in Complex Environments from Temporal Logic Specifications

  2022influential citation
• Leveraging class abstraction for commonsense reinforcement learning via residual policy gradient methods

  2022cites this paper
• Deep Reinforcement Learning Under Signal Temporal Logic Constraints Using Lagrangian Relaxation

  2022cites this paper
• Exploiting Transformer in Sparse Reward Reinforcement Learning for Interpretable Temporal Logic Motion Planning

  2022influential citation
• Overcoming Exploration: Deep Reinforcement Learning for Continuous Navigation in Complex Environments from Temporal Logic Specifications

  2022cites this paper
• Compositional Reinforcement Learning from Logical Specifications

  2021cites this paper
• Compositional RL Agents That Follow Language Commands in Temporal Logic

  2021cites this paper
• Safe-Critical Modular Deep Reinforcement Learning with Temporal Logic through Gaussian Processes and Control Barrier Functions

  2021cites this paper
• In a Nutshell, the Human Asked for This: Latent Goals for Following Temporal Specifications

  2021influential citation
• Towards Continual Reinforcement Learning: A Review and Perspectives

  2020cites this paper
• Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning

  2020influential citation
• Symbolic AI and Big Data

  year unknowncites this paper
• Finding the FrameStack: Learning What to Remember for Non-Markovian Reinforcement Learning

  year unknowncites this paper
• Automata Conditioned Reinforcement Learning with Experience Replay

  year unknowncites this paper
• Instruction Following in Text-Based Games

  year unknowncites this paper
• Relational Deep Reinforcement Learning and Latent Goals for Following Instructions in Temporal Logic

  year unknowninfluential citation