Algorithmic and Human Teaching of Sequential Decision Tasks

A helpful teacher can significantly improve the learning rate of a learning agent. Teaching algorithms have been formally studied within the field of Algorithmic Teaching. These give important insights into how a teacher can select the most informative examples while teachinga new concept. However the field has so far focused purely on classification tasks. In this paper we introducea novel method for optimally teaching sequential decision tasks. We present an algorithm that automatically selects the set of most informative demonstrations andevaluate it on several navigation tasks. Next, we explore the idea of using this algorithm to produce instructions for humans on how to choose examples when teaching sequential decision tasks. We present a user study that demonstrates the utility of such instructions.

• Publication year

  2012

• Venue

  AAAI Conference on Artificial Intelligence

• Publication date

  2012-07-22

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1609/aaai.v26i1.8333
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• An Introduction to Reinforcement Learning

  2013cited by this paper
• Reasoning in teaching and misleading situations

  2011cited by this paper
• Comparing Action-Query Strategies in Semi-Autonomous Agents

  2011cited by this paper
• How Do Humans Teach: On Curriculum Learning and Teaching Dimension

  2011cited by this paper
• Preference Learning: An Introduction

  2010cited by this paper
• Optimality of human teachers for robot learners

  2010cited by this paper
• Using training regimens to teach expanding function approximators

  2010cited by this paper
• Optimal Bayesian Recommendation Sets and Myopically Optimal Choice Query Sets

  2010cited by this paper
• Recent Developments in Algorithmic Teaching

  2009cited by this paper
• Assisting Transfer-Enabled Machine Learning Algorithms: Leveraging Human Knowledge for Curriculum Design

  2009cited by this paper
• Training parsers by inverse reinforcement learning

  2009cited by this paper
• Active Learning for Reward Estimation in Inverse Reinforcement Learning

  2009cited by this paper
• Active Learning Literature Survey

  2009cited by this paper
• Curriculum learning

  2009cited by this paper
• Teaching Games : Statistical Sampling Assumptions for Learning in Pedagogical Situations

  2008cited by this paper
• Measuring teachability using variants of the teaching dimension

  2008cited by this paper
• Confidence-based policy learning from demonstration using Gaussian mixture models

  2007cited by this paper
• Dogged Learning for Robots

  2007cited by this paper
• Apprenticeship Learning using Inverse Reinforcement Learning and Gradient Methods

  2007cited by this paper
• Near-optimal Nonmyopic Value of Information in Graphical Models

  2005cited by this paper
• A Note on the Budgeted Maximization of Submodular Functions

  2005cited by this paper
• Apprenticeship learning via inverse reinforcement learning

  2004cited by this paper
• Algorithms for Inverse Reinforcement Learning

  2000influential reference
• Reinforcement Learning: An Introduction

  1998cited by this paper
• On Specifying Boolean Functions by Labelled Examples

  1995cited by this paper
• Learning and development in neural networks: the importance of starting small.

  1993cited by this paper
• On the complexity of teaching

  1991cited by this paper
• Robotics and autonomous systems

  1988cited by this paper
• Queries and concept learning

  1988cited by this paper
• On learning Boolean functions

  1987cited by this paper
• An analysis of approximations for maximizing submodular set functions—I

  1978cited by this paper

• How Can Everyday Users Efficiently Teach Robots by Demonstration?

  2025cites this paper
• A survey of communicating robot learning during human-robot interaction

  2024cites this paper
• Observation Interference in Partially Observable Assistance Games

  2024cites this paper
• On the Complexity of Teaching a Family of Linear Behavior Cloning Learners

  2024cites this paper
• Can humans teach machines to code?

  2024influential citation
• Understanding Robot Minds: Leveraging Machine Teaching for Transparent Human-Robot Collaboration Across Diverse Groups

  2024cites this paper
• (X)AI as a Teacher: Learning with Explainable Artificial Intelligence

  2024cites this paper
• Transferring Domain Knowledge with (X)AI-Based Learning Systems

  2024cites this paper
• Interactively Teaching an Inverse Reinforcement Learner with Limited Feedback

  2023influential citation
• A Review of Communicating Robot Learning during Human-Robot Interaction

  2023cites this paper
• Optimally Teaching a Linear Behavior Cloning Agent

  2023cites this paper
• Towards Effective Human-AI Decision-Making: The Role of Human Learning in Appropriate Reliance on AI Advice

  2023cites this paper
• Teaching to Learn: Sequential Teaching of Learners with Internal States

  2023cites this paper
• ML-Based Teaching Systems: A Conceptual Framework

  2023cites this paper
• Towards Modeling and Influencing the Dynamics of Human Learning

  2023cites this paper
• How Can Everyday Users Efficiently Teach Robots by Demonstrations?

  2023cites this paper
• Know Thy Student: Interactive Learning with Gaussian Processes

  2022cites this paper
• Reciprocal MIND MELD: Improving Learning From Demonstration via Personalized, Reciprocal Teaching

  2022cites this paper
• Communicating Robot Conventions through Shared Autonomy

  2022cites this paper
• Autonomous Assessment of Demonstration Sufficiency via Bayesian Inverse Reinforcement Learning

  2022cites this paper
• Counterfactual Examples for Human Inverse Reinforcement Learning

  2022cites this paper
• Learning Curricula for Humans: An Empirical Study with Puzzles from The Witness

  2022cites this paper
• Teaching Multiple Inverse Reinforcement Learners

  2021influential citation
• Bias Adaptive Statistical Inference Learning Agents for Learning from Human Feedback

  2021cites this paper
• Physical interaction as communication: Learning robot objectives online from human corrections

  2021influential citation
• Machine Teaching for Human Inverse Reinforcement Learning

  2021cites this paper
• Gradient-based Algorithms for Machine Teaching

  2021cites this paper
• Curriculum Design for Teaching via Demonstrations: Theory and Applications

  2021cites this paper
• Teaching via Best-Case Counterexamples in the Learning-with-Equivalence-Queries Paradigm

  2021cites this paper
• Hybrid social learning in human-algorithm cultural transmission

  2021cites this paper
• Teaching an Active Learner with Contrastive Examples

  2021cites this paper
• Explaining Reward Functions to Humans for Better Human-Robot Collaboration

  2021cites this paper
• Iterative Teacher-Aware Learning

  2021influential citation
• The Sample Complexity of Teaching by Reinforcement on Q-Learning

  2021cites this paper
• Fair Clustering Using Antidote Data

  2021cites this paper
• Teaching Unknown Learners to Classify via Feature Importance

  2021cites this paper
• AutoPreview: A Framework for Autopilot Behavior Understanding

  2021cites this paper
• Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environments

  2021cites this paper
• Integrating an Observer in Interactive Reinforcement Learning to Learn Legible Trajectories

  2020cites this paper
• Open Problems in Cooperative AI

  2020cites this paper
• The Teaching Dimension of Kernel Perceptron

  2020cites this paper
• Human-in-the-Loop Imitation Learning using Remote Teleoperation

  2020cites this paper
• Value Alignment Verification

  2020cites this paper
• The Teaching Dimension of Q-learning

  2020cites this paper
• Should artificial agents ask for help in human-robot collaborative problem-solving?

  2020cites this paper
• Policy Teaching in Reinforcement Learning via Environment Poisoning Attacks

  2020cites this paper
• Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learning

  2020cites this paper
• Teaching to Learn: Sequential Teaching of Agents with Inner States

  2020cites this paper
• Interaction-limited Inverse Reinforcement Learning

  2020cites this paper
• Using Machine Teaching to Investigate Human Assumptions when Teaching Reinforcement Learners

  2020cites this paper
• Towards the Learning, Perception, and Effectiveness of Teachable Conversational Agents

  2019cites this paper
• Learning to Collaborate in Markov Decision Processes

  2019cites this paper
• Interactive Teaching Algorithms for Inverse Reinforcement Learning

  2019cites this paper
• Learner-aware Teaching: Inverse Reinforcement Learning with Preferences and Constraints

  2019cites this paper
• Ranking-Based Reward Extrapolation without Rankings

  2019cites this paper
• L G ] 2 0 Ju n 20 19 Learning to Collaborate in Markov Decision Processes

  2019cites this paper
• Fake It Till You Make It: Learning-Compatible Performance Support

  2019cites this paper
• Policy Poisoning in Batch Reinforcement Learning and Control

  2019influential citation
• Preference-Based Batch and Sequential Teaching: Towards a Unified View of Models

  2019cites this paper
• End-user Robot Programming in Cobotic Environments

  2019cites this paper
• Machines Learning Trends, Perspectives and Prospects in Education Sector

  2019cites this paper
• Class Teaching for Inverse Reinforcement Learners

  2019influential citation
• Adapt, Explain, Engage—A Study on How Social Robots Can Scaffold Second-language Learning of Children

  2019cites this paper
• Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations

  2019cites this paper
• Optimizing for Robot Transparency

  2019influential citation
• Integrating Socially Assistive Robots into Language Tutoring Systems. A Computational Model for Scaffolding Young Children's Foreign Language Learning

  2019cites this paper
• Teacher Improves Learning by Selecting a Training Subset

  2018cites this paper
• Machine Teaching of Active Sequential Learners

  2018cites this paper
• Understanding the Role of Adaptivity in Machine Teaching: The Case of Version Space Learners

  2018cites this paper
• Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications

  2018influential citation
• An Overview of Machine Teaching

  2018cites this paper
• Simultaneous End-User Programming of Goals and Actions for Robotic Shelf Organization

  2018cites this paper
• Assisted Inverse Reinforcement Learning

  2018cites this paper
• Iterative Classroom Teaching

  2018cites this paper
• Teaching Inverse Reinforcement Learners via Features and Demonstrations

  2018cites this paper
• A User Study on Robot Skill Learning Without a Cost Function: Optimization of Dynamic Movement Primitives via Naive User Feedback

  2018cites this paper
• Including Uncertainty when Learning from Human Corrections

  2018cites this paper
• On-line Dialogue Policy Learning with Companion Teaching

  2017cites this paper
• Enabling robots to communicate their objectives

  2017cites this paper
• Adaptive Robot Language Tutoring Based on Bayesian Knowledge Tracing and Predictive Decision-Making

  2017cites this paper
• Second Language Tutoring using Social Robots

  2017influential citation
• Artificial learners adopting normative conventions from human teachers

  2017cites this paper
• Adaptively selecting biology questions generated from a semantic network

  2017cites this paper
• POLICY ADVICE, NON-CONVEX AND DISTRIBUTED OPTIMIZATION IN REINFORCEMENT LEARNING

  2016cites this paper
• Supportive Behaviors for Human-Robot Teaming

  2016cites this paper
• An Intelligent Robot and Augmented Reality Instruction System

  2016cites this paper
• Showing versus doing: Teaching by demonstration

  2016cites this paper
• Cooperative Inverse Reinforcement Learning

  2016cites this paper
• Exploration from Demonstration for Interactive Reinforcement Learning

  2016cites this paper
• Theoretically-Grounded Policy Advice from Multiple Teachers in Reinforcement Learning Settings with Applications to Negative Transfer

  2016cites this paper
• Becoming the expert - interactive multi-class machine teaching

  2015cites this paper
• Learning behaviors via human-delivered discrete feedback: modeling implicit feedback strategies to speed up learning

  2015cites this paper
• Policy Shaping with Human Teachers

  2015cites this paper
• Language and Policy Learning from Human-delivered Feedback

  2015cites this paper
• Machine Teaching: An Inverse Problem to Machine Learning and an Approach Toward Optimal Education

  2015cites this paper
• Online Transfer Learning in Reinforcement Learning Domains

  2015cites this paper
• A Strategy-Aware Technique for Learning Behaviors from Discrete Human Feedback

  2014cites this paper
• Learning something from nothing: Leveraging implicit human feedback strategies

  2014cites this paper
• Developmental learning for Intelligent Tutoring Systems

  2014cites this paper
• A social learning formalism for learners trying to figure out what a teacher wants them to do

  2014cites this paper