As computational agents are increasingly used beyond research labs, their success will depend on their ability to learn new skills and adapt to their dynamic, complex environments. If human users---without programming skills---can transfer their task knowledge to agents, learning can accelerate dramatically, reducing costly trials. The tamer framework guides the design of agents whose behavior can be shaped through signals of approval and disapproval, a natural form of human feedback. More recently, tamer+rl was introduced to enable human feedback to augment a traditional reinforcement learning (RL) agent that learns from a Markov decision process's (MDP) reward signal. We address limitations of prior work on tamer and tamer+rl, contributing in two critical directions. First, the four successful techniques for combining human reward with RL from prior tamer+rl work are tested on a second task, and these techniques' sensitivities to parameter changes are analyzed. Together, these examinations yield more general and prescriptive conclusions to guide others who wish to incorporate human knowledge into an RL algorithm. Second, tamer+rl has thus far been limited to a sequential setting, in which training occurs before learning from MDP reward. In this paper, we introduce a novel algorithm that shares the same spirit as tamer+rl but learns simultaneously from both reward sources, enabling the human feedback to come at any time during the reinforcement learning process. We call this algorithm simultaneous tamer+rl. To enable simultaneous learning, we introduce a new technique that appropriately determines the magnitude of the human model's influence on the RL algorithm throughout time and state-action space.
Reinforcement learning from simultaneous human and MDP reward
Published 2012 in Adaptive Agents and Multi-Agent Systems
ABSTRACT
PUBLICATION RECORD
- Publication year
2012
- Venue
Adaptive Agents and Multi-Agent Systems
- Publication date
2012-06-04
- Fields of study
Computer Science
- Identifiers
- External record
- Source metadata
Semantic Scholar
CITATION MAP
EXTRACTION MAP
CLAIMS
CONCEPTS
- human feedback
Human-provided approval or disapproval signals used as a learning signal for the agent.
Aliases: human reward
- human model influence
The estimated strength of the human feedback model's contribution to the RL update process.
- mdp reward
The reward signal supplied by the Markov decision process that the RL agent normally learns from.
Aliases: environment reward
- parameter sensitivity analysis
An analysis of how the performance of a method changes when its parameter settings are varied.
- reinforcement learning
A learning setting in which an agent improves its policy from reward signals.
Aliases: RL
- simultaneous tamer+rl
A variant of tamer+rl designed to learn from human feedback and MDP reward simultaneously during training.
- state-action space
The space of all possible state and action pairs considered by the reinforcement-learning agent.
- tamer framework
A framework for shaping an agent's behavior through approval and disapproval signals from humans.
Aliases: tamer
- tamer+rl
A reinforcement-learning variant that combines human feedback with reward from a Markov decision process.
REFERENCES
Showing 1-22 of 22 references · Page 1 of 1