Evolutionary Policy Transfer and Search Methods for Boosting Behavior Quality: RoboCup Keep-Away Case Study

This study evaluates various evolutionary search methods to direct neural controller evolution in company with policy (behavior) transfer across increasingly complex collective robotic (RoboCup keep-away) tasks. That is, where robot behaviors are first evolved in a source task and then transferred for further evolution to more complex target tasks. Evolutionary search methods tested include objective-based search (fitness function), behavioral and genotypic diversity maintenance and hybrids of such diversity maintenance and objective-based search. Evolved behavior quality is evaluated according to effectiveness and efficiency. Effectiveness is the average task performance of transferred and evolved behaviors, where task performance is the average time the ball is controlled by a keeper team. Efficiency is the average number of generations taken for the fittest evolved behaviors to reach a minimum task performance threshold given policy transfer. Results indicate that policy transfer coupled with evolutionary search directed by hybridized behavioral diversity maintenance and objective-based search addresses the bootstrapping problem for increasingly complex keep-away tasks, in that this method evolves collective behaviors that could not be evolved by comparative evolutionary methods (with and without policy transfer).

• Publication year

  2017

• Venue

  Frontiers in Robotics and AI

• Publication date

  2017-11-29

• Fields of study

  Computer Science, Engineering

• Identifiers
  DOI 10.3389/frobt.2017.00062
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Introduction to Reinforcement Learning

  2020cited by this paper
• Novelty-Driven Cooperative Coevolution

  2017influential reference
• Maintaining Genetic Diversity in Multimodal Evolutionary Algorithms using Population Injection

  2016cited by this paper
• Multi-agent Behavior-Based Policy Transfer

  2016influential reference
• Validating the Grid Diversity Operator: An Infusion Technique for Diversity Maintenance in Population-Based Optimisation Algorithms

  2016cited by this paper
• Distributed Age-Layered Novelty Search

  2016cited by this paper
• Comparison of Constraint-Handling Mechanisms for the (1,λ)-ES on a Simple Constrained Problem

  2016cited by this paper
• Hybridizing novelty search for transfer learning

  2016cited by this paper
• Evolution of Collective Behaviors for a Real Swarm of Aquatic Surface Robots

  2015cited by this paper
• Evolutionary Robotics: What, Why, and Where to

  2015cited by this paper
• Unsupervised Cross-Domain Transfer in Policy Gradient Reinforcement Learning via Manifold Alignment

  2015cited by this paper
• Devising Effective Novelty Search Algorithms: A Comprehensive Empirical Study

  2015influential reference
• Evolving Generalised Maze Solvers

  2015influential reference
• Constrained Novelty Search: A Study on Game Content Generation

  2015cited by this paper
• Robots that can adapt like animals

  2014influential reference
• Knowledge Extraction from Learning Traces in Continuous Domains

  2014cited by this paper
• Family bootstrapping: A genetic transfer learning approach for onsetting the evolution for a set of related robotic tasks

  2014cited by this paper
• Novelty search creates robots with general skills for exploration

  2014influential reference
• Beyond black-box optimization: a review of selective pressures for evolutionary robotics

  2014cited by this paper
• Avoiding convergence in cooperative coevolution with novelty search

  2014cited by this paper
• Confronting the challenge of learning a flexible neural controller for a diversity of morphologies

  2013cited by this paper
• Effective diversity maintenance in deceptive domains

  2013influential reference
• Improving Grammatical Evolution in Santa Fe Trail using Novelty Search

  2013cited by this paper
• Scalable multiagent learning through indirect encoding of policy geometry

  2013cited by this paper
• Single-unit pattern generators for quadruped locomotion

  2013cited by this paper
• Evolving a Behavioral Repertoire for a Walking Robot

  2013influential reference
• Evolution of swarm robotics systems with novelty search

  2013influential reference
• Exploration and exploitation in evolutionary algorithms: A survey

  2013cited by this paper
• Critical factors in the performance of hyperNEAT

  2013cited by this paper
• Generic behaviour similarity measures for evolutionary swarm robotics

  2013influential reference
• Reinforcement learning transfer via sparse coding

  2012cited by this paper
• Translation techniques in cross-language information retrieval

  2012cited by this paper
• Progressive Minimal Criteria Novelty Search

  2012cited by this paper
• An examination of different fitness and novelty based selection methods for the evolution of neural networks

  2012cited by this paper
• Encouraging Behavioral Diversity in Evolutionary Robotics: An Empirical Study

  2012influential reference
• Novelty-based restarts for evolution strategies

  2011influential reference
• Transfer Learning in Multi-Agent Reinforcement Learning Domains

  2011cited by this paper
• Effective Task Transfer Through Indirect Encoding

  2011influential reference
• Constraining connectivity to encourage modularity in HyperNEAT

  2011cited by this paper
• Enhancing es-hyperneat to evolve more complex regular neural networks

  2011cited by this paper
• Evolutionary Robotics: Exploring New Horizons

  2011cited by this paper
• Transfer Learning via Multiple Inter-task Mappings

  2011cited by this paper
• When Novelty Is Not Enough

  2011influential reference
• Novelty Search and the Problem with Objectives

  2011influential reference
• Abandoning Objectives: Evolution Through the Search for Novelty Alone

  2011influential reference
• Efficiently evolving programs through the search for novelty

  2010influential reference
• A Survey on Transfer Learning

  2010cited by this paper
• Evolving Static Representations for Task Transfer

  2010influential reference
• Sustaining behavioral diversity in NEAT

  2010influential reference
• Revising the evolutionary computation abstraction: minimal criteria novelty search

  2010influential reference
• Evolving plastic neural networks with novelty search

  2010cited by this paper
• Evolutionary Function Approximation

  2010influential reference
• Overcoming the bootstrap problem in evolutionary robotics using behavioral diversity

  2009cited by this paper
• Transfer Learning

  2009cited by this paper
• Using behavioral exploration objectives to solve deceptive problems in neuro-evolution

  2009cited by this paper
• Transfer Learning for Reinforcement Learning Domains: A Survey

  2009influential reference
• A Hypercube-Based Encoding for Evolving Large-Scale Neural Networks

  2009cited by this paper
• A Hypercube-Based Indirect Encoding for Evolving Large-Scale Neural Networks

  2009influential reference
• Transfer of evolved pattern-based heuristics in games

  2008influential reference
• Analysis of an evolutionary reinforcement learning method in a multiagent domain

  2008cited by this paper
• Neuroevolution: from architectures to learning

  2008influential reference
• A Case Study on the Critical Role of Geometric Regularity in Machine Learning

  2008cited by this paper
• Transfer Learning via Inter-Task Mappings for Temporal Difference Learning

  2007influential reference
• Reinforcement learning in the robocup-soccer keepaway

  2007cited by this paper
• Compositional pattern producing networks: A novel abstraction of development

  2007cited by this paper
• BI OM A 20 06 P roo fs ENTROPY-DRIVEN EXPLORATION AND EXPLOITATION IN EVOLUTIONARY ALGORITHMS

  2006cited by this paper
• Evolutionary Function Approximation for Reinforcement Learning

  2006cited by this paper
• Transfer Learning for Policy Search Methods

  2006cited by this paper
• Keepaway Soccer: From Machine Learning Testbed to Benchmark

  2005cited by this paper
• Evolving Soccer Keepaway Players Through Task Decomposition

  2005cited by this paper
• Explicit Control of Diversity and Effective Variation Distance in Linear Genetic Programming

  2002influential reference
• Evolving Neural Networks through Augmenting Topologies

  2002influential reference
• Maintaining the Diversity of Genetic Programs

  2002influential reference
• Multi-objective optimization using evolutionary algorithms

  2001cited by this paper
• Fitness sharing and niching methods revisited

  1998cited by this paper
• Incremental Evolution of Complex General Behavior

  1997cited by this paper
• Vector quantization and signal compression

  1991cited by this paper
• Numerical recipes

  1990influential reference
• The self-organizing map

  1990influential reference
• Information theory, complexity and neural networks

  1989cited by this paper

• Energy Costs and Neural Complexity Evolution in Changing Environments

  2025cites this paper
• Bioengineering hybrid artificial life

  2025cites this paper
• Automating Robot Design with Multi-Level Evolution

  2024cites this paper
• Body and Brain Quality-Diversity in Robot Swarms

  2024influential citation
• Multi-Objective Evolution for Chemical Product Design

  2024cites this paper
• Morpho-Material Evolution for Automated Robot Design

  2024cites this paper
• Evolving Folding Bodies and Brains in Origami Robots

  2023cites this paper
• The Environment and Body-Brain Complexity

  2021cites this paper
• Exploiting abstractions for grammar‐based learning of complex multi‐agent behaviours

  2021cites this paper
• A survey of benchmarking frameworks for reinforcement learning

  2020cites this paper
• Evolutionary Automation of Coordinated Autonomous Vehicles

  2020influential citation
• Energy and Complexity in Evolving Collective Robot Bodies and Brains

  2020cites this paper
• The expense of neuro-morpho functional machines

  2020cites this paper
• Automating Coordinated Autonomous Vehicle Control

  2020influential citation
• The Cost of Big Brains in Groups

  2019influential citation
• Objective versus Non-Objective Search in Evolving Morphologically Robust Robot Controllers

  2018cites this paper
• Policy transfer methods in RoboCup keep-away

  2018cites this paper
• Neuro-evolution behavior transfer for collective behavior tasks

  2018cites this paper