Emergent Coordination Through Competition

We study the emergence of cooperative behaviors in reinforcement learning agents by introducing a challenging competitive multi-agent soccer environment with continuous simulated physics. We demonstrate that decentralized, population-based training with co-play can lead to a progression in agents' behaviors: from random, to simple ball chasing, and finally showing evidence of cooperation. Our study highlights several of the challenges encountered in large scale multi-agent training in continuous control. In particular, we demonstrate that the automatic optimization of simple shaping rewards, not themselves conducive to co-operative behavior, can lead to long-horizon team behavior. We further apply an evaluation scheme, grounded by game theoretic principals, that can assess agent performance in the absence of pre-defined evaluation tasks or human baselines.

• Publication year

  2019

• Venue

  International Conference on Learning Representations

• Publication date

  2019-02-19

• Fields of study

  Physics, Computer Science

• Identifiers
  arXiv 1902.07151
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Stochastic games

  2019cited by this paper
• A Generalised Method for Empirical Game Theoretic Analysis

  2018cited by this paper
• Overlapping layered learning

  2018cited by this paper
• Learning an Embedding Space for Transferable Robot Skills

  2018cited by this paper
• Re-evaluating evaluation

  2018influential reference
• Learning by Playing - Solving Sparse Reward Tasks from Scratch

  2018influential reference
• DeepMind Control Suite

  2018cited by this paper
• Emergent Complexity via Multi-Agent Competition

  2017influential reference
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments

  2017cited by this paper
• Counterfactual Multi-Agent Policy Gradients

  2017cited by this paper
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning

  2017cited by this paper
• Emergence of Grounded Compositional Language in Multi-Agent Populations

  2017cited by this paper
• StarCraft II: A New Challenge for Reinforcement Learning

  2017cited by this paper
• DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker

  2017cited by this paper
• A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning

  2017cited by this paper
• Population Based Training of Neural Networks

  2017influential reference
• Emergence of Locomotion Behaviours in Rich Environments

  2017cited by this paper
• Data-efficient Deep Reinforcement Learning for Dexterous Manipulation

  2017cited by this paper
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments

  2017cited by this paper
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning

  2016cited by this paper
• Safe and Efficient Off-Policy Reinforcement Learning

  2016cited by this paper
• Mastering the game of Go with deep neural networks and tree search

  2016cited by this paper
• Learning Multiagent Communication with Backpropagation

  2016cited by this paper
• Learning and Transfer of Modulated Locomotor Controllers

  2016cited by this paper
• OpenAI Gym

  2016cited by this paper
• Cooperation and communication in multiagent deep reinforcement learning

  2016cited by this paper
• Memory-based control with recurrent neural networks

  2015influential reference
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

  2015cited by this paper
• Learning Continuous Control Policies by Stochastic Value Gradients

  2015influential reference
• Human-level control through deep reinforcement learning

  2015cited by this paper
• Adam: A Method for Stochastic Optimization

  2014cited by this paper
• Deterministic Policy Gradient Algorithms

  2014cited by this paper
• Optimal rewards in multiagent teams

  2012cited by this paper
• MuJoCo: A physics engine for model-based control

  2012influential reference
• Reinforcement learning for robot soccer

  2009cited by this paper
• On Local Rewards and Scaling Distributed Reinforcement Learning

  2005cited by this paper
• R-MAX - A General Polynomial Time Algorithm for Near-Optimal Reinforcement Learning

  2001cited by this paper
• Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping

  1999cited by this paper
• Reinforcement Learning: An Introduction

  1998cited by this paper
• Long Short-Term Memory

  1997influential reference
• RoboCup: The Robot World Cup Initiative

  1997cited by this paper
• Some Studies in Machine Learning Using the Game of Checkers

  1995cited by this paper
• Temporal Difference Learning and TD-Gammon

  1995cited by this paper
• Markov Games as a Framework for Multi-Agent Reinforcement Learning

  1994cited by this paper
• The rating of chessplayers, past and present

  1978influential reference

• SoccerSynth-Detection: A Synthetic Dataset for Soccer Player Detection

  2025cites this paper
• SetDQN: An Agent Number Invariant Reinforcement Learning Model Using Set Transformers

  2025cites this paper
• SoccerSynth Field: enhancing field detection with synthetic data from virtual soccer simulator

  2025cites this paper
• Learning Behaviours for Decentralised Multi-Robot Collision Avoidance in Constrained Pathways Using Curriculum Reinforcement Learning

  2025cites this paper
• Scalable Multi-Robot Cooperation for Multi-Goal Tasks Using Reinforcement Learning

  2025cites this paper
• Making Experience Replay Useful in Multiagent Scenarios

  2025cites this paper
• Who's the Mole? Modeling and Detecting Intention-Hiding Malicious Agents in LLM-Based Multi-Agent Systems

  2025cites this paper
• Multiagent reinforcement learning with quantified information-decision content measurement

  2025cites this paper
• Exploring the Potential of Robotic Coaching in eSports: A Pilot Study on Social Robots for Gaming Performance Enhancement

  2025cites this paper
• Agile Flight Emerges from Multi-Agent Competitive Racing

  2025cites this paper
• Event-driven Multi-Agent Concurrent and Collaborative Coordination

  2025cites this paper
• Generalized Population-Based Training for Hyperparameter Optimization in Reinforcement Learning

  2024cites this paper
• Emergent Program Synthesis Based on Reinforsment Learning and Computer Vision Techniques

  2024cites this paper
• A survey on multi-agent reinforcement learning and its application

  2024cites this paper
• Reinforcement Learning Within the Classical Robotics Stack: A Case Study in Robot Soccer

  2024cites this paper
• Security and Privacy Issues in Deep Reinforcement Learning: Threats and Countermeasures

  2024cites this paper
• Fuzzy Feedback Multiagent Reinforcement Learning for Adversarial Dynamic Multiteam Competitions

  2024cites this paper
• From Agents to Robots: A Training and Evaluation Platform for Multi-robot Reinforcement Learning

  2024cites this paper
• Enabling Multi-Robot Collaboration from Single-Human Guidance

  2024cites this paper
• MARLadona - Towards Cooperative Team Play Using Multi-Agent Reinforcement Learning

  2024cites this paper
• Subequivariant Reinforcement Learning in 3D Multi-Entity Physical Environments

  2024cites this paper
• Wildfire Autonomous Response and Prediction Using Cellular Automata (WARP-CA)

  2024cites this paper
• Scaling Up Multi-Agent Reinforcement Learning: An Extensive Survey on Scalability Issues

  2024cites this paper
• Distributed Online Planning for Min-Max Problems in Networked Markov Games

  2024cites this paper
• CompetEvo: Towards Morphological Evolution from Competition

  2024cites this paper
• Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learning

  2024cites this paper
• Manipulative basic Movement Learning Model in Class 6 Primary School Students

  2024cites this paper
• Improving Multiagent Actor-Critic Architectures, with Opponent Approximation and Dropout for Control

  2024cites this paper
• Neural Population Learning beyond Symmetric Zero-sum Games

  2024cites this paper
• Human-Timescale Adaptation in an Open-Ended Task Space

  2023cites this paper
• Evaluating Emergent Coordination in Multi-Agent Task Allocation Through Causal Inference and Sub-Team Identification

  2023cites this paper
• Dynamic traffic signal control based on multi-agent curricular transfer learning

  2023cites this paper
• Team-PSRO for Learning Approximate TMECor in Large Team Games via Cooperative Reinforcement Learning

  2023cites this paper
• Adversarial Dynamics in Centralized Versus Decentralized Intelligent Systems

  2023cites this paper
• Impact of Heterogeneity on Multi-Agent Reinforcement Learning

  2023cites this paper
• Iteratively Learn Diverse Strategies with State Distance Information

  2023cites this paper
• EvoX: A Distributed GPU-accelerated Library towards Scalable Evolutionary Computation

  2023cites this paper
• Replication of Multi-agent Reinforcement Learning for the "Hide and Seek" Problem

  2023cites this paper
• A strategic framework for optimal decisions in football 1-vs-1 shot-taking situations: an integrated approach of machine learning, theory-based modeling, and game theory

  2023cites this paper
• A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decision

  2023influential citation
• Learning Diverse Risk Preferences in Population-based Self-play

  2023cites this paper
• Learning agile soccer skills for a bipedal robot with deep reinforcement learning

  2023cites this paper
• Scaling multi-agent reinforcement learning to full 11 versus 11 simulated robotic football

  2023influential citation
• Evolutionary Reinforcement Learning: A Survey

  2023cites this paper
• Diverse Policy Optimization for Structured Action Space

  2023cites this paper
• TiZero: Mastering Multi-Agent Football with Curriculum Learning and Self-Play

  2023cites this paper
• A Review of Cooperation in Multi-agent Learning

  2023cites this paper
• Towards Skilled Population Curriculum for Multi-Agent Reinforcement Learning

  2023cites this paper
• Multi-Agent Interplay in a Competitive Survival Environment

  2023cites this paper
• Beyond Rewards: a Hierarchical Perspective on Offline Multiagent Behavioral Analysis

  2022cites this paper
• Assisting Reinforcement Learning in Real-time Strategy Environments with SCENIC

  2022cites this paper
• Negotiation and honesty in artificial intelligence methods for the board game of Diplomacy

  2022cites this paper
• Rethinking Trajectory Prediction via "Team Game"

  2022cites this paper
• Lamarckian Platform: Pushing the Boundaries of Evolutionary Reinforcement Learning Toward Asynchronous Commercial Games

  2022cites this paper
• Developing, evaluating and scaling learning agents in multi-agent environments

  2022cites this paper
• D ESIGNING N EURAL N ETWORK C OLLECTIVES

  2022cites this paper
• MA-Dreamer: Coordination and communication through shared imagination

  2022influential citation
• Continuously Discovering Novel Strategies via Reward-Switching Policy Optimization

  2022cites this paper
• Step-by-Step Acquisition of Cooperative Behavior in Soccer Task

  2022cites this paper
• NeuPL: Neural Population Learning

  2022cites this paper
• S2RL: Do We Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?

  2022cites this paper
• Learning Collective Action under Risk Diversity

  2022cites this paper
• Hindsight-aware deep reinforcement learning algorithm for multi-agent systems

  2022cites this paper
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems

  2022cites this paper
• From Multi-agent to Multi-robot: A Scalable Training and Evaluation Platform for Multi-robot Reinforcement Learning

  2022cites this paper
• Cooperation and Learning Dynamics under Wealth Inequality and Diversity in Individual Risk

  2022cites this paper
• Motivating Physical Activity via Competitive Human-Robot Interaction

  2022cites this paper
• Modelling Behavioural Diversity for Learning in Open-Ended Games

  2021cites this paper
• Optimal Planning of Emergency Communication Network Using Deep Reinforcement Learning

  2021cites this paper
• Multi-Agent Coordination in Adversarial Environments through Signal Mediated Strategies

  2021cites this paper
• Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems

  2021cites this paper
• Deep Latent Competition: Learning to Race Using Visual Control Policies in Latent Space

  2021cites this paper
• Dealing with Non-Stationarity in Multi-Agent Reinforcement Learning via Trust Region Decomposition

  2021cites this paper
• Learning to Play Soccer From Scratch: Sample-Efficient Emergent Coordination Through Curriculum-Learning and Competition

  2021influential citation
• Multi-robot Coordination and Planning in Uncertain and Adversarial Environments

  2021cites this paper
• On the Critical Role of Conventions in Adaptive Human-AI Collaboration

  2021cites this paper
• Two-stage training algorithm for AI robot soccer

  2021cites this paper
• Living systems are smarter bots: Slime mold semiosis versus AI symbol manipulation

  2021cites this paper
• Cooperation between Independent Reinforcement Learners under Wealth Inequality and Collective Risks

  2021cites this paper
• Pick Your Battles: Interaction Graphs as Population-Level Objectives for Strategic Diversity

  2021cites this paper
• Multi-agent deep reinforcement learning: a survey

  2021cites this paper
• From Motor Control to Team Play in Simulated Humanoid Football

  2021influential citation
• MAIN: A Real-world Multi-agent Indoor Navigation Benchmark for Cooperative Learning

  2021influential citation
• rSoccer: A Framework for Studying Reinforcement Learning in Small and Very Small Size Robot Soccer

  2021cites this paper
• Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL

  2021cites this paper
• Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot

  2021cites this paper
• Evaluating Strategic Structures in Multi-Agent Inverse Reinforcement Learning

  2021cites this paper
• Acquisition of Cooperative Behavior in a Soccer Task Using Reward Shaping

  2021cites this paper
• TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations

  2021cites this paper
• Local Advantage Actor-Critic for Robust Multi-Agent Deep Reinforcement Learning

  2021cites this paper
• How does AI play football? An analysis of RL and real-world football strategies

  2021influential citation
• Collective intelligence for deep learning: A survey of recent developments

  2021cites this paper
• Multi-agent reinforcement learning for character control

  2021cites this paper
• Capacity-Limited Decentralized Actor-Critic for Multi-Agent Games

  2021cites this paper
• Dealing with Non-Stationarity in MARL via Trust-Region Decomposition

  2021cites this paper
• Deep multiagent reinforcement learning: challenges and directions

  2021cites this paper
• PettingZoo: Gym for Multi-Agent Reinforcement Learning

  2021cites this paper
• Repeated Interactions Convention Dependence HighLow ρi ρ 2 ρ 3 Rule representation Convention representation 4 player chess Friendly Rock Paper Scissors time gt gp

  2021cites this paper
• Physically Embedded Planning Problems: New Challenges for Reinforcement Learning

  2020cites this paper
• A survey on multi-agent deep reinforcement learning: from the perspective of challenges and applications

  2020cites this paper