TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations

Deep reinforcement learning (DRL) has achieved super-human performance on complex video games (e.g., StarCraft II and Dota II). However, current DRL systems still suffer from challenges of multi-agent coordination, sparse rewards, stochastic environments, etc. In seeking to address these challenges, we employ a football video game, e.g., Google Research Football (GRF), as our testbed and develop an end-to-end learning-based AI system (denoted as TiKick) to complete this challenging task. In this work, we first generated a large replay dataset from the self-playing of single-agent experts, which are obtained from league training. We then developed a distributed learning system and new offline algorithms to learn a powerful multi-agent AI from the fixed single-agent dataset. To the best of our knowledge, Tikick is the first learning-based AI system that can take over the multi-agent Google Research Football full game, while previous work could either control a single agent or experiment on toy academic scenarios. Extensive experiments further show that our pre-trained model can accelerate the training process of the modern multi-agent algorithm and our method achieves state-of-the-art performances on various academic scenarios.

• Publication year

  2021

• Venue

  arXiv.org

• Publication date

  2021-10-09

• Fields of study

  Computer Science

• Identifiers
  arXiv 2110.04507
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• The Surprising Effectiveness of MAPPO in Cooperative, Multi-Agent Games

  2021influential reference
• Open-Ended Learning Leads to Generally Capable Agents

  2021cited by this paper
• Simplifying Deep Reinforcement Learning via Self-Supervision

  2021cited by this paper
• Unifying Behavioral and Response Diversity for Open-ended Learning in Zero-sum Games

  2021influential reference
• Celebrating Diversity in Shared Multi-Agent Reinforcement Learning

  2021influential reference
• From Motor Control to Team Play in Simulated Humanoid Football

  2021cited by this paper
• NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning

  2021cited by this paper
• On Bonus Based Exploration Methods In The Arcade Learning Environment

  2020cited by this paper
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning

  2020influential reference
• BeBold: Exploration Beyond the Boundary of Explored Regions

  2020cited by this paper
• Supervised Learning Achieves Human-Level Performance in MOBA Games: A Case Study of Honor of Kings

  2020cited by this paper
• RL Unplugged: Benchmarks for Offline Reinforcement Learning

  2020cited by this paper
• Conservative Q-Learning for Offline Reinforcement Learning

  2020cited by this paper
• MOPO: Model-based Offline Policy Optimization

  2020cited by this paper
• MOReL : Model-Based Offline Reinforcement Learning

  2020cited by this paper
• An Optimistic Perspective on Offline Reinforcement Learning

  2019cited by this paper
• Emergent Coordination Through Competition

  2019cited by this paper
• Google Research Football: A Novel Reinforcement Learning Environment

  2019influential reference
• MAVEN: Multi-Agent Variational Exploration

  2019cited by this paper
• PyTorch: An Imperative Style, High-Performance Deep Learning Library

  2019cited by this paper
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

  2019cited by this paper
• Grandmaster level in StarCraft II using multi-agent reinforcement learning

  2019influential reference
• Behavior Regularized Offline Reinforcement Learning

  2019cited by this paper
• Dota 2 with Large Scale Deep Reinforcement Learning

  2019cited by this paper
• Off-Policy Deep Reinforcement Learning without Exploration

  2018cited by this paper
• Self-Imitation Learning

  2018influential reference
• Multi-task Deep Reinforcement Learning with PopArt

  2018cited by this paper
• Policy Optimization with Demonstrations

  2018cited by this paper
• Multi-Agent Generative Adversarial Imitation Learning

  2018cited by this paper
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

  2018influential reference
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments

  2017cited by this paper
• Focal Loss for Dense Object Detection

  2017cited by this paper
• Mastering the game of Go with deep neural networks and tree search

  2016cited by this paper
• A Concise Introduction to Decentralized POMDPs

  2016cited by this paper
• Generative Adversarial Imitation Learning

  2016cited by this paper
• High-Dimensional Continuous Control Using Generalized Advantage Estimation

  2015cited by this paper
• Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

  2014cited by this paper
• Adam: A Method for Stochastic Optimization

  2014cited by this paper
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

  2013cited by this paper
• Playing Atari with Deep Reinforcement Learning

  2013cited by this paper
• TrueSkill™: A Bayesian Skill Rating System

  2007cited by this paper
• TrueSkillTM: A Bayesian Skill Rating System

  2006cited by this paper
• RoboCup: The Robot World Cup Initiative

  1997cited by this paper

• Distributed Policy Space Response Oracles in Two-Player Zero-Sum Games

  2025cites this paper
• Enhancing AI-Bot Strength and Strategy Diversity in Adversarial Games: A Novel Deep Reinforcement Learning Framework

  2025cites this paper
• Construction of Football Agents by Inverse Reinforcement Learning Using Relative Positional Information Among Players

  2025cites this paper
• Reinforcement Learning Within the Classical Robotics Stack: A Case Study in Robot Soccer

  2024cites this paper
• Multi-Agent Diagnostics for Robustness via Illuminated Diversity

  2024cites this paper
• From mimic to counteract: a two-stage reinforcement learning algorithm for Google research football

  2024cites this paper
• A Survey on Self-play Methods in Reinforcement Learning

  2024cites this paper
• Collaborative Decision-Making Technique for Wireless Communication Means Based on Multi-Agent Reinforcement Learning

  2024cites this paper
• Towards Skilled Population Curriculum for Multi-Agent Reinforcement Learning

  2023cites this paper
• An Empirical Study on Google Research Football Multi-agent Scenarios

  2023cites this paper
• Fictitious Cross-Play: Learning Global Nash Equilibrium in Mixed Cooperative-Competitive Games

  2023influential citation
• Diversity is Strength: Mastering Football Full Game with Interactive Reinforcement Learning of Multiple AIs

  2023cites this paper
• A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decision

  2023cites this paper
• Heterogeneous-graph Attention Reinforcement Learning for Football Matches

  2023cites this paper
• Diverse Policies Converge in Reward-free Markov Decision Processe

  2023cites this paper
• Boosting Studies of Multi-Agent Reinforcement Learning on Google Research Football Environment: the Past, Present, and Future

  2023influential citation
• Offensive and Defensive Attention Reward in Reinforcement Learning for Football Game

  2023cites this paper
• Learning Cooperative Behaviours in Adversarial Multi-agent Systems

  2023cites this paper
• Synthesizing Physical Character-Scene Interactions

  2023cites this paper
• TiZero: Mastering Multi-Agent Football with Curriculum Learning and Self-Play

  2023influential citation
• DGPO: Discovering Multiple Strategies with Diversity-Guided Policy Optimization

  2022cites this paper
• Assisting Reinforcement Learning in Real-time Strategy Environments with SCENIC

  2022cites this paper
• Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualization

  2022cites this paper
• The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games

  2021influential citation
• JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learning

  2021cites this paper
• The Application of Reinforcement Learning in Video Games

  year unknowncites this paper
• Exploring the Role of Reinforcement Learning in Video Game Environments

  year unknowncites this paper
• Learning Global Nash Equilibrium in Team Competitive Games with Generalized Fictitious Cross-Play

  year unknowninfluential citation