Decentralised Learning in Systems with Many, Many Strategic Agents

Although multi-agent reinforcement learning can tackle systems of strategically interacting entities, it currently fails in scalability and lacks rigorous convergence guarantees. Crucially, learning in multi-agent systems can become intractable due to the explosion in the size of the state-action space as the number of agents increases. In this paper, we propose a method for computing closed-loop optimal policies in multi-agent systems that scales independently of the number of agents. This allows us to show, for the first time, successful convergence to optimal behaviour in systems with an unbounded number of interacting adaptive learners. Studying the asymptotic regime of N-player stochastic games, we devise a learning protocol that is guaranteed to converge to equilibrium policies even when the number of agents is extremely large. Our method is model-free and completely decentralised so that each agent need only observe its local state information and its realised rewards. We validate these theoretical results by showing convergence to Nash-equilibrium policies in applications from economics and control theory with thousands of strategically interacting agents.

• Publication year

  2018

• Venue

  AAAI Conference on Artificial Intelligence

• Publication date

  2018-03-13

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1609/aaai.v32i1.11586arXiv 1803.05028
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Multi-agent Reinforcement Learning in Sequential Social Dilemmas

  2017cited by this paper
• The Master Equation and the Convergence Problem in Mean Field Games

  2015cited by this paper
• Learning in mean field games: The fictitious play

  2015influential reference
• Potential-based difference rewards for multiagent reinforcement learning

  2014cited by this paper
• Explicit solutions of some linear-quadratic mean field games

  2012cited by this paper
• Multiagent Learning: Basics, Challenges, and Prospects

  2012cited by this paper
• Spectrum Sharing as Spatial Congestion Games

  2010cited by this paper
• Discrete Time, Finite State Space Mean Field Games

  2010cited by this paper
• Multiagent learning in large anonymous games

  2009cited by this paper
• Multiagent Systems: Introduction to Noncooperative Game Theory: Games in Normal Form

  2008cited by this paper
• A Comprehensive Survey of Multiagent Reinforcement Learning

  2008cited by this paper
• Mean field games

  2007cited by this paper
• Uniformly Lipschitz feedback optimal controls in a linear-quadratic framework☆

  2007cited by this paper
• Generalised weakened fictitious play

  2006cited by this paper
• Nash Q-Learning for General-Sum Stochastic Games

  2003cited by this paper
• Friend-or-Foe Q-learning in General-Sum Games

  2001cited by this paper
• Reinforcement Learning: An Introduction

  1998cited by this paper
• Markov Games as a Framework for Multi-Agent Reinforcement Learning

  1994cited by this paper
• Potential Games

  1994influential reference
• ACTIVITY ANALYSIS OF PRODUCTION AND ALLOCATION

  1952cited by this paper
• Funzione caratteristica di un fenomeno aleatorio

  year unknowncited by this paper

• Towards Bio-inspired Heuristically Accelerated Reinforcement Learning for Adaptive Underwater Multi-Agents Behaviour

  2025cites this paper
• Reward design in multi-agent systems using successor features and multi-information source bayesian optimization

  2025cites this paper
• The Traitors: Deception and Trust in Multi-Agent Language Model Simulations

  2025cites this paper
• LLT: Learning Latent Tactics in Uncertainty Cooperative Multi-Agent Reinforcement Learning

  2025cites this paper
• Mean field games for urban mobility: a review

  2025cites this paper
• Convergence of Actor-Critic Learning for Mean Field Games and Mean Field Control in Continuous Spaces

  2025cites this paper
• Socially-Minded Intelligence: How Individuals, Groups, and AI Systems Can Make Each-Other Smarter (or Not)

  2024cites this paper
• Scalable Learning for Spatiotemporal Mean Field Games Using Physics-Informed Neural Operator

  2024cites this paper
• A Single Online Agent Can Efficiently Learn Mean Field Games

  2024cites this paper
• Why does the two-timescale Q-learning converge to different mean field solutions? A unified convergence analysis

  2024cites this paper
• LJIR: Learning Joint-Action Intrinsic Reward in cooperative multi-agent reinforcement learning

  2023cites this paper
• Mean-Field Learning for Day-to-Day Departure Time Choice with Mode Switching

  2023cites this paper
• Convergence of Multi-Scale Reinforcement Q-Learning Algorithms for Mean Field Game and Control Problems

  2023cites this paper
• Decision making in open agent systems

  2023cites this paper
• Hierarchical Mean-Field Deep Reinforcement Learning for Large-Scale Multiagent Systems

  2023cites this paper
• Networked Communication for Decentralised Agents in Mean-Field Games

  2023cites this paper
• A Hybrid Framework of Reinforcement Learning and Physics-Informed Deep Learning for Spatiotemporal Mean Field Games

  2023cites this paper
• MANSA: Learning Fast and Slow in Multi-Agent Systems

  2023cites this paper
• Population-size-Aware Policy Optimization for Mean-Field Games

  2023cites this paper
• A Survey on Large-Population Systems and Scalable Multi-Agent Reinforcement Learning

  2022cites this paper
• Implementing a Hierarchical Deep Learning Approach for Simulating Multilevel Auction Data

  2022cites this paper
• Make Smart Decisions Faster: Deciding D2D Resource Allocation via Stackelberg Game Guided Multi-Agent Deep Reinforcement Learning

  2022cites this paper
• Learning Sparse Graphon Mean Field Games

  2022cites this paper
• Learning Mean Field Games: A Survey

  2022cites this paper
• Survey on Machine Learning for Traffic-Driven Service Provisioning in Optical Networks

  2022cites this paper
• Reinforcement Learning for Intra-and-Inter-Bank Borrowing and Lending Mean Field Control Game

  2022cites this paper
• Reinforcement Learning Algorithm for Mixed Mean Field Control Games

  2022cites this paper
• Semi-Centralised Multi-Agent Reinforcement Learning with Policy-Embedded Training

  2022cites this paper
• Individual-Level Inverse Reinforcement Learning for Mean Field Games

  2022cites this paper
• Taming Multi-Agent Reinforcement Learning with Estimator Variance Reduction

  2022cites this paper
• Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergence

  2022cites this paper
• Learning Graphon Mean Field Games and Approximate Nash Equilibria

  2021influential citation
• Reinforcement Learning for Mean Field Games with Strategic Complementarities

  2021cites this paper
• Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learning

  2021cites this paper
• Equilibrium Inverse Reinforcement Learning for Ride-hailing Vehicle Network

  2021cites this paper
• Learning in Nonzero-Sum Stochastic Games with Potentials

  2021cites this paper
• Discrete-Time Mean Field Control with Environment States

  2021cites this paper
• Concave Utility Reinforcement Learning: the Mean-field Game viewpoint

  2021cites this paper
• Reinforcement Learning for Mean Field Games, with Applications to Economics

  2021cites this paper
• Generalization in Mean Field Games by Learning Master Policies

  2021cites this paper
• LIGS: Learnable Intrinsic-Reward Generation Selection for Multi-Agent Learning

  2021cites this paper
• Decentralized Mean Field Games

  2021cites this paper
• Multi Type Mean Field Reinforcement Learning

  2020cites this paper
• A game theoretical approach to self-assembly in swarm robotics

  2020influential citation
• Model Free Reinforcement Learning Algorithm for Stationary Mean field Equilibrium for Multiple Types of Agents

  2020cites this paper
• Partially Observable Mean Field Reinforcement Learning

  2020cites this paper
• Exploratory LQG Mean Field Games with Entropy Regularization

  2020cites this paper
• Multiagent Adversarial Collaborative Learning via Mean-Field Theory

  2020cites this paper
• Multi-agent Distributed Data Mining: Challenges and Research Directions

  2020cites this paper
• Fictitious Play for Mean Field Games: Continuous Time Analysis and Applications

  2020cites this paper
• The Evolutionary Dynamics of Independent Learning Agents in Population Games

  2020cites this paper
• Unified reinforcement Q-learning for mean field game and control problems

  2020cites this paper
• Learning Trembling Hand Perfect Mean Field Equilibrium for Dynamic Mean Field Games

  2020cites this paper
• Stochastic Potential Games

  2020cites this paper
• Mean Field Game Guided Deep Reinforcement Learning for Task Placement in Cooperative Multiaccess Edge Computing

  2020cites this paper
• A General Framework for Learning Mean-Field Games

  2020cites this paper
• Deep Fictitious Play for Stochastic Differential Games

  2019cites this paper
• On the Convergence of Model Free Learning in Mean Field Games

  2019cites this paper
• Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning

  2019cites this paper
• Actor-Critic Provably Finds Nash Equilibria of Linear-Quadratic Mean-Field Games

  2019cites this paper
• Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methods

  2019cites this paper
• Modelling the Dynamics of Multiagent Q-Learning in Repeated Symmetric Games: a Mean Field Theoretic Approach

  2019cites this paper
• Reinforcement Learning in Stationary Mean-field Games

  2019cites this paper
• Learning Mean-Field Games

  2019cites this paper
• Coordinating the Crowd: Inducing Desirable Equilibria in Non-Cooperative Systems

  2019cites this paper
• Balancing Two-Player Stochastic Games with Soft Q-Learning

  2018cites this paper
• An Information-Theoretic Optimality Principle for Deep Reinforcement Learning

  2017cites this paper
• Mean-Field Game and Control of Large Population Systems: From Theory to Practice Mean field games for urban mobility: a review

  year unknowncites this paper