Distributed W-Learning: Multi-Policy Optimization in Self-Organizing Systems

Large-scale agent-based systems are required to self-optimize towards multiple, potentially conflicting, policies of varying spatial and temporal scope. As a result, not all agents may be implementing all policies at all times, resulting in agent heterogeneity. As agents share their operating environment, significant dependencies can arise between agents and therefore between policy implementations. To address self-optimization in the presence of agent heterogeneity, policy dependency and the lack of global knowledge that is inherent in large-scale decentralized environments, we propose Distributed W-Learning (DWL). DWL is a reinforcement learning (RL)-based algorithm for collaborative agent-based self-optimization towards multiple policies, which relies only on local interactions and learning. We have evaluated the DWL algorithm in a simulation of a self-organizing urban traffic control (UTC) system and show that using DWL can improve the performance of multiple policies deployed simultaneously, even over corresponding single-policy deployments. For example, in UTC, optimizing simultaneously for cars and public transport vehicles reduces the waiting times of cars to 78% of their waiting times in the best-performing single-policy deployment that optimizes for cars only, while also outperforming the widely-deployed round-robin and saturation balancing traffic controllers that we used as baselines.

• Publication year

  2009

• Venue

  2009 Third IEEE International Conference on Self-Adaptive and Self-Organizing Systems

• Publication date

  2009-09-14

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1109/SASO.2009.23
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Using distributed w-learning for multi-policy optimization in decentralized autonomic systems

  2009cited by this paper
• Using Reinforcement Learning for Multi-policy Optimization in Decentralized Autonomic Systems - An Experimental Evaluation

  2009influential reference
• Searching and sharing information in networks of heterogeneous agents

  2008cited by this paper
• A Collaborative Reinforcement Learning Approach to Urban Traffic Control Optimization

  2008cited by this paper
• Digital Evolution of Behavioral Models for Autonomic Systems

  2008cited by this paper
• Simulation and evaluation of urban bus-networks using a multiagent approach

  2007cited by this paper
• Reinforcement Learning in Autonomic Computing: A Manifesto and Case Studies

  2007cited by this paper
• Learning multi-goal dialogue strategies using reinforcement learning with reduced state-action spaces

  2006cited by this paper
• Learning Road Traffic Control: Towards Practical Traffic Control Using Policy Gradients

  2006cited by this paper
• Requirements for an ubiquitous computing simulation and emulation environment

  2006cited by this paper
• Cooperative Multi-Agent Learning: The State of the Art

  2005cited by this paper
• A Distributed Approach for Coordination of Traffic Signal Agents

  2005cited by this paper
• MAKING WAY FOR EMERGENCY VEHICLES

  2005cited by this paper
• The Decentralised Coordination of Self-Adaptive Components for Autonomic Distributed Systems

  2005cited by this paper
• Self-organization in multi-agent systems

  2005cited by this paper
• Technical Note: Q-Learning

  2004cited by this paper
• Cooperative multiagent systems for the optimization of urban traffic

  2004cited by this paper
• Multi Agent Reinforcement Learning Independent vs Cooperative Agents

  2003cited by this paper
• Reinforcement learning for true adaptive traffic signal control

  2003cited by this paper
• A particle swarm model for swarm-based networked sensor systems

  2002cited by this paper
• Messor: Load-Balancing through a Swarm of Autonomous Agents

  2002cited by this paper
• Multi-Agent Reinforcement Leraning for Traffic Light Control

  2000cited by this paper
• Distributed reinforcement learning for a traffic engineering application

  2000cited by this paper
• Multi-Agent Reinforcement Learning for Traffic Light control

  2000cited by this paper
• Multiagent systems: a modern approach to distributed artificial intelligence

  1999cited by this paper
• MAKING WAY FOR EMERGENCY VEHICLES

  1999cited by this paper
• Reinforcement Learning: An Introduction

  1998cited by this paper
• Action selection methods using reinforcement learning

  1997influential reference
• The Sydney coordinated adaptive traffic system - principles, methodology, algorithms

  1982cited by this paper

• Intelligent Traffic Management System using Multi-Agent Reinforcement Learning

  2025cites this paper
• Multi-Agent Adaptive Traffic Signal Control Based on Q-Learning and Speed Transition Matrices

  2025cites this paper
• MTLight: Efficient Multi-Task Reinforcement Learning for Traffic Signal Control

  2024cites this paper
• Reinforcement learning in urban network traffic signal control: A systematic literature review

  2022cites this paper
• Multi-objective reward generalization: improving performance of Deep Reinforcement Learning for applications in single-asset trading

  2022cites this paper
• A literature review on optimization techniques for adaptation planning in adaptive systems: State of the art and research directions

  2022cites this paper
• Spatial-Temporal Traffic Flow Control on Motorways Using Distributed Multi-Agent Reinforcement Learning

  2021influential citation
• MetaVIM: Meta Variationally Intrinsic Motivated Reinforcement Learning for Decentralized Traffic Signal Control

  2021cites this paper
• A practical guide to multi-objective reinforcement learning and planning

  2021cites this paper
• Action Selection for Composable Modular Deep Reinforcement Learning

  2021cites this paper
• Self-Adaptive Systems: A Systematic Literature Review Across Categories and Domains

  2021influential citation
• Coordination of Electric Vehicle Charging Through Multiagent Reinforcement Learning

  2020influential citation
• Extended Variable Speed Limit control using Multi-agent Reinforcement Learning

  2020cites this paper
• Cluster-Based Social Reinforcement Learning

  2020cites this paper
• Multi-objective multi-agent decision making: a utility-based analysis and survey

  2019cites this paper
• Multi-objective Optimisation in Hybrid Collaborating Adaptive Systems

  2019cites this paper
• On Learning in Collective Self-Adaptive Systems: State of Practice and a 3D Framework

  2019cites this paper
• Urban Traffic Control Using Distributed Multi-agent Deep Reinforcement Learning

  2019cites this paper
• Decentralized Collective Learning for Self-managed Sharing Economies

  2018cites this paper
• Q-Learning versus SVM Study for Green Context-Aware Multimodal ITS Stations

  2018cites this paper
• Participant Selection for Short-term Collaboration in Open Multi-agent systems

  2017cites this paper
• Self-Adaptive Learning in Decentralized Combinatorial Optimization - A Design Paradigm for Sharing Economies

  2017cites this paper
• Decentralization of Control Loop for Self-Adaptive Software through Reinforcement Learning

  2017cites this paper
• A Mutual Influence-based Learning Algorithm

  2016influential citation
• Goal-based Multi-agent Collaboration Community Formation : A Conceptual Model

  2016cites this paper
• Autonomic Transport Management Systems—Enabler for Smart Cities, Personalized Medicine, Participation and Industry Grid/Industry 4.0

  2016cites this paper
• Multi-objective multiagent credit assignment in reinforcement learning and NSGA-II

  2016cites this paper
• Context-aware multi-modal traffic management in ITS: A Q-learning based algorithm

  2015cites this paper
• Parallel Transfer Learning: Accelerating Reinforcement Learning in Multi-Agent Systems

  2015cites this paper
• Accelerating Learning in multi-objective systems through Transfer Learning

  2014cites this paper
• A self-learning motorway traffic control system for ramp metering

  2014cites this paper
• Knowledge Engineering Tools in Planning : State-ofthe-art and Future Challenges

  2014cites this paper
• A dynamic forecasting method for small scale residential electrical demand

  2014cites this paper
• Self-organising algorithms for residential demand response

  2014cites this paper
• Reinforcement Learning for Coverage Optimization Through PTZ Camera Alignment in Highly Dynamic Environments

  2014cites this paper
• A Survey of Multi-Objective Sequential Decision-Making

  2013cites this paper
• Autonomic multi-policy optimization in pervasive systems: Overview and evaluation

  2012influential citation
• Context-Aware Pervasive Services for Smart Cities

  2011cites this paper
• A New COST Action: Autonomic Road Transport Support (ARTS) Systems

  2011cites this paper
• Soilse: A decentralized approach to optimization of fluctuating urban traffic using Reinforcement Learning

  2010cites this paper
• Designing Comprehensible Self-Organising Systems

  2010cites this paper
• Multi-policy Optimization in Self-organizing Systems

  2009influential citation
• Diagrammatic reasoning (cont.)

  year unknowncites this paper