Multi-Agent Reinforcement Learning for Energy Harvesting Two-Hop Communications with Full Cooperation

We focus on energy harvesting (EH) two-hop communications since they are the essential building blocks of more complicated multi-hop networks. The scenario consists of three nodes, where an EH transmitter wants to send data to a receiver through an EH relay. The harvested energy is used exclusively for data transmission and we address the problem of how to efficiently use it. As in practical scenarios, we assume only causal knowledge at the EH nodes, i.e., in each time interval, the transmitter and the relay know their own current and past amounts of incoming energy, battery levels, data buffer levels and channel coefficients for their own transmit channels. Our goal is to find transmission policies which aim at maximizing the throughput considering that the EH nodes fully cooperate with each other to exchange their causal knowledge during a signaling phase. We model the problem as a Markov game and propose a multi-agent reinforcement learning algorithm to find the transmission policies. Furthermore, we show the trade-off between the achievable throughput and the signaling required, and provide convergence guarantees for the proposed algorithm. Results show that even when the signaling overhead is taken into account, the proposed algorithm outperforms other approaches that do not consider cooperation among the nodes.

• Publication year

  2017

• Venue

  arXiv.org

• Publication date

  2017-02-08

• Fields of study

  Mathematics, Computer Science, Engineering

• Identifiers
  arXiv 1702.06185
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Adaptive Power Management in Solar Energy Harvesting Sensor Node Using Reinforcement Learning

  2017cited by this paper
• Delay-Constrained Data Transmission for Energy Harvesting Traansmitter

  2017cited by this paper
• Reinforcement learning for energy harvesting point-to-point communications

  2016influential reference
• A Learning Based Solution for Energy Harvesting Decode-and-Forward Two-Hop Communications

  2016influential reference
• Delay minimal policies in energy harvesting broadcast channels

  2016cited by this paper
• Advances in Energy Harvesting Communications: Past, Present, and Future Challenges

  2016cited by this paper
• Energy Harvesting Wireless Communications: A Review of Recent Advances

  2015cited by this paper
• Energy Harvesting Two-Hop Communication Networks

  2015cited by this paper
• Multi-Access Communications With Energy Harvesting: A Multi-Armed Bandit Model and the Optimality of the Myopic Policy

  2015cited by this paper
• Online power control for cooperative relaying with energy harvesting

  2015cited by this paper
• Throughput maximization in two-hop energy harvesting communications

  2015cited by this paper
• Online Resource Allocation for Energy Harvesting Downlink Multiuser Systems: Precoding With Modulation, Coding Rate, and Subchannel Selection

  2015cited by this paper
• Bayesian reinforcement learning for energy harvesting communication systems with uncertainty

  2015cited by this paper
• Iterative Dynamic Water-Filling for Fading Multiple-Access Channels With Energy Harvesting

  2014cited by this paper
• Energy Harvesting Cooperative Communication Systems

  2014cited by this paper
• The Multiway Relay Channel

  2013cited by this paper
• Throughput maximization for energy harvesting two-hop networks

  2013cited by this paper
• Delay-constrained distortion minimization for energy harvesting transmission over a fading channel

  2013cited by this paper
• The energy harvesting two-way decode-and-forward relay channel with stochastic data arrivals

  2013cited by this paper
• A Tutorial on Linear Function Approximators for Dynamic Programming and Reinforcement Learning

  2013influential reference
• Optimum transmission policies for energy harvesting two-way relay channels

  2013cited by this paper
• Two-hop networks with energy harvesting: The (non-)impact of buffer size

  2013cited by this paper
• Optimal packet scheduling in a multiple access channel with energy harvesting transmitters

  2012cited by this paper
• Optimal Scheduling and Power Allocation for Two-Hop Energy Harvesting Communication Systems

  2012cited by this paper
• Optimal Broadcast Scheduling for an Energy Harvesting Rechargeable Transmitter with a Finite Capacity Battery

  2012cited by this paper
• Throughput maximization for an energy harvesting communication system with processing cost

  2012cited by this paper
• Optimal transmission policies for energy harvesting two-hop networks

  2012cited by this paper
• A Learning Theoretic Approach to Energy Harvesting Communication System Optimization

  2012cited by this paper
• Optimal Save-Then-Transmit Protocol for Energy Harvesting Wireless Transmitters

  2012cited by this paper
• Throughput Optimal Policies for Energy Harvesting Wireless Transmitters with Non-Ideal Circuit Power

  2012cited by this paper
• Two-hop communication with energy harvesting

  2011cited by this paper
• Transmission with Energy Harvesting Nodes in Fading Wireless Channels: Optimal Policies

  2011cited by this paper
• Analysis of hybrid adaptive/non-adaptive multi-user OFDMA systems with imperfect channel knowledge

  2011cited by this paper
• Fundamentals of Wireless Sensor Networks: Theory and Practice

  2010cited by this paper
• Optimum Transmission Policies for Battery Limited Energy Harvesting Nodes

  2010influential reference
• Multi-Pair Two-Way Relay Channel with Multiple Antenna Relay Station

  2010cited by this paper
• A generic model for optimizing single-hop transmission policy of replenishable sensors

  2009cited by this paper
• The multi-way relay channel

  2009cited by this paper
• Kalman-filter channel estimator for OFDM systems in time and frequency-selective fading environment

  2004cited by this paper
• Energy-efficient collision-free medium access control for wireless sensor networks

  2003cited by this paper
• Value-function reinforcement learning in Markov games

  2001cited by this paper
• Reinforcement learning with function approximation converges to a region

  2001cited by this paper
• An Algorithm for Distributed Reinforcement Learning in Cooperative Multi-Agent Systems

  2000influential reference
• Reinforcement Learning with Function Approximation Converges to a Region

  2000influential reference
• Reinforcement Learning: An Introduction

  1998influential reference
• Markov Games as a Framework for Multi-Agent Reinforcement Learning

  1994cited by this paper
• Microwave Mobile Communications

  1975cited by this paper

• Multiple Time Blocks Energy Harvesting Relay Optimizing with Time-Switching Structure and Decoding Cost

  2019cites this paper
• Grouping and Cooperating among APs for Energy Efficiency in 5G UUDN

  2019cites this paper
• Link Scheduling Algorithms For In-Band Full-Duplex Wireless Networks

  2019cites this paper
• Distributed Power Control for Large Energy Harvesting Networks: A Multi-Agent Deep Reinforcement Learning Approach

  2019cites this paper
• Deep Reinforcement Learning for Router Selection in Network With Heavy Traffic

  2019cites this paper
• Multi -Agent Deep Reinforcement Learning based Power Control for Large Energy Harvesting Networks

  2019cites this paper
• Power Control in Energy Harvesting Multiple Access System With Reinforcement Learning

  2019influential citation
• Optimization of Energy Harvesting Mobile Nodes Within Scalable Converter System Based on Reinforcement Learning

  2019cites this paper
• 07 . 2019 Optimization of Energy Harvesting Mobile Nodes Within Scalable Converter System Based on Reinforcement Learning

  2019cites this paper
• Reinforcement Learning Based Multi-Access Control with Energy Harvesting

  2018influential citation
• Reinforcement Learning-Based Multiaccess Control and Battery Prediction With Energy Harvesting in IoT Systems

  2018influential citation