αPOMDP: POMDP-based user-adaptive decision-making for social robots

Abstract In this work we present αPOMDP: a User-Adaptive Decision-Making technique for social robots. This technique is based on the classical POMDP formulation which we extend with novel aspects inspired by Reward Shaping and Model-Based Reinforcement Learning. Our technique innovates in two main ways: by applying a novel set of rewarding schemes based on the state of the user and by employing a novel execution loop that enables the system to learn the impact of its actions on the user on-the-fly. Our technique has been tested with multiple POMDP solvers and reward formulations in simulations and with real users through the GrowMu social robot. Results show that our technique is able to correctly decide which actions to take, maintaining the user in positive states which interacting with the robot and methodically exploring and learning their characteristics, activities and behaviors.

• Publication year

  2019

• Venue

  Pattern Recognition Letters

• Publication date

  2019-02-01

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1016/j.patrec.2018.03.011
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• BUM: Bayesian user model for distributed social robots

  2017cited by this paper
• POMDPs for Risk-Aware Autonomy

  2016cited by this paper
• Adaptive artificial companions learning from users’ feedback

  2016cited by this paper
• Automated Planning and Acting

  2016cited by this paper
• Personality-based consistent robot behavior

  2016cited by this paper
• Target Surveillance in Adversarial Environments Using POMDPs

  2016cited by this paper
• The GrowMeUp Project and the Applicability of Action Recognition Techniques

  2015cited by this paper
• Multi-Objective POMDPs with Lexicographic Reward Preferences

  2015cited by this paper
• Robot recommender system using affection-based episode ontology for personalization

  2013cited by this paper
• Assessment of adaptive human-robot interactions

  2013cited by this paper
• Probabilistic Approaches to Robotic Perception

  2013cited by this paper
• Evaluating POMDP rewards for active perception

  2012cited by this paper
• Bayesian reasoning and machine learning

  2012cited by this paper
• User adaptable robot behavior

  2011cited by this paper
• A POMDP framework for modelling human interaction with assistive robots

  2011cited by this paper
• Evolving policies for multi-reward partially observable markov decision processes (MR-POMDPs)

  2011cited by this paper
• A POMDP Extension with Belief-dependent Rewards

  2010cited by this paper
• Model-based reinforcement learning with nearly tight exploration complexity bounds

  2010cited by this paper
• User—robot personality matching and assistive robot behavior adaptation for post-stroke rehabilitation therapy

  2008cited by this paper
• SARSOP: Efficient Point-Based POMDP Planning by Approximating Optimally Reachable Belief Spaces

  2008cited by this paper
• Point-Based POMDP Algorithms: Improved Analysis and Implementation

  2005cited by this paper
• Point-based value iteration: an anytime algorithm for POMDPs

  2003cited by this paper
• Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping

  1999cited by this paper
• Planning and Acting in Partially Observable Stochastic Domains

  1998cited by this paper
• PDDL-the planning domain definition language

  1998cited by this paper
• Incremental Pruning: A Simple, Fast, Exact Method for Partially Observable Markov Decision Processes

  1997cited by this paper
• Learning Policies for Partially Observable Environments: Scaling Up

  1997cited by this paper
• The Shannon information entropy of protein sequences.

  1996cited by this paper
• Planning in Stochastic Domains: Problem Characteristics and Approximation

  1996cited by this paper
• Computing Optimal Policies for Partially Observable Decision Processes Using Compact Representations

  1996influential reference
• Acting Optimally in Partially Observable Stochastic Domains

  1994cited by this paper
• Solution Procedures for Partially Observed Markov Decision Processes

  1989cited by this paper
• State of the Art—A Survey of Partially Observable Markov Decision Processes: Theory, Models, and Algorithms

  1982cited by this paper
• OPTIMAL CONTROL FOR PARTIALLY OBSERVABLE MARKOV DECISION PROCESSES OVER AN INFINITE HORIZON

  1978cited by this paper
• Planning and Acting

  1978cited by this paper
• The Optimal Control of Partially Observable Markov Processes over the Infinite Horizon: Discounted Costs

  1978cited by this paper
• The Optimal Control of Partially Observable Markov Processes over a Finite Horizon

  1973cited by this paper
• A mathematical theory of communication

  1948cited by this paper

• Parley+: Uncertainty Reduction in Self-Adaptive Systems

  2025cites this paper
• Model-Based RL Decision-Making for UAVs Operating in GNSS-Denied, Degraded Visibility Conditions with Limited Sensor Capabilities

  2025cites this paper
• Integrated decision-control for social robot autonomous navigation considering nonlinear dynamics model

  2025cites this paper
• SHIFT: An Interdisciplinary Framework for Scaffolding Human Attention and Understanding in Explanatory Tasks

  2025cites this paper
• Formal Synthesis of Uncertainty Reduction Controllers

  2024cites this paper
• A Systematic Literature Review of Decision-Making and Control Systems for Autonomous and Social Robots

  2023cites this paper
• Uncertainty Aware Task Allocation for Human-Automation Cooperative Recognition in Autonomous Driving Systems

  2023cites this paper
• Intervention Request Planning with Operator Capability Model for Human-Automation Cooperative Recognition

  2023cites this paper
• Personalized Behaviour Models: A Survey Focusing on Autism Therapy Applications

  2022cites this paper
• On Proactive Human-AI Systems

  2022cites this paper
• Two ways to make your robot proactive: Reasoning about human intentions or reasoning about possible futures

  2022cites this paper
• Methods for Robot Behavior Adaptation for Cognitive Neurorehabilitation

  2021cites this paper
• Desire-Driven Reasoning considering Status-Based Knowledge Description for Personal Care Robots

  2020cites this paper
• Reinforcement Learning Approaches in Social Robotics

  2020influential citation
• Dynamic Emotional Language Adaptation in Multiparty Interactions with Agents

  2020cites this paper
• Adaptivity as a Service (AaaS): Personalised Assistive Robotics for Ambient Assisted Living

  2020cites this paper
• Modeling and verification of contingency resolution strategies for multi-robot missions using temporal logic

  2019cites this paper
• An Extended Bayesian User Model (BUM) for Capturing Cultural Attributes with a Social Robot

  2018cites this paper