Automatic Action Space Curriculum Learning with Dynamic Per-Step Masking

Curriculum learning (CL) based on action space is an effective approach to deep reinforcement learning (DRL) in high-dimensional action spaces with limited training data. However, it can be challenging to determine which part of the action space to focus on during training. We propose an automatic CL approach that dynamically changes the action space using masking. Our approach selects the part of the action space based on its learning progress at each step of the training process, rather than every episode, which encourages the policy to explore actions that are learning faster. At the same time, it allows the policy to learn coordination among other actions. To evaluate the proposed CL approach, we tested it on an industrial robot performing the anchor bolt insertion task, which is in high demand for automation in the construction industry. The results indicate that our approach improved the learning performance of the DRL policy, resulting in better task execution. The policies trained with our approach outperformed those trained from scratch and those trained with similar action-space CL approaches. Although we only tested our approach on the anchor bolt insertion task, we believe it can be applied to other DRL policies with high-dimensional discrete action spaces. Overall, our automated CL approach shows promise for improving the automation of field tasks with real robots.

• Publication year

  2023

• Venue

  2023 IEEE 19th International Conference on Automation Science and Engineering (CASE)

• Publication date

  2023-08-26

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1109/CASE56687.2023.10260397
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Visual Spatial Attention and Proprioceptive Data-Driven Reinforcement Learning for Robust Peg-in-Hole Task Under Variable Conditions

  2023cited by this paper
• Curriculum-based Offline Network Training for Improvement of Peg-in-hole Task Performance for Holes in Concrete

  2022influential reference
• A Peg-in-hole Task Strategy for Holes in Concrete

  2021influential reference
• Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation

  2021cited by this paper
• Growing Action Spaces

  2019cited by this paper
• Mix&Match - Agent Curricula for Reinforcement Learning

  2018cited by this paper
• Accuracy-based Curriculum Learning in Deep Reinforcement Learning

  2018cited by this paper
• Sim-to-Real Reinforcement Learning for Deformable Object Manipulation

  2018cited by this paper
• Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning

  2018cited by this paper
• CASSL: Curriculum Accelerated Self-Supervised Learning

  2017cited by this paper
• Teacher–Student Curriculum Learning

  2017cited by this paper
• Automatic Goal Generation for Reinforcement Learning Agents

  2017cited by this paper
• Deep Reinforcement Learning with Double Q-Learning

  2015cited by this paper
• Dropout: a simple way to prevent neural networks from overfitting

  2014cited by this paper
• Playing Atari with Deep Reinforcement Learning

  2013cited by this paper
• Active learning of inverse models with intrinsically motivated goal exploration in robots

  2013cited by this paper
• Curriculum learning

  2009cited by this paper
• Construction robotics and automation: past-present-future

  2004cited by this paper
• Learning and development in neural networks: the importance of starting small.

  1993cited by this paper
• Learning

  1965cited by this paper
• Curriculum

  1956cited by this paper

• Unlocking Efficiency: Adaptive Masking for Gene Transformer Models

  2024cites this paper
• Applying Action Masking and Curriculum Learning Techniques to Improve Data Efficiency and Overall Performance in Operational Technology Cyber Security using Reinforcement Learning

  2024cites this paper