Acquiring Grounded Representations of Words with Situated Interactive Instruction

We present an approach for acquiring grounded representations of words from mixed-initiative, situated interactions with a human instructor. The work focuses on the acquisition of diverse types of knowledge including perceptual, semantic, and procedural knowledge along with learning grounded meanings. Interactive learning allows the agent to control its learning by requesting instructions about unknown concepts, making learning efficient. Our approach has been instantiated in Soar and has been evaluated on a table-top robotic arm capable of manipulating small objects.

• Publication year

  2025

• Venue

  arXiv.org

• Publication date

  2025-02-28

• Fields of study

  Computer Science, Education

• Identifiers
  DOI 10.48550/arXiv.2502.20754arXiv 2502.20754
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• The Soar Cognitive Architecture

  2012cited by this paper
• Situated comprehension of imperative sentences in embodied, cognitive agents

  2012influential reference
• A Joint Model of Language and Perception for Grounded Attribute Learning

  2012cited by this paper
• Proceedings of the Twenty-Ninth International Conference on Machine Learning

  2012cited by this paper
• Learning Grounded Language through Situated Interactive Instruction

  2012cited by this paper
• Grounded Semantic Composition for Visual Scenes

  2011cited by this paper
• Learning to Interpret Natural Language Navigation Instructions from Observations

  2011cited by this paper
• Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation

  2011cited by this paper
• Learning actions from human-robot dialogues

  2011cited by this paper
• Developing high-level cognitive functions for service robots

  2010cited by this paper
• PLOW: A Collaborative Task Learning Agent

  2007cited by this paper
• Formulating Queries for Assessing Clinical Trial Eligibility

  2006cited by this paper
• Grounding words in perception and action: computational insights.

  2005cited by this paper
• COLLAGEN: A Collaboration Manager for Software Interface Agents

  1998cited by this paper
• Parsing English with a Link Grammar

  1995cited by this paper
• Mapping Explanation-Based Generalization onto Soar

  1986cited by this paper

• Energy-Efficient Domain-Specific Artificial Intelligence Models and Agents: Pathways and Paradigms

  2025cites this paper
• Towards visual-symbolic integration in the Soar cognitive architecture

  2025cites this paper
• Using Natural Language for Human-Robot Collaboration in the Real World

  2025cites this paper
• Improving Knowledge Extraction from LLMs for Task Learning through Agent Analysis

  2023cites this paper
• Cognitive Architectures for Language Agents

  2023cites this paper
• Improving Knowledge Extraction from LLMs for Robotic Task Learning through Agent Analysis

  2023cites this paper
• A Demonstration of Compositional, Hierarchical Interactive Task Learning

  2022cites this paper
• Analogical Concept Memory for Architectures Implementing the Common Model of Cognition

  2022influential citation
• Interactive Grounding and Inference in Learning by Instruction

  2021cites this paper
• Unpacking Human Teachers’ Intentions for Natural Interactive Task Learning

  2021cites this paper
• Robots that Help Humans Build Better Mental Models of Robots

  2021cites this paper
• Conversational Multi-Hop Reasoning with Neural Commonsense Knowledge and Symbolic Logic Rules

  2021cites this paper
• Exploring Learning Capability of an Agent in SOAR: Using 8-Queens Problem

  2020cites this paper
• Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog

  2020cites this paper
• Characterizing an Analogical Concept Memory for Architectures Implementing the Common Model of Cognition

  2020influential citation
• Interactive Grounding and Inference in Instruction Following

  2020cites this paper
• Conversational Learning

  2020cites this paper
• Characterizing an Analogical Concept Memory for Newellian Cognitive Architectures

  2020influential citation
• GRIP: Generative Robust Inference and Perception for Semantic Robot Manipulation in Adversarial Environments

  2019cites this paper
• Building Jarvis - A Learner-Aware Conversational Trainer

  2019cites this paper
• Teaching Robots New Tasks through Natural Interaction

  2019cites this paper
• Learning Hierarchical Symbolic Representations to Support Interactive Task Learning and Knowledge Transfer

  2019cites this paper
• A Heuristic Strategy for Identifying Misclassified Data Using Classification Labels

  2019cites this paper
• Towards using transparency mechanisms to build better mental models

  2019cites this paper
• Learning a Policy for Opportunistic Active Learning

  2018cites this paper
• A Framework for Natural Cognitive System Training Interactions

  2018cites this paper
• Goal-directed robot manipulation through axiomatic scene estimation

  2017cites this paper
• Semantic Robot Programming for Goal-Directed Manipulation in Cluttered Scenes

  2017cites this paper
• Interactive Task Learning

  2017cites this paper
• Opportunistic Active Learning for Grounding Natural Language Descriptions

  2017cites this paper
• Scene-level Programming by Demonstration

  2017cites this paper
• A Demonstration of Interactive Task Learning

  2016cites this paper
• Towards an Indexical Model of Situated Language Comprehension for Cognitive Agents in Physical Worlds

  2016influential citation
• Five Dimensions of Reasoning in the Wild

  2016cites this paper
• Continuously Improving Natural Language Understanding for Robotic Systems through Semantic Parsing, Dialog, and Multi-modal Perception

  2016cites this paper
• Learning General and Efficient Representations of Novel Games Through Interactive Instruction

  2016cites this paper
• Learning task goals interactively with visual demonstrations

  2016cites this paper
• A Computational Model for Situated Task Learning with Interactive Instruction

  2016cites this paper
• Axiomatic particle filtering for goal-directed robotic manipulation

  2015cites this paper
• From Verbs to Tasks: An Integrated Account of Learning Tasks from Situated Interactive Instruction

  2015cites this paper
• Ontology-Based Information Extraction with a Cognitive Agent

  2015cites this paper
• Interactive Task Learning for Simple Games

  2014cites this paper
• A Case Study of Knowledge Integration Across Multiple Memories in Soar

  2014cites this paper
• Learning Goal-Oriented Hierarchical Tasks from Situated Interactive Instruction

  2014influential citation
• Towards an Indexical Model of Situated Language Comprehension for Real-World Cognitive Agents

  2013cites this paper
• Reflections on Abstractions for General Artificial Intelligence

  2013cites this paper
• Learning Task Formulations through Situated Interactive Instruction

  2013cites this paper
• An architecture for observational learning and decision making based on internal models

  2013cites this paper