The 19 Unifying Questionnaire Constructs of Artificial Social Agents: An IVA Community Analysis

In this paper, we report on the multi-year Intelligent Virtual Agents (IVA) community effort, involving more than 80 researchers worldwide, researching the IVA community interests and practises in evaluating human interaction with an artificial social agent (ASA). The effort is driven by previous IVA workshops and plenary IVA discussions related to the methodological crisis on the evaluation of ASAs. A previous literature review showed a continuous practise of creating new questionnaires instead of reusing validated questionnaires. We address this issue by examining questionnaire measurement constructs used in empirical studies between 2013 to 2018 published in the IVA conference. We identified 189 constructs used in 89 questionnaires that are reported across 81 studies. Although these constructs have different names, they often measure the same thing. In this paper, we, therefore, present a unifying set of 19 constructs that captures more than 80% of the 189 constructs initially identified. We established this set in two steps. First, 49 researchers classified the constructs in broad theoretically based categories. Next, 23 researchers grouped the constructs in each category on their similarity. The resulting 19 groups form a unifying set of constructs, which will be the basis for the future questionnaire instrument of human-ASA interaction.

• Publication year

  2020

• Venue

  International Conference on Intelligent Virtual Agents

• Publication date

  2020-10-19

• Fields of study

  Sociology, Computer Science, Psychology

• Identifiers
  DOI 10.1145/3383652.3423873
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• What are We Measuring Anyway?: - A Literature Survey of Questionnaires Used in Studies Reported in the Intelligent Virtual Agent Conferences

  2019influential reference
• Coordination, Organizations, Institutions, and Norms in Agent System VII

  2012cited by this paper
• Explanation in Human-Agent Teamwork

  2011cited by this paper
• Social Situated Agents in Virtual, Real and Mixed Reality Environments

  2005cited by this paper
• Applying artificial intelligence to virtual reality: Intelligent virtual environments

  2000cited by this paper
• Understanding Attitudes and Predicting Social Behavior

  1980cited by this paper

• Identifying Barriers Hindering the Acceptance of Generative AI as a Work Associate, measured with the new AGAWA scale

  2025cites this paper
• Does Robot Clothing Really Help? User Preferences and Effects in Simulated Domestic Scenarios

  2025cites this paper
• Psychological and behavioural responses in human-agent vs. human-human interactions: a systematic review and meta-analysis

  2025cites this paper
• The Artificial Social Agent Questionnaire (ASAQ) - Development and evaluation of a validated instrument for capturing human interaction experiences with artificial social agents

  2025cites this paper
• The Impact of Appearance and Task Outcome on User Psychology in Interactions with LLM-Based Agents

  2025cites this paper
• Generating Unexpected yet Relevant User Dialog Acts

  2024cites this paper
• Response Generation for Cognitive Behavioral Therapy with Large Language Models: Comparative Study with Socratic Questioning

  2024cites this paper
• Beyond Functionality: Unveiling Dimensions of User Experience in Embodied Conversational Agents for Customer Service

  2024cites this paper
• Towards Asynchronous VR Collaboration with Virtual Agents as Placeholder Users

  2024cites this paper
• Simulating Patient Oral Dialogues: A Study on Naturalness and Coherence of Conditioned Large Language Models

  2024cites this paper
• German and Dutch Translations of the Artificial-Social-Agent Questionnaire Instrument for Evaluating Human-Agent Interactions

  2024cites this paper
• MITHOS: Interactive Mixed Reality Training to Support Professional Socio-Emotional Interactions at Schools

  2024cites this paper
• Barriers and enabling factors for error analysis in NLG research

  2023cites this paper
• ASAP: Endowing Adaptation Capability to Agent in Human-Agent Interaction

  2023cites this paper
• Mandarin Chinese translation of the Artificial-Social-Agent questionnaire instrument for evaluating human-agent interaction

  2023cites this paper
• Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisations

  2023influential citation
• Perspectives on Socially Intelligent Conversational Agents

  2022cites this paper
• Didn't see that coming: a survey on non-verbal social human behavior forecasting

  2022cites this paper
• Evaluating data-driven co-speech gestures of embodied conversational agents through real-time interaction

  2022cites this paper
• The artificial-social-agent questionnaire: establishing the long and short questionnaire versions

  2022cites this paper
• Toward Believable Acting for Autonomous Animated Characters

  2022cites this paper
• Towards Building Rapport with a Human Support Robot

  2021cites this paper
• eXtended Artificial Intelligence: New Prospects of Human-AI Interaction Research

  2021cites this paper
• Psychology in Virtual Reality: Toward a Validated Measure of Social Presence

  2021influential citation
• Empirical Analysis of Training Strategies of Transformer-Based Japanese Chit-Chat Systems

  2021cites this paper
• Questionnaire Items for Evaluating Artificial Social Agents - Expert Generated, Content Validated and Reliability Analysed

  2021cites this paper
• Linking Theory of Mind in Human-Agent Interactions to Validated Evaluations: Can Explicit Questionnaires Measure Implicit Behaviour?

  2021cites this paper
• VR Realism Scale—Revalidation of contemporary VR headsets on a Polish sample

  2021cites this paper
• Taming the Interaction Jungle: Embodiment is All You Need

  2021cites this paper
• Mixed Reality Methods. Preliminary Considerations for Multimodal Analysis of Human-Agent Interactions

  2021cites this paper
• Pretest or not to pretest? A preliminary version of a tool for the virtual character standardization

  2021cites this paper
• Mixed Reality Methods for Analysis of Multimodal Human-Agent Interactions

  2021cites this paper