Do I trust my machine teammate?: an investigation from perception to decision

In the human-machine collaboration context, understanding the reason behind each human decision is critical for interpreting the performance of the human-machine team. Via an experimental study of a system with varied levels of accuracy, we describe how human trust interplays with system performance, human perception and decisions. It is revealed that humans are able to perceive the performance of automatic systems and themselves, and adjust their trust levels according to the accuracy of systems. The 70% system accuracy suggests to be a threshold between increasing and decreasing human trust and system usage. We have also shown that trust can be derived from a series of users' decisions rather than from a single one, and relates to the perceptions of users. A general framework depicting how trust and perception affect human decision making is proposed, which can be used as future guidelines for human-machine collaboration design.

• Publication year

  2019

• Venue

  International Conference on Intelligent User Interfaces

• Publication date

  2019-03-17

• Fields of study

  Computer Science, Psychology

• Identifiers
  DOI 10.1145/3301275.3302277
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• User Trust Dynamics: An Investigation Driven by Differences in System Performance

  2017cited by this paper
• Effects of Faults, Experience, and Personality on Trust in a Robot Co-Worker

  2017cited by this paper
• User Trust in Intelligent Systems: A Journey Over Time

  2016cited by this paper
• Trust and Reliance Based on System Accuracy

  2016cited by this paper
• Experience of automation failures in training: effects on trust, automation bias, complacency and performance

  2016cited by this paper
• “Why Should I Trust You?”: Explaining the Predictions of Any Classifier

  2016cited by this paper
• Effects of the Advisor and Environment on Requesting and Complying With Automated Advice

  2016cited by this paper
• Algorithm aversion: people erroneously avoid algorithms after seeing them err.

  2015cited by this paper
• Decision-making in abstract trust games: A user interface perspective

  2014cited by this paper
• Using System-Wide Trust Theory to Make Predictions About Dependence on Four Diagnostic Aids

  2010cited by this paper
• A User Behavior Based Trust Model for Mobile Applications

  2008cited by this paper
• Not All Trust Is Created Equal: Dispositional and History-Based Trust in Human-Automation Interactions

  2008cited by this paper
• Similarities and differences between human–human and human–automation trust: an integrative review

  2007cited by this paper
• Perception and misperception of bias in human judgment.

  2007cited by this paper
• Mining Trust Values from Recommendation Errors

  2006cited by this paper
• Supporting Trust Calibration and the Effective Use of Decision Aids by Presenting Dynamic System Confidence Information

  2006cited by this paper
• An integrated theory of the mind.

  2004cited by this paper
• Type of Automation Failure: The Effects on Trust and Reliance in Automation

  2004cited by this paper
• Trust in Automation: Designing for Appropriate Reliance

  2004cited by this paper
• A MODEL FOR PREDICTING HUMAN TRUST IN AUTOMATED SYSTEMS

  2003cited by this paper
• Trust in computer technology and the implications for design and evaluation

  2002cited by this paper
• Self-Confidence and Personal Motivation

  2002cited by this paper
• Measuring Trust

  2000cited by this paper
• Adaptive automation, trust, and self-confidence in fault management of time-critical tasks.

  2000cited by this paper
• Trust And Distrust: New Relationships and Realities

  1998cited by this paper
• Modelling self-confidence in users of a computer-based system showing unrepresentative design

  1998cited by this paper
• Trust in automation. Part II. Experimental studies of trust and human intervention in a process control simulation.

  1996influential reference
• Are humans good intuitive statisticians after all

  1996cited by this paper
• Behind human error : cognitive systems, computers, and hindsight : state-of-the-art report

  1994cited by this paper
• Trust in automation. I: Theoretical issues in the study of trust and human intervention in automated systems

  1994cited by this paper
• Trust, self-confidence, and operators' adaptation to automation

  1994cited by this paper
• Trust, control strategies and allocation of function in human-machine systems.

  1992cited by this paper
• In the Age of the Smart Machine

  1988cited by this paper
• The Logic and Limits of Trust. By Bernard Barber. (New Brunswick, N.J.: Rutgers University Press, 1983. Pp. 190. $27.50, cloth; $9.95, paper.)

  1984cited by this paper
• The Logic and Limits of Trust

  1983influential reference
• The Logic and Limits of Trust

  1983cited by this paper
• PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 48th ANNUAL MEETING—2004 439 EFFECT OF ERROR SEVERITY ON HUMAN TRUST IN HYBRID SYSTEMS

  year unknowncited by this paper

• Under what influence: Measuring AI influence to fit user profiles in decision-making

  2026cites this paper
• 'The AI is uncertain, so am I. What now?': Navigating Shortcomings of Uncertainty Representations in Human-AI Collaboration with Capability-focused Guidance

  2025cites this paper
• Essential Skills for Lean Six Sigma 4.0 Implementation: Insights From a TOE-Based Mixed-Method Analysis

  2025cites this paper
• A Systematic Review and Taxonomy of Human–Agent Teaming Testbeds

  2025cites this paper
• Does Calibration Affect Human Actions?

  2025cites this paper
• To Solicit or Not to Solicit? Impact of AI Assistance Delivery Mechanisms on Decision-Making

  2025cites this paper
• What Needs Attention? Prioritizing Drivers of Developers' Trust and Adoption of Generative AI

  2025cites this paper
• Trust in AI is dynamically updated based on users' expectations

  2025cites this paper
• Trust, distrust, and appropriate reliance in (X)AI: A conceptual clarification of user trust and survey of its empirical evaluation

  2025cites this paper
• Presenting Large Language Models as Companions Affects What Mental Capacities People Attribute to Them

  2025cites this paper
• Navigating challenges/opportunities in developing smart agricultural extension platforms: Multi-media data mining techniques

  2025cites this paper
• The impact of AI recommendation quality on service satisfaction: the moderating roles of standardization and customization

  2025cites this paper
• Machine learning systems as mentors in human learning: A user study on machine bias transmission in medical training

  2025cites this paper
• Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies

  2025cites this paper
• Human-Machine Interaction Design in Adaptive Automation

  2024cites this paper
• Confirmation bias in AI-assisted decision-making: AI triage recommendations congruent with expert judgments increase psychologist trust and recommendation acceptance

  2024cites this paper
• Does More Advice Help? The Effects of Second Opinions in AI-Assisted Decision Making

  2024cites this paper
• Mitigative Strategies for Recovering From Large Language Model Trust Violations

  2024cites this paper
• Do We Learn From Each Other: Understanding the Human-AI Co-Learning Process Embedded in Human-AI Collaboration

  2024influential citation
• ContractMind: Trust-calibration interaction design for AI contract review tools

  2024cites this paper
• Do We Use Relatively Bad (Algorithmic) Advice? The Effects of Performance Feedback and Advice Representation on Advice Usage

  2024cites this paper
• Systematic Literature Review in the Development of Datasets and Fact Verification Models for Indonesian Language

  2024cites this paper
• Understanding the Evolvement of Trust Over Time within Human-AI Teams

  2024cites this paper
• Towards Human-centered Design of Explainable Artificial Intelligence (XAI): A Survey of Empirical Studies

  2024cites this paper
• What Guides Our Choices? Modeling Developers' Trust and Behavioral Intentions Towards Genai

  2024cites this paper
• Behave Yourself! Behavioral Indicators of Trust in Human-Agent Teams

  2024cites this paper
• Why Would You Suggest That? Human Trust in Language Model Responses

  2024cites this paper
• Trust in Human-Agent Teams: A Multilevel Perspective and Future Research Agenda

  2024cites this paper
• Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making

  2024influential citation
• Establishing Appropriate Trust in AI through Transparency and Explainability

  2024cites this paper
• Missing Pieces: How Do Designs that Expose Uncertainty Longitudinally Impact Trust in AI Decision Aids? An In Situ Study of Gig Drivers

  2024cites this paper
• Decoding AI's Nudge: A Unified Framework to Predict Human Behavior in AI-assisted Decision Making

  2024cites this paper
• Enhancing AI-Assisted Group Decision Making through LLM-Powered Devil's Advocate

  2024cites this paper
• Complementarity in human-AI collaboration: concept, sources, and evidence

  2024cites this paper
• Are Two Heads Be�er Than One in AI-Assisted Decision Making? Comparing the Behavior and Performance of Groups and Individuals in Human-AI Collaborative Recidivism Risk Assessment

  2023cites this paper
• A User Interface for Explaining Machine Learning Model Explanations

  2023cites this paper
• From Data Processing to Blockchain Networking: A Recount and Projection of Information Systems Research

  2023cites this paper
• CoArgue : Fostering Lurkers’ Contribution to Collective Arguments in Community-based QA Platforms

  2023cites this paper
• Measuring and Understanding Trust Calibrations for Automated Systems: A Survey of the State-Of-The-Art and Future Directions

  2023influential citation
• Are Two Heads Better Than One in AI-Assisted Decision Making? Comparing the Behavior and Performance of Groups and Individuals in Human-AI Collaborative Recidivism Risk Assessment

  2023cites this paper
• Incorporating Experts' Judgment into Machine Learning Models

  2023cites this paper
• Sonographer interaction with artificial intelligence: collaboration or conflict?

  2023cites this paper
• Humans, AI, and Context: Understanding End-Users’ Trust in a Real-World Computer Vision Application

  2023cites this paper
• In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Making

  2023cites this paper
• Towards a Science of Human-AI Decision Making: An Overview of Design Space in Empirical Human-Subject Studies

  2023influential citation
• A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making

  2023cites this paper
• A Literature Review of Human-AI Synergy in Decision Making: From the Perspective of Affordance Actualization Theory

  2023cites this paper
• May I Ask a Follow-up Question? Understanding the Benefits of Conversations in Neural Network Explainability

  2023influential citation
• Assessing the relationship between subjective trust, confidence measurements, and mouse trajectory characteristics in an online task

  2023cites this paper
• Human-AI collaboration is not very collaborative yet: a taxonomy of interaction patterns in AI-assisted decision making from a systematic review

  2023cites this paper
• Meaningful Explanation Effect on User’s Trust in an AI Medical System: Designing Explanations for Non-Expert Users

  2023cites this paper
• Trust, distrust, and appropriate reliance in (X)AI: a survey of empirical evaluation of user trust

  2023influential citation
• AI for human assessment: What do professional assessors need?

  2022cites this paper
• Exploring the Effects of Machine Learning Literacy Interventions on Laypeople’s Reliance on Machine Learning Models

  2022cites this paper
• Developing A Dynamic Testbed: Designing for Trust in Complex Human Agent Teams

  2022cites this paper
• Explanations Can Reduce Overreliance on AI Systems During Decision-Making

  2022influential citation
• “It would work for me too”: How Online Communities Shape Software Developers’ Trust in AI-Powered Code Generation Tools

  2022cites this paper
• COVID-19 masks increase the influence of face recognition algorithm decisions on human decisions in unfamiliar face matching

  2022cites this paper
• Reliance and Automation for Human-AI Collaborative Data Labeling Conflict Resolution

  2022cites this paper
• When Do XAI Methods Work? A Cost-Benefit Approach to Human-AI Collaboration

  2022cites this paper
• Tangible Interaction with In-Car Smart Intelligence

  2022cites this paper
• Designing for Conversational System Trustworthiness: The Designing for Conversational System Trustworthiness: The Impact of Model Transparency on Trust and Task Performance Impact of Model Transparency on Trust and Task Performance

  2022cites this paper
• Designing for Conversational System Trustworthiness: The Impact of Model Transparency on Trust and Task Performance

  2022influential citation
• The risk of algorithm transparency: How algorithm complexity drives the effects on the use of advice

  2022cites this paper
• A Meta-Analysis of the Utility of Explainable Artificial Intelligence in Human-AI Decision-Making

  2022cites this paper
• Algorithm aversion? On the influence of advice accuracy on trust in algorithmic advice

  2022cites this paper
• You Complete Me: Human-AI Teams and Complementary Expertise

  2022cites this paper
• A Human-machine Collaborative Framework for Evaluating Malevolence in Dialogues

  2021cites this paper
• DAPPER

  2021influential citation
• EUCA: the End-User-Centered Explainable AI Framework

  2021cites this paper
• Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies

  2021influential citation
• Cognitive computing based ethical principles for improving organisational reputation: A B2B digital marketing perspective

  2021cites this paper
• A Case Study of Using Analytic Provenance to Reconstruct User Trust in a Guided Visual Analytics System

  2021cites this paper
• How Mock Model Training Enhances User Perceptions of AI Systems

  2021cites this paper
• Hierarchical Decision Ensembles- An inferential framework for uncertain Human-AI collaboration in forensic examinations

  2021influential citation
• Learn, Generate, Rank, Explain: A Case Study of Visual Explanation by Generative Machine Learning

  2021cites this paper
• Confirmation bias and trust: Human factors that influence teachers' attitudes towards AI-based educational technology

  2021cites this paper
• EUCA: A Practical Prototyping Framework towards End-User-Centered Explainable Artificial Intelligence

  2021cites this paper
• How to Evaluate Trust in AI-Assisted Decision Making? A Survey of Empirical Methodologies

  2021cites this paper
• CACTUS: Detecting and Resolving Conflicts in Objective Functions

  2021cites this paper
• Calibrating trust toward an autonomous image classifier.

  2021influential citation
• Implementing an Intelligent Collaborative Agent as Teammate in Collaborative Writing: toward a Synergy of Humans and AI

  2021cites this paper
• Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making

  2020cites this paper
• Soliciting Human-in-the-Loop User Feedback for Interactive Machine Learning Reduces User Trust and Impressions of Model Accuracy

  2020cites this paper
• Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance

  2020cites this paper
• Towards Trustworthy Human-AI Teaming under Uncertainty

  2019cites this paper
• How students justify the usability of self-built machine learning models

  year unknowncites this paper
• Scholarship Repository @ Florida Tech Scholarship Repository @ Florida Tech

  year unknowncites this paper
• Measuring Trust Dynamics in AI-Assisted Decision-Making: Measuring Trust Dynamics in AI-Assisted Decision-Making: Insights from an Experimental Study Insights from an Experimental Study

  year unknowncites this paper
• International Journal of Human - Computer Studies

  year unknowncites this paper