SCIENCE@home

Comparing Humans and Large Language Models on an Experimental Protocol Inventory for Theory of Mind Evaluation (EPITOME)

Cameron R. Jones,Sean Trott,Benjamin K. Bergen

Published 2024 in Transactions of the Association for Computational Linguistics

ABSTRACT

Abstract We address a growing debate about the extent to which large language models (LLMs) produce behavior consistent with Theory of Mind (ToM) in humans. We present EPITOME: a battery of six experiments that tap diverse ToM capacities, including belief attribution, emotional inference, and pragmatic reasoning. We elicit a performance baseline from human participants for each task. We use the dataset to ask whether distributional linguistic information learned by LLMs is sufficient to explain ToM in humans. We compare performance of five LLMs to a baseline of responses from human comprehenders. Results are mixed. LLMs display considerable sensitivity to mental states and match human performance in several tasks. Yet, they commit systematic errors in others, especially those requiring pragmatic reasoning on the basis of mental state information. Such uneven performance indicates that human-level ToM may require resources beyond distributional information.

PUBLICATION RECORD

  • Publication year

    2024

  • Venue

    Transactions of the Association for Computational Linguistics

  • Publication date

    2024-06-01

  • Fields of study

    Computer Science, Linguistics, Psychology

  • Identifiers
    DOI 10.1162/tacl_a_00674
  • External record

    Open on Semantic Scholar

  • Source metadata

    Semantic Scholar

CITATION MAP

EXTRACTION MAP

CLAIMS

  • No claims are published for this paper.

CONCEPTS

  • No concepts are published for this paper.

REFERENCES

Showing 1-72 of 72 references · Page 1 of 1

CITED BY

Showing 1-14 of 14 citing papers · Page 1 of 1