SCIENCE@home

Adversarial NLI: A New Benchmark for Natural Language Understanding

Yixin Nie,Adina Williams,Emily Dinan,Mohit Bansal,J. Weston,Douwe Kiela

Published 2019 in Annual Meeting of the Association for Computational Linguistics

ABSTRACT

We introduce a new large-scale NLI benchmark dataset, collected via an iterative, adversarial human-and-model-in-the-loop procedure. We show that training models on this new dataset leads to state-of-the-art performance on a variety of popular NLI benchmarks, while posing a more difficult challenge with its new test set. Our analysis sheds light on the shortcomings of current state-of-the-art models, and shows that non-expert annotators are successful at finding their weaknesses. The data collection method can be applied in a never-ending learning scenario, becoming a moving target for NLU, rather than a static benchmark that will quickly saturate.

PUBLICATION RECORD

CITATION MAP

EXTRACTION MAP

CLAIMS

CONCEPTS

  • adversarial nli
    dataset, benchmark

    A large-scale NLI benchmark dataset collected via an iterative adversarial human-and-model-in-the-loop procedure introduced in this paper.

    Aliases: ANLI

    뀨 (7c402c1b98) extraction
  • human-and-model-in-the-loop
    method

    An iterative data collection procedure where human annotators and models interact adversarially to generate challenging examples.

    뀨 (7c402c1b98) extraction
  • never-ending learning
    learning paradigm

    A continual learning scenario in which the benchmark evolves as a moving target rather than remaining a fixed static evaluation.

    뀨 (7c402c1b98) extraction
  • nli benchmarks
    evaluation setting

    Existing natural language inference evaluation datasets used to measure model performance in this paper.

    Aliases: NLI datasets

    뀨 (7c402c1b98) extraction
  • nlu models
    model

    Neural models for natural language understanding whose weaknesses are probed by non-expert annotators in this paper.

    Aliases: state-of-the-art models

    뀨 (7c402c1b98) extraction

REFERENCES

Showing 1-64 of 64 references · Page 1 of 1

CITED BY

Showing 1-100 of 1167 citing papers · Page 1 of 12

123412
Next