SCIENCE@home

Long Document Classification in the Transformer Era: A Survey on Challenges, Advances, and Open Issues

Renzo Arturo Alva Principe,Nicola Chiarini,Marco Viviani

Published 2025 in WIREs Data. Mining. Knowl. Discov.

ABSTRACT

Automatic Document Classification (ADC) refers to the process of automatically categorizing or labeling documents into predefined classes or categories. Its effectiveness may depend on various factors, including the models used for the formal representation of documents, the classification techniques applied, or a combination of both. Recently, Transformer models have gained popularity due to their pre‐training on large corpora, allowing for flexible knowledge transfer to downstream tasks, such as ADC. However, such models can face challenges when handling “long” documents, particularly due to input sequence length constraints, which can have knock‐on effects on the task we refer to as Automatic Long Document Classification (ALDC). Distinct models for tackling this limitation of Transformers have been proposed over the past few years, and employed to perform ALDC; however, their application to this task has resulted in some inconsistent outcomes, struggles to surpass simple baselines, and difficulties in generalizing across diverse datasets and scenarios. That is why this survey aims to illustrate these limitations, by: (i) presenting current long document representation issues and solutions proposed in the literature; (ii) based on such solutions, illustrating a comprehensive analysis of their application in ALDC and their effectiveness; and (iii) discussing current evaluation strategies in ALDC with particular reference to suitable baselines and actual long‐document benchmark datasets.

PUBLICATION RECORD

  • Publication year

    2025

  • Venue

    WIREs Data. Mining. Knowl. Discov.

  • Publication date

    2025-05-08

  • Fields of study

    Computer Science

  • Identifiers
    DOI 10.1002/widm.70019
  • External record

    Open on Semantic Scholar

  • Source metadata

    Semantic Scholar

CITATION MAP

EXTRACTION MAP

CLAIMS

  • No claims are published for this paper.

CONCEPTS

  • No concepts are published for this paper.

REFERENCES

Showing 1-88 of 88 references · Page 1 of 1

CITED BY