Bidirectional Context-Aware Hierarchical Attention Network for Document Understanding

The Hierarchical Attention Network (HAN) has made great strides, but it suffers a major limitation: at level 1, each sentence is encoded in complete isolation. In this work, we propose and compare several modifications of HAN in which the sentence encoder is able to make context-aware attentional decisions (CAHAN). Furthermore, we propose a bidirectional document encoder that processes the document forwards and backwards, using the preceding and following sentences as context. Experiments on three large-scale sentiment and topic classification datasets show that the bidirectional version of CAHAN outperforms HAN everywhere, with only a modest increase in computation time. While results are promising, we expect the superiority of CAHAN to be even more evident on tasks requiring a deeper understanding of the input documents, such as abstractive summarization. Code is publicly available.

• Publication year

  2019

• Venue

  arXiv.org

• Publication date

  2019-08-16

• Fields of study

  Computer Science

• Identifiers
  arXiv 1908.06006
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Dialogue Act Classification with Context-Aware Self-Attention

  2019influential reference
• Context-Aware Self-Attention Networks

  2019cited by this paper
• Energy-based Self-attentive Learning of Abstractive Communities for Spoken Language Understanding

  2019cited by this paper
• Deep Communicating Agents for Abstractive Summarization

  2018cited by this paper
• Context-Aware Neural Machine Translation Learns Anaphora Resolution

  2018influential reference
• A disciplined approach to neural network hyper-parameters: Part 1 - learning rate, batch size, momentum, and weight decay

  2018cited by this paper
• Improving the Transformer Translation Model with Document-Level Context

  2018cited by this paper
• Notes on Deep Learning for NLP

  2018cited by this paper
• Get To The Point: Summarization with Pointer-Generator Networks

  2017cited by this paper
• Three Factors Influencing Minima in SGD

  2017cited by this paper
• Exploiting Cross-Sentence Context for Neural Machine Translation

  2017influential reference
• Attention is All you Need

  2017cited by this paper
• A Structured Self-attentive Sentence Embedding

  2017cited by this paper
• Does Neural Machine Translation Benefit from Larger Context?

  2017cited by this paper
• Context Gates for Neural Machine Translation

  2016cited by this paper
• Hierarchical Attention Networks for Document Classification

  2016influential reference
• Modeling Coverage for Neural Machine Translation

  2016influential reference
• Character-level Convolutional Networks for Text Classification

  2015cited by this paper
• A Hierarchical Neural Autoencoder for Paragraphs and Documents

  2015cited by this paper
• Hierarchical Recurrent Neural Network for Document Modeling

  2015cited by this paper
• Effective Approaches to Attention-based Neural Machine Translation

  2015cited by this paper
• Document Modeling with Gated Recurrent Neural Network for Sentiment Classification

  2015cited by this paper
• Cyclical Learning Rates for Training Neural Networks

  2015cited by this paper
• Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

  2014influential reference
• Dropout: a simple way to prevent neural networks from overfitting

  2014cited by this paper
• Efficient Estimation of Word Representations in Vector Space

  2013cited by this paper

• Position-Wise Gated Res2Net-Based Convolutional Network with Selective Fusing for Sentiment Analysis

  2023cites this paper
• Hierarchical multi-attention networks for document classification

  2021cites this paper
• ScopeIt: Scoping Task Relevant Sentences in Documents

  2020cites this paper
• Hierarchical Interaction Networks with Rethinking Mechanism for Document-level Sentiment Analysis

  2020influential citation
• Mining Actuarial Risk Predictors in Accident Descriptions Using Recurrent Neural Networks

  2020cites this paper