Learning text representation using recurrent convolutional neural network with highway layers

Recently, the rapid development of word embedding and neural networks has brought new inspiration to various NLP and IR tasks. In this paper, we describe a staged hybrid model combining Recurrent Convolutional Neural Networks (RCNN) with highway layers. The highway network module is incorporated in the middle takes the output of the bi-directional Recurrent Neural Network (Bi-RNN) module in the first stage and provides the Convolutional Neural Network (CNN) module in the last stage with the input. The experiment shows that our model outperforms common neural network models (CNN, RNN, Bi-RNN) on a sentiment analysis task. Besides, the analysis of how sequence length influences the RCNN with highway layers shows that our model could learn good representation for the long text.

• Publication year

  2016

• Venue

  Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

• Publication date

  2016-06-22

• Fields of study

  Computer Science

• Identifiers
  arXiv 1606.06905
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Recurrent Convolutional Neural Networks for Text Classification

  2015influential reference
• A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification

  2015cited by this paper
• Adam: A Method for Stochastic Optimization

  2014cited by this paper
• A Convolutional Neural Network for Modelling Sentences

  2014cited by this paper
• Convolutional Neural Network Architectures for Matching Natural Language Sentences

  2014cited by this paper
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

  2014cited by this paper
• Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

  2014cited by this paper
• Dropout: a simple way to prevent neural networks from overfitting

  2014cited by this paper
• Convolutional Neural Networks for Sentence Classification

  2014influential reference
• Distributed Representations of Words and Phrases and their Compositionality

  2013cited by this paper
• Advances in optimizing recurrent networks

  2012cited by this paper
• Representation Learning: A Review and New Perspectives

  2012cited by this paper
• Learning Word Vectors for Sentiment Analysis

  2011cited by this paper
• Search Engines - Information Retrieval in Practice

  2009cited by this paper
• Neural Probabilistic Language Models

  2006cited by this paper
• On the computational basis of learning and cognition: Arguments from LSA

  2002cited by this paper
• Long Short-Term Memory

  1997influential reference
• Bidirectional recurrent neural networks

  1997influential reference

• Text classification using SVD, BERT, and GRU optimized by improved Seagull optimization (ISO) algorithm

  2025cites this paper
• Neural Architecture Search Based on Bipartite Graphs for Text Classification

  2024cites this paper
• Reassessing Glove Embeddings in Deep Learning: A Comparative Study with Classical ML Approaches

  2024cites this paper
• Predicting suicidal and self-injurious events in a correctional setting using AI algorithms on unstructured medical notes and structured data.

  2023cites this paper
• No Labels?: No Problem! Experiments with active learning strategies for multi-class classification in imbalanced low-resource settings

  2023cites this paper
• Multi-task recommendation system for scientific papers with high-way networks

  2022cites this paper
• Clustering-based Aggregations for Prediction in Event Streams

  2022cites this paper
• Context-aware sentiment analysis with attention-enhanced features from bidirectional transformers

  2022cites this paper
• A comparative study on deep learning models for text classification of unstructured medical notes with various levels of class imbalance

  2022cites this paper
• APIRO: A Framework for Automated Security Tools API Recommendation

  2022cites this paper
• Online Prediction of Aggregated Retailer Consumer Behaviour

  2021cites this paper
• A Comparison of Different Source Code Representation Methods for Vulnerability Prediction in Python

  2021cites this paper
• Identifying and Predicting Intentional Self-Harm in Electronic Health Record Clinical Notes: Deep Learning Approach

  2020cites this paper
• Hybrid Tiled Convolutional Neural Networks for Text Sentiment Classification

  2020cites this paper
• Benchmark Performance of Machine And Deep Learning Based Methodologies for Urdu Text Document Classification

  2020cites this paper
• A Precisely Xtreme-Multi Channel Hybrid Approach for Roman Urdu Sentiment Analysis

  2020cites this paper
• Hybrid Tiled Convolutional Neural Networks (HTCNN) Text Sentiment Classification

  2020cites this paper
• Exploring The Dimension of DNN Techniques For Text Categorization Using NLP

  2020cites this paper
• Improved Text Classification using Long Short-Term Memory and Word Embedding Technique

  2020cites this paper
• Chinese Question Classification Based on ERNIE and Feature Fusion

  2020cites this paper
• Hybrid convolutional bidirectional recurrent neural network based sentiment analysis on movie reviews

  2020influential citation
• ANALYSIS FOR CREDIT CARD FRAUD DETECTION THROUGH DEEP LEARNING

  2020cites this paper
• Efficient and Accurate Hybrid Deep Learning Model for Multimodal Disease Risk Prediction

  2020cites this paper
• Improving Convolutional Network using K-max mechanism for Sentiment Analysis Tasks

  2020cites this paper
• Deep Learning--based Text Classification

  2020cites this paper
• Benchmarking performance of machine and deep learning-based methodologies for Urdu text document classification

  2020cites this paper
• KNN-Based Pseudo-supervised RCNN Framework for Text Clustering

  2019cites this paper
• Incorporating Sub-Word Level Information in Language Invariant Neural Event Detection

  2019cites this paper
• A Probabilistic Matrix Factorization Recommendation Method Based on Deep Learning

  2019cites this paper
• Disease Prediction by Machine Learning from Healthcare Communities

  2019cites this paper
• Multi-layer Convolutional Neural Network Model Based on Prior Knowledge of Knowledge Graph for Text Classification

  2019cites this paper
• Analyzing Disease using Machine Learning Approach over Big Data in Healthcare Sector

  2019cites this paper
• A Two-Stage Deep Neural Network for Sequence Labeling

  2019cites this paper
• Study on text representation method based on deep learning and topic information

  2019cites this paper
• Comparative Study ofConvolutional Neural Network with Word Embedding Technique for Text Classification

  2019influential citation
• Explaining Sentiment Classification

  2019cites this paper
• Semi-Supervised Convolutional Neural Network for Law Advice Online

  2019cites this paper
• Review on Text Sequence Processing with use of different Deep Neural Network Model

  2019cites this paper
• A Short Tour on Improving Disease Prediction by Machine Learning

  2018cites this paper
• An overview of neural networks for drug discovery and the inputs used

  2018cites this paper
• Adversarial Product Review Generation with Word Replacements

  2018cites this paper
• Evaluating deep learning models for sentiment classification

  2018cites this paper
• Pseudo-Supervised Approach for Text Clustering Based on Consensus Analysis

  2018cites this paper
• Persian text classification via character-level convolutional neural networks

  2018cites this paper
• Generative Adversarial Nets for Information Retrieval: Fundamentals and Advances

  2018cites this paper
• Party Matters: Enhancing Legislative Embeddings with Author Attributes for Vote Prediction

  2018cites this paper
• Combinations of Text Preprocessing and Word Embedding Suitable for Neural Network Models for Document Classification

  2018cites this paper
• A Joint Model based on CNN-LSTMs in Dialogue Understanding

  2018cites this paper
• Report on the SIGIR 2016 Workshop on Neural Information Retrieval (Neu-IR)

  2017cites this paper
• Topic classification based on distributed document representation and latent topic information

  2017cites this paper
• Sentiments Analysis of Reviews Based on ARCNN Model

  2017cites this paper
• Text mining to detect indications of fraud in annual reports worldwide

  2017cites this paper
• Comparative Study of CNN and RNN for Natural Language Processing

  2017cites this paper
• Text feature extraction based on deep learning: a review

  2017cites this paper
• Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling

  2016cites this paper