Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems

We propose a simplified model of attention which is applicable to feed-forward neural networks and demonstrate that the resulting model can solve the synthetic "addition" and "multiplication" long-term memory problems for sequence lengths which are both longer and more widely varying than the best published results for these tasks.

• Publication year

  2015

• Venue

  arXiv.org

• Publication date

  2015-12-29

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 1512.08756
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Pruning subsequence search with attention-based embedding

  2016cited by this paper
• Unitary Evolution Recurrent Neural Networks

  2015cited by this paper
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

  2015cited by this paper
• End-To-End Memory Networks

  2015cited by this paper
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units

  2015influential reference
• Molding CNNs for text: non-linear, non-consecutive convolutions

  2015influential reference
• End-to-end attention-based large vocabulary speech recognition

  2015cited by this paper
• Regularizing RNNs by Stabilizing Activations

  2015cited by this paper
• Lasagne: First release.

  2015cited by this paper
• Recommending music on Spotify with deep learning – Sander Dieleman

  2015cited by this paper
• Weakly Supervised Memory Networks

  2015cited by this paper
• Convolutional LSTM Networks for Subcellular Localization of Proteins

  2015cited by this paper
• Neural Turing Machines

  2014cited by this paper
• RECURRENT NEURAL NETWORKS

  2014cited by this paper
• Adam: A Method for Stochastic Optimization

  2014cited by this paper
• Neural Machine Translation by Jointly Learning to Align and Translate

  2014influential reference
• Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

  2014cited by this paper
• Generating Sequences With Recurrent Neural Networks

  2013cited by this paper
• On the importance of initialization and momentum in deep learning

  2013influential reference
• Rectifier Nonlinearities Improve Neural Network Acoustic Models

  2013cited by this paper
• Network In Network

  2013cited by this paper
• Long Short-Term Memory in Echo State Networks: Details of a Simulation Study

  2012influential reference
• Theano: new features and speed improvements

  2012cited by this paper
• On the difficulty of training recurrent neural networks

  2012cited by this paper
• Supervised Sequence Labelling with Recurrent Neural Networks

  2012cited by this paper
• Learning Recurrent Neural Networks with Hessian-Free Optimization

  2011influential reference
• Text Categorization with Support Vector Machines: Learning with Many Relevant Features

  1999cited by this paper
• Long Short-Term Memory

  1997influential reference
• Learning long-term dependencies with gradient descent is difficult

  1994cited by this paper
• Backpropagation Through Time: What It Does and How to Do It

  1990cited by this paper

• Investigation for Relative Voice Impression Estimation

  2026cites this paper
• Research on Online Log Anomaly Detection Model Based on Informer

  2025cites this paper
• RadSee: See Your Handwriting Through Walls Using FMCW Radar

  2025cites this paper
• Research on patent classification technology based on deep learning

  2025cites this paper
• Predicting the Remaining Useful Life of Aircraft Engines Using Spatial and Temporal Attention Mechanisms

  2025cites this paper
• Asthma and Chronic Obstructive Pulmonary Disease (COPD) Prediction Using PEEP Expiratory Occlusion Data: A Hybrid Approach with Random Forest and Feedforward Neural Network

  2025cites this paper
• Deep Learning-Based Gesture Recognition for Myoelectric Prosthetic Control Using Surface Electromyography

  2025cites this paper
• Quantity versus diversity: Influence of data on detecting EEG pathology with advanced ML models

  2025cites this paper
• CNAttention: an attention-based deep multiple-instance method for uncovering copy number aberration signatures across cancers

  2025cites this paper
• Attention-Aided Channel Prediction for Efficient Resource Management in Industrial IoT Subnetworks

  2025cites this paper
• Psychomedical named entity recognition method based on multi-level feature extraction and multi-granularity embedding fusion

  2025cites this paper
• Multi-interaction TTS toward professional recording reproduction

  2025cites this paper
• Efficient deep neural networks for cancer detection on histopathology combining attention and image downsampling

  2025cites this paper
• Camscribe: Enhanced Dashcam Video Descriptions Through Multimodal Spatiotemporal and Object Detection for Autonomous Vehicles

  2025cites this paper
• A Deep-Learning Method for Remaining Useful Life Prediction of Power Machinery via Dual-Attention Mechanism

  2025cites this paper
• An explainable artificial intelligence - human collaborative model for investigating patent novelty

  2025cites this paper
• Multivariable Environmental Prediction for Chinese Solar Greenhouse IoT System Based on FB-LSTM Model

  2025cites this paper
• Attention-Based Generative Adversarial Network Model for RC Column Backbone Curve Identification

  2025cites this paper
• Enhancing Wind Power Forecasting at Local Peak Points: A Novel Seq2LPP Model

  2025cites this paper
• Multiple Instance Learning With Instance-Level Positive-Unlabeled Learning in Anomaly Detection

  2025cites this paper
• Voice Impression Control in Zero-Shot TTS

  2025cites this paper
• Human Activity Recognition via Attention-Augmented TCN-BiGRU Fusion

  2025cites this paper
• Can Attention Improve Sequence-to-Point Load Disaggregation? A Comparative Assessment

  2025cites this paper
• Wearable Inertial Sensor-Based Recognition of Swimming Movements and Postures

  2025cites this paper
• Task-Level Insights from Eigenvalues across Sequence Models

  2025cites this paper
• Enhancing Human Activity Recognition in Smart Homes with Self-Supervised Learning and Self-Attention

  2024cites this paper
• Determining community happiness index with transformers and attention-based deep learning

  2024cites this paper
• An Effective Multi-Time Series Model of RC Column Backbone Curve Identification

  2024cites this paper
• Survival Prediction via Deep Attention-Based Multiple-Instance Learning Networks with Instance Sampling

  2024cites this paper
• Device Position-Independent Human Activity Recognition with Wearable Sensors Using Deep Neural Networks

  2024cites this paper
• Comfort-Aware Lane Change Planning With Exit Strategy for Autonomous Vehicle

  2024cites this paper
• Language Models for Hierarchical Classification of Radiology Reports With Attention Mechanisms, BERT, and GPT-4

  2024cites this paper
• Predicting the Unpredictable: Advancements in Earthquake Forecasting Using Artificial Intelligence and LSTM Networks

  2024cites this paper
• Enhanced industrial text classification via hyper variational graph-guided global context integration

  2024cites this paper
• A novel approach for APT attack detection based on feature intelligent extraction and representation learning

  2024cites this paper
• Description-Based Controllable Text-to-Speech With Cross-Lingual Voice Control

  2024cites this paper
• Learning patterns of HIV-1 resistance to broadly neutralizing antibodies with reduced subtype bias using multi-task learning

  2024cites this paper
• A novel approach for APT attack detection based on an advanced computing

  2024cites this paper
• Emotional Polarity Attention Mechanism for Text Sentiment Analysis

  2024cites this paper
• Enhancing Apparent Personality Trait Analysis with Cross-Modal Embeddings

  2024cites this paper
• A Systematic Review of Attention Models in Natural Language Processing

  2024cites this paper
• Extracting problem and method sentence from scientific papers: a context-enhanced transformer using formulaic expression desensitization

  2024cites this paper
• Open-Circuit Fault Diagnosis for Grid-tied T-Type Inverters Using an LSTM Autoencoder

  2023cites this paper
• Event-Based Contrastive Learning for Medical Time Series

  2023cites this paper
• Multi-Task Learning for Game Review Classification with Emotion and Sarcasm Detection

  2023cites this paper
• Demographic Information Fusion Using Attentive Pooling In CNN-GRU Model For Systolic Blood Pressure Estimation

  2023cites this paper
• Learning patterns of HIV-1 co-resistance to broadly neutralizing antibodies with reduced subtype bias using multi-task learning

  2023cites this paper
• Enhancing Instance-Level Image Classification with Set-Level Labels

  2023cites this paper
• DeepASDPred: a CNN-LSTM-based deep learning method for Autism spectrum disorders risk RNA identification

  2023cites this paper
• DGA domain name detection model based on multiscale feature

  2023cites this paper
• A writing style-based multi-task model with the hierarchical attention for rumor detection

  2023cites this paper
• Left-right brain interaction inspired bionic deep network for forecasting significant wave height

  2023cites this paper
• Knowledge-Enriched Multi-Cross Attention Network for Legal Judgment Prediction

  2023cites this paper
• Attention-Based Residual BiLSTM Networks for Human Activity Recognition

  2023influential citation
• A Systematic Review of Deep Learning-based Research on Radiology Report Generation

  2023cites this paper
• Deep Citywide Multisource Data Fusion-Based Air Quality Estimation

  2023cites this paper
• DeepLBCEPred: A Bi-LSTM and multi-scale CNN-based deep learning method for predicting linear B-cell epitopes

  2023influential citation
• Fine-Grained Motion Recognition in At-Home Fitness Monitoring with Smartwatch: A Comparative Analysis of Explainable Deep Neural Networks

  2023cites this paper
• Augmenting Feature Representation with Gradient Penalty for Robust Text Categorization

  2023cites this paper
• Improved Grey Wolf Optimization- (IGWO-) Based Feature Selection on Multiview Features and Enhanced Multimodal-Sequential Network Intrusion Detection Approach

  2023cites this paper
• Remaining Useful Life Prediction of Turbofan Engines Using CNN-LSTM-SAM Approach

  2023cites this paper
• Named Entity Recognition Model Based on Feature Fusion

  2023cites this paper
• Assessment of few-hits machine learning classification algorithms for low-energy physics in liquid argon detectors

  2023cites this paper
• An Advanced Attention-Enhanced Hybrid Deep Learning Model for WBSNs’ Gait Pattern Recognition

  2022cites this paper
• An empirical evaluation of deep learning‐based source code vulnerability detection: Representation versus models

  2022cites this paper
• Image-text interaction graph neural network for image-text sentiment analysis

  2022cites this paper
• Taxi demand forecasting based on the temporal multimodal information fusion graph neural network

  2022cites this paper
• SMU Data

  2022cites this paper
• Inductive Representation Learning on Dynamic Stock Co-Movement Graphs for Stock Predictions

  2022cites this paper
• A Survey on Dynamic Graph Neural Networks Modeling

  2022cites this paper
• ProcGuard: Process Injection Behaviours Detection Using Fine-grained Analysis of API Call Chain with Deep Learning

  2022cites this paper
• Non-Autoregressive Speech Recognition with Error Correction Module

  2022cites this paper
• LMFE: Learning-Based Multiscale Feature Engineering in Partial Discharge Detection

  2022cites this paper
• Language Identification-Based Evaluation of Single Channel Speech Separation of Overlapped Speeches

  2022cites this paper
• Deep learning with an attention mechanism for continuous biomechanical motion estimation across varied activities

  2022influential citation
• Dynamic global structure enhanced multi-channel graph neural network for session-based recommendation

  2022cites this paper
• Multi-label annotation of text reports from computed tomography of the chest, abdomen, and pelvis using deep learning

  2022cites this paper
• ArmanEmo: a Persian dataset for text-based emotion detection

  2022cites this paper
• A Combined Model for Multivariate Time Series Forecasting Based on MLP-Feedforward Attention-LSTM

  2022influential citation
• Joint Speaker Encoder and Neural Back-end Model for Fully End-to-End Automatic Speaker Verification with Multiple Enrollment Utterances

  2022cites this paper
• Deep Feature Learning Based Fault Detection with High-Frequency Signals

  2022cites this paper
• Enhancer-LSTMAtt: A Bi-LSTM and Attention-Based Deep Learning Method for Enhancer Recognition

  2022cites this paper
• Spoofing-Aware Attention based ASV Back-end with Multiple Enrollment Utterances and a Sampling Strategy for the SASV Challenge 2022

  2022influential citation
• Emotion Recognition of Students’ Bilingual Tweets during COVID-19 Pandemic using Attention-based Bi-GRU

  2022cites this paper
• Privacy of Vehicular Time Series Data

  2022cites this paper
• An Efficient Lightweight Hybrid Model with Attention Mechanism for Enhancer Sequence Recognition

  2022cites this paper
• A Multi-Head Convolutional Neural Network Based Non-Intrusive Load Monitoring Algorithm Under Dynamic Grid Voltage Conditions

  2022cites this paper
• A deep learning approach for detecting drill bit failures from a small sound dataset

  2022cites this paper
• Multitasking Learning Model Based on Hierarchical Attention Network for Arabic Sentiment Analysis Classification

  2022cites this paper
• Attention-Based RU-BiLSTM Sentiment Analysis Model for Roman Urdu

  2022cites this paper
• N-ACT: An Interpretable Deep Learning Model for Automatic Cell Type and Salient Gene Identification

  2022cites this paper
• Confidence-based laboratory test reduction recommendation algorithm

  2022cites this paper
• A New Method for WebShell Detection Based on Bidirectional GRU and Attention Mechanism

  2022cites this paper
• Language vs Speaker Change: A Comparative Study

  2022cites this paper
• Investigation of Training Multiple Instance Learning Networks with Instance Sampling

  2022cites this paper
• Lite Unified Modeling for Discriminative Reading Comprehension

  2022cites this paper
• A hybrid method for online cycle life prediction of lithium‐ion batteries

  2022cites this paper
• Multi-channel word embeddings for sentiment analysis

  2022cites this paper
• Software Systems Security Vulnerabilities Management by Exploring the Capabilities of Language Models Using NLP

  2021cites this paper
• NCYPred: A Bidirectional LSTM Network With Attention for Y RNA and Short Non-Coding RNA Classification

  2021cites this paper