Effective Approaches to Attention-based Neural Machine Translation

An attentional mechanism has lately been used to improve neural machine translation (NMT) by selectively focusing on parts of the source sentence during translation. However, there has been little work exploring useful architectures for attention-based NMT. This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time. We demonstrate the effectiveness of both approaches on the WMT translation tasks between English and German in both directions. With local attention, we achieve a significant gain of 5.0 BLEU points over non-attentional systems that already incorporate known techniques such as dropout. Our ensemble model using different attention architectures yields a new state-of-the-art result in the WMT’15 English to German translation task with 25.9 BLEU points, an improvement of 1.0 BLEU points over the existing best system backed by NMT and an n-gram reranker. 1

• Publication year

  2015

• Venue

  Conference on Empirical Methods in Natural Language Processing

• Publication date

  2015-08-17

• Fields of study

  Linguistics, Computer Science

• Identifiers
  DOI 10.18653/v1/D15-1166arXiv 1508.04025
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• Neural Machine Translation of Rare Words with Subword Units

  20152 semantic links2 concept links0 claim links

  • wmt 15 part of · The WMT English-German translation task is an evaluation task within the WMT 15 benchmark.
  • bleu related to · BLEU and BLEU score refer to the same automatic machine-translation evaluation metric for translation quality.

• An ensemble combining different attention architectures achieves a new state-of-the-art result of 25.9 BLEU on the WMT'15 English-to-German task, exceeding the previous best by 1.0 BLEU.

  Confidence 0.98

  q (76h6bfydm6) extractionAnonymous (12632b8b5f) review
• Local attention yields a 5.0 BLEU-point gain over non-attentional systems that already use dropout.

  Confidence 0.97

  q (76h6bfydm6) extractionAnonymous (12632b8b5f) review
• Global and local attention architectures are both effective on the WMT English-German translation tasks in both directions.

  Confidence 0.95

  q (76h6bfydm6) extractionAnonymous (12632b8b5f) review

• bleu score

  evaluation metric

  An automatic metric used here to evaluate translation quality.

  Aliases: BLEU

  q (76h6bfydm6) extractionAnonymous (12632b8b5f) review
• dropout

  regularization technique

  A regularization technique used in the baseline translation systems described in the abstract.

  q (76h6bfydm6) extractionAnonymous (12632b8b5f) review
• ensemble model

  method

  A combined model that uses multiple attention architectures together for translation.

  Aliases: ensemble

  q (76h6bfydm6) extractionAnonymous (12632b8b5f) review
• global attention

  method

  An attentional mechanism that always attends to all source words during translation.

  Aliases: global attentional mechanism, global approach

  q (76h6bfydm6) extractionAnonymous (12632b8b5f) review
• local attention

  method

  An attentional mechanism that focuses on a subset of source words at each translation step.

  Aliases: local attentional mechanism, local approach

  q (76h6bfydm6) extractionAnonymous (12632b8b5f) review
• n-gram reranker

  component

  An existing reranking component used in the previously best system cited for comparison.

  q (76h6bfydm6) extractionAnonymous (12632b8b5f) review
• non-attentional system

  baseline

  A neural machine translation baseline that does not use an attention mechanism but includes other known techniques.

  Aliases: non-attentional systems

  q (76h6bfydm6) extractionAnonymous (12632b8b5f) review
• wmt english-german translation task

  evaluation setting

  The machine translation evaluation setting involving translation between English and German in the WMT benchmark.

  Aliases: WMT'15 English-to-German translation task, English-German translation task, WMT translation tasks between English and German

  q (76h6bfydm6) extractionAnonymous (12632b8b5f) review

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

  2015influential reference
• DRAW: A Recurrent Neural Network For Image Generation

  2015influential reference
• End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results

  2014cited by this paper
• Addressing the Rare Word Problem in Neural Machine Translation

  2014influential reference
• Sequence to Sequence Learning with Neural Networks

  2014influential reference
• N-gram Counts and Language Models from the Common Crawl

  2014influential reference
• Neural Machine Translation by Jointly Learning to Align and Translate

  2014influential reference
• Recurrent Models of Visual Attention

  2014cited by this paper
• Recurrent Neural Network Regularization

  2014cited by this paper
• On Using Very Large Target Vocabulary for Neural Machine Translation

  2014influential reference
• Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

  2014influential reference
• Recurrent Continuous Translation Models

  2013cited by this paper
• Squibs and Discussions: Measuring Word Alignment Quality for Statistical Machine Translation

  2007cited by this paper
• Alignment by Agreement

  2006cited by this paper
• Statistical Phrase-Based Translation

  2003cited by this paper
• Bleu: a Method for Automatic Evaluation of Machine Translation

  2002cited by this paper

• A dynamic hybrid attention-based autoencoder model with adaptive contextual attention for grammatical error correction

  2026cites this paper
• Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention

  2026cites this paper
• DP-MDLA Net: Detection of smooth pursuit abnormalities in Parkinson’s disease

  2026cites this paper
• Organ-Aware Attention Improves CT Triage and Classification

  2026cites this paper
• Arabic speech command recognition using an enhanced CNN-LSTM model with attention and data augmentation

  2026cites this paper
• PatientFlow: Learning to generate mixed-type longitudinal clinical data with flow matching

  2026cites this paper
• Learning object representations through amortized inference over probabilistic programs

  2026cites this paper
• A Neural Network Informer in Algorithmic Investment Strategies on High-Frequency Bitcoin Data

  2026cites this paper
• Multi-level dynamic fusion and temporal role-aware network for diagnosis prediction.

  2026cites this paper
• You Need Better Attention Priors

  2026cites this paper
• Enhancing Explainable Sequential Recommendation With Disentangled Representations and Auxiliary Review Explanations

  2026cites this paper
• FFCA-UNet: Feature Fusion and Cross-Attention Mechanism for Remote Sensing Image Semantic Segmentation

  2026cites this paper
• EGAM: Extended Graph Attention Model for Solving Routing Problems

  2026cites this paper
• Enhanced Adaptive LSTM Framework for Robust Early-Life Prediction of Lithium-Ion Battery Degradation

  2026cites this paper
• A Dual-Branch Deep Learning Approach for Passive Sonar Underwater Target Classification

  2026cites this paper
• ReaGP: integrating residual units and attention mechanisms in convolution neural network for genomic prediction

  2026cites this paper
• Escaping the forest: a sparse, interpretable, and foundational neural network alternative for tabular data

  2026cites this paper
• A robust natural language text-to-SQL generation framework with dynamic strategies based on LLMs

  2026cites this paper
• Attention–based HAPS–to–ground nodes optimization for differential privacy towards secure semantic communications

  2026cites this paper
• Importance of temporal encoding in data-driven estimation of downhole variables

  2026cites this paper
• Advances in Driver Behavior Classification: Traditional Paradigms, Deep Learning, Generative AI, and Explainable Systems

  2026cites this paper
• Multitask Learning and BERT Embedding: A Comprehensive Approach to Subjectivity Detection and Aspect-Based Sentiment Analysis

  2026cites this paper
• EASLT: Emotion-Aware Sign Language Translation

  2026cites this paper
• Towards Explainable Deep Learning for Ship Trajectory Prediction in Inland Waterways

  2026cites this paper
• Comparative Study of Machine Learning Models for Weather Prediction Using Historical Temperature Data

  2026cites this paper
• Artificial intelligence in wind turbine fault diagnosis: A systematic knowledge mapping and trend analysis

  2026cites this paper
• Multiphysics Modeling of Porous Media Using Temporal Attention Based Deep Learning Framework

  2026cites this paper
• Classification of epileptic seizure using hybrid deep learning framework with time and time-frequency Hjorth features

  2026cites this paper
• Mobile Auslan: A multimodal dialogue-centered sign language learning system

  2026cites this paper
• Fine-grained sensitive node hardening for graph convolutional network systems

  2026cites this paper
• Mechanism Informed Image Feature Decoding for Melt Pool Morphology Evolution Prediction in Laser Powder Bed Fusion

  2026cites this paper
• Progressive multi-level learning for gloss-free sign language translation

  2026cites this paper
• Attention mechanisms in neural networks

  2026cites this paper
• CAPTLDA: Predicting LncRNA-Disease Associations Based on Capsule Networks and Transformers

  2026cites this paper
• A model of errors in transformers

  2026cites this paper
• Cross-Domain Deep Reinforcement Learning for Real-Time Resource Allocation in Transportation Hubs: From Airport Gates to Seaport Berths

  2026cites this paper
• Product Interaction: An Algebraic Formalism for Deep Learning Architectures

  2026cites this paper
• Rational Transductors

  2026cites this paper
• GFASNet: Gait feature attention-driven deep sequential network for dementia-related gait pattern analysis

  2026cites this paper
• Low-Resource Neural Machine Translation Using Recurrent Neural Networks and Transfer Learning: A Case Study on English-to-Igbo

  2025influential citation
• Implementation of Single Candidate Loss Optimization Algorithm for Loss Optimization of Bhojpuri-English Machine Translation Model

  2025cites this paper
• Clean-to-clean: pretraining vision transformers without additional data

  2025cites this paper
• The Cognitive Diagnostic Model Incorporating Convolutional Neural Networks and Attention Mechanisms

  2025cites this paper
• A multi-criteria attention-LSTM approach for enhancing privacy and accuracy in recommender systems

  2025cites this paper
• How Machine Learning Helps in Combating Antimicrobial Resistance: A Review of AMP Analysis and Generation Methods

  2025cites this paper
• Harmonizing Product Strategy with Development Execution: A Comprehensive Study on Quality Engineering Excellence

  2025cites this paper
• Vehicle Acceleration Prediction Considering Environmental Influence and Individual Driving Behavior

  2025cites this paper
• Intelligent Bilingual Reading Translation System Based on Natural Language Processing

  2025cites this paper
• Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translation

  2025cites this paper
• Predicting explainable dementia types with LLM-aided feature engineering

  2025cites this paper
• Imitating Radiological Scrolling: A Global-Local Attention Model for 3D Chest CT Volumes Multi-Label Anomaly Classification

  2025cites this paper
• TRA: Better Length Generalisation with Threshold Relative Attention

  2025cites this paper
• Study on the standardization method of radiotelephony communication in low-altitude airspace based on BART

  2025cites this paper
• Spatio-Temporal Traffic Prediction for Efficient ITS Management

  2025cites this paper
• EnsembleSleepNet: a novel ensemble deep learning model based on transformers and attention mechanisms using multimodal data for sleep stages classification

  2025cites this paper
• Domain-Generalized Emotion Recognition on German Text Corpora

  2025cites this paper
• Predicting Wave Dynamics using Deep Learning with Multistep Integration Inspired Attention and Physics-Based Loss Decomposition

  2025cites this paper
• Out-of-Distribution Image Detection with Residual Spiking Neural Network Architectures

  2025cites this paper
• Rethinking Natural Language Generation with Layer-Wise Multi-View Decoding

  2025cites this paper
• RNN Under the Lens: Attention, Confidence, and Feature Importance

  2025cites this paper
• Detection of AI-Generated Texts: A Bi-LSTM and Attention-Based Approach

  2025influential citation
• Aspect-Based Sentiment Analysis on Movie Reviews using Machine Learning

  2025cites this paper
• Gemma 3 Technical Report

  2025cites this paper
• WatchoutPed: A dataset and model for Vulnerable Pedestrian Anticipation in surveillance videos

  2025cites this paper
• KPQDG: a key sentence prompt-tuning for automatic question-distractor pairs generation

  2025cites this paper
• Cross-Lingual Transfer Learning for Low-Resource Languages: Expanding NLP’s Global Reach

  2025cites this paper
• When deep learning deciphers silent video: a survey on automatic deep lip reading

  2025cites this paper
• Attentional Mechanisms for Enhancement in the Convolutional Neural Network Task of Recognizing Handwritten Chinese Character Styles

  2025cites this paper
• Temporal-contextual Event Learning for Pedestrian Crossing Intent Prediction

  2025cites this paper
• Adaptive Head Pruning for Attention Mechanism in the Maritime Domain

  2025cites this paper
• Multidemand Forecasting for Electric Vehicle Charging Stations Under Time-of-Use Strategy via Attention-Based Deep Neural Network

  2025influential citation
• TokenButler: Token Importance is Predictable

  2025cites this paper
• Bandwidth-Efficient Communication Modelling for Autonomous Vehicle Collaborative Perception

  2025cites this paper
• A multi-stage fusion deep learning framework merging local patterns with attention-driven contextual dependencies for cancer detection

  2025cites this paper
• Spatio–temporal enhanced anomaly detection in FRP bridge monitoring using MPCA, biGRU, and attention mechanisms

  2025influential citation
• Graph convolutional network based on self-attention variational autoencoder and capsule contrastive learning for aspect-based sentiment analysis

  2025cites this paper
• Automatic Paraphrase Generation at Phrasal, and Sentence Level for Urdu Language: Data and Methods

  2025cites this paper
• Real-Time Posture Correction of Squat Exercise: A Deep Learning Approach for Performance Analysis and Error Correction

  2025cites this paper
• RNA structure prediction using deep learning - A comprehensive review

  2025cites this paper
• Unveiling Reasoning Thresholds in Language Models: Scaling, Fine-Tuning, and Interpretability through Attention Maps

  2025cites this paper
• Sign language recognition using LSTM

  2025cites this paper
• A Dual-Stage Time-Context Network for Speech-Based Alzheimer’s Disease Detection

  2025cites this paper
• Positional Encoding in Transformer-Based Time Series Models: A Survey

  2025cites this paper
• A two-stage multimodal emotion analysis using body actions and facial features

  2025cites this paper
• MISS: Multiple information span scoring for Chinese named entity recognition

  2025cites this paper
• Deep learning methods for clinical workflow phase-based prediction of procedure duration: a benchmark study

  2025influential citation
• Long-Term Sales Forecasting for Apparel Sales Based on Transformer

  2025cites this paper
• Generative Artificial Intelligence: Evolving Technology, Growing Societal Impact, and Opportunities for Information Systems Research

  2025cites this paper
• AlphaRouter: Bridging the Gap Between Reinforcement Learning and Optimization for Vehicle Routing with Monte Carlo Tree Searches

  2025cites this paper
• ProvGuard: Detecting SDN Control Policy Manipulation via Contextual Semantics of Provenance Graphs

  2025cites this paper
• Stress Detection using Multimodal Representation Learning, Fusion Techniques, and Applications

  2025cites this paper
• Complex Networks for Pattern-Based Data Classification

  2025cites this paper
• EKBSA: A Chinese Sentiment Analysis Model by Enhancing K-BERT

  2025cites this paper
• AMuSE: Attentive Multilingual Speech Encoding for Zero-Prior ASR

  2025cites this paper
• An empirical investigation of the neural base approaches based on the sentence length using low-resource language: English-to-Nyishi

  2025influential citation
• Capturing spatiotemporal dependencies with competitive set attention for video summarization

  2025cites this paper
• LatentDE: latent-based directed evolution for protein sequence design

  2025cites this paper
• Sentiment-Aware Machine Translation for Indic Languages

  2025cites this paper
• Language Model Uncertainty Quantification with Attention Chain

  2025cites this paper
• A Patch-Wise Mechanism for Enhancing Sparse Radar Echo Extrapolation in Precipitation Nowcasting

  2025cites this paper