Statistical Machine Translation Features with Multitask Tensor Networks

We present a three-pronged approach to improving Statistical Machine Translation (SMT), building on recent success in the application of neural networks to SMT. First, we propose new features based on neural networks to model various non-local translation phenomena. Second, we augment the architecture of the neural network with tensor layers that capture important higher-order interaction among the network units. Third, we apply multitask learning to estimate the neural network parameters jointly. Each of our proposed methods results in significant improvements that are complementary. The overall improvement is +2.7 and +1.8 BLEU points for Arabic-English and Chinese-English translation over a state-of-the-art system that already includes neural network features.

• Publication year

  2015

• Venue

  Annual Meeting of the Association for Computational Linguistics

• Publication date

  2015-06-01

• Fields of study

  Computer Science

• Identifiers
  DOI 10.3115/v1/P15-1004arXiv 1506.00698
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

  2016cited by this paper
• Sequence to Sequence Learning with Neural Networks

  2014cited by this paper
• Fast and Robust Neural Network Joint Models for Statistical Machine Translation

  2014influential reference
• Max-Margin Tensor Neural Network for Chinese Word Segmentation

  2014cited by this paper
• A Convolutional Neural Network for Modelling Sentences

  2014cited by this paper
• Neural Machine Translation by Jointly Learning to Align and Translate

  2014cited by this paper
• Translation Modeling with Bidirectional Recurrent Neural Networks

  2014cited by this paper
• Joint inference of entities, relations, and coreference

  2013cited by this paper
• Factored Soft Source Syntactic Constraints for Hierarchical Machine Translation

  2013cited by this paper
• Tensor Deep Stacking Networks

  2013cited by this paper
• Joint Language and Translation Modeling with Recurrent Neural Networks

  2013influential reference
• Two-Neighbor Orientation Model with Cross-Boundary Global Contexts

  2013cited by this paper
• Reasoning With Neural Tensor Networks for Knowledge Base Completion

  2013cited by this paper
• Morphological Analysis and Disambiguation for Dialectal Arabic

  2013cited by this paper
• Recurrent Continuous Translation Models

  2013cited by this paper
• Continuous Space Translation Models with Neural Networks

  2012cited by this paper
• Continuous Space Translation Models for Phrase-Based Statistical Machine Translation

  2012cited by this paper
• Large Vocabulary Speech Recognition Using Deep Tensor Neural Networks

  2012cited by this paper
• Trait-Based Hypothesis Selection For Machine Translation

  2012cited by this paper
• Parsing Natural Scenes and Natural Language with Recursive Neural Networks

  2011cited by this paper
• Natural Language Processing (Almost) from Scratch

  2011cited by this paper
• BBN System Description for WMT10 System Combination Task

  2010cited by this paper
• String-to-Dependency Statistical Machine Translation

  2010influential reference
• Lexical Features for Statistical Machine Translation

  2009cited by this paper
• Joint Parsing and Named Entity Recognition

  2009cited by this paper
• 11,001 New Features for Statistical Machine Translation

  2009cited by this paper
• Language and Translation Model Adaptation using Comparable Corpora

  2008cited by this paper
• A unified architecture for natural language processing: deep neural networks with multitask learning

  2008cited by this paper
• Improving Statistical Machine Translation Using Word Sense Disambiguation

  2007cited by this paper
• Continuous Space Language Models for Statistical Machine Translation

  2006cited by this paper
• A Unigram Orientation Model for Statistical Machine Translation

  2004cited by this paper
• A Neural Probabilistic Language Model

  2003cited by this paper
• A Systematic Comparison of Various Statistical Alignment Models

  2003cited by this paper
• An Empirical Evaluation of Knowledge Sources and Learning Algorithms for Word Sense Disambiguation

  2002cited by this paper
• Multitask Learning

  1997cited by this paper
• Large vocabulary speech recognition

  1995influential reference
• The Mathematics of Statistical Machine Translation: Parameter Estimation

  1993influential reference

• Deep Learning for Natural Language Processing: A Survey

  2023cites this paper
• Neural Translation System of Meta-Domain Transfer Based on Self-Ensemble and Self-Distillation

  2022cites this paper
• PASTA: a parallel sparse tensor algorithm benchmark suite

  2019cites this paper
• Reasoning over Arabic WordNet Relations with Neural Tensor Network

  2019cites this paper
• Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization

  2019cites this paper
• Transferring Informal Text in Arabic as Low Resource Languages: State-of-the-Art and Future Research Directions

  2019cites this paper
• From Feature to Paradigm: Deep Learning in Machine Translation (Extended Abstract)

  2018cites this paper
• BBN’s low-resource machine translation for the LoReHLT 2016 evaluation

  2018cites this paper
• Word Emotion Induction for Multiple Languages as a Deep Multi-Task Learning Problem

  2018cites this paper
• Review of state-of-the-arts in artificial intelligence with application to AI safety problem

  2016cites this paper
• Research on Intelligent Automatic Translation System in Chinese and English Based on Integration Technology

  2016cites this paper
• Convolution-Enhanced Bilingual Recursive Neural Network for Bilingual Semantic Modeling

  2016cites this paper
• A Continuous Space Rule Selection Model for Syntax-based Statistical Machine Translation

  2016cites this paper
• Left-to-Right Hierarchical Phrase-based Machine Translation

  2016cites this paper
• Multilingual Word Embeddings using Multigraphs

  2016cites this paper
• First Result on Arabic Neural Machine Translation

  2016cites this paper
• On the Expressive Power of Deep Learning: A Tensor Analysis

  2015cites this paper