Natural Language Multitasking: Analyzing and Improving Syntactic Saliency of Hidden Representations

We train multi-task autoencoders on linguistic tasks and analyze the learned hidden sentence representations. The representations change significantly when translation and part-of-speech decoders are added. The more decoders a model employs, the better it clusters sentences according to their syntactic similarity, as the representation space becomes less entangled. We explore the structure of the representation space by interpolating between sentences, which yields interesting pseudo-English sentences, many of which have recognizable syntactic structure. Lastly, we point out an interesting property of our models: The difference-vector between two sentences can be added to change a third sentence with similar features in a meaningful way.

• Publication year

  2018

• Venue

  Neural Information Processing Systems

• Publication date

  2018-01-18

• Fields of study

  Mathematics, Linguistics, Computer Science

• Identifiers
  arXiv 1801.06024
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Exploiting Linguistic Resources for Neural Machine Translation Using Multi-task Learning

  2017cited by this paper
• Unsupervised Neural Machine Translation

  2017cited by this paper
• Image Style Transfer Using Convolutional Neural Networks

  2016cited by this paper
• Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval

  2015cited by this paper
• Multi-task Sequence to Sequence Learning

  2015cited by this paper
• Sequence to Sequence Learning with Neural Networks

  2014cited by this paper
• Distributed Representations of Sentences and Documents

  2014cited by this paper
• Show and tell: A neural image caption generator

  2014cited by this paper
• Distributed Representations of Words and Phrases and their Compositionality

  2013cited by this paper
• NLTK: The Natural Language Toolkit

  2006cited by this paper
• Europarl: A Parallel Corpus for Statistical Machine Translation

  2005cited by this paper

• P.STAR: Parameter-free Structure Analysis via Reordering for Lossless Visualization of Layers in Language Models

  2025cites this paper
• Panorama des outils de visualisation pour l'explicabilité en apprentissage profond pour le traitement automatique de la langue

  2023cites this paper
• Post-hoc Interpretability for Neural NLP: A Survey

  2021cites this paper
• A Survey of the Usages of Deep Learning for Natural Language Processing

  2020cites this paper
• Explaining Word Embeddings via Disentangled Representation

  2020cites this paper
• Understanding EFL Linguistic Models through Relationship between Natural Language Processing and Artificial Intelligence Applications

  2019cites this paper
• Long short-term memory for a model-free estimation of macronutrient ion concentrations of root-zone in closed-loop soilless cultures

  2019cites this paper
• On internal language representations in deep learning: an analysis of machine translation and speech recognition

  2018cites this paper
• Analysis Methods in Neural Language Processing: A Survey

  2018cites this paper
• A Survey of the Usages of Deep Learning in Natural Language Processing

  2018cites this paper
• Identifying and Controlling Important Neurons in Neural Machine Translation

  2018cites this paper
• Adversarial Input in Games and Markets

  2016cites this paper