Not All Contexts Are Created Equal: Better Word Representations with Variable Attention

We introduce an extension to the bag-ofwords model for learning words representations that take into account both syntactic and semantic properties within language. This is done by employing an attention model that finds within the contextual words, the words that are relevant for each prediction. The general intuition of our model is that some words are only relevant for predicting local context (e.g. function words), while other words are more suited for determining global context, such as the topic of the document. Experiments performed on both semantically and syntactically oriented tasks show gains using our model over the existing bag of words model. Furthermore, compared to other more sophisticated models, our model scales better as we increase the size of the context of the model.

• Publication year

  2015

• Venue

  Conference on Empirical Methods in Natural Language Processing

• Publication date

  2015-09-01

• Fields of study

  Computer Science

• Identifiers
  DOI 10.18653/v1/D15-1161
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation

  2015influential reference
• Unsupervised POS Induction with Word Embeddings

  2015influential reference
• Two/Too Simple Adaptations of Word2Vec for Syntax Problems

  2015cited by this paper
• word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method

  2014cited by this paper
• A Dependency Parser for Tweets

  2014cited by this paper
• Revisiting Embedding Features for Simple Semi-supervised Learning

  2014cited by this paper
• Fast and Robust Neural Network Joint Models for Statistical Machine Translation

  2014cited by this paper
• Tailoring Continuous Word Representations for Dependency Parsing

  2014cited by this paper
• A Fast and Accurate Dependency Parser using Neural Networks

  2014cited by this paper
• Sequence to Sequence Learning with Neural Networks

  2014cited by this paper
• Neural Machine Translation by Jointly Learning to Align and Translate

  2014cited by this paper
• A Recursive Recurrent Neural Network for Statistical Machine Translation

  2014cited by this paper
• Distributed Representations of Words and Phrases and their Compositionality

  2013influential reference
• Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

  2013cited by this paper
• Recurrent Continuous Translation Models

  2013cited by this paper
• Learning Generative Models with Visual Attention

  2013cited by this paper
• Improving Word Representations via Global Context and Multiple Word Prototypes

  2012cited by this paper
• A Better Way to Pretrain Deep Boltzmann Machines

  2012cited by this paper
• Natural Language Processing (Almost) from Scratch

  2011cited by this paper
• Word Representations: A Simple and General Method for Semi-Supervised Learning

  2010cited by this paper
• Why Does Unsupervised Pre-training Help Deep Learning?

  2010cited by this paper
• The CoNLL 2007 Shared Task on Dependency Parsing

  2007cited by this paper
• V-Measure: A Conditional Entropy-Based External Cluster Evaluation Measure

  2007cited by this paper

• Effective document summarization: a hybrid clustering approach using transformer model

  2026cites this paper
• HalluJudge: A Reference-Free Hallucination Detection for Context Misalignment in Code Review Automation

  2026cites this paper
• Concatenated Vector Representation with the Asymmetric GloVe Model

  2025cites this paper
• Polarity-aware deep attention network for aspect-based sentiment analysis

  2024cites this paper
• MANE: Organizational Network Embedding With Multiplex Attentive Neural Networks

  2023cites this paper
• Classification of Documents with Variable Length Using Likelihood Extracted Discriminative Attributes

  2023cites this paper
• Integrating graph embedding and neural models for improving transition-based dependency parsing

  2023cites this paper
• A longitudinal investigation of the impact of code-switching versus target language only on Saudi EFL students’ written receptive vocabulary knowledge development in terms of breadth

  2023cites this paper
• Automatic Question Generation based on Sentence Structure Analysis

  2023cites this paper
• Advances in monolingual and crosslingual automatic disability annotation in Spanish

  2023cites this paper
• Multi-View Multi-Task Campaign Embedding for Cold-Start Conversion Rate Forecasting

  2023cites this paper
• Polarity enriched attention network for aspect-based sentiment analysis

  2022cites this paper
• Multi-Label Annotation and Classification of Arabic Texts Based on Extracted Seed Keyphrases and Bi-Gram Alphabet Feed Forward Neural Networks Model

  2022cites this paper
• Robust and Informative Text Augmentation (RITA) via Constrained Worst-Case Transformations for Low-Resource Named Entity Recognition

  2022cites this paper
• Attentive Dual Embedding for Understanding Medical Concept in Electronic Health Record

  2021cites this paper
• Low Resource Neural Machine Translation: Assamese to/from Other Indo-Aryan (Indic) Languages

  2021cites this paper
• Morpheme Embedding for Bahasa Indonesia Using Modified Byte Pair Encoding

  2021cites this paper
• BE-BLC: BERT-ELMO-Based Deep Neural Network Architecture for English Named Entity Recognition Task

  2021cites this paper
• A Survey On Neural Word Embeddings

  2021cites this paper
• A survey on training and evaluation of word embeddings

  2021influential citation
• Shallow learning for MTL in end-to-end RNN for basic sequence tagging

  2021cites this paper
• Learning chinese word embeddings from character structural information

  2020cites this paper
• Citation Recommendations Considering Content and Structural Context Embedding

  2020cites this paper
• Universal Adversarial Attack on Attention and the Resulting Dataset DAmageNet

  2020cites this paper
• Deep Learning Architectures for Named Entity Recognition: A Survey

  2020cites this paper
• Revisiting the Context Window for Cross-lingual Word Embeddings

  2020cites this paper
• Attention Word Embedding

  2020influential citation
• DocCit2Vec: Citation Recommendation via Embedding of Content and Structural Contexts

  2020cites this paper
• Aspect terms grouping via fusing concepts and context information

  2020cites this paper
• Dual Attention Model for Citation Recommendation

  2020cites this paper
• Word embeddings for biomedical natural language processing: A survey

  2020cites this paper
• Patient Representation From Structured Electronic Medical Records Based on Embedding Technique: Development and Validation Study

  2020cites this paper
• A Survey on Recent Advances in Sequence Labeling from Deep Learning Models

  2020cites this paper
• Modeling both Intra- and Inter-modal Influence for Real-Time Emotion Detection in Conversations

  2020cites this paper
• Bi-Directional Recurrent Attentional Topic Model

  2020cites this paper
• Effective attention-based network for syndrome differentiation of AIDS

  2020cites this paper
• Measuring the effect of different types of unsupervised word representations on Medical Named Entity Recognition

  2019cites this paper
• TEALS: Time-aware Text Embedding Approach to Leverage Subgraphs

  2019cites this paper
• TEAGS: time-aware text embedding approach to generate subgraphs

  2019cites this paper
• Learning word hierarchical representations with neural networks for document modeling

  2019cites this paper
• Vector-Space Models of Semantic Representation From a Cognitive Perspective: A Discussion of Common Misconceptions

  2019cites this paper
• Temporal Self-Attention Network for Medical Concept Embedding

  2019cites this paper
• Automatic Classification and Reporting of Multiple Common Thorax Diseases Using Chest Radiographs

  2019cites this paper
• Identifying Implicit Polarity of Events by Using an Attention-Based Neural Network Model

  2019cites this paper
• ICNet: Incorporating Indicator Words and Contexts to Identify Functional Description Information

  2019cites this paper
• Hierarchical Multi-dimensional Attention Model for Answer Selection

  2019cites this paper
• Attentive Dual Embedding for Understanding Medical Concepts in Electronic Health Records

  2019cites this paper
• Improving Word Representations: A Sub-sampled Unigram Distribution for Negative Sampling

  2019cites this paper
• SoulMate: Short-Text Author Linking Through Multi-Aspect Temporal-Textual Embedding

  2019cites this paper
• Contextualized Word Representations from Distant Supervision with and for NER

  2019cites this paper
• An Improved Adaptive and Structured Sentence Embedding

  2019cites this paper
• Machine Learning Models for Efficient and Robust Natural Language Processing

  2019cites this paper
• Real-Time Emotion Recognition via Attention Gated Hierarchical Memory Network

  2019influential citation
• Understanding Human and Predictive Moderation of Online Science Discourse

  2019cites this paper
• Unsupervised Training of Automatic Dialogue Systems for Customer Support

  2019cites this paper
• Analysis of Mir-22 Relative Expression in Breast Cancer Patients

  2019cites this paper
• Deep Learning and Convolutional Neural Networks for Medical Imaging and Clinical Informatics

  2019cites this paper
• An Efficient Framework for Sentence Similarity Modeling

  2019cites this paper
• Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts

  2019cites this paper
• AILA: Attentive Interactive Labeling Assistant for Document Classification through Attention-Based Deep Neural Networks

  2019cites this paper
• Multi-Entity Aspect-Based Sentiment Analysis with Context, Entity, Aspect Memory and Dependency Information

  2019cites this paper
• Understanding prediction of low-quality comments in online science discourse

  2019cites this paper
• Parsing Coordination For Spoken Language Understanding

  2018cites this paper
• Diacritization of Maghrebi Arabic Sub-Dialects

  2018cites this paper
• Understanding Actors and Evaluating Personae with Gaussian Embeddings

  2018cites this paper
• Double Attention Mechanism for Sentence Embedding

  2018cites this paper
• Improving Word Embeddings by Emphasizing Co-hyponyms

  2018cites this paper
• Directional Skip-Gram: Explicitly Distinguishing Left and Right Context for Word Embeddings

  2018influential citation
• TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays

  2018cites this paper
• Bio-SimVerb and Bio-SimLex: wide-coverage evaluation sets of word similarity in biomedicine

  2018cites this paper
• ConceptVector: Text Visual Analytics via Interactive Lexicon Building Using Word Embedding

  2018cites this paper
• Deep Learning in Social Computing

  2018cites this paper
• Learning Joint Gaussian Representations for Movies, Actors, and Literary Characters

  2018cites this paper
• Named Entity Recognition Based on BiRHN and CRF

  2018cites this paper
• Recurrent neural network-based models for recognizing requisite and effectuation parts in legal texts

  2018cites this paper
• Leveraging Conceptualization for Short-Text Embedding

  2018cites this paper
• Medical Concept Embedding with Time-Aware Attention

  2018cites this paper
• Robust Lexical Features for Improved Neural Network Named-Entity Recognition

  2018influential citation
• Model-Free Context-Aware Word Composition

  2018cites this paper
• Embedded-State Latent Conditional Random Fields for Sequence Labeling

  2018cites this paper
• Question Passage Question & Passage Encoding Question-Passage Matching Passage Self-Matching Word Character Answer Prediction

  2017cites this paper
• Bib2vec: An Embedding-based Search System for Bibliographic Information

  2017cites this paper
• Reconstruction of Word Embeddings from Sub-Word Parameters

  2017cites this paper
• Modelling Semantic Context of OOV Words in Large Vocabulary Continuous Speech Recognition

  2017cites this paper
• Fast and Accurate Sequence Labeling with Iterated Dilated Convolutions

  2017cites this paper
• Low-Rank Hidden State Embeddings for Viterbi Sequence Labeling

  2017cites this paper
• Nested Named Entity Recognition Using Multilayer Recurrent Neural Networks

  2017cites this paper
• A Bidirectional LSTM and Conditional Random Fields Approach to Medical Named Entity Recognition

  2017cites this paper
• Applying Existing Named Entity Taggers at BARR IBEREVAL 2017 Task

  2017cites this paper
• Word Ordering as Unsupervised Learning Towards Syntactically Plausible Word Representations

  2017cites this paper
• Syllable-aware Neural Language Models: A Failure to Beat Character-aware Ones

  2017cites this paper
• Japanese Sentiment Classification using a Tree-Structured Long Short-Term Memory with Attention

  2017cites this paper
• Recurrent Attentional Topic Model

  2017cites this paper
• Dependency Parsing with Dilated Iterated Graph CNNs

  2017cites this paper
• Gated Self-Matching Networks for Reading Comprehension and Question Answering

  2017cites this paper
• A Structured Self-attentive Sentence Embedding

  2017cites this paper
• Capturing Long-range Contextual Dependencies with Memory-enhanced Conditional Random Fields

  2017cites this paper
• Fast and Accurate Entity Recognition with Iterated Dilated Convolutions

  2017cites this paper
• Stacking With Auxiliary Features for Entity Linking in the Medical Domain

  2017cites this paper
• Episodic CAMN: Contextual Attention-Based Memory Networks with Iterative Feedback for Scene Labeling

  2017cites this paper