On the Robustness of Self-Attentive Models

This work examines the robustness of self-attentive neural networks against adversarial input perturbations. Specifically, we investigate the attention and feature extraction mechanisms of state-of-the-art recurrent neural networks and self-attentive architectures for sentiment analysis, entailment and machine translation under adversarial attacks. We also propose a novel attack algorithm for generating more natural adversarial examples that could mislead neural models but not humans. Experimental results show that, compared to recurrent neural models, self-attentive models are more robust against adversarial perturbation. In addition, we provide theoretical explanations for their superior robustness to support our claims.

• Publication year

  2019

• Venue

  Annual Meeting of the Association for Computational Linguistics

• Publication date

  2019-07-01

• Fields of study

  Computer Science

• Identifiers
  DOI 10.18653/v1/P19-1147
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

  2019cited by this paper
• Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples

  2018cited by this paper
• Adversarial Example Generation with Syntactically Controlled Paraphrase Networks

  2018cited by this paper
• Generating Natural Language Adversarial Examples

  2018cited by this paper
• Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures

  2018cited by this paper
• Greedy Attack and Gumbel Attack: Generating Adversarial Examples for Discrete Data

  2018cited by this paper
• Attention is All you Need

  2017influential reference
• A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

  2017cited by this paper
• Synthetic and Natural Noise Both Break Neural Machine Translation

  2017cited by this paper
• Generating Natural Adversarial Examples

  2017cited by this paper
• Findings of the 2017 Conference on Machine Translation (WMT17)

  2017cited by this paper
• Deep Text Classification Can be Fooled

  2017cited by this paper
• HotFlip: White-Box Adversarial Examples for Text Classification

  2017cited by this paper
• Adversarial Examples for Evaluating Reading Comprehension Systems

  2017cited by this paper
• Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder

  2016cited by this paper
• Crafting adversarial input sequences for recurrent neural networks

  2016cited by this paper
• Understanding Neural Networks through Representation Erasure

  2016cited by this paper
• Character-level Convolutional Networks for Text Classification

  2015cited by this paper
• Effective Approaches to Attention-based Neural Machine Translation

  2015cited by this paper
• Neural Machine Translation by Jointly Learning to Align and Translate

  2014cited by this paper
• Sequence to Sequence Learning with Neural Networks

  2014cited by this paper
• Explaining and Harnessing Adversarial Examples

  2014cited by this paper
• Intriguing properties of neural networks

  2013cited by this paper
• Bleu: a Method for Automatic Evaluation of Machine Translation

  2002cited by this paper
• LSTM can Solve Hard Long Time Lag Problems

  1996cited by this paper

• A Provable Expressiveness Hierarchy in Hybrid Linear-Full Attention

  2026cites this paper
• CVAE-GAN & STFT-CNN for low resolution geomagnetic data reconstruction in Pi2 pulsation identification

  2026cites this paper
• ANROT-HELANet: adverserially and naturally robust attention-based aggregation network via the hellinger distance for few-shot classification

  2026cites this paper
• RedHerring Attack: Testing the Reliability of Attack Detection

  2025influential citation
• Overcoming Black-box Attack Inefficiency with Hybrid and Dynamic Select Algorithms

  2025cites this paper
• Multiclass Arrhythmia Classification from Imbalanced ECG Data Using Encoded Transformer based CNN-LSTM Hybrid Model

  2025cites this paper
• Efficient Gradient-Oriented Optimization for Black-box Hard-label Textual Adversarial Attack

  2025cites this paper
• Integrated gradients-based defense against adversarial word substitution attacks

  2025cites this paper
• Trick or Neat: Adversarial Ambiguity and Language Model Evaluation

  2025cites this paper
• Language Models That Walk the Talk: A Framework for Formal Fairness Certificates

  2025cites this paper
• Strong Eye Closure Detection in Children with Profound Intellectual and Multiple Disabilities Using Robust Temporal Difference Features

  2025cites this paper
• VertAttack: Taking advantage of Text Classifiers' horizontal vision

  2024cites this paper
• Development of an Attention Mechanism for Task-Adaptive Heterogeneous Robot Teaming

  2024cites this paper
• Transformer Encoder Satisfiability: Complexity and Impact on Formal Reasoning

  2024cites this paper
• Adversarial Robustness of Vision Transformers Versus Convolutional Neural Networks

  2024cites this paper
• Thermal Error Transfer Prediction Modeling of Machine Tool Spindle with Self-Attention Mechanism-Based Feature Fusion

  2024cites this paper
• A More Context-Aware Approach for Textual Adversarial Attacks Using Probability Difference-Guided Beam Search

  2024cites this paper
• VertAttack: Taking Advantage of Text Classifiers’ Horizontal Vision

  2024cites this paper
• The Computational Complexity of Formal Reasoning for Encoder-Only Transformers

  2024cites this paper
• Vision Transformers in Domain Adaptation and Generalization: A Study of Robustness

  2024cites this paper
• CARL: Unsupervised Code-Based Adversarial Attacks for Programming Language Models via Reinforcement Learning

  2024cites this paper
• PE-Attack: On the Universal Positional Embedding Vulnerability in Transformer-Based Models

  2024influential citation
• Transformer-aided dynamic causal model for scalable estimation of effective connectivity

  2024cites this paper
• Vision transformers in domain adaptation and domain generalization: a study of robustness

  2024cites this paper
• BinarySelect to Improve Accessibility of Black-Box Attack Research

  2024influential citation
• HQA-Attack: Toward High Quality Black-Box Hard-Label Adversarial Attack on Text

  2024cites this paper
• Too Few Bug Reports? Exploring Data Augmentation for Improved Changeset-based Bug Localization

  2023cites this paper
• An experimental study measuring the generalization of fine‐tuned language representation models across commonsense reasoning benchmarks

  2023cites this paper
• AF2-Mutation: Adversarial Sequence Mutations against AlphaFold2 on Protein Tertiary Structure Prediction

  2023cites this paper
• Improving Classifier Robustness through Active Generation of Pairwise Counterfactuals

  2023cites this paper
• Expanding Scope: Adapting English Adversarial Attacks to Chinese

  2023cites this paper
• Towards Robust Aspect-based Sentiment Analysis through Non-counterfactual Augmentations

  2023cites this paper
• Temporal Dynamics-Aware Adversarial Attacks on Discrete-Time Dynamic Graph Models

  2023cites this paper
• A Survey of Adversarial Defenses and Robustness in NLP

  2023cites this paper
• The Efficacy of Transformer-based Adversarial Attacks in Security Domains

  2023cites this paper
• Towards a Robust Deep Neural Network Against Adversarial Texts: A Survey

  2023cites this paper
• On the Adversarial Robustness of Vision Transformers

  2022cites this paper
• A Survey in Adversarial Defences and Robustness in NLP

  2022cites this paper
• Is NLP Ready for Standardization?

  2022cites this paper
• TABS: Efficient Textual Adversarial Attack for Pre-trained NL Code Model Using Semantic Beam Search

  2022cites this paper
• Black-Box Adversarial Attacks on Deep Neural Networks: A Survey

  2022cites this paper
• Towards Explainable Evaluation of Language Models on the Semantic Similarity of Visual Concepts

  2022cites this paper
• Aliasing black box adversarial attack with joint self-attention distribution and confidence probability

  2022cites this paper
• TreeDRNet: A Robust Deep Model for Long Term Time Series Forecasting

  2022cites this paper
• CodeAttack: Code-based Adversarial Attacks for Pre-Trained Programming Language Models

  2022cites this paper
• BERT Probe: A python package for probing attention based robustness evaluation of BERT models

  2022cites this paper
• Text Adversarial Attacks and Defenses: Issues, Taxonomy, and Perspectives

  2022cites this paper
• On the Robustness of Offensive Language Classifiers

  2022influential citation
• A Survey of Adversarial Defences and Robustness in NLP

  2022cites this paper
• Hindi/Bengali Sentiment Analysis Using Transfer Learning and Joint Dual Input Learning with Self Attention

  2022cites this paper
• White-Box Attacks on Hate-speech BERT Classifiers in German with Explicit and Implicit Character Level Defense

  2022influential citation
• Robustness Verification for Attention Networks using Mixed Integer Programming

  2022cites this paper
• Faith: An Efficient Framework for Transformer Verification on GPUs

  2022cites this paper
• Improving Robustness through Pairwise Generative Counterfactual Data Augmentation

  2021cites this paper
• CIFS: Improving Adversarial Robustness of CNNs via Channel-wise Importance-based Feature Selection

  2021cites this paper
• Token-modification adversarial attacks for natural language processing: A survey

  2021influential citation
• Enhancing Neural Recommender Models through Domain-Specific Concordance

  2021cites this paper
• Understanding Robustness of Transformers for Image Classification

  2021cites this paper
• On the Robustness of Vision Transformers to Adversarial Examples

  2021cites this paper
• Evaluating Neural Model Robustness for Machine Comprehension

  2021cites this paper
• Double Perturbation: On the Robustness of Robustness and Counterfactual Bias Evaluation

  2021cites this paper
• Targeted Adversarial Training for Natural Language Understanding

  2021cites this paper
• Certified Robustness to Text Adversarial Attacks by Randomized [MASK]

  2021influential citation
• Fast and precise certification of transformers

  2021cites this paper
• On Adversarial Robustness of Synthetic Code Generation

  2021cites this paper
• Predicting Inductive Biases of Pre-Trained Models

  2021cites this paper
• Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNs

  2021cites this paper
• Robust Learning for Text Classification with Multi-source Noise Simulation and Hard Example Mining

  2021cites this paper
• eMLM: A New Pre-training Objective for Emotion Related Tasks

  2021cites this paper
• NLP_UIOWA at Semeval-2021 Task 5: Transferring Toxic Sets to Tag Toxic Spans

  2021influential citation
• Perceptual Models of Machine-Edited Text

  2021cites this paper
• Demystifying Neural Language Models' Insensitivity to Word-Order

  2021cites this paper
• Searching for an Effective Defender: Benchmarking Defense against Adversarial Word Substitution

  2021cites this paper
• Breaking BERT: Understanding its Vulnerabilities for Named Entity Recognition through Adversarial Attack

  2021cites this paper
• Back in Black: A Comparative Evaluation of Recent State-Of-The-Art Black-Box Attacks

  2021cites this paper
• Breaking BERT: Understanding its Vulnerabilities for Biomedical Named Entity Recognition through Adversarial Attack

  2021cites this paper
• Multi-Layer Random Perturbation Training for improving Model Generalization Efficiently

  2021cites this paper
• Knowledge Distillation with BERT for Image Tag-Based Privacy Prediction

  2021influential citation
• Better constraints of imperceptibility, better adversarial examples in the text

  2021cites this paper
• Robustness and Adversarial Examples in Natural Language Processing

  2021cites this paper
• Local Structure Matters Most: Perturbation Study in NLU

  2021cites this paper
• Robustness Verification for Transformers

  2020cites this paper
• Stress Test Evaluation of Transformer-based Models in Natural Language Understanding Tasks

  2020cites this paper
• On the Transferability of Adversarial Attacks against Neural Text Classifier

  2020cites this paper
• Writing Polishment with Simile: Task, Dataset and A Neural Approach

  2020cites this paper
• Do Fine-tuned Commonsense Language Models Really Generalize?

  2020cites this paper
• How Does Context Matter? On the Robustness of Event Detection with Context-Selective Mask Generalization

  2020cites this paper
• Improving Grammatical Error Correction Models with Purpose-Built Adversarial Examples

  2020cites this paper
• Posterior Differential Regularization with f-divergence for Improving Model Robustness

  2020cites this paper
• I NFORMATION - THEORETIC P ROBING E XPLAINS R E - LIANCE ON S PURIOUS H EURISTICS

  2020cites this paper
• Does Data Augmentation Improve Generalization in NLP

  2020cites this paper
• Tasty Burgers, Soggy Fries: Probing Aspect Robustness in Aspect-Based Sentiment Analysis

  2020cites this paper
• Generating Adversarial Examples for Topic-Dependent Argument Classification

  2020cites this paper
• TinySpeech: Attention Condensers for Deep Speech Recognition Neural Networks on Edge Devices

  2020cites this paper
• What’s in a Name? Are BERT Named Entity Representations just as Good for any other Name?

  2020cites this paper
• Inner Attention Modeling for Flexible Teaming of Heterogeneous Multi Robots Using Multi-Agent Reinforcement Learning

  2020cites this paper
• Evaluating and Enhancing the Robustness of Neural Network-based Dependency Parsing Models with Adversarial Examples

  2020cites this paper
• Data augmentation and the role of hardness for feature learning in NLP

  2020cites this paper
• Inner Attention Supported Adaptive Cooperation for Heterogeneous Multi Robots Teaming based on Multi-agent Reinforcement Learning

  2020cites this paper
• When does data augmentation help generalization in NLP?

  2020cites this paper