A Survey of Transformers

Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from academic and industry researchers. Up to the present, a great variety of Transformer variants (a.k.a. X-formers) have been proposed, however, a systematic and comprehensive literature review on these Transformer variants is still missing. In this survey, we provide a comprehensive review of various X-formers. We first briefly introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the various X-formers from three perspectives: architectural modification, pre-training, and applications. Finally, we outline some potential directions for future research.

• Publication year

  2021

• Venue

  AI Open

• Publication date

  2021-06-08

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1016/j.aiopen.2022.10.001arXiv 2106.04554
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• ERNIE-Doc: A Retrospective Long-Document Modeling Transformer

  2021cited by this paper
• Conditional Positional Encodings for Vision Transformers

  2021influential reference
• On Position Embeddings in BERT

  2021cited by this paper
• Hash Layers For Large Sparse Models

  2021cited by this paper
• Luna: Linear Unified Nested Attention

  2021cited by this paper
• Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modeling

  2021cited by this paper
• Memory-Efficient Differentiable Transformer Architecture Search

  2021cited by this paper
• Exploring Sparse Expert Models and Beyond

  2021cited by this paper
• Accelerating BERT Inference for Sequence Labeling via Early-Exit

  2021cited by this paper
• Early Exiting with Ensemble Internal Classifiers

  2021cited by this paper
• CogView: Mastering Text-to-Image Generation via Transformers

  2021cited by this paper
• Poolingformer: Long Document Modeling with Pooling Attention

  2021cited by this paper
• Learning Shared Semantic Space for Speech-to-Text Translation

  2021cited by this paper
• SETransformer: Speech Enhancement Transformer

  2021cited by this paper
• RoFormer: Enhanced Transformer with Rotary Position Embedding

  2021cited by this paper
• ViViT: A Video Vision Transformer

  2021cited by this paper
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

  2021cited by this paper
• Mask Attention Networks: Rethinking and Strengthen Transformer

  2021influential reference
• Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth

  2021cited by this paper
• Random Feature Attention

  2021influential reference
• Transformer in Transformer

  2021cited by this paper
• M6: A Chinese Multimodal Pretrainer

  2021cited by this paper
• LazyFormer: Self Attention with Lazy Update

  2021cited by this paper
• Zero-Shot Text-to-Image Generation

  2021cited by this paper
• Linear Transformers Are Secretly Fast Weight Memory Systems

  2021influential reference
• TransGAN: Two Transformers Can Make One Strong GAN

  2021cited by this paper
• Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention

  2021cited by this paper
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

  2021influential reference
• Transformers in Vision: A Survey

  2021influential reference
• On the Sub-Layer Functionalities of Transformer Decoder

  2020cited by this paper
• Transformer

  2020cited by this paper
• Reformer: The Efficient Transformer

  2020cited by this paper
• How Much Position Information Do Convolutional Neural Networks Encode?

  2020cited by this paper
• On Layer Normalization in the Transformer Architecture

  2020cited by this paper
• GLU Variants Improve Transformer

  2020cited by this paper
• Low-Rank Bottleneck in Multi-head Attention Models

  2020cited by this paper
• Controlling Computation versus Quality for Neural Sequence Models

  2020cited by this paper
• Sparse Sinkhorn Attention

  2020cited by this paper
• Talking-Heads Attention

  2020cited by this paper
• ReZero is All You Need: Fast Convergence at Large Depth

  2020cited by this paper
• Efficient Content-Based Sparse Attention with Routing Transformers

  2020cited by this paper
• Pre-trained models for natural language processing: A survey

  2020cited by this paper
• Lite Transformer with Long-Short Range Attention

  2020cited by this paper
• Learning to Encode Position for Transformer with Continuous Dynamical Model

  2020cited by this paper
• SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive Connection

  2020influential reference
• Longformer: The Long-Document Transformer

  2020cited by this paper
• Understanding the Difficulty of Training Transformers

  2020cited by this paper
• FLAT: Chinese NER Using Flat-Lattice Transformer

  2020cited by this paper
• Improving End-to-End Speech Synthesis with Local Recurrent Neural Network Enhanced Transformer

  2020cited by this paper
• DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference

  2020cited by this paper
• Synthesizer: Rethinking Self-Attention for Transformer Models

  2020cited by this paper
• Hard-Coded Gaussian Attention for Neural Machine Translation

  2020cited by this paper
• Conformer: Convolution-augmented Transformer for Speech Recognition

  2020cited by this paper
• End-to-End Object Detection with Transformers

  2020cited by this paper
• Language Models are Few-Shot Learners

  2020cited by this paper
• Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers

  2020influential reference
• Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing

  2020cited by this paper
• Linformer: Self-Attention with Linear Complexity

  2020cited by this paper
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention

  2020cited by this paper
• BERT Loses Patience: Fast and Robust Inference with Early Exit

  2020cited by this paper
• Generative Pretraining From Pixels

  2020cited by this paper
• Rethinking Positional Encoding in Language Pre-training

  2020cited by this paper
• PowerNorm: Rethinking Batch Normalization in Transformers

  2020cited by this paper
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

  2020cited by this paper
• Multi-Head Attention: Collaborate Instead of Concatenate

  2020cited by this paper
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

  2020cited by this paper
• Fast Transformers with Clustered Attention

  2020cited by this paper
• Big Bird: Transformers for Longer Sequences

  2020influential reference
• DeLighT: Very Deep and Light-weight Transformer

  2020cited by this paper
• Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context Size

  2020cited by this paper
• Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data

  2020cited by this paper
• ETC: Encoding Long and Structured Inputs in Transformers

  2020cited by this paper
• Rethinking Attention with Performers

  2020influential reference
• Temporal Context Aggregation for Video Retrieval with Contrastive Learning

  2020cited by this paper
• Guiding Attention for Self-Supervised Learning with Transformers

  2020cited by this paper
• Deformable DETR: Deformable Transformers for End-to-End Object Detection

  2020cited by this paper
• Memformer: The Memory-Augmented Transformer

  2020cited by this paper
• Developing Real-Time Streaming Transformer Transducer for Speech Recognition on Large-Scale Dataset

  2020cited by this paper
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

  2020influential reference
• Reformer-TTS: Neural Speech Synthesis with Reformer Network

  2020cited by this paper
• End-to-End Object Detection with Adaptive Clustering Transformer

  2020cited by this paper
• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting

  2020cited by this paper
• RealFormer: Transformer Likes Residual Attention

  2020influential reference
• UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning

  2020cited by this paper
• Synthesizer: Rethinking Self-Attention in Transformer Models

  2020cited by this paper
• Switching Poisson Gamma Dynamical Systems

  2020influential reference
• Understanding and Improving Layer Normalization

  2019cited by this paper
• The Evolved Transformer

  2019influential reference
• Insertion-based Decoding with Automatically Inferred Generation Order

  2019cited by this paper
• Information Aggregation for Multi-Head Attention with Routing-by-Agreement

  2019cited by this paper
• VideoBERT: A Joint Model for Video and Language Representation Learning

  2019cited by this paper
• Generating Long Sequences with Sparse Transformers

  2019influential reference
• Very Deep Self-Attention Networks for End-to-End Speech Recognition

  2019cited by this paper
• Language Modeling with Deep Transformers

  2019cited by this paper
• HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization

  2019cited by this paper
• Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

  2019cited by this paper
• Adaptive Attention Span in Transformers

  2019cited by this paper
• Language Models are Unsupervised Multitask Learners

  2019cited by this paper
• Hierarchical Transformers for Multi-Document Summarization

  2019cited by this paper
• Learning Deep Transformer Models for Machine Translation

  2019cited by this paper

• TAP++: a tier-aware transformer for adaptive prefetching system in hybrid memory systems

  2026cites this paper
• A survey of transformer networks for time series forecasting

  2026cites this paper
• Transformers for Graph-based Recommender Systems: A Survey

  2026cites this paper
• The Evolution of the Brain From Analog to Digital: The Birth of Artificial Intelligence.

  2026cites this paper
• A Systematic Review of Diffusion Models for Medical Image-Based Diagnosis: Methods, Taxonomies, Clinical Integration, Explainability, and Future Directions

  2026cites this paper
• Revolutionizing predictive maintenance: Remaining useful life forecasting based on cognitive artificial intelligence using knowledge growing system

  2026cites this paper
• A Dual-Decomposition Graph-Mamba-Transformer Framework for Ultra-Short-Term Wind Power Forecasting

  2026cites this paper
• Enhancing transparency understanding using machine learning and visual analytics

  2026cites this paper
• Transformer-based fusion of acoustic and textual cues with proportional augmentation for emotion recognition

  2026cites this paper
• Sequence-based framework for construction worker action recognition using context-aware synthetic data

  2026cites this paper
• Aligning incentives and resilience: joint node selection and resource allocation in the lightning network

  2026cites this paper
• Hyperbolic Heterogeneous Graph Transformer

  2026cites this paper
• Whip strike detection using high-sampling-rate audio by evaluating convolutional recurrent neural network configurations and class imbalance strategies

  2026cites this paper
• Introdução às redes neurais para Físicos

  2026cites this paper
• Machine learning-based automatic detection and prediction of cracks and corrosion using spatiotemporal measurements from distributed fiber optic sensors

  2026cites this paper
• Reconstruction and Projection of Near‐Surface Wind Speed in China Based on Transformer Neural Network

  2026cites this paper
• SGMT-IDS: A Dual-Branch Semi-Supervised Intrusion Detection Model Based on Graphs and Transformers

  2026cites this paper
• Frontier AI in computational civil engineering: a review of graph, sequence, physics-informed deep learning, and beyond (2020–2025)

  2026cites this paper
• Review of transformers applied to wind turbine data: Exploring the potential of transformers in revolutionizing wind turbine data analysis

  2026cites this paper
• Importance of temporal encoding in data-driven estimation of downhole variables

  2026cites this paper
• TLDA: A lightweight and adaptive deep learning framework for accurate monthly runoff forecasting

  2026cites this paper
• Advancing international business research through artificial intelligence and machine learning applications

  2026cites this paper
• A statistical perspective on transformers for small longitudinal cohort data

  2026cites this paper
• MuDeNet: A multi-patch descriptor network for anomaly modeling

  2026cites this paper
• Remote dispersion scan: transformer-network retrieval of ultrafast pulses after non-linear propagation

  2026cites this paper
• Transformer: Attention History May Matter

  2026cites this paper
• Emotion recognition based on spatio-temporal connectivity of prefrontal EEG signals

  2026cites this paper
• Enhancing clustering stability, compactness, and separation in multimodal data environments

  2026cites this paper
• DCTDSE: A Bimodal Design Space Exploration Flow via Discrete–Continuous Transformation

  2026cites this paper
• Time-Frequency Domain Lightweight Dual-Branch MSCFormer for PMSM ITSC Fault Diagnosis

  2026cites this paper
• Deep learning approach for short-term entry passenger flow forecasting in urban rail transit stations

  2026cites this paper
• Transformer-based network for enhanced wireless service differentiation in 6G

  2026cites this paper
• AIoT for human emotion recognition: Potentials, challenges, and healthcare applications

  2026cites this paper
• SegAuth: Semantic-Aware Multimotion Behavioral Biometric-Based Implicit Authentication

  2026cites this paper
• Vision-Language Model for Accurate Crater Detection

  2026cites this paper
• Characteristic prediction model TCN-informer for gasifier furnace temperature

  2026cites this paper
• The role of fibration symmetries in geometric deep learning.

  2026cites this paper
• Customising Czech Phonetic Alignment using HuBERT and manual segmentation

  2026cites this paper
• From Words to Frameworks: Transformer Models for Metal–Organic Framework Design in Nanotheranostics

  2026influential citation
• VibrationLLM: Enabling personalized risk management for hand-arm vibration exposure in construction

  2026cites this paper
• DNN-Enabled Multi-User Beamforming for Throughput Maximization under Adjustable Fairness

  2026cites this paper
• Improving smart contract security with transformer-based anomaly detection

  2026cites this paper
• Stochastic Neural Networks for Quantum Devices

  2026cites this paper
• TiledAttention: a CUDA Tile SDPA Kernel for PyTorch

  2026cites this paper
• Transformer-based artificial intelligence for forecasting energy demand in irrigation districts

  2026cites this paper
• Improving speech-to-text for the Indonesian language using a modified transformer

  2026cites this paper
• Nonparametric Teaching of Attention Learners

  2026cites this paper
• Exploring potential of large language models for automated essay scoring in education

  2026cites this paper
• Transformers meet hyperspectral imaging: A comprehensive study of models, challenges and open problems

  2026cites this paper
• Deep Sketch‐Based 3D Modeling: A Survey

  2026cites this paper
• Swin-AFR-FPN: Attention and fusion refined object detection model with swin-transformer backbone for Escherichia coli (E. coli) bacterial cell cycle identification and antibiotic phenotyping

  2026cites this paper
• Softmax Linear Attention: Reclaiming Global Competition

  2026cites this paper
• Time-series InSAR monitoring and evolution predictive analysis of filling-area subsidence in Yan’an New District

  2026cites this paper
• TPSformer: knowledge graph representation learning with text and position structure for link prediction

  2026cites this paper
• Bidirectional Translation of ASL and English Using Machine Vision and CNN and Transformer Networks

  2026cites this paper
• Temporal dynamics of animal vocal behavior affect semi-automatic identifications

  2026cites this paper
• The multi-GPU Wetland DEM Ponding Model

  2025cites this paper
• Code Clone Detection Techniques Based on Large Language Models

  2025cites this paper
• Contrastive learning for one-shot building shape recognition using vector polygon transformers

  2025cites this paper
• SGNetPose+: Stepwise Goal-Driven Networks with Pose Information for Trajectory Prediction in Autonomous Driving

  2025cites this paper
• How to Write Effective Prompts for Screening Biomedical Literature Using Large Language Models

  2025cites this paper
• Distributed Multiagent Resource Allocation Based on Transformer and DRL for Cloud XR Hybrid Content Transmission

  2025cites this paper
• PNetGPT: Proprietary Protocol Network Traffic Generation with Pre-trained Transformer

  2025cites this paper
• Evaluating open-source Large Language Models for automated fact-checking

  2025cites this paper
• Geodesign in the era of artificial intelligence

  2025cites this paper
• Introdu\c{c}\~ao a rede neural para F\'isicos

  2025cites this paper
• TSNet: a foundation model for wireless network status prediction in digital twins

  2025cites this paper
• SwinVision: Detecting Small Objects in Low-Light Environments

  2025cites this paper
• Deadline-Aware Online Job Scheduling for Distributed Training in Heterogeneous Clusters

  2025cites this paper
• Evaluating Machine Learning-Based Soft Sensors for Effluent Quality Prediction in Wastewater Treatment Under Variable Weather Conditions

  2025influential citation
• AI Computing Systems for Large Language Models Training

  2025cites this paper
• Transformer-Based Power Optimization for Max-Min Fairness in Cell-Free Massive MIMO

  2025influential citation
• Attention Condensation via Sparsity Induced Regularized Training

  2025cites this paper
• Learning Precoding in Multi-User Multi-Antenna Systems: Transformer or Graph Transformer?

  2025cites this paper
• Evaluating Transformers Learning by Representing Self-Attention Weights as a Graph

  2025cites this paper
• Transformer Meets Twicing: Harnessing Unattended Residual Information

  2025cites this paper
• CrystalFramer: Rethinking the Role of Frames for SE(3)-Invariant Crystal Structure Modeling

  2025cites this paper
• Deep Causal Behavioral Policy Learning: Applications to Healthcare

  2025cites this paper
• Integrating Biological and Machine Intelligence: Attention Mechanisms in Brain-Computer Interfaces

  2025cites this paper
• Large Model Era: Deep Learning in Osteoporosis Drug Discovery

  2025cites this paper
• ColorDynamic: Generalizable, Scalable, Real-time, End-to-end Local Planner for Unstructured and Dynamic Environments

  2025cites this paper
• On the Privacy Risks of Spiking Neural Networks: A Membership Inference Analysis

  2025cites this paper
• An attention-based neural ordinary differential equation framework for modeling inelastic processes

  2025cites this paper
• The Multifaceted Impact of Chess: Cognitive Enhancement, Educational Applications, and Technological Innovations

  2025cites this paper
• Revisiting Kernel Attention with Correlated Gaussian Process Representation

  2025cites this paper
• Solvent Screening for Separation Processes Using Machine Learning and High-Throughput Technologies

  2025cites this paper
• Concentrating Harder for Faster Audio Transformer

  2025cites this paper
• Exploring LoRA for parameter-efficient fine-tuning of LLMs in enhanced algorithm-to-python-source-code translation task

  2025cites this paper
• A Comparative Study of Fine-Tuning Deep Learning Models for Leaf Disease Identification and Classification

  2025cites this paper
• Corn pose estimation using 3D object detection and stereo images

  2025cites this paper
• Addressing myocardial infarction in South-Asian populations: risk factors and machine learning approaches

  2025cites this paper
• Advancements in the Application of Convolutional Neural Networks in Ultrasound Imaging for Breast Cancer Diagnosis and Treatment

  2025cites this paper
• Machine learning method for As-Is tunnel information model reconstruction

  2025cites this paper
• Advances in Remote Sensing and Deep Learning in Coastal Boundary Extraction for Erosion Monitoring

  2025cites this paper
• AI-Driven Parametrization of Michaelis-Menten Maximal Velocity: Advancing in silico New Approach Methodologies (NAMs)

  2025cites this paper
• Self-Distillation Attention for Efficient and Accurate Motion Prediction in Autonomous Driving

  2025cites this paper
• Integration of natural language processing methods and machine learning model for malicious webpage detection based on web contents

  2025cites this paper
• TriSwinUNETR lobe segmentation model for computing DIR-free CT-ventilation

  2025cites this paper
• Impact of EU non-financial reporting regulation on Spanish companies’ environmental disclosure: a cutting-edge natural language processing approach

  2025cites this paper
• Gamma Log Inversion of Seismic Data Based on Transformer With Stratigraphic Position Encoding

  2025cites this paper