On Layer Normalizations and Residual Connections in Transformers

No abstract is available for this paper.

• Publication year

  2022

• Venue

  arXiv.org

• Publication date

  Unknown publication date

• Fields of study

  Computer Science, Engineering

• Identifiers
  DOI 10.48550/arXiv.2206.00330
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• DeepNet: Scaling Transformers to 1,000 Layers

  2022cited by this paper
• Rethinking Perturbations in Encoder-Decoders for Fast Training

  2021cited by this paper
• Improving Transformer Optimization Through Better Initialization

  2020influential reference
• Very Deep Transformers for Neural Machine Translation

  2020cited by this paper
• Language Models are Few-Shot Learners

  2020cited by this paper
• On Layer Normalization in the Transformer Architecture

  2020cited by this paper
• Understanding the Difficulty of Training Transformers

  2020influential reference
• Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq

  2020influential reference
• Fixup Initialization: Residual Learning Without Normalization

  2019cited by this paper
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling

  2019cited by this paper
• Positional Encoding to Control Output Sequence Length

  2019cited by this paper
• Defending Against Neural Fake News

  2019cited by this paper
• Learning Deep Transformer Models for Machine Translation

  2019influential reference
• Energy and Policy Considerations for Deep Learning in NLP

  2019cited by this paper
• Improving Deep Transformer with Depth-Scaled Initialization and Merged Attention

  2019cited by this paper
• Findings of the 2019 Conference on Machine Translation (WMT19)

  2019cited by this paper
• Understanding and Improving Layer Normalization

  2019cited by this paper
• Tensor2Tensor for Neural Machine Translation

  2018cited by this paper
• A Call for Clarity in Reporting BLEU Scores

  2018cited by this paper
• Scaling Neural Machine Translation

  2018cited by this paper
• SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

  2018cited by this paper
• Training Deeper Neural Machine Translation Models with Transparent Attention

  2018cited by this paper
• Adaptive Input Representations for Neural Language Modeling

  2018influential reference
• Exploiting Deep Representations for Neural Machine Translation

  2018cited by this paper
• Attention is All you Need

  2017cited by this paper
• OpenNMT: Open-Source Toolkit for Neural Machine Translation

  2017cited by this paper
• Identity Mappings in Deep Residual Networks

  2016cited by this paper
• Pointer Sentinel Mixture Models

  2016influential reference
• Deep Residual Learning for Image Recognition

  2015cited by this paper
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

  2015cited by this paper
• Training Very Deep Networks

  2015cited by this paper
• Librispeech: An ASR corpus based on public domain audio books

  2015influential reference
• A Neural Attention Model for Abstractive Sentence Summarization

  2015influential reference
• Neural Machine Translation of Rare Words with Subword Units

  2015influential reference
• Annotated Gigaword

  2012cited by this paper

• Understanding Transformer Optimization via Gradient Heterogeneity

  2025cites this paper
• Stack Transformer Based Spatial-Temporal Attention Model for Dynamic Sign Language and Fingerspelling Recognition

  2025cites this paper
• Model Hemorrhage and the Robustness Limits of Large Language Models

  2025cites this paper
• Stack Transformer Based Spatial-Temporal Attention Model for Dynamic Multi-Culture Sign Language Recognition

  2025cites this paper
• A Survey of Vision Transformers in Autonomous Driving: Current Trends and Future Directions

  2024cites this paper
• SpecFormer: Guarding Vision Transformer Robustness via Maximum Singular Value Penalization

  2024cites this paper
• Why are Sensitive Functions Hard for Transformers?

  2024cites this paper
• SRFLOW‐Attention: Super Resolution with Multi‐Head Attention for Effective Use of Low‐Resolution Information

  2024cites this paper
• Resolving Lexical Bias in Model Editing

  2024cites this paper
• Shuffle Mixing: An Efficient Alternative to Self Attention

  2023cites this paper
• Vision transformer architecture and applications in digital health: a tutorial and survey

  2023cites this paper
• ResiDual: Transformer with Dual Residual Connections

  2023cites this paper
• Predicting Chemical Reaction Product by Graph Transformer

  2023cites this paper
• NT5 at WMT 2022 General Translation Task

  2022influential citation
• Convexifying Transformers: Improving optimization and understanding of transformer networks

  2022cites this paper