In-Context Learning Without Copying

Induction heads are attention heads that perform inductive copying by matching patterns from earlier context and copying their continuations verbatim. As models develop induction heads, they experience a sharp drop in training loss, a phenomenon cited as evidence that induction heads may underlie a wide range of in-context learning (ICL) capabilities. In this work, we investigate whether induction heads are a necessary building block for learning abstractive ICL capabilities (i.e., tasks where the answer is not contained in the input context), or whether such capabilities can emerge independently. We propose Hapax, a training regime that omits the loss contribution of tokens predictable by induction heads. Despite a significant reduction in inductive copying, abstractive ICL capabilities are preserved, with the model achieving higher accuracy than the vanilla model on 13 out of 21 tasks, even though 31.7% of tokens are omitted from the loss. Furthermore, our model achieves lower loss values on token positions that induction heads cannot predict. Mechanistic analysis shows that models trained with Hapax develop fewer and weaker induction heads despite preserving abstractive ICL capabilities. Our findings suggest that the developmental link between induction heads and abstractive ICL capabilities is weaker than previously hypothesized.

• Publication year

  2025

• Venue

  arXiv.org

• Publication date

  2025-11-07

• Fields of study

  Computer Science

• Identifiers
  DOI 10.48550/arXiv.2511.05743arXiv 2511.05743
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders

  2025cited by this paper
• Empirical Evaluation of Loss Masking to Selectively Prevent Memorization

  2025cited by this paper
• The emergence of sparse attention: impact of data distribution and benefits of repetition

  2025cited by this paper
• Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergence

  2025cited by this paper
• Induction Head Toxicity Mechanistically Explains Repetition Curse in Large Language Models

  2025cited by this paper
• Understanding the Repeat Curse in Large Language Models from a Feature Perspective

  2025cited by this paper
• The Dual-Route Model of Induction

  2025influential reference
• Strategy Coopetition Explains the Emergence and Transience of In-Context Learning

  2025cited by this paper
• Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Models

  2025cited by this paper
• Which Attention Heads Matter for In-Context Learning?

  2025cited by this paper
• Language Models "Grok" to Copy

  2024cited by this paper
• In-Context Language Learning: Architectures and Algorithms

  2024cited by this paper
• The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains

  2024cited by this paper
• Identifying Semantic Induction Heads to Understand In-Context Learning

  2024cited by this paper
• What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation

  2024cited by this paper
• Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs

  2024cited by this paper
• The Remarkable Robustness of LLMs: Stages of Inference?

  2024cited by this paper
• Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs

  2024cited by this paper
• Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning

  2024cited by this paper
• Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers

  2024cited by this paper
• From Tokens to Words: On the Inner Lexicon of LLMs

  2024cited by this paper
• Repetition Neurons: How Do Language Models Produce Repetitions?

  2024cited by this paper
• Copy Suppression: Comprehensively Understanding a Motif in Language Model Attention Heads

  2024cited by this paper
• RedPajama: an Open Dataset for Training Large Language Models

  2024cited by this paper
• NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internals

  2024cited by this paper
• Function Vectors in Large Language Models

  2023influential reference
• The mechanistic basis of data dependence and abrupt learning in an in-context classification task

  2023cited by this paper
• Finding Neurons in a Haystack: Case Studies with Sparse Probing

  2023cited by this paper
• Birth of a Transformer: A Memory Viewpoint

  2023cited by this paper
• In-Context Learning Creates Task Vectors

  2023cited by this paper
• Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

  2023cited by this paper
• Data Distributional Properties Drive Emergent In-Context Learning in Transformers

  2022cited by this paper
• In-context Learning and Induction Heads

  2022influential reference
• Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small

  2022cited by this paper
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling

  2020cited by this paper
• Language Models are Few-Shot Learners

  2020cited by this paper
• Neural Text Generation with Unlikelihood Training

  2019cited by this paper
• The Context

  2019cited by this paper
• (Preprint)

  2018cited by this paper
• Pointer Sentinel Mixture Models

  2016cited by this paper
• Extending Zipf’s law to n-grams for large corpora

  2009cited by this paper
• NLTK: The Natural Language Toolkit

  2006influential reference
• Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms

  1998cited by this paper

• No citing papers are available for this paper.