The Sufficiency-Conciseness Trade-off in LLM Self-Explanation from an Information Bottleneck Perspective

Large Language Models increasingly rely on self-explanations, such as chain of thought reasoning, to improve performance on multi step question answering. While these explanations enhance accuracy, they are often verbose and costly to generate, raising the question of how much explanation is truly necessary. In this paper, we examine the trade-off between sufficiency, defined as the ability of an explanation to justify the correct answer, and conciseness, defined as the reduction in explanation length. Building on the information bottleneck principle, we conceptualize explanations as compressed representations that retain only the information essential for producing correct answers.To operationalize this view, we introduce an evaluation pipeline that constrains explanation length and assesses sufficiency using multiple language models on the ARC Challenge dataset. To broaden the scope, we conduct experiments in both English, using the original dataset, and Persian, as a resource-limited language through translation. Our experiments show that more concise explanations often remain sufficient, preserving accuracy while substantially reducing explanation length, whereas excessive compression leads to performance degradation.

• Publication year

  2026

• Venue

  Unknown venue

• Publication date

  2026-02-15

• Fields of study

  Computer Science

• Identifiers
  arXiv 2602.14002
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• A Generalized Information Bottleneck Theory of Deep Learning

  2025cited by this paper
• Length Controlled Generation for Black-box LLMs

  2025cited by this paper
• Explain Yourself, Briefly! Self-Explaining Neural Networks with Concise Sufficient Reasons

  2025cited by this paper
• Are self-explanations from Large Language Models faithful?

  2024cited by this paper
• GPT-4o System Card

  2024influential reference
• Sufficient and Necessary Explanations (and What Lies in Between)

  2024cited by this paper
• Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations

  2023cited by this paper
• How Does Information Bottleneck Help Deep Learning?

  2023cited by this paper
• Large Language Models are Zero-Shot Reasoners

  2022cited by this paper
• Least-to-Most Prompting Enables Complex Reasoning in Large Language Models

  2022cited by this paper
• Towards Faithful Model Explanation in NLP: A Survey

  2022cited by this paper
• Language Models are Multilingual Chain-of-Thought Reasoners

  2022cited by this paper
• Measuring and Narrowing the Compositionality Gap in Language Models

  2022cited by this paper
• Consistent Sufficient Explanations and Minimal Local Rules for explaining the decision of any classifier or regressor

  2022cited by this paper
• When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data

  2021cited by this paper
• Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness?

  2020cited by this paper
• ERASER: A Benchmark to Evaluate Rationalized NLP Models

  2019cited by this paper
• Attention is not not Explanation

  2019cited by this paper
• On the information bottleneck theory of deep learning

  2018cited by this paper
• Attention is All you Need

  2017cited by this paper
• A Unified Approach to Interpreting Model Predictions

  2017cited by this paper
• Axiomatic Attribution for Deep Networks

  2017cited by this paper
• Deep learning and the information bottleneck principle

  2015cited by this paper
• Neural Machine Translation by Jointly Learning to Align and Translate

  2014cited by this paper
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps

  2013cited by this paper
• The information bottleneck method

  2000influential reference
• Claude 3.5 Sonnet Model Card Addendum

  year unknowncited by this paper

• No citing papers are available for this paper.