Asymptotic Log-Loss of Prequential Maximum Likelihood Codes

We analyze the Dawid-Rissanen prequential maximum likelihood codes relative to one-parameter exponential family models M. If data are i.i.d. according to an (essentially) arbitrary P, then the redundancy grows at rate 1/2 c ln n. We show that c = σ 2 1 /σ 2 2 , where σ 2 1 is the variance of P, and σ 2 2 is the variance of the distribution M* ∈ M that is closest to P in KL divergence. This shows that prequential codes behave quite differently from other important universal codes such as the 2-part MDL, Shtarkov and Bayes codes, for which c = 1. This behavior is undesirable in an MDL model selection setting.

• Publication year

  2005

• Venue

  Annual Conference Computational Learning Theory

• Publication date

  2005-02-01

• Fields of study

  Mathematics, Computer Science

• Identifiers
  DOI 10.1007/11503415_44arXiv cs/0502004
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Elements of Information Theory

  2005cited by this paper
• An Empirical Study of MDL Model Selection with Infinite Parametric Complexity

  2005cited by this paper
• Predicting a binary sequence almost as well as the optimal biased coin

  2003cited by this paper
• Geometrical foundations of asymptotic inference

  2002cited by this paper
• Comparing Prequential Model Selection Criteria in Supervised Learning of Mixture Models

  2001cited by this paper
• On predictive distributions and Bayesian networks

  2000cited by this paper
• Iterated logarithmic expansions of the pathwise code lengths for exponential families

  2000cited by this paper
• Worst-Case Bounds for the Logarithmic Loss of Predictors

  1999cited by this paper
• Relative Loss Bounds for On-Line Density Estimation with the Exponential Family of Distributions

  1999cited by this paper
• Prequential and Cross-Validated Regression Estimation

  1998cited by this paper
• The Minimum Description Length Principle in Coding and Modeling

  1998influential reference
• Geometrical Foundations of Asymptotic Inference

  1997influential reference
• Geometrical Foundations of Asymptotic Inference: Kass/Geometrical

  1997cited by this paper
• Statistical Decision Theory and Bayesian Analysis, Second Edition

  1993cited by this paper
• On Predictive Least Squares Principles

  1992cited by this paper
• Information-theoretic asymptotics of Bayes methods

  1990cited by this paper
• Strong Consistency of the PLS Criterion for Order Determination of Autoregressive Processes

  1989cited by this paper
• Stochastic Complexity in Statistical Inquiry

  1989cited by this paper
• Mathematical Statistics and Data Analysis

  1988cited by this paper
• Statistical Decision Theory and Bayesian Analysis. Second Edition (James O. Berger)

  1987cited by this paper
• Stochastic Complexity and Modeling

  1986cited by this paper
• A Predictive Least-Squares Principle

  1986cited by this paper
• Bayes theory

  1984cited by this paper
• Universal coding, information, prediction, and estimation

  1984cited by this paper
• Bounds for the Moments of Linear and Quadratic Forms in Independent Variables

  1960cited by this paper
• Advances in Minimum Description

  year unknowncited by this paper

• Fine-Tuned In-Context Learners for Efficient Adaptation

  2025cites this paper
• E-Values for Exponential Families: the General Case

  2024influential citation
• Evaluating Representations with Readout Model Switching

  2023cites this paper
• Sequential prediction under log-loss and misspecification

  2021cites this paper
• Near-optimality of sequential joint detection and estimation via online mirror descent

  2017cites this paper
• Almost the Best of Three Worlds: Risk, Consistency and Optional Stopping for the Switch Criterion in Nested Model Selection

  2014cites this paper
• Essays on Price Dynamics and Market Selection

  2013cites this paper
• The switch model selection criterion for single-parameter exponential families

  2013cites this paper
• Sequential normalized maximum likelihood in log-loss prediction

  2012cites this paper
• Maximum Likelihood vs. Sequential Normalized Maximum Likelihood in On-line Density Estimation

  2011cites this paper
• Bounds on Individual Risk for Log-loss Predictors

  2011cites this paper
• Following the Flattened Leader

  2010cites this paper
• Prequential plug-in codes that achieve optimal redundancy rates even if the model is wrong

  2010influential citation
• When Data Compression and Statistics Disagree: Two Frequentist Challenges for the Minimum Description Length Principle

  2010cites this paper
• Minimum Description Length Model Selection

  2008cites this paper
• An empirical study of minimum description length model selection with infinite parametric complexity

  2006cites this paper
• The Momentum Problem in MDL and Bayesian Prediction

  2006cites this paper
• Model selection by Normalized Maximum Likelihood

  2006cites this paper
• MDL model selection using the ML plug-in code

  2005influential citation
• MDL model selection using the ML plug-in code

  2005cites this paper
• An Empirical Study of MDL Model Selection with Infinite Parametric Complexity

  2005cites this paper