MML Is Not Consistent for Neyman-Scott

Strict Minimum Message Length (SMML) is an information-theoretic statistical inference method widely cited (but only with informal arguments) as providing estimations that are consistent for general estimation problems. It is, however, almost invariably intractable to compute, for which reason only approximations of it (known as MML algorithms) are ever used in practice. Using novel techniques that allow for the first time direct, non-approximated analysis of SMML solutions, we investigate the Neyman-Scott estimation problem, an oft-cited showcase for the consistency of MML, and show that even with a natural choice of prior neither SMML nor its popular approximations are consistent for it, thereby providing a counterexample to the general claim. This is the first known explicit construction of an SMML solution for a natural, high-dimensional problem.

• Publication year

  2016

• Venue

  IEEE Transactions on Information Theory

• Publication date

  2016-10-14

• Fields of study

  Mathematics, Computer Science

• Identifiers
  DOI 10.1109/TIT.2019.2943464arXiv 1610.04336
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Econometric Analysis of Panel Data

  2020cited by this paper
• Subspace Clustering Under Complex Noise

  2019cited by this paper
• Information-Theoretic Inference of an Optimal Dictionary of Protein Supersecondary Structures.

  2019cited by this paper
• A taxonomy of estimator consistency on discrete estimation problems

  2019cited by this paper
• Color image segmentation with bounded generalized Gaussian mixture model and feature selection

  2018cited by this paper
• Coding Ockham's Razor

  2018cited by this paper
• MML-Based Approach for Determining the Number of Topics in EDCM Mixture Models

  2018cited by this paper
• Bayesian Sparse Global-Local Shrinkage Regression for Selection of Grouped Variables

  2017cited by this paper
• Statistical Compression of Protein Folding Patterns for Inference of Recurrent Substructural Themes

  2017cited by this paper
• Minimum message length analysis of multiple short time series

  2016cited by this paper
• Nonlinear Dynamical Systems Analysis For The Behavioral Sciences Using Real Data

  2016cited by this paper
• Environmental change over 28 years in a subtropical salt marsh: optimal classification and pictures from the exposition

  2016cited by this paper
• Bayesian Model Selection

  2016cited by this paper
• Minimum message length estimation of mixtures of multivariate Gaussian and von Mises-Fisher distributions

  2015cited by this paper
• Estimating the reproducibility of psychological science

  2015influential reference
• MLE’s Bias Pathology, Model Updated MLE, and Wallace’s Minimum Message Length Method

  2015cited by this paper
• Using visual statistical inference to better understand random class separations in high dimension, low sample size data

  2014cited by this paper
• On Rereading

  2013cited by this paper
• Nearly universal consistency of maximum likelihood in discrete models

  2013cited by this paper
• MML Logistic Regression with Translation and Rotation Invariant Priors

  2012cited by this paper
• SMML Estimators for 1-Dimensional Continuous Data

  2012cited by this paper
• The Likelihood Principle

  2011cited by this paper
• MML, hybrid Bayesian network graphical models, statistical consistency, invarianc

  2010cited by this paper
• Minimum Description Length Principle.

  2010cited by this paper
• Minimum Message Length Shrinkage Estimation

  2009cited by this paper
• The Minimum Description Length Principle (Adaptive Computation and Machine Learning)

  2007cited by this paper
• Bayes not Bust! Why Simplicity is no Problem for Bayesians1

  2007cited by this paper
• Information and Complexity in Statistical Modeling

  2006cited by this paper
• Bayesian Model Selection

  2006cited by this paper
• Annals of Statistics

  2006cited by this paper
• Invited review of the book Statistical and Inductive Inference by Minimum Message Length

  2006influential reference
• Minimum message length and generalized Bayesian nets with asymmetric languages

  2005cited by this paper
• Statistical and Inductive Inference by Minimum Message Length (Information Science and Statistics)

  2005influential reference
• MML Inference of Decision Graphs with Multi-way Joins and Dynamic Attributes

  2003cited by this paper
• Unsupervised Learning of Correlated Multivariate Gaussian Mixture Models Using MML

  2003cited by this paper
• The Complexity of Strict Minimum Message Length Inference

  2002cited by this paper
• MML Inference of Decision Graphs with Multi-way Joins and Dynamic Attributes

  2002cited by this paper
• MML clustering of multi-state, Poisson, von Mises circular and Gaussian distributions

  2000cited by this paper
• Ridge Regression: Biased Estimation for Nonorthogonal Problems

  2000cited by this paper
• The incidental parameter problem since 1948

  2000cited by this paper
• Finding overlapping components with MML

  2000cited by this paper
• Statistical Distributions

  2000cited by this paper
• Refinements of MDL and MML Coding

  1999cited by this paper
• Hypothesis Selection and Testing by the MDL Principle

  1999cited by this paper
• Modified Likelihood and Related Methods for Handling Nuisance Parameters in the Linear Regression Model

  1998cited by this paper
• Point Estimation Using the Kullback-Leibler Loss Function and MML

  1998cited by this paper
• Minimum Message Length Segmentation

  1998cited by this paper
• The Bayesian Approach

  1998influential reference
• Single Factor Analysis in MML Mixture Modelling

  1998cited by this paper
• Resolving the Neyman-Scott problem by minimum message length

  1997cited by this paper
• The lasso method for variable selection in the Cox model.

  1997cited by this paper
• MML Estimation of the Parameters of the Sherical Fisher Distribution

  1996cited by this paper
• A Catalog of Noninformative Priors

  1996cited by this paper
• Bayesian Estimation of the Von Mises Concentration Parameter

  1996cited by this paper
• The Kindest Cut: Minimum Message Length Segmentation

  1996cited by this paper
• Numerical Methods for the Solution of Ill-Posed Problems

  1995cited by this paper
• MDL and Categorical Theories (Continued)

  1995cited by this paper
• Foresight: Its Logical Laws, Its Subjective Sources

  1992cited by this paper
• Minimum complexity density estimation

  1991cited by this paper
• Fisher Consistency - the Evolution of a Concept: It's Hard to Get It Right the First Time

  1989cited by this paper
• Stochastic Complexity in Statistical Inquiry

  1989cited by this paper
• Stochastic complexity and the mdl principle

  1987cited by this paper
• Estimation and Inference by Compact Coding

  1987cited by this paper
• Statistical distributions

  1983cited by this paper
• A Subjectivist’s Guide to Objective Chance

  1980influential reference
• Complexity-based induction systems: Comparisons and convergence theorems

  1978cited by this paper
• Estimating the Dimension of a Model

  1978cited by this paper
• On the Elimination of Nuisance Parameters

  1977cited by this paper
• On Rereading R. A. Fisher

  1976cited by this paper
• Information Theory and an Extension of the Maximum Likelihood Principle

  1973cited by this paper
• On the strong consistency of approximate maximum likelihood estimators

  1972cited by this paper
• The Logic of Decision

  1968cited by this paper
• An Information Measure for Classification

  1968cited by this paper
• A Formal Theory of Inductive Inference. Part II

  1964cited by this paper
• On the Foundations of Statistical Inference

  1962cited by this paper
• Theory of point estimation

  1950cited by this paper
• Note on the Consistency of the Maximum Likelihood Estimate

  1949cited by this paper
• A mathematical theory of communication

  1948cited by this paper
• Consistent Estimates Based on Partially Consistent Observations

  1948cited by this paper
• An invariant form for the prior probability in estimation problems

  1946cited by this paper
• Probability and statistics

  1934cited by this paper
• The Theory of Probability

  1922cited by this paper
• A Review of Consistency and Convergence of Posterior Distribution

  year unknowncited by this paper
• On the Mathematical Foundations of Theoretical Statistics

  year unknowncited by this paper
• Theory of Probability

  year unknowncited by this paper

• A taxonomy of estimator consistency on discrete estimation problems

  2019cites this paper
• Risk-averse estimation, an axiomatic approach to inference, and Wallace-Freeman without MML

  2018cites this paper