Estimating the Shannon Entropy Using the Pitman--Yor Process

The Shannon entropy is a fundamental measure for quantifying diversity and model complexity in fields such as information theory, ecology, and genetics. However, many existing studies assume that the number of species is known, an assumption that is often unrealistic in practice. In recent years, efforts have been made to relax this restriction. Motivated by these developments, this study proposes an entropy estimation method based on the Pitman--Yor process, a representative approach in Bayesian nonparametrics. By approximating the true distribution as an infinite-dimensional process, the proposed method enables stable estimation even when the number of observed species is smaller than the true number of species. This approach provides a principled way to deal with the uncertainty in species diversity and enhances the reliability and robustness of entropy-based diversity assessment. In addition, we investigate the convergence property of the Shannon entropy for regularly varying distributions and use this result to establish the consistency of the proposed estimator. Finally, we demonstrate the effectiveness of the proposed method through numerical experiments.

• Publication year

  2026

• Venue

  Unknown venue

• Publication date

  2026-02-09

• Fields of study

  Biology, Mathematics

• Identifiers
  arXiv 2602.08347
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Estimating the Entropy of Linguistic Distributions

  2022cited by this paper
• Fundamentals of Nonparametric Bayesian Inference

  2017cited by this paper
• The Entropy of Words - Learnability and Expressivity across More than 1000 Languages

  2017cited by this paper
• Minimax Estimation of Functionals of Discrete Distributions

  2014cited by this paper
• Minimax Rates of Entropy Estimation on Large Alphabets via Best Polynomial Approximation

  2014cited by this paper
• Bayesian entropy estimation for countable discrete distributions

  2013influential reference
• Asymptotic normality of a nonparametric estimator of sample coverage

  2009influential reference
• Entropy Inference and the James-Stein Estimator, with Application to Nonlinear Gene Association Networks

  2008cited by this paper
• Entropy and diversity

  2006cited by this paper
• Heavy-Tail Phenomena: Probabilistic and Statistical Modeling

  2006cited by this paper
• Estimating entropy on m bins given fewer than m samples

  2004cited by this paper
• Nonparametric estimation of Shannon’s index of diversity when there are unseen species in sample

  2004cited by this paper
• Beta-Diversity in Tropical Forest Trees

  2002cited by this paper
• On the Convergence Rate of Good-Turing Estimators

  2000influential reference
• The two-parameter Poisson-Dirichlet distribution derived from a stable subordinator

  1997cited by this paper
• NewsWeeder: Learning to Filter Netnews

  1995cited by this paper
• Estimating the Number of Classes via Sample Coverage

  1992influential reference
• A Normal Limit Law for a Nonparametric Estimator of the Coverage of a Random Sample

  1983influential reference
• Diversity as a Concept and its Measurement

  1982cited by this paper
• Π-regular variation

  1981influential reference
• Note on the bias of information estimates

  1955cited by this paper
• THE POPULATION FREQUENCIES OF SPECIES AND THE ESTIMATION OF POPULATION PARAMETERS

  1953cited by this paper
• ournal of Statistical Mechanics: J Theory and Experiment Infinite Shannon entropy

  year unknowninfluential reference

• No citing papers are available for this paper.