Functional Bregman Divergence and Bayesian Estimation of Distributions

A class of distortions termed functional Bregman divergences is defined, which includes squared error and relative entropy. A functional Bregman divergence acts on functions or distributions, and generalizes the standard Bregman divergence for vectors and a previous pointwise Bregman divergence that was defined for functions. A recent result showed that the mean minimizes the expected Bregman divergence. The new functional definition enables the extension of this result to the continuous case to show that the mean minimizes the expected functional Bregman divergence over a set of functions or distributions. It is shown how this theorem applies to the Bayesian estimation of distributions. Estimation of the uniform distribution from independent and identically drawn samples is presented as a case study.

• Publication year

  2006

• Venue

  IEEE Transactions on Information Theory

• Publication date

  2006-11-23

• Fields of study

  Mathematics, Computer Science

• Identifiers
  DOI 10.1109/TIT.2008.929943arXiv cs/0611123
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• An Introduction to Functional Derivatives

  2008cited by this paper
• Bayesian Quadratic Discriminant Analysis

  2007cited by this paper
• An Analysis of Logistic Models: Exponential Family Connections and Online Performance

  2007cited by this paper
• On weighting clustering

  2006cited by this paper
• Structured Prediction, Dual Extragradient and Bregman Projections

  2006cited by this paper
• Distribution-based Bayesian Minimum Expected Risk for Discriminant Analysis

  2006cited by this paper
• Unifying Divergence Minimization and Statistical Inference Via Convex Duality

  2006cited by this paper
• Axiomatic geometry of conditional models

  2005cited by this paper
• On the optimality of conditional expectation as a Bregman predictor

  2005cited by this paper
• Elements of Information Theory

  2005cited by this paper
• Clustering with Bregman Divergences

  2005cited by this paper
• Information Geometry of U-Boost and Bregman Divergence

  2004cited by this paper
• The Elements of Statistical Learning

  2003cited by this paper
• Duality and Auxiliary Functions for Bregman Distances

  2001cited by this paper
• Logistic Regression, AdaBoost and Bregman Distances

  2000cited by this paper
• Methods of information geometry

  2000cited by this paper
• A new look at entropy for solving linear inverse problems

  1999cited by this paper
• Additive models, boosting, and inference for generalized divergences

  1999cited by this paper
• Relative Loss Bounds for Multidimensional Regression Problems

  1997cited by this paper
• About distances of discrete distributions satisfying the data processing theorem of information theory

  1997cited by this paper
• Generalized projections for non-negative functions

  1995cited by this paper
• General entropy criteria for inverse problems, with applications to data compression, pattern classification, and cluster analysis

  1990cited by this paper
• The Geometry of Asymptotic Inference

  1989cited by this paper
• Advanced calculus of several variables

  1973cited by this paper
• Integrals which are convex functionals. II

  1968influential reference
• Calculus of Variations

  1954cited by this paper
• Theory of point estimation

  1950cited by this paper

• Calibration without Ground Truth

  2026cites this paper
• Bias-variance decompositions: the exclusive privilege of Bregman divergences

  2025cites this paper
• A Unified Representation of Density-Power-Based Divergences Reducible to M-Estimation

  2025cites this paper
• Relationship between Hölder Divergence and Functional Density Power Divergence: Intersection and Generalization

  2025cites this paper
• Fault-tolerant distributed fusion of PDFs using KLDs-induced functional FCM clustering

  2025cites this paper
• Boosting Statistic Learning with Synthetic Data from Pretrained Large Models

  2025cites this paper
• On the Structure of Information

  2024cites this paper
• A Rate-Distortion View of Uncertainty Quantification

  2024cites this paper
• Learning Bregman Divergences with Application to Robustness

  2024cites this paper
• Preconditioners based on Voronoi quantizers of random variable coefficients for stochastic elliptic partial differential equations

  2024cites this paper
• Risk Bounds for Mixture Density Estimation on Compact Domains via the h-Lifted Kullback-Leibler Divergence

  2024cites this paper
• Subsampling Suffices for Adaptive Data Analysis

  2023cites this paper
• Quasi-Arithmetic Mixtures, Divergence Minimization, and Bregman Information

  2023cites this paper
• Distributed Bayesian Estimation in Sensor Networks: Consensus on Marginal Densities

  2023cites this paper
• Comptes Rendus Mécanique

  2023cites this paper
• Bregman divergences for physically informed discrepancy measures for learning and computation in thermomechanics

  2023cites this paper
• A study on fault diagnosis in nonlinear dynamic systems with uncertainties

  2023cites this paper
• Learning Empirical Bregman Divergence for Uncertain Distance Representation

  2023cites this paper
• A Dimensionality Reduction Method for Finding Least Favorable Priors with a Focus on Bregman Divergence

  2022cites this paper
• Generalization of Bayesian inversion with application to PP reflection in a vertical transverse isotropy medium

  2022cites this paper
• Nonasymptotic Concentration Rates in Cooperative Learning–Part I: Variational Non-Bayesian Social Learning

  2022cites this paper
• Variational representations of annealing paths: Bregman information under monotonic embedding

  2022cites this paper
• Uncertainty Estimates of Predictions via a General Bias-Variance Decomposition

  2022cites this paper
• A robust multivariate linear non-parametric maximum likelihood model for ties

  2021cites this paper
• Projected Pseudo-Mirror Descent in Reproducing Kernel Hilbert Space

  2021cites this paper
• Regression using localised functional Bregman divergence

  2021cites this paper
• ϕ-Informational Measures: Some Results and Interrelations

  2021cites this paper
• The Information Projection in Moment Inequality Models: Existence, Dual Representation, and Approximation

  2021cites this paper
• Deep Divergence Learning

  2020influential citation
• Learning sub-patterns in piecewise continuous functions

  2020cites this paper
• A Class of Lower Bounds for Bayesian Risk with a Bregman Loss

  2020cites this paper
• Reasoning About Generalization via Conditional Mutual Information

  2020cites this paper
• Marginal Density Averaging for Distributed Node Localization from Local Edge Measurements

  2020cites this paper
• A Spectral Estimation Framework for Phase Retrieval via Bregman Divergence Minimization

  2020cites this paper
• Sparse Representations of Positive Functions via First- and Second-Order Pseudo-Mirror Descent

  2020cites this paper
• Teaching Experience

  2020cites this paper
• Sparse Representations of Positive Functions via Projected Pseudo-Mirror Descent

  2020cites this paper
• Nonlinear Cauchy problem and identification in contact mechanics: a solving method based on Bregman-gap

  2020cites this paper
• Nonlinear nonlocal Douglas identity

  2020cites this paper
• Update of a conditional probability by minimal divergence

  2019cites this paper
• Update of Prior Probabilities by Minimal Divergence

  2019influential citation
• Prototypical Bregman Networks

  2019cites this paper
• Bayesian influence diagnostics using normalized functional Bregman divergence

  2019cites this paper
• Uncoupled Regression from Pairwise Comparison Data

  2019influential citation
• Transfer metric learning: algorithms, applications and outlooks

  2018cites this paper
• Necessary and sufficient conditions of proper estimators based on self density ratio for unnormalized statistical models

  2018cites this paper
• Copula Variational Bayes inference via information geometry

  2018cites this paper
• Neural language models: Dealing with large vocabularies. (Modèles de langue neuronaux: Gestion des grands vocabulaires)

  2018cites this paper
• Detecting influential observations in spatial models using Bregman divergence

  2018cites this paper
• Bayesian and robust Bayesian analysis in a general setting

  2018cites this paper
• Bregman divergences based on optimal design criteria and simplicial measures of dispersion

  2018cites this paper
• Arbitrage-free regularization, geometric learning, and non-Euclidean filtering in finance

  2018cites this paper
• Calibrating Noise to Variance in Adaptive Data Analysis

  2017cites this paper
• Postquantum Br\`{e}gman relative entropies and nonlinear resource theories

  2017cites this paper
• Supervoxel‐based statistical analysis of diffusion tensor imaging in schizotypal personality disorder

  2017cites this paper
• Quantum Brègman distances and categories

  2017cites this paper
• Iterative Receivers for Downlink MIMO-SCMA: Message Passing and Distributed Cooperative Detection

  2017cites this paper
• Planning of step-stress accelerated degradation test based on the inverse Gaussian process

  2016cites this paper
• Launch and Iterate: Reducing Prediction Churn

  2016cites this paper
• Statistical distances and probability metrics for multivariate data, ensembles and probability distributions

  2015cites this paper
• Level Sets Based Distances for Probability Measures and Ensembles with Applications

  2015cites this paper
• Proper Scoring Rules and Bregman Divergences

  2015cites this paper
• Influence modeling in behavioral data

  2015cites this paper
• On two extensions of the vector quantization scheme

  2015cites this paper
• Applications of Bregman Divergence Measures in

  2015cites this paper
• Information Fusion with Functional Bregman Divergence

  2015cites this paper
• On the Identifiability of Transmission Dynamic Models for Infectious Diseases

  2015cites this paper
• Applications of Bregman Divergence Measures in Bayesian Modeling

  2015cites this paper
• A Bregman Matrix and the Gradient of Mutual Information for Vector Poisson and Gaussian Channels

  2014cites this paper
• Improving Bregman k-means

  2014cites this paper
• Revisiting Stein's paradox: multi-task averaging

  2014cites this paper
• Maximum a posteriori probability estimates in infinite-dimensional Bayesian inverse problems

  2014cites this paper
• A Measure-Theoretic Variational Bayesian Algorithm for Large Dimensional Problems

  2014cites this paper
• The Application of Divergences in Prototype Based Vector Quantization

  2014influential citation
• Bayesian model diagnostics using functional Bregman divergence

  2014cites this paper
• Clustering Longitudinal Health Data Using Hidden Markov Models

  2014cites this paper
• Generalized Bregman divergence and gradient of mutual information for vector Poisson channels

  2013cites this paper
• Tighter PAC-Bayes bounds through distribution-dependent priors

  2013cites this paper
• Information-Theoretic Dissimilarities for Graphs

  2013cites this paper
• Approximated Bayesian Inference for Massive Streaming Data

  2013influential citation
• Divergence measures for statistical data processing - An annotated bibliography

  2013cites this paper
• Generalized minimizers of convex integral functionals, Bregman distance, Pythagorean identities

  2012cites this paper
• Cost-Sensitive Sequences of Bregman Divergences

  2012cites this paper
• Inversion Bayésienne : illustration sur des problèmes tomographiques et astrophysiques

  2012cites this paper
• $\gamma$-SUP: A clustering algorithm for cryo-electron microscopy images of asymmetric particles

  2012cites this paper
• Informative Energy Metric for Similarity Measure in Reproducing Kernel Hilbert Spaces

  2012cites this paper
• Divergence-Based Vector Quantization

  2011influential citation
• Bayes Risk Error is a Bregman Divergence

  2011cites this paper
• Theory and Use of the EM Algorithm

  2011cites this paper
• Wasserstein distances in the analysis of time series and dynamical systems

  2011cites this paper
• Kernels for Longitudinal Data with Variable Sequence Length and Sampling Intervals

  2011cites this paper
• Exploiting structure defined by data in machine learning : some new analyses

  2011cites this paper
• Bregman divergence as general framework to estimate unnormalized statistical models

  2011cites this paper
• Distance-based analysis of dynamical systems and time series by optimal transport

  2010cites this paper
• Divergence measures for statistical data processing

  2010cites this paper
• Families of Alpha- Beta- and Gamma- Divergences: Flexible and Robust Measures of Similarities

  2010cites this paper
• Parametric Bayesian Estimation of Differential Entropy and Relative Entropy

  2010cites this paper
• Quantization and clustering with Bregman divergences

  2010cites this paper
• Quantum theory as inductive inference

  2010cites this paper
• Completely Lazy Learning

  2010cites this paper