Interpretation and Generalization of Score Matching

Score matching is a recently developed parameter learning method that is particularly effective to complicated high dimensional density models with intractable partition functions. In this paper, we study two issues that have not been completely resolved for score matching. First, we provide a formal link between maximum likelihood and score matching. Our analysis shows that score matching finds model parameters that are more robust with noisy training data. Second, we develop a generalization of score matching. Based on this generalization, we further demonstrate an extension of score matching to models of discrete data.

• Publication year

  2009

• Venue

  Conference on Uncertainty in Artificial Intelligence

• Publication date

  2009-06-18

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 1205.2629
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• A generalization of score matching is developed and used to extend score matching to models of discrete data.

  Confidence 0.93

  박진우 (dztg5apj7m) extractionB (s683577b42) reviewAnonymous (12632b8b5f) review
• Score matching is shown to find model parameters that are more robust to noisy training data.

  Confidence 0.91

  박진우 (dztg5apj7m) extractionB (s683577b42) reviewAnonymous (12632b8b5f) review
• A formal link is established between maximum likelihood and score matching.

  Confidence 0.95

  박진우 (dztg5apj7m) extractionB (s683577b42) reviewAnonymous (12632b8b5f) review

• discrete data models

  model class

  Statistical models defined for variables that take discrete values rather than continuous ones.

  Aliases: models of discrete data, discrete models

  박진우 (dztg5apj7m) extractionB (s683577b42) reviewAnonymous (12632b8b5f) review
• generalization of score matching

  method, learning criterion

  An extended form of score matching introduced to broaden the original criterion beyond its standard continuous setting.

  Aliases: generalized score matching

  박진우 (dztg5apj7m) extractionB (s683577b42) reviewAnonymous (12632b8b5f) review
• intractable partition functions

  model property

  Normalization terms in probabilistic models that are difficult or impossible to compute exactly.

  Aliases: intractable normalizing constants

  박진우 (dztg5apj7m) extractionB (s683577b42) reviewAnonymous (12632b8b5f) review
• maximum likelihood

  learning criterion, method

  A standard parameter-estimation principle that chooses parameters to maximize the probability of the observed data under a model.

  Aliases: ML

  박진우 (dztg5apj7m) extractionB (s683577b42) reviewAnonymous (12632b8b5f) review
• noisy training data

  data quality, condition

  Training examples whose observed values are perturbed or corrupted by noise.

  Aliases: noisy data, corrupted training data

  박진우 (dztg5apj7m) extractionB (s683577b42) reviewAnonymous (12632b8b5f) review
• score matching

  learning criterion, method

  A parameter-learning criterion that fits models by matching the score function of the data distribution to the model.

  Aliases: SM

  박진우 (dztg5apj7m) extractionB (s683577b42) reviewAnonymous (12632b8b5f) review

• Estimating Markov Random Field Potentials for Natural Images

  2009cited by this paper
• Optimal Approximation of Signal Priors

  2008cited by this paper
• Learning Transformational Invariants from Natural Movies

  2008cited by this paper
• Some extensions of score matching

  2007cited by this paper
• Connections Between Score Matching, Contrastive Divergence, and Pseudolikelihood for Continuous-Valued Variables

  2007cited by this paper
• Image Analysis, Random Fields and Markov Chain Monte Carlo Methods: A Mathematical Introduction (Stochastic Modelling and Applied Probability)

  2006cited by this paper
• Learning to be Bayesian without Supervision

  2006cited by this paper
• Elements of Information Theory

  2005cited by this paper
• Estimation of Non-Normalized Statistical Models by Score Matching

  2005cited by this paper
• Image Analysis, Random Fields and Markov Chain Monte Carlo Methods: A Mathematical Introduction

  2002cited by this paper
• Training Products of Experts by Minimizing Contrastive Divergence

  2002cited by this paper
• Clustering by Scale-Space Filtering

  2000cited by this paper
• Scale-Space Theory in Computer Vision

  1993cited by this paper
• Image analysis

  1989cited by this paper
• 7 – Image Analysis

  1984cited by this paper
• Spatial Interaction and the Statistical Analysis of Lattice Systems

  1974cited by this paper
• On the distinction between the conditional probability and the joint probability approaches in the specification of nearest-neighbour systems

  1964cited by this paper

• Ordering-based Causal Discovery via Generalized Score Matching

  2026influential citation
• Provably Reliable Classifier Guidance via Cross-Entropy Control

  2026cites this paper
• Provably Reliable Classifier Guidance through Cross-entropy Error Control

  2026cites this paper
• SCORENF: Score-based Normalizing Flows for Sampling Unnormalized distributions

  2025cites this paper
• New closed-form estimators for discrete distributions

  2025cites this paper
• Learning density ratios in causal inference using Bregman-Riesz regression

  2025influential citation
• Identifying Causal Mechanism Shifts Under Additive Models with Arbitrary Noise

  2025cites this paper
• Composition and Alignment of Diffusion Models using Constrained Learning

  2025cites this paper
• Geometric regularity in deterministic sampling dynamics of diffusion-based generative models

  2025cites this paper
• DGAE: Diffusion-Guided Autoencoder for Efficient Latent Representation Learning

  2025cites this paper
• Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces

  2025cites this paper
• Simulating Fokker-Planck equations via mean field control of score-based normalizing flows

  2025cites this paper
• Unconditional CNN denoisers contain sparse semantic representation of images

  2025cites this paper
• Uni-Instruct: One-step Diffusion Model through Unified Diffusion Divergence Instruction

  2025cites this paper
• Fast and scalable score-based kernel calibration tests

  2025cites this paper
• AmorLIP: Efficient Language-Image Pretraining via Amortization

  2025cites this paper
• Preference Optimization by Estimating the Ratio of the Data Distribution

  2025cites this paper
• Discrete Neural Flow Samplers with Locally Equivariant Transformer

  2025cites this paper
• Generalized Score Matching: Bridging $f$-Divergence and Statistical Estimation Under Correlated Noise

  2025cites this paper
• Monte Carlo Score Matching for Image Generation

  2025cites this paper
• Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Models

  2025cites this paper
• DICE: Distilling Classifier-Free Guidance into Text Embeddings

  2025cites this paper
• Diffusion Generative Modeling on Lie Group Representations

  2025influential citation
• Robust and Conjugate Spatio-Temporal Gaussian Processes

  2025cites this paper
• Score-based physics-informed learning framework for stochastic dynamics

  2025cites this paper
• Robust inference using density-powered Stein operators

  2025cites this paper
• Unified Discrete Diffusion for Categorical Data

  2024cites this paper
• Long and Short Guidance in Score identity Distillation for One-Step Text-to-Image Generation

  2024cites this paper
• Energy-Based Modelling for Discrete and Mixed Data via Heat Equations on Structured Spaces

  2024cites this paper
• Preference Alignment for Diffusion Model via Explicit Denoised Distribution Estimation

  2024cites this paper
• High-Dimensional Differential Parameter Inference in Exponential Family using Time Score Matching

  2024cites this paper
• Simple and Fast Distillation of Diffusion Models

  2024cites this paper
• Conditional Image Synthesis with Diffusion Models: A Survey

  2024cites this paper
• LightPure: Realtime Adversarial Image Purification for Mobile Devices Using Diffusion Models

  2024cites this paper
• DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents

  2024cites this paper
• Kolmogorov-Smirnov GAN

  2024cites this paper
• Guiding a Diffusion Model with a Bad Version of Itself

  2024cites this paper
• Guided Score identity Distillation for Data-Free One-Step Text-to-Image Generation

  2024cites this paper
• On the Trajectory Regularity of ODE-based Diffusion Sampling

  2024cites this paper
• Discrete-state Continuous-time Diffusion for Graph Generation

  2024cites this paper
• Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation

  2024cites this paper
• Momentum-Imbued Langevin Dynamics (MILD) for Faster Sampling

  2024cites this paper
• UniCtrl: Improving the Spatiotemporal Consistency of Text-to-Video Diffusion Models via Training-Free Unified Attention Control

  2024cites this paper
• Sound Source Separation Using Latent Variational Block-Wise Disentanglement

  2024cites this paper
• Diffusion-Based Graph Generative Methods

  2024cites this paper
• Fast Sampling via Discrete Non-Markov Diffusion Models with Predetermined Transition Time

  2023cites this paper
• DSMBind: SE(3) denoising score matching for unsupervised binding energy prediction and nanobody design

  2023cites this paper
• Fit Like You Sample: Sample-Efficient Generalized Score Matching from Fast Mixing Diffusions

  2023cites this paper
• Fast ODE-based Sampling for Diffusion Models in Around 5 Steps

  2023cites this paper
• The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing

  2023cites this paper
• Robust and Conjugate Gaussian Process Regression

  2023cites this paper
• TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Models

  2023cites this paper
• Robust and Scalable Bayesian Online Changepoint Detection

  2023cites this paper
• Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation

  2023influential citation
• Sobolev Space Regularised Pre Density Models

  2023cites this paper
• Training Discrete Energy-Based Models with Energy Discrepancy

  2023cites this paper
• Energy Discrepancies: A Score-Independent Loss for Energy-Based Models

  2023cites this paper
• Self-supervised learning with diffusion-based multichannel speech enhancement for speaker verification under noisy conditions

  2023cites this paper
• Fit Like You Sample: Sample-Efficient Generalized Score Matching from Fast Mixing Markov Chains

  2023influential citation
• Score-based Quickest Change Detection for Unnormalized Models

  2023cites this paper
• Insights into Closed-form IPM-GAN Discriminator Guidance for Diffusion Modeling

  2023cites this paper
• Make-Your-Video: Customized Video Generation Using Textual and Structural Guidance

  2023cites this paper
• A Geometric Perspective on Diffusion Models

  2023cites this paper
• Stein's method of moments

  2023cites this paper
• Training Energy-Based Normalizing Flow with Score-Matching Objectives

  2023cites this paper
• Generalized Precision Matrix for Scalable Estimation of Nonparametric Markov Networks

  2023influential citation
• Controlling Posterior Collapse by an Inverse Lipschitz Constraint on the Decoder Network

  2023cites this paper
• Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models

  2023cites this paper
• Generalized score matching

  2023influential citation
• Understanding the Diffusion Objective as a Weighted Integral of ELBOs

  2023influential citation
• A Survey on Generative Diffusion Model

  2022cites this paper
• Robustness and Accuracy Could Be Reconcilable by (Proper) Definition

  2022cites this paper
• Learning Set Functions Under the Optimal Subset Oracle via Equivariant Variational Inference

  2022cites this paper
• Differentially Private Learning Needs Hidden State (Or Much Faster Convergence)

  2022cites this paper
• Generalized Score Matching for Regression

  2022influential citation
• Diffusion Models for Adversarial Purification

  2022cites this paper
• gDDIM: Generalized denoising diffusion implicit models

  2022cites this paper
• Generalized Bayesian Inference for Discrete Intractable Likelihood

  2022cites this paper
• DPGEN: Differentially Private Generative Energy-Guided Network for Natural Image Synthesis

  2022cites this paper
• Learning Neural Set Functions Under the Optimal Subset Oracle

  2022cites this paper
• Statistical Efficiency of Score Matching: The View from Isoperimetry

  2022cites this paper
• Gradient-Guided Importance Sampling for Learning Binary Energy-Based Models

  2022cites this paper
• Concrete Score Matching: Generalized Score Matching for Discrete Data

  2022influential citation
• Score-based Continuous-time Discrete Diffusion Models

  2022cites this paper
• A Diffusion-ReFinement Model for Sketch-to-Point Modeling

  2022cites this paper
• Denoising Score Matching via Random Fourier Features

  2022cites this paper
• Exponential Family Model-Based Reinforcement Learning via Score Matching

  2021cites this paper
• An effective SteinGLM initialization scheme for training multi-layer feedforward sigmoidal neural networks

  2021cites this paper
• Oops I Took A Gradient: Scalable Sampling for Discrete Distributions

  2021cites this paper
• On Maximum Likelihood Training of Score-Based Generative Models

  2021cites this paper
• Denoising Score Matching with Random Fourier Features

  2021cites this paper
• How to Train Your Energy-Based Models

  2021cites this paper
• Tackling the Generative Learning Trilemma with Denoising Diffusion GANs

  2021cites this paper
• A review on generative Boltzmann networks applied to dynamic systems

  2021cites this paper
• Feature Shift Detection: Localizing Which Features Have Shifted via Conditional Distribution Tests

  2021cites this paper
• Score-Based Generative Modeling with Critically-Damped Langevin Diffusion

  2021cites this paper
• Standardisation-function Kernel Stein Discrepancy: A Unifying View on Kernel Stein Discrepancy Tests for Goodness-of-fit

  2021cites this paper
• Speed Limits for Macroscopic Transitions

  2021influential citation
• Interaction Models and Generalized Score Matching for Compositional Data

  2021cites this paper
• Generalised Kernel Stein Discrepancy (GKSD): A Unifying Approach for Non-parametric Goodness-of-fit Testing

  2021cites this paper