The Shape of Data: Intrinsic Distance for Data Distributions

The ability to represent and compare machine learning models is crucial in order to quantify subtle model changes, evaluate generative models, and gather insights on neural network architectures. Existing techniques for comparing data distributions focus on global data properties such as mean and covariance; in that sense, they are extrinsic and uni-scale. We develop a first-of-its-kind intrinsic and multi-scale method for characterizing and comparing data manifolds, using a lower-bound of the spectral variant of the Gromov-Wasserstein inter-manifold distance, which compares all data moments. In a thorough experimental study, we demonstrate that our method effectively discerns the structure of data manifolds even on unaligned data of different dimensionalities; moreover, we showcase its efficacy in evaluating the quality of generative models.

• Publication year

  2019

• Venue

  International Conference on Learning Representations

• Publication date

  2019-05-27

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 1905.11141
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Generative Adversarial Networks

  2021cited by this paper
• On Stabilizing Generative Adversarial Training With Noise

  2019cited by this paper
• Sphere Generative Adversarial Network Based on Geometric Moment Matching

  2019cited by this paper
• Similarity of Neural Network Representations Revisited

  2019cited by this paper
• ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithms

  2018cited by this paper
• Is Generator Conditioning Causally Related to GAN Performance?

  2018cited by this paper
• Geometry Score: A Method For Comparing Generative Adversarial Networks

  2018cited by this paper
• Pros and Cons of GAN Evaluation Measures

  2018cited by this paper
• Estimating the Spectral Density of Large Implicit Matrices

  2018cited by this paper
• On the Dimensionality of Word Embedding

  2018cited by this paper
• Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation

  2018cited by this paper
• Insights on representational similarity in neural networks with canonical correlation

  2018cited by this paper
• NetLSD: Hearing the Shape of a Graph

  2018cited by this paper
• Noise-Tempered Generative Adversarial Networks

  2018cited by this paper
• GILBO: One Metric to Measure Them All

  2018cited by this paper
• Quantitatively Evaluating GANs With Divergences Proposed for Training

  2018cited by this paper
• Demystifying MMD GANs

  2018influential reference
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability

  2017cited by this paper
• Improved Training of Wasserstein GANs

  2017cited by this paper
• Wasserstein Generative Adversarial Networks

  2017influential reference
• Fast Estimation of tr(f(A)) via Stochastic Lanczos Quadrature

  2017cited by this paper
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

  2017cited by this paper
• MMD GAN: Towards Deeper Understanding of Moment Matching Network

  2017cited by this paper
• Improved Techniques for Training GANs

  2016cited by this paper
• Revisiting Classifier Two-Sample Tests

  2016cited by this paper
• Very Deep Convolutional Networks for Large-Scale Image Recognition

  2014cited by this paper
• A fast divide-and-conquer algorithm for computing the spectra of real symmetric tridiagonal matrices

  2013cited by this paper
• Improved Bounds on Sample Size for Implicit Matrix Trace Estimators

  2013cited by this paper
• A Kernel Two-Sample Test

  2012cited by this paper
• Efficient k-nearest neighbor graph construction for generic similarity measures

  2011influential reference
• A spectral notion of Gromov–Wasserstein distance and related methods

  2011cited by this paper
• Randomized algorithms for estimating the trace of an implicit symmetric positive semi-definite matrix

  2011cited by this paper
• Software Framework for Topic Modelling with Large Corpora

  2010cited by this paper
• An Analysis of the Convergence of Graph Laplacians

  2010cited by this paper
• Sample Complexity of Testing the Manifold Hypothesis

  2010cited by this paper
• Matrices, Moments and Quadrature with Applications

  2009cited by this paper
• On the Laplacian Eigenvalues of Gn,p

  2007cited by this paper
• Graph Laplacians and their Convergence on Random Neighborhood Graphs

  2006cited by this paper
• Convergence of Laplacian Eigenmaps

  2006cited by this paper
• Connectivity of random k-nearest-neighbour graphs

  2005cited by this paper
• The Spectra of Random Graphs with Given Expected Degrees

  2004cited by this paper
• Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering

  2001cited by this paper
• On Krylov Subspace Approximations to the Matrix Exponential Operator

  1997cited by this paper
• A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines

  1989cited by this paper
• Calculation of Gauss quadrature rules

  1967cited by this paper
• Mathematics for the Physical Sciences

  1959cited by this paper
• Eurographics Symposium on Geometry Processing 2009 a Concise and Provably Informative Multi-scale Signature Based on Heat Diffusion

  year unknowninfluential reference

• Topological Alternatives for Precision and Recall in Generative Models

  2025cites this paper
• Manifold Approximation leads to Robust Kernel Alignment

  2025cites this paper
• Computational topology in high-dimensional data clustering and manifold learning

  2025cites this paper
• MAYA: Addressing Inconsistencies in Generative Password Guessing through a Unified Benchmark

  2025cites this paper
• k-NN as a Simple and Effective Estimator of Transferability

  2025cites this paper
• RTD-Lite: Scalable Topological Analysis for Comparing Weighted Graphs in Learning Tasks

  2025influential citation
• Approximating Latent Manifolds in Neural Networks via Vanishing Ideals

  2025cites this paper
• Connecting Neural Models Latent Geometries with Relative Geodesic Representations

  2025cites this paper
• TexTile: A Differentiable Metric for Texture Tileability

  2024cites this paper
• SuperLoRA: Parameter-Efficient Unified Adaptation of Multi-Layer Attention Modules

  2024cites this paper
• SuperLoRA: Parameter-Efficient Unified Adaptation for Large Vision Models

  2024cites this paper
• On the Use of Perceptual Loss for Fine Structure Generation : Illustration on Lung MR to CT Synthesis

  2024cites this paper
• SuperLoRA: Parameter-Efficient Unified Adaptation of Large Foundation Models

  2024cites this paper
• Latent Functional Maps

  2024cites this paper
• ReSi: A Comprehensive Benchmark for Representational Similarity Measures

  2024cites this paper
• Recent Trends in Text-to-Image GAN

  2024cites this paper
• Interpoint Inception Distance: Gaussian-Free Evaluation of Deep Generative Models

  2024cites this paper
• Neural Lineage

  2024cites this paper
• Learning to predict perceptual visibility of rendering deterioration in computer games

  2024cites this paper
• Normalized Space Alignment: A Versatile Metric for Representation Analysis

  2024cites this paper
• Latent Space Translation via Inverse Relative Projection

  2024cites this paper
• Do Vision and Language Encoders Represent the World Similarly?

  2024cites this paper
• Identifying Sub-networks in Neural Networks via Functionally Similar Representations

  2024cites this paper
• Mapping the Multiverse of Latent Representations

  2024cites this paper
• FedAnchor: Enhancing Federated Semi-Supervised Learning with Label Contrastive Loss for Unlabeled Clients

  2024cites this paper
• Latent Space Translation via Semantic Alignment

  2023cites this paper
• Bootstrapping Parallel Anchors for Relative Representations

  2023cites this paper
• MR to CT Synthesis Using GANs: A Practical Guide Applied to Thoracic Imaging

  2023cites this paper
• Similarity of Neural Network Models: A Survey of Functional and Representational Measures

  2023influential citation
• Neural FIM for learning Fisher Information Metrics from point cloud data

  2023cites this paper
• Understanding Deep Generative Models with Generalized Empirical Likelihoods

  2023cites this paper
• Adaptive Indexing of Objects with Spatial Extent

  2023cites this paper
• Adaptive Indexing in High-Dimensional Metric Spaces

  2023cites this paper
• Metrics to Quantify Global Consistency in Synthetic Medical Images

  2023cites this paper
• Lung CT Synthesis Using GANs with Conditional Normalization on Registered Ultrashort Echo-Time MRI

  2023cites this paper
• Neural Operator Variational Inference Based on Regularized Stein Discrepancy for Deep Gaussian Processes

  2023cites this paper
• Using Skew to Assess the Quality of GAN-generated Image Features

  2023cites this paper
• Quality Ranking for Synthetically Generated Images

  2023cites this paper
• On the Direct Alignment of Latent Spaces

  2023cites this paper
• Blind Drifting: Diffusion models with a linear SDE drift term for blind image restoration tasks

  2022cites this paper
• Log-Euclidean Signatures for Intrinsic Distances Between Unaligned Datasets

  2022influential citation
• Relative representations enable zero-shot latent space communication

  2022cites this paper
• PyTorch Image Quality: Metrics for Image Quality Assessment

  2022cites this paper
• GM Score: Incorporating Inter-Class and Intra-Class Generator Diversity, Discriminability of Latent Space, and Sample Fidelity for Evaluating GANs

  2022cites this paper
• Image Quality Assessment for Magnetic Resonance Imaging

  2022influential citation
• Evaluation Metrics for Graph Generative Models: Problems, Pitfalls, and Practical Solutions

  2021cites this paper
• Manifold Topology Divergence: a Framework for Comparing Data Manifolds

  2021influential citation
• A Method for Evaluating Deep Generative Models of Images via Assessing the Reproduction of High-order Spatial Context

  2021cites this paper
• Pros and Cons of GAN Evaluation Measures: New Developments

  2021cites this paper
• Representation Topology Divergence: A Method for Comparing Neural Network Representations

  2021influential citation
• Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes

  2021cites this paper
• Subsampling and Aggregation: A Solution to the Scalability Problem in Distance-Based Prediction for Mixed-Type Data

  2021cites this paper
• Generalized Spectral Clustering via Gromov-Wasserstein Learning

  2020cites this paper
• Just SLaQ When You Approximate: Accurate Spectral Distances for Web-Scale Graphs

  2020cites this paper
• Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling

  2020cites this paper
• FREDE: Linear-Space Anytime Graph Embeddings

  2020cites this paper
• Do Vision and Language Encoders Represent the World Similarly?

  year unknowncites this paper