Rethinking LDA: Moment Matching for Discrete ICA

We consider moment matching techniques for estimation in latent Dirichlet allocation (LDA). By drawing explicit links between LDA and discrete versions of independent component analysis (ICA), we first derive a new set of cumulant-based tensors, with an improved sample complexity. Moreover, we reuse standard ICA techniques such as joint diagonalization of tensors to improve over existing methods based on the tensor power method. In an extensive set of experiments on both synthetic and real datasets, we show that our new combination of tensors and orthogonal joint diagonalization techniques outperforms existing moment matching methods.

• Publication year

  2015

• Venue

  Neural Information Processing Systems

• Publication date

  2015-07-07

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 1507.01784
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• FINDING STRUCTURE WITH RANDOMNESS : PROBABILISTIC ALGORITHMS FOR CONSTRUCTING

  2016cited by this paper
• Tensor Factorization via Matrix Factorization

  2015cited by this paper
• A Provably Correct Learning Algorithm for Latent-Variable PCFGs

  2014cited by this paper
• Concentration Inequalities: A Nonasymptotic Theory of Independence

  2013cited by this paper
• A Spectral Algorithm for Latent Dirichlet Allocation

  2012cited by this paper
• A Practical Algorithm for Topic Modeling with Provable Guarantees

  2012cited by this paper
• Tensor decompositions for learning latent variable models

  2012influential reference
• Handbook of Blind Source Separation: Independent Component Analysis and Applications

  2010cited by this paper
• Introduction to the Dirichlet Distribution and Related Processes

  2010cited by this paper
• Tensor Decompositions and Applications

  2009cited by this paper
• Rethinking LDA: Why Priors Matter

  2009cited by this paper
• Presenter : HMW Category : graphical models Preference : Oral Evaluation Methods for Topic Models

  2009cited by this paper
• Latent Dirichlet Allocation

  2009influential reference
• Independent Component Analysis

  2005cited by this paper
• GaP: a factor model for discrete data

  2004cited by this paper
• Applying Discrete PCA in Data Analysis

  2004cited by this paper
• Numerical optimization

  2004cited by this paper
• Euclidean Embedding of Co-occurrence Data

  2004cited by this paper
• Kernel independent component analysis

  2003cited by this paper
• Blind separation of sources, Part I" An adaptive algorithm based on neuromimetic architecture

  2002cited by this paper
• Variational Extensions to EM and Multinomial PCA

  2002cited by this paper
• High-Order Contrasts for Independent Component Analysis

  1999cited by this paper
• Probabilistic Principal Component Analysis

  1999cited by this paper
• Fast and robust fixed-point algorithms for independent component analysis

  1999cited by this paper
• Blind Separation of Sources

  1999cited by this paper
• EM Algorithms for PCA and SPCA

  1997cited by this paper
• Jacobi Angles for Simultaneous Diagonalization

  1996cited by this paper
• Independent component analysis, a survey of some algebraic methods

  1996cited by this paper
• Independent component analysis, A new concept?

  1994cited by this paper
• Numerical Methods for Simultaneous Diagonalization

  1993cited by this paper
• Blind beamforming for non-gaussian signals

  1993influential reference
• Eigen-structure of the fourth-order cumulant tensor with application to the blind source separation problem

  1990cited by this paper
• Source separation using higher order moments

  1989cited by this paper
• Calcul neuromimétique et traitement du signal : analyse en composantes indépendantes

  1987cited by this paper
• Supplementary References

  year unknowncited by this paper

• Time and Memory Trade-off of KV-Cache Compression in Tensor Transformer Decoding

  2025cites this paper
• Training Tensor Attention Efficiently: From Cubic to Almost Linear Time

  2024cites this paper
• Theoretical Constraints on the Expressive Power of RoPE-based Tensor Attention Transformers

  2024cites this paper
• Nonlinear independent component analysis for principled disentanglement in unsupervised deep learning

  2023cites this paper
• Statistical learning with high-cardinality string categorical variables

  2022influential citation
• Binary independent component analysis: a non-stationarity-based approach

  2021cites this paper
• Topic-based Video Analysis

  2021cites this paper
• Binary Independent Component Analysis via Non-stationarity

  2021cites this paper
• Apprentissage statistique à partir de variables catégorielles non-uniformisées

  2019influential citation
• The latent topic block model for the co-clustering of textual interaction data

  2019cites this paper
• Modèle de mélange et modèles linéaires généralisés, application aux données de co-infection (arbovirus & paludisme)

  2018cites this paper
• Fast and Sample-Efficient Learning for Latent Dirichlet Allocation

  2018cites this paper
• Sequential Local Learning for Latent Graphical Models

  2017cites this paper
• Relative Error Tensor Low Rank Approximation

  2017cites this paper
• Weighted Tensor Decomposition for Learning Latent Variables with Partial Data

  2017influential citation
• Supplementary Material for Combinatorial Topic Models using Small-Variance Asymptotics

  2017cites this paper
• Sur deux problèmes d'apprentissage automatique : la détection de communautés et l'appariement adaptatif. (On two problems in machine learning : community detection and adaptive matching)

  2017cites this paper
• Small-Variance Asymptotics for Bayesian Models

  2017influential citation
• Learning from Sequences with Point Processes

  2017cites this paper
• Inference and applications for topic models

  2017cites this paper
• Beyond CCA: Moment Matching for Multi-View Models

  2016influential citation
• On the Method of Moments for Estimation in Latent Variable Models. (Sur la méthode des moments pour d'estimation des modèles à variable latentes)

  2016cites this paper
• Combinatorial Topic Models using Small-Variance Asymptotics

  2016influential citation
• On the method of moments for estimation in latent linear models

  2016cites this paper
• Grasp the implicit features: Hierarchical emotion classification based on topic model and SVM

  2016cites this paper
• Uncovering Causality from Multivariate Hawkes Integrated Cumulants

  2016cites this paper
• Generative adversarial networks

  2014cites this paper