Least-Squares Independent Component Analysis

Accurately evaluating statistical independence among random variables is a key element of independent component analysis (ICA). In this letter, we employ a squared-loss variant of mutual information as an independence measure and give its estimation method. Our basic idea is to estimate the ratio of probability densities directly without going through density estimation, thereby avoiding the difficult task of density estimation. In this density ratio approach, a natural cross-validation procedure is available for hyperparameter selection. Thus, all tuning parameters such as the kernel width or the regularization parameter can be objectively optimized. This is an advantage over recently developed kernel-based independence measures and is a highly useful property in unsupervised learning problems such as ICA. Based on this novel independence measure, we develop an ICA algorithm, named least-squares independent component analysis.

• Publication year

  2011

• Venue

  Neural Computation

• Publication date

  2011-01-01

• Fields of study

  Mathematics, Computer Science, Medicine

• Identifiers
  DOI 10.1162/NECO_a_00062PMID 20964543
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• Computational complexity of kernel-based density-ratio estimation: a condition number analysis

  2012cited by this paper
• Density-ratio matching under the Bregman divergence: a unified framework of density-ratio estimation

  2012cited by this paper
• Transactions on Information and Systems

  2012cited by this paper
• Feature Selection via L1-Penalized Squared-Loss Mutual Information

  2012cited by this paper
• Machine Learning with Squared-Loss Mutual Information

  2012cited by this paper
• Relative Density-Ratio Estimation for Robust Distribution Comparison

  2011cited by this paper
• Kernel dimension reduction in regression

  2009cited by this paper
• X. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling

  2009cited by this paper
• A Least-squares Approach to Direct Importance Estimation

  2009cited by this paper
• Nonnegative Matrix and Tensor Factorizations - Applications to Exploratory Multi-way Data Analysis and Blind Source Separation

  2009cited by this paper
• Direct importance estimation for covariate shift adaptation

  2008cited by this paper
• Estimating Divergence Functionals and the Likelihood Ratio by Convex Risk Minimization

  2008cited by this paper
• Approximating Mutual Information by Maximum Likelihood Density Ratio Estimation

  2008cited by this paper
• Sequential Fixed-Point ICA Based on Mutual Information Minimization

  2008influential reference
• Kernel Measures of Conditional Dependence

  2007cited by this paper
• On Divergences and Informations in Statistics and Information Theory

  2006cited by this paper
• Independent Component Analysis

  2005influential reference
• Measuring Statistical Dependence with Hilbert-Schmidt Norms

  2005cited by this paper
• Learning algorithms utilizing quasi-geodesic flows on the Stiefel manifold

  2005cited by this paper
• Kernel Methods for Measuring Independence

  2005cited by this paper
• Kernel independent component analysis

  2003cited by this paper
• Estimation of Entropy and Mutual Information

  2003influential reference
• Adaptive Blind Signal and Image Processing - Learning Algorithms and Applications

  2002cited by this paper
• Blind separation of sources, Part I" An adaptive algorithm based on neuromimetic architecture

  2002influential reference
• Fast and robust fixed-point algorithms for independent component analysis

  1999cited by this paper
• Natural Gradient Works Efficiently in Learning

  1998cited by this paper
• A New Learning Algorithm for Blind Signal Separation

  1995cited by this paper
• A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection

  1995influential reference
• Blind beamforming for non-gaussian signals

  1993influential reference
• The Predictive Sample Reuse Method with Applications

  1975influential reference
• A General Class of Coefficients of Divergence of One Distribution from Another

  1966influential reference
• On Information and Sufficiency

  1951cited by this paper
• On the Criterion that a Given System of Deviations from the Probable in the Case of a Correlated System of Variables is Such that it Can be Reasonably Supposed to have Arisen from Random Sampling

  1900cited by this paper
• Natural Gradient Works Eciently in Learning

  year unknowncited by this paper

• Self-sufficient Independent Component Analysis via KL Minimizing Flows

  2025influential citation
• Constant-Time Algorithms for Continuous Optimization Problems

  2021cites this paper
• Non-negative matrix factorisation is the most appropriate method for extraction of muscle synergies in walking and running

  2020cites this paper
• A noise reduction method based on EWT-ICA for spectrum induced polarisation data

  2020cites this paper
• Rank based Least-squares Independent Component Analysis

  2018influential citation
• On scalable inference and learning in spike-and-slab sparse coding

  2017cites this paper
• Minimizing Quadratic Functions in Constant Time

  2016cites this paper
• Minimum dependency key frames selection via quadratic mutual information

  2015cites this paper
• Cross-Domain Matching with Squared-Loss Mutual Information

  2015cites this paper
• Registration of infrared transmission images using squared-loss mutual information

  2015cites this paper
• Gradient of Probability Density Functions based Contrasts for Blind Source Separation (BSS)

  2015cites this paper
• Narrow-Band Interference Suppression for SAR Based on Independent Component Analysis

  2013cites this paper
• Direct Divergence Approximation between Probability Distributions and Its Applications in Machine Learning

  2013cites this paper
• Learning under Non-stationarity : Covariate Shift Adaptation by Importance Weighting Masashi Sugiyama

  2013cites this paper
• Direct Approximation of Divergences Between Probability Distributions

  2013cites this paper
• Divergence estimation for machine learning and signal processing

  2013cites this paper
• Improved variability classification of CoRoT targets with Giraffe spectra

  2013cites this paper
• between Probability Distributions: Recent Advances in Machine Learning

  2013cites this paper
• Studies on Kernel Learning and Independent Component Analysis

  2013cites this paper
• 0 Perceived Age Estimation from Face Images

  2013cites this paper
• Feature Selection via L1-Penalized Squared-Loss Mutual Information

  2012cites this paper
• Forecast of Employment Based on Independent Component Analysis

  2012cites this paper
• Density-ratio matching under the Bregman divergence: a unified framework of density-ratio estimation

  2012cites this paper
• Machine Learning with Squared-Loss Mutual Information

  2012influential citation
• Learning Under Non-stationarity: Covariate Shift Adaptation by Importance Weighting

  2012cites this paper
• Computational complexity of kernel-based density-ratio estimation: a condition number analysis

  2012cites this paper
• A truncated EM approach for spike-and-slab sparse coding

  2012influential citation
• On Kernel Parameter Selection in Hilbert-Schmidt Independence Criterion

  2012cites this paper
• A Truncated Variational EM Approach for Spike-and-Slab Sparse Coding

  2012cites this paper
• Transactions on Information and Systems

  2012cites this paper
• On nonparametric measures of dependence and conditional independence: Theory and applications

  2011cites this paper
• Perceived Age Estimation from Face Images

  2011cites this paper
• A Closed-Form EM Algorithm for Sparse Coding

  2011influential citation
• Relative Density-Ratio Estimation for Robust Distribution Comparison

  2011cites this paper
• Closed-Form EM for Sparse Coding and Its Application to Source Separation

  2011influential citation
• Direct Importance Estimation with a Mixture of Probabilistic Principal Component Analyzers

  2010cites this paper