A Large Dimensional Analysis of Least Squares Support Vector Machines

In this paper, a large dimensional performance analysis of kernel least squares support vector machines (LS-SVMs) is provided under the assumption of a two-class Gaussian mixture model for the input data. Building upon recent advances in a random matrix theory, we show, when the dimension of data <inline-formula><tex-math notation="LaTeX">$p$</tex-math></inline-formula> and their number <inline-formula><tex-math notation="LaTeX">$n$</tex-math></inline-formula> are both large, that the LS-SVM decision function can be well approximated by a normally distributed random variable, the mean and variance of which depend explicitly on a local behavior of the kernel function. This theoretical result is then applied to the MNIST and Fashion-MNIST datasets which, despite their non-Gaussianity, exhibit a convincingly close behavior. Most importantly, our analysis provides a deeper understanding of the mechanism into play in SVM-type methods and in particular of the impact on the choice of the kernel function as well as some of their theoretical limits in separating high-dimensional Gaussian vectors.

• Publication year

  2017

• Venue

  IEEE Transactions on Signal Processing

• Publication date

  2017-01-11

• Fields of study

  Mathematics, Computer Science

• Identifiers
  DOI 10.1109/TSP.2018.2889954arXiv 1701.02967
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Random Matrix Asymptotics of Inner Product Kernel Spectral Clustering

  2018cited by this paper
• Classification Asymptotics in the Random Matrix Regime

  2018cited by this paper
• A Random Matrix Approach to Neural Networks

  2017cited by this paper
• Harnessing neural networks: A random matrix approach

  2017cited by this paper
• The counterintuitive mechanism of graph-based semi-supervised learning in the big data regime

  2017cited by this paper
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms

  2017cited by this paper
• A random matrix analysis and improvement of semi-supervised learning for large dimensional data

  2017cited by this paper
• Random matrix improved subspace clustering

  2016cited by this paper
• Kernel spectral clustering of large dimensional data

  2015cited by this paper
• THE SPECTRUM OF RANDOM INNER-PRODUCT KERNEL MATRICES

  2012cited by this paper
• Machine learning - a probabilistic perspective

  2012cited by this paper
• Extreme Learning Machine for Regression and Multiclass Classification

  2012cited by this paper
• The spectrum of kernel random matrices

  2010influential reference
• Probability and Measure

  2009cited by this paper
• Optimal Rates for Regularized Least Squares Regression

  2009cited by this paper
• Spectral Analysis of Large Dimensional Random Matrices

  2009cited by this paper
• Relaxed online SVMs for spam filtering

  2007cited by this paper
• Optimal Rates for the Regularized Least-Squares Algorithm

  2007cited by this paper
• SVM versus Least Squares SVM

  2007cited by this paper
• A GA-based feature selection and parameters optimizationfor support vector machines

  2006cited by this paper
• Multicategory Proximal Support Vector Machine Classifiers

  2005cited by this paper
• Automatic model selection for the optimization of SVM kernels

  2005cited by this paper
• Practical selection of SVM parameters and noise estimation for SVM regression

  2004cited by this paper
• A tutorial on support vector regression

  2004cited by this paper
• Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond

  2003cited by this paper
• Bayesian Framework for Least-Squares Support Vector Machine Classifiers, Gaussian Processes, and Kernel Fisher Discriminant Analysis

  2002cited by this paper
• Choosing Multiple Parameters for Support Vector Machines

  2002cited by this paper
• RSVM: Reduced Support Vector Machines

  2001cited by this paper
• Regularization Networks and Support Vector Machines

  2000cited by this paper
• Generalized Discriminant Analysis Using a Kernel Approach

  2000cited by this paper
• The Nature of Statistical Learning Theory

  2000cited by this paper
• Image representations for object detection using kernel classifiers

  2000cited by this paper
• Feature Selection for SVMs

  2000cited by this paper
• A Trainable System for Object Detection

  2000cited by this paper
• Least Squares Support Vector Machine Classifiers

  1999cited by this paper
• Support vector machines for spam categorization

  1999cited by this paper
• Fisher discriminant analysis with kernels

  1999cited by this paper
• Neural Networks for Signal Processing IX : proceedings of the 1999 IEEE Signal Processing Society Workshop

  1999cited by this paper
• Text Categorization with Support Vector Machines: Learning with Many Relevant Features

  1999cited by this paper
• Gradient-based learning applied to document recognition

  1998influential reference
• Training support vector machines: an application to face detection

  1997cited by this paper
• Learning algorithms for classification: A comparison on handwritten digit recognition

  1995cited by this paper
• Support-Vector Networks

  1995cited by this paper

• Automated Extraction of Scientific Statements for Integrated Assessment of Animal Welfare using Language Models

  2026cites this paper
• Concept activation vectors: a unifying view and adversarial attacks

  2025influential citation
• Classifying Overlapping Gaussian Mixtures in High Dimensions: From Optimal Classifiers to Neural Nets

  2024cites this paper
• Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixtures

  2024cites this paper
• The Breakdown of Gaussian Universality in Classification of High-dimensional Linear Factor Mixtures

  2024cites this paper
• Random Matrix Analysis to Balance between Supervised and Unsupervised Learning under the Low Density Separation Assumption

  2023influential citation
• Adversarial Learning-Based Sentiment Analysis for Socially Implemented IoMT Systems

  2023cites this paper
• Large Dimensional Analysis of LS-SVM Transfer Learning: Application to Polsar Classification

  2023influential citation
• Optimizing Spca-based Continual Learning: A Theoretical Approach

  2023cites this paper
• Security provisions in smart edge computing devices using blockchain and machine learning algorithms: a novel approach

  2022cites this paper
• Apprentissage par transfert en grande dimension : application aux images SAR polarimétriques

  2022influential citation
• An Efficient Algorithm for a Class of Large-Scale Support Vector Machines Exploiting Hidden Sparsity

  2022cites this paper
• Accurate on-line support vector regression incorporated with compensated prior knowledge

  2021cites this paper
• Federated Learning Meets Blockchain in Edge Computing: Opportunities and Challenges

  2021cites this paper
• A Random Matrix Perspective on Random Tensors

  2021cites this paper
• What causes the test error? Going beyond bias-variance via ANOVA

  2020cites this paper
• M L ] 3 S ep 2 02 0 Large Dimensional Multi Task Learning Large Dimensional Analysis and Improvement of Multi Task Learning

  2020influential citation
• Large Dimensional Asymptotics of Multi-Task Learning

  2020influential citation
• On the Precise Error Analysis of Support Vector Machines

  2020cites this paper
• Real-Time Embedded EMG Signal Analysis for Wrist-Hand Pose Identification

  2020cites this paper
• Risk Convergence of Centered Kernel Ridge Regression with Large Dimensional Data

  2020cites this paper
• Asymptotics of Ridge(less) Regression under General Source Condition

  2020cites this paper
• Word Representations Concentrate and This is Good News!

  2020influential citation
• Deciphering and Optimizing Multi-Task and Transfer Learning: a Random Matrix Approach

  2020cites this paper
• Large Dimensional Analysis and Improvement of Multi Task Learning

  2020influential citation
• Implementation and Evaluation of LS-SVM Optimization Methods for Estimating DoAs

  2020cites this paper
• High Dimensional Classification via Regularized and Unregularized Empirical Risk Minimization: Precise Error and Optimal Loss.

  2019cites this paper
• Risk Convergence of Centered Kernel Ridge Regression With Large Dimensional Data

  2019cites this paper
• A Large Scale Analysis of Logistic Regression: Asymptotic Performance and New Insights

  2019cites this paper
• Spectral distribution of large generalized random kernel matrices

  2019cites this paper
• High Dimensional Classification via Empirical Risk Minimization: Improvements and Optimality

  2019cites this paper
• A Flexible EM-Like Clustering Algorithm for Noisy Data

  2019cites this paper
• Standard Condition Number of Hessian Matrix for Neural Networks

  2019cites this paper
• Inner-product Kernels are Asymptotically Equivalent to Binary Discrete Kernels

  2019cites this paper
• High Dimensional Robust Classification: A Random Matrix Analysis

  2019cites this paper
• A Random Matrix Analysis and Optimization Framework to Large Dimensional Transfer Learning

  2019cites this paper
• A random matrix framework for large dimensional machine learning and neural networks

  2019cites this paper
• Methods of random matrices for large dimensional statistical learning

  2019cites this paper
• Classification Asymptotics in the Random Matrix Regime

  2018cites this paper
• Nouvelles méthodes pour l'apprentissage non-supervisé en grandes dimensions. (New methods for large-scale unsupervised learning)

  2018cites this paper
• A Random Matrix and Concentration Inequalities Framework for Neural Networks Analysis

  2018cites this paper
• Random Matrix-Improved Kernels For Large Dimensional Spectral Clustering

  2018cites this paper
• A Random Matrix Approach to Neural Networks

  2017cites this paper
• A random matrix analysis and improvement of semi-supervised learning for large dimensional data

  2017influential citation