Ladder Polynomial Neural Networks

Polynomial functions have plenty of useful analytical properties, but they are rarely used as learning models because their function class is considered to be restricted. This work shows that when trained properly polynomial functions can be strong learning models. Particularly this work constructs polynomial feedforward neural networks using the product activation, a new activation function constructed from multiplications. The new neural network is a polynomial function and provides accurate control of its polynomial order. It can be trained by standard training techniques such as batch normalization and dropout. This new feedforward network covers several previous polynomial models as special cases. Compared with common feedforward neural networks, the polynomial feedforward network has closed-form calculations of a few interesting quantities, which are very useful in Bayesian learning. In a series of regression and classification tasks in the empirical study, the proposed model outperforms previous polynomial models.

• Publication year

  2019

• Venue

  arXiv.org

• Publication date

  2019-09-25

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 2106.13834
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Fast and Provable Algorithms for Learning Two-Layer Polynomial Neural Networks

  2019cited by this paper
• On the Expressive Power of Deep Polynomial Neural Networks

  2019cited by this paper
• On the Power of Over-parametrization in Neural Networks with Quadratic Activation

  2018influential reference
• Towards Provable Learning of Polynomial Neural Networks Using Low-Rank Matrix Estimation

  2018cited by this paper
• How Does Batch Normalization Help Optimization? (No, It Is Not About Internal Covariate Shift)

  2018cited by this paper
• Multi-output Polynomial Networks and Factorization Machines

  2017cited by this paper
• The Second Order Linear Model

  2017cited by this paper
• Attention is All you Need

  2017cited by this paper
• Polynomial Networks and Factorization Machines: New Insights and Efficient Training Algorithms

  2016influential reference
• Parallelized Tensor Train Learning of Polynomial Classifiers

  2016influential reference
• Deep Residual Learning for Image Recognition

  2015cited by this paper
• Weight Uncertainty in Neural Networks

  2015cited by this paper
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

  2015cited by this paper
• On the Computational Efficiency of Training Neural Networks

  2014cited by this paper
• Dropout: a simple way to prevent neural networks from overfitting

  2014cited by this paper
• Tensor-Train Decomposition

  2011cited by this paper
• Factorization Machines

  2010influential reference

• Identifiability of Deep Polynomial Neural Networks

  2025cites this paper
• FedVS: Straggler-Resilient and Privacy-Preserving Vertical Federated Learning for Split Models

  2023cites this paper