Neural Networks and Rational Functions

Neural networks and rational functions efficiently approximate each other. In more detail, it is shown here that for any ReLU network, there exists a rational function of degree $O(\text{polylog}(1/\epsilon))$ which is $\epsilon$-close, and similarly for any rational function there exists a ReLU network of size $O(\text{polylog}(1/\epsilon))$ which is $\epsilon$-close. By contrast, polynomials need degree $\Omega(\text{poly}(1/\epsilon))$ to approximate even a single ReLU. When converting a ReLU network to a rational function as above, the hidden constants depend exponentially on the number of layers, which is shown to be tight; in other words, a compositional representation can be beneficial even for rational functions.

• Publication year

  2017

• Venue

  International Conference on Machine Learning

• Publication date

  2017-06-11

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 1706.03301
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks

  2016influential reference
• Error bounds for approximations with deep ReLU networks

  2016influential reference
• Reliably Learning the ReLU in Polynomial Time

  2016cited by this paper
• Why Deep Neural Networks for Function Approximation?

  2016cited by this paper
• Benefits of Depth in Neural Networks

  2016cited by this paper
• Why and when can deep-but not shallow-networks avoid the curse of dimensionality: A review

  2016cited by this paper
• Depth Separation in ReLU Networks for Approximating Smooth Non-Linear Functions

  2016influential reference
• The Power of Depth for Feedforward Neural Networks

  2015cited by this paper
• Representation Benefits of Deep Feedforward Networks

  2015influential reference
• On the Expressive Power of Deep Learning: A Tensor Analysis

  2015cited by this paper
• Neural Network Learning: Theoretical Foundations

  1999cited by this paper
• Constructive approximation : advanced problems

  1996influential reference
• Approximating Threshold Circuits by Rational Functions

  1994cited by this paper
• Splines, Rational Functions and Neural Networks

  1991cited by this paper
• Approximation by superpositions of a sigmoidal function

  1989cited by this paper
• Log Depth Circuits for Division and Related Problems

  1984influential reference
• The Rational Approximation of Real Functions

  1978influential reference
• Rational approximation to | x

  1964influential reference

• Rational ANOVA Networks

  2026cites this paper
• Rational Neural Networks have Expressivity Advantages

  2026cites this paper
• Symbolic recovery of PDEs from measurement data

  2026cites this paper
• Quantitative Approximation Rates for Group Equivariant Learning

  2026influential citation
• KAN Policy: Learning Efficient and Smooth Robotic Trajectories via Kolmogorov-Arnold Networks

  2025cites this paper
• $\mathcal{C}^1$-approximation with rational functions and rational neural networks

  2025cites this paper
• PO-CKAN:Physics Informed Deep Operator Kolmogorov Arnold Networks with Chunk Rational Structure

  2025cites this paper
• A Quantifier-Reversal Approximation Paradigm for Recurrent Neural Networks

  2025cites this paper
• On the Design of Rational Polynomial State Feedback Controllers

  2025cites this paper
• On the algorithmic construction of deep ReLU networks

  2025cites this paper
• Establishing a Scale for Kullback--Leibler Divergence in Language Models Across Various Settings

  2025cites this paper
• Expressive Power of Implicit Models: Rich Equilibria and Test-Time Scaling

  2025cites this paper
• Don't Forget the Nonlinearity: Unlocking Activation Functions in Efficient Fine-Tuning

  2025cites this paper
• Neural network-based singularity detection and applications

  2025cites this paper
• Graph Neural Networks Need Cluster-Normalize-Activate Modules

  2024cites this paper
• Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models

  2024influential citation
• Statistical guarantees for denoising reflected diffusion models

  2024influential citation
• On the Geometry and Optimization of Polynomial Convolutional Networks

  2024cites this paper
• Cauchy activation function and XNet

  2024cites this paper
• Kolmogorov-Arnold Transformer

  2024cites this paper
• rKAN: Rational Kolmogorov-Arnold Networks

  2024cites this paper
• Learning smooth functions in high dimensions: from sparse polynomials to deep neural networks

  2024cites this paper
• An optical mechanism-based deep learning approach for deriving water trophic state of China's lakes from Landsat images.

  2024cites this paper
• Efficient Deep Learning

  2024cites this paper
• A comparison of rational and neural network based approximations

  2023influential citation
• Covering Number of Real Algebraic Varieties and Beyond: Improved Bounds and Applications

  2023cites this paper
• STENCIL-NET for equation-free forecasting from data

  2023cites this paper
• Rational Neural Network Controllers

  2023cites this paper
• Convolutional Neural Operators for robust and accurate learning of PDEs

  2023cites this paper
• Direct Poisson neural networks: learning non-symplectic mechanical systems

  2023cites this paper
• VB-DeepONet: A Bayesian operator learning framework for uncertainty quantification

  2023cites this paper
• Learning to Optimize for Reinforcement Learning

  2023influential citation
• Neural network expression rates and applications of the deep parametric PDE method in counterparty credit risk

  2023cites this paper
• On the Frequency-bias of Coordinate-MLPs

  2022cites this paper
• Transformers with Learnable Activation Functions

  2022cites this paper
• Interpretability of ReLU for Inversion

  2022cites this paper
• Aliasing-Free Convolutional Nonlinear Networks Using Implicitly Defined Functions

  2022cites this paper
• Rational activation functions in neural network with uniform norm based loss function and its application in classification

  2022influential citation
• Why Robust Generalization in Deep Learning is Difficult: Perspective of Expressive Power

  2022influential citation
• SignReLU neural network and its approximation ability

  2022cites this paper
• Approximation Algorithm Applications in Artificial Intelligence: A Review

  2022cites this paper
• Application of a semivariogram based on a deep neural network to Ordinary Kriging interpolation of elevation data

  2022cites this paper
• On Regularizing Coordinate-MLPs

  2022cites this paper
• A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data

  2021cites this paper
• STENCIL-NET: Data-driven solution-adaptive discretization of partial differential equations

  2021influential citation
• The difficulty of computing stable and accurate neural networks: On the barriers of deep learning and Smale’s 18th problem

  2021cites this paper
• Recurrent Rational Networks

  2021cites this paper
• Convergence rate of DeepONets for learning operators arising from advection-diffusion equations

  2021cites this paper
• Consistent Sparse Deep Learning: Theory and Computation

  2021cites this paper
• Parametric Complexity Bounds for Approximating PDEs with Neural Networks

  2021cites this paper
• Neural tensor contractions and the expressive power of deep neural quantum states

  2021cites this paper
• Learning Computable Models From Data

  2021cites this paper
• Exponential ReLU DNN Expression of Holomorphic Maps in High Dimension

  2021cites this paper
• Bridging the Gap between Spatial and Spectral Domains: A Unified Framework for Graph Neural Networks

  2021cites this paper
• A Comprehensive Survey and Performance Analysis of Activation Functions in Deep Learning

  2021cites this paper
• Rational activation functions in neural networks with uniform based loss functions and its application in classification

  2021cites this paper
• Deep Rational Reinforcement Learning

  2021cites this paper
• Adaptive Rational Activations to Boost Deep Reinforcement Learning

  2021cites this paper
• Activation functions in deep learning: A comprehensive survey and benchmark

  2021cites this paper
• Rational neural networks

  2020influential citation
• Neural Networks and Polynomial Regression. Demystifying the Overparametrization Phenomena

  2020cites this paper
• Deep ReLU networks and high-order finite element methods

  2020cites this paper
• Sparse Approximation of Triangular Transports, Part II: The Infinite-Dimensional Case

  2020cites this paper
• Supplementary Material of Rational neural networks

  2020influential citation
• On Polynomial Approximations for Privacy-Preserving and Verifiable ReLU Networks

  2020cites this paper
• Deep ReLU Neural Network Expression Rates for Data-to-QoI Maps in Bayesian PDE Inversion

  2020influential citation
• Learning safe neural network controllers with barrier certificates

  2020cites this paper
• ReLU nets adapt to intrinsic dimensionality beyond the target domain

  2020cites this paper
• Expressivity of Deep Neural Networks

  2020cites this paper
• Data Augmentation for Bayesian Deep Learning

  2019cites this paper
• Efficient Approximation of Solutions of Parametric PDEs by ReLU Neural Networks

  2019cites this paper
• Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks

  2019influential citation
• The phase diagram of approximation rates for deep neural networks

  2019cites this paper
• Smooth Function Approximation by Deep Neural Networks with General Activation Functions

  2019cites this paper
• SPARSE LATENT STRUCTURE

  2019influential citation
• A Theoretical Analysis of Deep Neural Networks and Parametric PDEs

  2019cites this paper
• Scalable Data Augmentation for Deep Learning

  2019cites this paper
• Theoretical guarantees for sampling and inference in generative models with latent diffusions

  2019cites this paper
• Representation Learning with Weighted Inner Product for Universal Approximation of General Similarities

  2019cites this paper
• On representation power of neural network-based graph embedding and beyond

  2018influential citation
• A probabilistic framework for multi-view feature learning with many-to-many associations via neural networks

  2018cites this paper
• Material : A probabilistic framework for multiview feature learning with many-to-many associations via neural networks

  2018cites this paper
• Negative results for approximation using single layer and multilayer feedforward neural networks

  2018cites this paper
• Calibrating Multivariate Lévy Processes with Neural Networks

  2018cites this paper
• Polynomial Regression As an Alternative to Neural Nets

  2018cites this paper
• Representational Power of ReLU Networks and Polynomial Kernels: Beyond Worst-Case Analysis

  2018influential citation
• Posterior Concentration for Sparse Deep Learning

  2018cites this paper
• Rational Neural Networks for Approximating Graph Convolution Operator on Jump Discontinuities

  2018cites this paper
• Some negative results for Neural Networks

  2018influential citation
• Some negative results for single layer and multilayer feedforward neural networks

  2018cites this paper
• Graph Embedding with Shifted Inner Product Similarity and Its Improved Approximation Capability

  2018influential citation
• Rational Neural Networks for Approximating Jump Discontinuities of Graph Convolution Operator

  2018cites this paper
• Deep Learning for Energy Markets

  2018cites this paper
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks

  2017cites this paper
• Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations

  2017cites this paper
• Sum-Product-Quotient Networks

  2017cites this paper
• Approximating Continuous Functions by ReLU Nets of Minimal Width

  2017cites this paper
• Expressivity of deep neural networks

  year unknowncites this paper
• Covering Number of Real Algebraic Varieties: Improved Bound and Applications

  year unknowncites this paper