High-order approximation rates for shallow neural networks with cosine and ReLUk activation functions

Jonathan W. Siegel, Jinchao Xu

Research output: Contribution to journalArticlepeer-review

19 Scopus citations

Abstract

We study the approximation properties of shallow neural networks with an activation function which is a power of the rectified linear unit. Specifically, we consider the dependence of the approximation rate on the dimension and the smoothness in the spectral Barron space of the underlying function f to be approximated. We show that as the smoothness index s of f increases, shallow neural networks with ReLUk activation function obtain an improved approximation rate up to a best possible rate of O(n−(k+1)log⁡(n)) in L2, independent of the dimension d. The significance of this result is that the activation function ReLUk is fixed independent of the dimension, while for classical methods the degree of polynomial approximation or the smoothness of the wavelets used would have to increase in order to take advantage of the dimension dependent smoothness of f. In addition, we derive improved approximation rates for shallow neural networks with cosine activation function on the spectral Barron space. Finally, we prove lower bounds showing that the approximation rates attained are optimal under the given assumptions.

Original languageEnglish (US)
Pages (from-to)1-26
Number of pages26
JournalApplied and Computational Harmonic Analysis
Volume58
DOIs
StatePublished - May 2022

All Science Journal Classification (ASJC) codes

  • Applied Mathematics

Fingerprint

Dive into the research topics of 'High-order approximation rates for shallow neural networks with cosine and ReLUk activation functions'. Together they form a unique fingerprint.

Cite this