Noncompact uniform universal approximation

Abstract

The universal approximation theorem is generalised to uniform convergence on the (noncompact) input space $\mathbb{R}^n$. All continuous functions that vanish at infinity can be uniformly approximated by neural networks with one hidden layer, for all activation functions $φ$ that are continuous, nonpolynomial, and asymptotically polynomial at $\pm\infty$. When $φ$ is moreover bounded, we exactly determine which functions can be uniformly approximated by neural networks, with the following unexpected results. Let $\overline{\mathcal{N}_φ^l(\mathbb{R}^n)}$ denote the vector space of functions that are uniformly approximable by neural networks with $l$ hidden layers and $n$ inputs. For all $n$ and all $l\geq2$, $\overline{\mathcal{N}_φ^l(\mathbb{R}^n)}$ turns out to be an algebra under the pointwise product. If the left limit of $φ$ differs from its right limit (for instance, when $φ$ is sigmoidal) the algebra $\overline{\mathcal{N}_φ^l(\mathbb{R}^n)}$ ($l\geq2$) is independent of $φ$ and $l$, and equals the closed span of products of sigmoids composed with one-dimensional projections. If the left limit of $φ$ equals its right limit, $\overline{\mathcal{N}_φ^l(\mathbb{R}^n)}$ ($l\geq1$) equals the (real part of the) commutative resolvent algebra, a C*-algebra which is used in mathematical approaches to quantum theory. In the latter case, the algebra is independent of $l\geq1$, whereas in the former case $\overline{\mathcal{N}_φ^2(\mathbb{R}^n)}$ is strictly bigger than $\overline{\mathcal{N}_φ^1(\mathbb{R}^n)}$.