Noncompact uniform universal approximation
share
The universal approximation theorem is generalised to uniform convergence on the (noncompact) input space ℝ^n. All continuous functions that vanish at infinity can be uniformly approximated by neural networks with one hidden layer, for all continuous activation functions φ≠0 with asymptotically linear behaviour at ±∞. When φ is moreover bounded, we exactly determine which functions can be uniformly approximated by neural networks, with the following unexpected results. Let 𝒩_φ^l(ℝ^n) denote the vector space of functions that are uniformly approximable by neural networks with l hidden layers and n inputs. For all n and all l≥2, 𝒩_φ^l(ℝ^n) turns out to be an algebra under the pointwise product. If the left limit of φ differs from its right limit (for instance, when φ is sigmoidal) the algebra 𝒩_φ^l(ℝ^n) (l≥2) is independent of φ and l, and equals the closed span of products of sigmoids composed with one-dimensional projections. If the left limit of φ equals its right limit, 𝒩_φ^l(ℝ^n) (l≥1) equals the (real part of the) commutative resolvent algebra, a C*-algebra which is used in mathematical approaches to quantum theory. In the latter case, the algebra is independent of l≥1, whereas in the former case 𝒩_φ^2(ℝ^n) is strictly bigger than 𝒩_φ^1(ℝ^n).
READ FULL TEXT