A pivotal result in deep learning stating that

feed-forward neural networks are universal estimators,

that is, for every continuous function ¹

\begin{equation*} f : \mathbb{R}^n \to \mathbb{R}^m \end{equation*}

there exists a feed-forward neural network \(\hat{f}\) with

• one hidden layer and
• non-polynomial activation functions

that approximates \(f\) such that

\begin{equation*} \boxed{| f(x) - \hat{f}(x) | < \epsilon} \end{equation*}

for

• all \(x\) in any given interval \([a, b] \subset \mathbb{R}^n\)
• all degrees of accuracy \(\epsilon\).