Universal approximation theorem

A pivotal result in deep learning theory stating that

that is, for every continuous function ²¹

\[\begin{equation*}
  f : \mathbb{R}^n \to \mathbb{R}^m
\end{equation*}
\]

there exists a feed-forward neural network \(\hat{f}\) with

• one hidden layer and
• non-polynomial activation functions

that approximates \(f\) such that

\[\begin{equation*}
  \boxed{| f(x) - \hat{f}(x) | < \epsilon}
\end{equation*}
\]

for

• all \(x\) in any given interval \([a, b] \subset \mathbb{R}^n\)
• all degrees of accuracy \(\epsilon\).