Deep Network With Approximation Error Being Reciprocal of Width to Power of Square Root of Depth

摘要

A new network with super-approximation power is introduced. This net­work is built with Floor (［x］) or ReLU (max{0, x}) activation function in each neuron; hence, we call such networks Floor-ReLU networks. For any hyperparameters N ∈ N~+ and L ∈ N~+, we show that Floor-ReLU net­works with width max{d, 5N + 13} and depth 64dL + 3 can uniformly ap­proximate a Hoelder function f on [0, 1]~d with an approximation error 3λd~(α/2)N~(-αL~(1/2)), where α ∈ (0,1] and λ are the Hoelder order and constant, respectively. More generally for an arbitrary continuous function f on [0, 1]~d with a modulus of continuity ω_f(·), the constructive approximation rate is ω_f(d~(1/2)N~(−L~(1/2))) +2_(ω_f)(d~(1/2))N~(−L~(1/2)). As a consequence, this new class of networks overcomes the curse of dimensionality in approximation power when the variation of ω_f(r) as r→0 is moderate (e.g., ω_f(r) (≤)r~α for Hoelder continuous functions), since the major term to be considered in our approximation rate is essentially d~(1/2) times a function of N and L independent of d within the modulus of continuity.