Reference Priors for Neural Networks: Laplace versus Gaussian

摘要

Motivated by the principle of maximum entropy, the Laplace prior has been introduced in the Bayesian inference approach to training feedforward neural networks as the prior distribution for network weights. In this paper, we examine in detail the arguments supporting the Laplace prior and argue that network weights, being continuous quantities with no obvious upper bound, complicate the application of the maximum entropy principle. On the other hand, we motivate the use of the Gaussian prior as reference prior in neural networks, basing on the assumptions of exchangeability and spherical symmetry.