Successful representation learning models appear to develop strikingly similar features to each other, raising the prospect of a fundamental underlying principle. We show that nonlinear Hebbian learning gives a parsimonious account for feature learning, underlying models such as sparse coding, neural networks and independent component analysis. For all datasets considered, the most hyper-Gaussian features are learned irrespective of the effective nonlinearity of the model. Particularly, it explains why Gabor filters are ubiquitously developed for image inputs. Our results reveal that feature learning is robust to normative assumptions, exposing a large class of models with comparable learning properties.
展开▼
