New Consideration on Criteria of Model Selection

摘要

Model selection is an important problem for intelligent systems. We visit the well known criteria such as AIC, BIC and MDL, and show new remarkable facts which remain unexplored. When we compare two statistical models of which one is a submodel of the larger one, we show there is a better or more economic way of describing given data. This leads us a new criterion of model selection, which sits between MDL and AIC. We also point out that, for a hierarchical family of models such as neural networks or Gaussian mixtures, the classic theories of AIC, BIC and MDL do not hold. A new consideration is again necessary. We give some hints on this problem.