An Efficient Method for Feature Selection in Linear RegressionBased on an Extended Akaike's Information Criterion

摘要

A method for feature selection in linear regression based on an extension of Akaike's infor-mation criterion is proposed. The use of classical Akaike's information criterion (AIC) for featureselection assumes the exhaustive search through all the subsets of features, which has unreasonablyhigh computational and time cost. A new information criterion is proposed that is a continuous exten-sion of AIC. As a result, the feature selection problem is reduced to a smooth optimization problem.An efficient procedure for solving this problem is derived. Experiments show that the proposedmethod enables one to efficiently select features in linear regression. In the experiments, the proposedprocedure is compared with the relevance vector machine, which is a feature selection method basedon Bayesian approach. It is shown that both procedures yield similar results. The main distinction ofthe proposed method is that certain regularization coefficients are identical zeros. This makes it pos-sible to avoid the underfitting effect, which is a characteristic feature of the relevance vector machine.A special case (the so-called nondiagonal regularization) is considered in which both methods areidentical.