Some Greedy Learning Algorithms for Sparse Regression and Classification with Mercer Kernels

摘要

We present greedy learning algorithms for building sparse nonlinear regression and classification models from observational data using Mercer kernels. Our objective is to develop efficient numerical schemes for reducing the training and runtime complexities of kernel-based algorithms applied to largedatasets. In the spirit of Natarajan's greedy algorithm (Natarajan, 1995),we iteratively minimize the L₂$ loss function subject to a specified constraint on the degree of sparsity required of the final model or till a specified stopping criterion is reached. We discuss various greedy criteria for basis selection and numerical schemes for improving the robustness and computational efficiency. Subsequently, algorithms based on residual minimization and thin QR factorization are presented for constructing sparse regression and classification models. During thecourse of the incremental model construction, the algorithms are terminatedusing model selection principles such as the minimum descriptive length (MDL) and Akaike's informationcriterion (AIC). Finally, experimental results onbenchmark data are presented to demonstrate the competitiveness of the algorithms developed in this paper.