Active Learning and Basis Selection for Kernel-Based Linear Models: A Bayesian Perspective

摘要

We develop an active learning algorithm for kernel-based linear regression and classification. The proposed greedy algorithm employs a minimum-entropy criterion derived using a Bayesian interpretation of ridge regression. We assume access to a matrix, ${bf Phi}in{BBR}^{Ntimes N}$, for which the $(i,j)$th element is defined by the kernel function $K(gamma_i,gamma_j)in{BBR}$, with the observed data $gamma_iin{BBR}^d$. We seek a model, ${cal M}:gamma_irightarrow y_i$, where $y_i$ is a real-valued response or integer-valued label, which we do not have access to a priori. To achieve this goal, a submatrix, ${bf Phi}_{I_l,I_b} in{BBR}^{ntimes m}$, is sought that corresponds to the intersection of $n$ rows and $m$ columns of ${bf Phi}$, indexed by the sets $I_l$ and $I_b$, respectively. Typically $mll N$ and $nll N$. We have two objectives: $(i)$ Determine the $m$ columns of ${bf Phi}$, indexed by the set $I_b$, that are the most informative for building a linear model, ${cal M}: [1 {bf Phi}_{i,I_b}]^T rightarrow y_i$ , without any knowledge of ${y_i}_{i=1}^N$ and $(ii)$ using active learning, sequentially determine which subset of $n$ elements of ${y_i}_{i=1}^N$ should be acquired; both stopping values, $vert I_bvert = m$ and