Superkernels for RBF Networks Initialization (Short Paper)

摘要

One of the basic tasks solved using artificial neural networks is the regression task. In its canonical form, one seeks for adjusting network's parameters so that its response on input training data fits the desired outputs reasonably well. Training data {x_i, y_i}_(i=1)~n, n ∈ N consists of points from R~(d+1) Euclidean space, i.e., x_i ∈ R~d,y_i ∈ R. The quality of the fit is typically measured in terms of the mean integrated squared error (MISE). Various regularization techniques are considered to prevent from overfitting. Optimal setting of parameters can be specified analytically in the linear model (linear computational units), however, for the nonlinear units, the network's parameters are set using different variants of stochastic optimization [1]. The formulation and solution of the regression task is relatively straightforward in the realm of probability theory. Training data are considered being a random sample from the distribution of the random vector (X,Y) : (Ω,A) → (R~(d+1), B(R~(d+1))). It is well known that the optimal MISE estimator of Y given X is the conditional expectation E[Y|X]. That is, given X = x, the regression function writes E[Y|X = x]. An explicit form of E[Y|X = x] : R~d → R is computed using the joint density f of the distribution of (X,Y). Having access to f(x,y) : R~d → [0,∞), it is the classical result that the conditional distribution of Y given X has the density f(y|x) = f(x,y)/f(x) and E[Y|X = x)= ∫yf(y|x)dy = ∫y_(f(x))~(f(x,y))dy=∫f(x,y)dx/∫yf(x,y)dy. Thus the regression function can be at least theoretically computed in the closed form (of course analytical integration can make problems). The key to this computation is the joint density f(x, y). Theory of non-parametric estimation [2] deals with the approximation of f(x,y) on basis of a random sample {x_i, y_i}_(i=1)~n ~ {X, Y). Namely, we work with the nonparametric approximation of E[Y|X = x] known as the Nadaraya-Watson estimator f_n~(NW), [2, Sec. 1.5]. Given the data {x_i,y_i}_(i=1)~n, the kernel estimate f(x,y) = 1/(nh~(d+1)) ∑_(i=1)~n K((x - x_i)/h_n) K((y - y_i)/h_n) of f is constructed for a suitable function K : R~(d+1) → R known as a kernel and a bandwidth h_n ＞ 0, which depends on the number of data n ∈ N. Approximating capabilities of the kernel are related to its order ℓ ∈ N. The Nadaraya-Watson estimator f_n~(NW) uses f to approximate f and consequently E[Y|X = x] as follows: f_n~(NW)(x)=∫f(x,y)dx/∫yf(x,y)dy=∑)(i=1)~nK((x-x_i)/h_n)/∑_(i=1)~ny_iK((x-x_i)/h_n). We presents the idea of using f_n~(NW) to initialize shallow RBF networks for further training to meet some regularization criterions. The straightforward approach to regularization is to limit the number of computation units. In RBF networks, selecting N 《 n units, their centers and widths can be specified on basis is of clustering the training data. Instead of setting the coefficients of a linear combination in the network using the training data, we linearly regress with respect to {x_k,f_n~(NW)(x_k)}_(k=1)~N', N' ∈ Ns, where {x_k}_(k=1)~N' regularly spans some region of interest, for example, [min_i{x_i~1}, max_i{x_i~1}] ×… × {min_i{x_i~d}, max_i{x_i~d}] with x_i = {x_i~1,…,x_i~d). The granularity of the span then determines the number of points N'. Other schemes for utilizing f_n~(NW) in initializing and learning RBF networks can be presented. The main issue discussed is how to deal with convergence of f to f in dependence on properties of f. The following upper bound applies on the MISE of the presented kernel density estimate [3, Theorem 3.5]: E[∫_(R~(d+1)) (f(x,y)- f{x,y))~2 dxdy] ≤ C·n~(-_(2β+d)~(2β)), where C is constant w.r.t. n and β ∈ N refers to the Sobolev character of the density f that relates to its smoothness. To have the bound valid, it is assumed that the order of kernel K meets β, i.e., that ℓ = β. Whilst the above upper bound increases with d, it decreases with β and lim_(β→∞) n~(-_(2β+d)~(2β)) = n~(-1) for the dimension d fixed. So, increasing smoothness can in some sense override the curse of dimensionality. However, the substantial issue here is that the Sobolev character of f is unknown when working with empirical data, and in consequence one cannot use some kernel K with the corresponding order ℓ = β to construct f_n~(NW). In the contribution, we discuss using the superkernels [2, p. 27] for constructing density kernel estimates and f_n~(NW) for RBF networks initialization. The superkernels are kernels which enjoy simultaneously all orders ℓ ∈ N. If a superkernel is used to construct f, then the maximal rate of convergence applies in the upper bound without exact specification of β, which overcomes the mentioned problem of the unknown Sobolev character. We discuss the construction of multidimensional superkernels, a relation to the Fourier transform and results from experiments showing performance in concrete tasks.