LEAST-SQUARES AND STOCHASTIC GRADIENT ALGORITHMS HAVE THE SAME CONVERGENCE RATE ORDER

摘要

This paper presents novel results on the almost sure convergence of the stochastic gradient based self-turning controller. The main focus of the paper is to evaluate convergence rate of the parameter estimates. It is proved that, surprisingly, this rate has the same order as the best one established for the least-squares algorithm, and it is the same as that in the laws of iterated logarithm. Although we consider the case of self-tuning controllers, presented results can easily be extended to some other adaptive processes.