An Overview of Improved Gradient Descent Algorithms for DNN Training within Significant Revolutions of Training Frameworks

摘要

We absolutely need to train parameters for a given Deep Neural Networks (DNN). One of the most common training algorithms is Gradient Descent (GD). This article looks to bring back classical and emerging GD-based algorithms, against poor convergence, for DNN training within transforming training framework—non-distribution/distribution. We also concentrate on asynchronous/synchronous training framework under distributed training. Additionally, we introduce some evolutionary improved methods for deep learning as alternatives to GD, which have proved to be superior to GD in terms of convergence.