AutoDLCon: An Approach for Controlling the Automated Tuning for Deep Learning Networks

摘要

Neural networks have become the main building block on revolutionizing the field of artificial intelligence aided applications. With the wide availability of data and the increasing capacity of computing resources, they triggered a new era of state-of-the-art results in diverse directions. However, building neural network models is domain-specific, and figuring out the best architecture and hyper-parameters in each problem is still an art. In practice, it is a highly iterative process that is very time-consuming, requires substantial computing resources, and needs deep knowledge and solid technical skills. To tackle this challenge, we introduce a new gradient-based technique, AutoDLCon, that aims to automate the design process of neural network architecture for the given classification task and dataset within a specified time budget using a controller neural network. In particular, the controller network predicts how good a model is and suggests trying an optimized model by back-propagating from a loss function through the controller network to the controller’s weights one time and to the controller’s inputs at another time. This approach mimics the exploration of the search space in a more efficient way by reducing the number of trials to the minimum. As a consequence, it can significantly reduce the time budget and computing resources to a minimum and controllable level. We evaluate our approach using MNIST and CIFAR-10 datasets with different settings based on the difficulty of the problem. The results of our experiments show that our approach is able to generate child models that are good enough to obtain competitive results on the validation data after only a few trials.