An effective algorithm for hyperparameter optimization of neural networks

摘要

A major challenge in designing neural network (NN) systems is to determinethe best structure and parameters for the network given the data for themachine learning problem at hand. Examples of parameters are the number oflayers and nodes, the learning rates, and the dropout rates. Typically, theseparameters are chosen based on heuristic rules and manually fine-tuned, whichmay be very time-consuming, because evaluating the performance of a singleparametrization of the NN may require several hours. This paper addresses theproblem of choosing appropriate parameters for the NN by formulating it as abox-constrained mathematical optimization problem, and applying aderivative-free optimization tool that automatically and effectively searchesthe parameter space. The optimization tool employs a radial basis functionmodel of the objective function (the prediction accuracy of the NN) toaccelerate the discovery of configurations yielding high accuracy. Candidateconfigurations explored by the algorithm are trained to a small number ofepochs, and only the most promising candidates receive full training. Theperformance of the proposed methodology is assessed on benchmark sets and inthe context of predicting drug-drug interactions, showing promising results.The optimization tool used in this paper is open-source.