A Surrogate Model Assisted Quantum-inspired Evolutionary Algorithm for Hyperparameter Optimization in Machine Learning

摘要

Machine learning techniques have achieved remarkable development in recent years. However, the performance of many machine learning models usually involves careful tuning of hyperparameters. The hyperparameter optimization (HPO) is usually a high-dimensional black box optimization problem and often faces expensive function evaluations. Besides, the task of HPO has gained great attention in academy and industry. In this paper, a novel method for hyperparameter optimization is proposed, referred to as surrogate model assisted quantum-inspired evolutionary algorithm (SA-QEA), which incorporates the principles of quantum-inspired evolutionary algorithm (QEA) and an efficient search framework based on a surrogate model. In the proposed algorithm, we adopt a single individual QEA with neighborhood exploration as the evolution scheme to generate the candidate solutions, and multivariate adaptive regression splines (MARS) is used as a surrogate to approximate the objective function around the individuals. Through conducting comprehensive experimental evaluations on two benchmark problems and three machine learning models, we test our proposed algorithm and compare it with other widely used methods. The results achieve competitive performance and demonstrate the effectiveness of SA-QEA for hyperparameter optimization.