IterationNet: Accelerating model incremental update for small datasets based on knowledge distillation

摘要

Model iteration with new data is important to improve generalization of the model. In general, there are two methods to deal with model incremental update: (a) retraining the model with merging all data together and (b) training a separate model with the new data based on transfer learning. However, the above methods are either time-consuming or suffering from over-fitting problems when the sample size of new data is small. To address this practical issue, we propose a new iteration model, the IterationNet, which can learn features of new data while maintain the performance on the old data. It is a new model iteration method based on knowledge distillation which adds consistency network and truncate LI regularization. In classifying fake avatar images of Weibo users, IterationNet extremely decreased training time from 8 hours to 5 minutes while the accuracy rate is only reduced from 96% to 91% comparing to training with merged data. Compared with transfer learning. IterationNet showed increased accuracy rate by 21 percent with similar training time.