BERT MODEL FINE-TUNING METHOD AND APPARATUS BASED ON CONVOLUTIONAL NEURAL NETWORK

摘要

Disclosed are a BERT model fine-tuning method and apparatus based on a convolutional neural network. The method comprises: constructing a first BERT model, a hidden layer of which is a transformer block network, and a second BERT model, a hidden layer of which is a convolutional neural network, wherein the number of layers of the hidden layer of the first BERT model is equal to the number of layers of the hidden layer of the second BERT model; training the first BERT model according to a first text set, and performing knowledge distillation on the second BERT model on the basis of the trained first BERT model, so as to obtain a knowledge distillation loss and a distribution loss of the second BERT model; inputting a second text set into the second BERT model, so as to obtain a cross entropy loss of the second BERT model; and updating a network parameter of the second BERT model according to the knowledge distillation loss and the cross entropy loss. The present application is based on neural network technology. By means of the method, a BERT model, a hidden layer of which is a convolutional neural network, is fine-tuned, and the number of parameters in the fine-tuned BERT model is also significantly reduced, thereby greatly improving the calculation speed of the model, and ensuring the accuracy of text classification of the model.