METHOD AND APPARATUS FOR TRAINING LANGUAGE MODEL ELECTRONIC DEVICE READABLE STORAGE MEDIUM AND COMPUTER PROGRAM PRODUCT

摘要

The present application discloses a training method, an apparatus, an electronic device and a readable storage medium of a language model, and relates to the technical field of natural language processing in artificial intelligence. A specific implementation method is to use a predetermined text corpus in the corpus base to perform prior training learning on the language model in advance; replacing at least one word in the sample text corpus with a word mask, respectively, to obtain a sample text corpus including at least one word mask, inputting the sample text corpus into the language model, and passing the at least one word through the language model output a context vector of each word mask in the mask; determine a word vector corresponding to each word mask based on a context vector and a word vector parameter matrix of each said word mask, respectively; training the language model based on a word vector corresponding to each of the word masks. By introducing the semantic information representation of a larger granularity, the learning ability of the word semantic information of the language model is enhanced, and the risk of information leakage that is likely caused by the whole word masking of the characters can be effectively avoided.