METHOD AND APPARATUS FOR TRAINING LANGUAGE MODEL, ELECTRONIC DEVICE, READABLE STORAGE MEDIUM AND COMPUTER PROGRAM PRODUCT

摘要

A method and apparatus for training a language model, an electronic device, a readable storage medium and a computer program product, which relate to the field of natural language processing technologies in artificial intelligence, are disclosed. The method may include pre-training the language model using preset text language materials in a corpus; replacing at least one word in a sample text language material with a word mask respectively to obtain a sample text language material including at least one word mask; inputting the sample text language material including the at least one word mask into the language model, and outputting a context vector of each of the at least one word mask via the language model; determining a word vector corresponding to each word mask based on the context vector of the word mask and a word vector parameter matrix; and training the language model based on the word vector corresponding to each word mask. Introduction of semantic information representation with greater granularity enhances the capacity of the language model to learn word meaning information and may effectively avoid an information leakage risk possibly caused by character-based full word coverage.