Using bidirectional LSTM with BERT for Chinese punctuation prediction

摘要

Punctuation prediction is an important step in the post processing of ASR systems. Lack of punctuation text is usually difficult to read and understand. In this paper, we propose a method based on Chinese punctuation prediction by combining the Bidirectional Long Short-Term Memory (BLSTM) and the Bidirectional Encoder Representations from Transformers (BERT), which makes the use of BERT as text encoding layers for learning contextualized word representations for improving the performance of BLSTM network. Compared with the previous punctuation prediction methods based on Recurrent Neural Network (RNN), our method improves the performance of punctuation prediction with the powerful ability of capturing semantics and long-distance dependencies in Chinese unsegmented text. Our experimental results on Chinese news datasets that our BERT-BLSTM based method outperforms the baseline by up to 31.07% absolute in overall micro-F1.