Sentence Segmentation from Unformatted Text using Language Modeling and Sequence Labeling Approaches

摘要

Current research devoted to the Natural Language Processing problem of sentence segmentation from raw text. The focus was directed to the task of segmentation of auto-generated transcripts for videos that do not have any punctuation and segmentation. Two general approaches to solve the problem of sentence segmentation were proposed and experiments concluded on a comparison of results of pre-trained transformer-based models. Research on how different approach of solving problem affects results were carried out. As a result, the sequence labeling approach turned out to be the most suitable.