SYSTEMS AND METHODS FOR WORD SEGMENTATION BASED ON A COMPETING NEURAL CHARACTER LANGUAGE MODEL

摘要

A computer-implemented method and system for word segmentation, comprising: a computer-implemented method for word segmentation, comprising: receiving a plurality of characters for segmentation; converting the characters of the plurality of characters into embedding vectors by using the embedding model; feed the embedding vector to the forward language model to retrieve the first output vector; feed the embedding vector to the reverse language model to retrieve the second output vector; compare the embedding vector with the first output vector and the second output vector; and dividing the plurality of characters based on the comparison, wherein comparing the embedding vector with the first output vector and the second output vector is an inverse of the Euclidean distance between the embedding vector and each of the first output vector and the second output vector. Including determining the exponent.