Corpus Specificity in LSA and Word2vec: The Role of Out-of-Domain Documents

摘要

Latent Semantic Analysis (LSA) and Word2vec are some of the most widely usedword embeddings. Despite the popularity of these techniques, the precisemechanisms by which they acquire new semantic relations between words remainunclear. In the present article we investigate whether LSA and Word2veccapacity to identify relevant semantic dimensions increases with size ofcorpus. One intuitive hypothesis is that the capacity to identify relevantdimensions should increase as the amount of data increases. However, if corpussize grow in topics which are not specific to the domain of interest, signal tonoise ratio may weaken. Here we set to examine and distinguish thesealternative hypothesis. To investigate the effect of corpus specificity andsize in word-embeddings we study two ways for progressive elimination ofdocuments: the elimination of random documents vs. the elimination of documentsunrelated to a specific task. We show that Word2vec can take advantage of allthe documents, obtaining its best performance when it is trained with the wholecorpus. On the contrary, the specialization (removal of out-of-domaindocuments) of the training corpus, accompanied by a decrease of dimensionality,can increase LSA word-representation quality while speeding up the processingtime. Furthermore, we show that the specialization without the decrease in LSAdimensionality can produce a strong performance reduction in specific tasks.From a cognitive-modeling point of view, we point out that LSA's word-knowledgeacquisitions may not be efficiently exploiting higher-order co-occurrences andglobal relations, whereas Word2vec does.