You Are What You Read: The Effect of Corpus and Training Task on Semantic Absorption in Recurrent Neural Architectures

摘要

Recurrent neural networks are able to capture semantic meaning within the geometry of the resultant embedding space, but the impact of training corpus and training curriculum on the structure of the learned representations remains poorly understood. This paper sheds some light on the mystery by comparing the performance of a simple recurrent model on three training tasks and two strongly divergent training corpora. The learned representations are compared on tasks including the Semantic Textual Similarity Benchmark, the Stanford Natural Language Inference Corpus, and Google’s Analogical Reasoning Test Set. Results show that context-based training produces the strongest semantic alignment within the embedding space, with reconstruction loss as an interesting close second. Pairwise comparisons of models trained on different corpora show that the choice of corpus also has powerful effects on the learned representations. Most importantly, we observe that the choice of input corpus and training task are not unilaterally independent, but instead interact with each other in interesting ways. This motivates a cautionary position against training neural models by simply throwing as many different training tasks as possible into the mix. Instead, it may be wiser to carefully select only tasks that are compatible with the chosen input corpus, and vice versa.