Language Models Learn POS First

摘要

The SVCCA results imply that early in training, representing syntax and POS is the natural way to get initial high performance. However, as training progresses, these low-level aspects of linguistic structure sees diminishing returns from committing more parameters to their representation. Instead, later training realizes more gains from refining representations of global topic.