Attention to Deep Structure in Recurrent Neural Networks

摘要

Deep recurrent networks can build complex representations of sequential data, enabling them to do things such as translate text from one language into another. These networks often utilize an attention mechanism that allows them to focus on important input representations. Network attention can be used to provide insight into network strategies for identifying structure in the input sequence. This study investigates attention in a deep recurrent network, trained to annotate text, to determine how the distribution of information in the text affects learning and network performance. Results suggest that reversing the input text makes it difficult for the network to discover higher-order linguistic structure. This study contributes to an understanding of how our brains might process sequential data.