In the literature, tensors have been effectively used for capturing the context information in language models. However, the existing methods usually adopt relatively-low order tensors, which have limited expressive power in modeling language. Developing a higher-order tensor representation is challenging, in terms of deriving an effective solution and showing its generality. In this paper, we propose a language model named Tensor Space Language Model (TSLM), by utilizing tensor networks and tensor decomposition. In TSLM, we build a high-dimensional semantic space constructed by the tensor product of word vectors. Theoretically, we prove that such tensor representation is a generalization of the n-gram language model. We further show that this high-order tensor representation can be decomposed to a recursive calculation of conditional probability for language modeling. The experimental results on Penn Tree Bank (PTB) dataset and Wiki-Text benchmark demonstrate the effectiveness of TSLM.
展开▼
机译：在文献中，张量已经有效地用于捕获语言模型中的上下文信息。然而，现有方法通常采用相对低阶的张量，在建模语言中具有有限的表现力。就衍生有效的解决方案并表达其一般性，开发更高阶的张量表示是具有挑战性的。在本文中，我们提出了一种名为Tensor空间语言模型（TSLM）的语言模型，利用张量网络和张量分解。在TSLM中，我们建立由字向量的张量产品构建的高维语义空间。从理论上讲，我们证明这种张量表示是n-gram语言模型的概括。我们进一步表明，这种高阶张量表示可以分解到语言建模的条件概率的递归计算。 Penn Tree Bank（PTB）数据集和维基文本基准的实验结果证明了TSLM的有效性。
展开▼
