Lake and Baroni (2018) recently introduced the SCAN data set, which consists of simple commands paired with action sequences and is intended to test the strong generalization abilities of recurrent sequence-to-sequence models. Their initial experiments suggested that such models may fail because they lack the ability to extract systematic rules. Here, we take a closer look at SCAN and show that it does not always capture the kind of generalization that it was designed for. To mitigate this we propose a complementary dataset, which requires mapping actions back to the original commands, called NACS. We show that models that do well on SCAN do not necessarily do well on NACS, and that NACS exhibits properties more closely aligned with realistic use-cases for sequence-to-sequence models.
展开▼
机译：Lake And Baroni（2018）最近推出了扫描数据集，它由与动作序列配对的简单命令组成，旨在测试经常性序列到序列模型的强普遍性能力。他们的初步实验表明，这种模型可能会失败，因为它们缺乏提取系统规则的能力。在这里，我们仔细看看扫描并表明它并不总是捕获它设计的概括。要缓解此方法，我们提出了一个互补的数据集，这需要映射映射到原始命令的映射，称为NAC。我们展示了扫描良好的模型不一定在NACS上做得好，并且NACS表现出与序列到序列模型的现实用例更紧密地对齐的性能。
展开▼
