...
首页> 外文期刊>Natural language engineering >Assessing user simulation for dialog systems using human judges and automatic evaluation measures
【24h】

Assessing user simulation for dialog systems using human judges and automatic evaluation measures

机译:使用人工判断和自动评估措施评估对话系统的用户模拟

获取原文
获取原文并翻译 | 示例
   

获取外文期刊封面封底 >>

       

摘要

While different user simulations are built to assist dialog system development, there is an increasing need to quickly assess the quality of the user simulations reliably. Previous studies have proposed several automatic evaluation measures for this purpose. However, the validity of these evaluation measures has not been fully proven. We present an assessment study in which human judgments are collected on user simulation qualities as the gold standard to validate automatic evaluation measures. We show that a ranking model can be built using the automatic measures to predict the rankings of the simulations in the same order as the human judgments. We further show that the ranking model can be improved by using a simple feature that utilizes time-series analysis.
机译:尽管构建了不同的用户仿真来辅助对话系统开发,但越来越需要快速可靠地评估用户仿真的质量。先前的研究为此提出了几种自动评估措施。但是,这些评估措施的有效性尚未得到充分证明。我们提供了一项评估研究,其中收集了用户对用户模拟质量的判断作为黄金标准,以验证自动评估措施。我们表明,可以使用自动测量来构建排名模型,以与人类判断相同的顺序预测模拟的排名。我们进一步表明,可以通过使用利用时间序列分析的简单功能来改善排名模型。

著录项

  • 来源
    《Natural language engineering》 |2011年第4期|p.511-540|共30页
  • 作者

    H U A A I; DIANE LITMAN;

  • 作者单位

    Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA 15260, USA;

    Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA 15260, USA;

  • 收录信息
  • 原文格式 PDF
  • 正文语种 eng
  • 中图分类
  • 关键词

相似文献

  • 外文文献
  • 中文文献
  • 专利
获取原文

客服邮箱:kefu@zhangqiaokeyan.com

京公网安备:11010802029741号 ICP备案号:京ICP备15016152号-6 六维联合信息科技 (北京) 有限公司©版权所有
  • 客服微信

  • 服务号