Web-based monitoring, logging and reporting tools for multi-service multi-modal systems

摘要

This paper describes MILER (Multi-modal data Logger for Evaluation and Report), a web-based multi-service monitoring, logging and reporting tool for advanced multi-modal dialog systems. MILER has been designed to directly arrange and synchronize logging data collected from live services and to provide real-time reports about service usage and system performance. Special attention has been given to the architecture design in order to achieve service and access-device independence and reliable synchronization of data from distributed logs. MILER allows researchers to analyze multi-modal interactions, analyze the call flow, reconstruct the system/user dialogue turns,play the recorded user utterances, and provide a preliminary dialogue performance evaluation. It also supports labeling and anaotation of the dialogue turns for further offline analysis. Once the user inputs (i.e. speech and othe rinput modalities) are manually transcribed and labeled, along with detailed log events from each dialog, MILER derives a set of objective mekasures. which includes word and concept accuracy, number of attempts per concept, dialog turn counts and duration, and task completin rates. Subjective measures extracted from user's surveys. Including perceived task success and ease of use measures,can be combined with the objective measures and the results used later for accuracy computation.