Multimodal Meeting Capture and Understanding with the CALO Meeting Assistant

摘要

The CALO Meeting Assistant is a multimodal meeting assistant technology that integrates speech, gestures, and multimodal data collected from multiparty interactions during meetings. Using machine learning and robust discourse processing, it provides a rich, browsable record of a meeting. For nearly as long as people have been holding meetings, they have been thinking up ways to make them more bearable. Recent advances include technologies that integrate the processing of speech, natural language understanding, vision, and multimodal interaction to produce tools that perceive what happens at a meeting, extract salient events, and produce a reliable record. The ICSI Meeting Project sought to produce automated and segmented transcripts from natural, multi-party speech during meetings, while the ISL Smart Meeting Room Task and the M4 and AMI projects instrumented meeting rooms to collect data on behaviors so the interactions of meeting participants could be analyzed to produce flexible records of their activities, while providing a supportive environment for collaboration. Akin to the latter, the CALO Meeting Assistant collects data about the behaviors of people in meetings, assimilating speech, movement, and note-taking data to create a rich representation of a meeting that can be analyzed and reviewed at many levels. In addition, the CALO Meeting Assistant integrates its observations with those of a larger system of agents, which assesses the meeting data in the context of the ongoing projects and workflow for each of the meeting participants. Thus, our meeting assistant aims to reach beyond an intelligent room that understands only the activities of people in meetings, and attempts to understand their overarching concerns and interpret their behaviors from the perspective of what their meetings mean to them.