Evaluating Casama: Contextualized semantic maps for summarization of lung cancer studies

摘要

Abstract Objective It is crucial for clinicians to stay up to date on current literature in order to apply recent evidence to clinical decision making. Automatic summarization systems can help clinicians quickly view an aggregated summary of literature on a topic. Casama, a representation and summarization system based on “contextualized semantic maps,” captures the findings of biomedical studies as well as the contexts associated with patient population and study design. This paper presents a user-oriented evaluation of Casama in comparison to a context-free representation, SemRep. Materials and methods The effectiveness of the representation was evaluated by presenting users with manually annotated Casama and SemRep summaries of ten articles on driver mutations in cancer. Automatic annotations were evaluated on a collection of articles on EGFR mutation in lung cancer. Seven users completed a questionnaire rating the summarization quality for various topics and applications. Results Casama had higher median scores than SemRep for the majority of the topics ( p ≤ 0.00032), all of the applications ( p ≤ 0.00089), and in overall summarization quality ( p ≤ 1.5e-05). Casama's manual annotations outperformed Casama's automatic annotations ( p ?=?0.00061). Discussion Casama performed particularly well in the representation of strength of evidence, which was highly rated both quantitatively and qualitatively. Users noted that Casama's less granular, more targeted representation improved usability compared to SemRep. Conclusion This evaluation demonstrated the benefits of a contextualized representation for summarizing biomedical literature on cancer. Iteration on specific areas of Casama's representation, further development of its algorithms, and a clinically-oriented evaluation are warranted. Highlights ? Summarization system Casama outperformed SemRep in a user-oriented evaluation. ? Capturing strength of evidence and patient attributes improved summarization quality. ? Including less granular concept types improved usability and searchability.