Towards Automatic Linkage of Knowledge Worker’s Claims with Associated Evidence from Screenshots

摘要

Knowledge workers are frequently subject to information overload. As a result, when looking to make analytic judgements, they may only have time to search for evidence that already matches their existing viewpoint, leading to confirmation bias. New computer systems are needed that can help users overcome this and other cognitive biases. As an enabling step towards such systems, the research community has developed instrumentation software that captures data to help better understand sensemaking processes and workflows. However, existing instrumentation approaches are limited by the need to write operating system-specific (and often application-specific) code to `see' what the user is doing inside different applications on their computer. This source code quickly becomes complex and brittle. Furthermore, this approach does not provide a holistic view of how the user is gleaning information from multiple applications at once. We propose an alternative approach to instrumentation based on automated analysis of desktop screenshots, and demonstrate this in the context of extraction of `claims' from reports that users are writing, and association of these claims with `evidence' obtained from web browsing. We evaluate our approach on a corpus of 121,000 screenshots obtained from a study of 150 participants carrying out a controlled analysis task. The topic of the task was previously unfamiliar to them (hence the need to search for evidence on the web). We report results from several variants of our approach using a human evaluation of extracted claim/evidence pairs, and find that a simple word matching metric (based on Jaccard similarity) can outperform more complex sentence similarity metrics. We also describe many of the difficulties inherent to screenshot analysis and our approaches to overcome them.