The Art of Creating an Informative Data Collection for Automated Deception Detection: A Corpus of Truths and Lies

摘要

One of the novel research directions in Natural LanguageProcessing and Machine Learning involves creating anddeveloping methods for automatic discernment of deceptivemessages from truthful ones. Mistaking intentionallydeceptive pieces of information for authentic ones (true tothe writer’s beliefs) can create negative consequences, sinceour everyday decision-making, actions, and mood are oftenimpacted by information we encounter. Such research isvital today as it aims to develop tools for the automatedrecognition of deceptive, disingenuous or fake information(the kind intended to create false beliefs or conclusions inthe reader’s mind). The ultimate goal is to supporttruthfulness ratings that signal the trustworthiness of theretrieved information, or alert information seekers topotential deception. To proceed with this agenda, werequire elicitation techniques for obtaining samples of bothdeceptive and truthful messages from study participants invarious subject areas. A data collection, or a corpus oftruths and lies, should meet certain basic criteria to allowfor meaningful analysis and comparison of socio-linguisticbehaviors. In this paper we propose solutions and weighpros and cons of various experimental set-ups in the art ofcorpus building. The outcomes of three experimentsdemonstrate certain limitations with using onlinecrowdsourcing for data collection of this type.Incorporating motivation in the task descriptions, and therole of visual context in creating deceptive narratives areother factors that should be addressed in future efforts tobuild a quality dataset.