What Matters for Chatbots? Analyzing Quality Measures for Facebook Messenger's 100 Most Popular Chatbots

摘要

Chatbots are becoming mainstream. This work aims at ascertaining what are the enablers behind this popularity. To this end, we introduce four quality attributes, namely, 'support of a minimal set of commands', 'foresee language variations', 'human-assistance provision' and 'timeliness'. These criteria are applied to the 100 most popular Facebook Messenger chatbots. We review and measure both capacities and performance in order to find correlations between quality attribute fulfilment and popularity (chatbots' 'likes'). Results show no significance correlations between quality attributes and chatbot popularity. However, the experiment comes up with three main contributions. First, a detailed description of how to measure these four quality attributes. Second, insights about how this assessment can be automatized, paving the way towards chatbot-evaluation platforms. Third, a checklist of frequently committed interaction errors as found in the revised chatbots. This might help developers to double-check their development.