Hot Topic Detection Using Twitter Streaming Data

摘要

With the increasing popularity and widespread use of social networks, it is becoming increasingly beneficial to analyse the data being shared to identify topics of public interest and specific social phenomena. This paper analyses the possibilities, challenges and difficulties of automatic periodic collection and analysis of data from popular social networks and explains in detail the acquisition and analysis of data from Twitter. The paper explains an implementation of a simple hot topic detection algorithm based on texts acquired from the Twitter’s official API. The texts collected are being pre-processed by removing stop-words and stemming the remaining words using Porter’s stemming algorithm. Words from pre-processed text are assigned ranks depending on a large-scale analysis using TF-IDF weight and grouped into a hot topic. The algorithm accuracy was evaluated by comparison with Twitter’s official hot topic detection algorithm. Appropriate user interface enabling configuring the process of data acquisition, analysis and viewing results in a geographic fashion was implemented.