A Text Preprocessing Framework for Text Mining on Big Data Infrastructure

摘要

There are many people using social media to comment appearance of a various event and post own activity in every day. The business organizations and industries can use social media to improve a produce by using the technique of text mining. However, a single machine cannot perform to compute a large amount of data because it has the resource limitation such as CPU, main memory, storage and so on. Moreover, a data also consists of structured and unstructured data that must be prepared before a computation with text mining or machine learning. Therefore, the text preprocessing becomes one of the most very important tasks because it includes many steps. In this paper, we design and develop an efficient text preprocessing framework on a big data infrastructure, which is proposed to support the text preprocessing task to reduce the computation time. The framework consists of the main four modules: Data collection module, storage module, data cleaning module and feature extraction module. We collected the sentiment data from Facebook page, named “Tasty” that has more than 90 million members for the performance evaluation. The data was separated into different three sizes and each dataset was tested on different the cluster system environments. Moreover, we also compared the computation time between the proposed our framework and the single machine. As the result, our framework can reduce the computation time significantly. Thus, this framework can apply to solve the problem of the complex text preprocessing in text mining areas.