Design of the web log analysis system based on Hadoop

摘要

As a result of rapid development, the Internet has become an indispensable tool in people's daily life, and web logs have been growing rapidly as well. How to deal with massive logs timely and extract information people need from the logs has become a problem; handling web log by single computer can not meet people's needs any more. Combining cloud computing and Hadoop technology, this paper established a new processing system for the collection of logs and remote parallelization analysis, which not only solved the issues of traditional systems that data handing and collection could not proceed simultaneously, but reduced the performance bottleneck of computing power and storage capacity, thereby saving much time and improving efficiency significantly.