基于Mahout框架的Hadoop平台作业日志分析平台设计与实现

摘要

With the wild use of Hadoop and appearance of Hadoop Yarn,the scale of clusters is getting larger.And the open source implementation of the clusters operating state monitor system in the Hadoop ecosystem has been very ma-ture,but there is not yet a platform for statistical analysis of the running trend of the Hadoop Jobs.In this paper,a job resource statistical analysis platform Yarn Hadoop is presented,which is for the Cluster Administrator and the ordinary users,with the double dimension of date and user to analyze the job,and get the standard of Hadoop operation.%随着Hadoop的流行与Hadoop Yarn的出现,集群的规模越来越大.在Hadoop生态圈中对集群运行状态的开源实现已经很成熟,但是尚未有对一个对 Hadoop 作业的运行趋势进行统计分析的平台.本文介绍了一个面向Hadoop Yarn的作业资源统计分析平台,面向集群管理员与普通用户,以时间、用户双维度对作业进行统计分析,得出一个Hadoop作业运行的标准.