Hadoop + Spark Platform Based on Big Data System Design of Agricultural Product Price Analysis and Prediction by HoltWinters

摘要

In the market of agricultural products, the price of agricultural products is affected by production cost, market supply and other factors. In order to obtain the market information of agricultural products, the price fluctuation can be analyzed and predicted. A distributed big data software platform based on Hadoop, Hive and Spark is proposed to analyze and forecast agricultural price data. Firstly, Hadoop, Hive and Spark big data frameworks were built to store the data information of agricultural products crawled into MYSQL. Secondly, the information of agricultural products crawled from MYSQL was exported to a text file, uploaded to HDFS, and mapped to spark SQL database. The data was cleaned and improved by Holt-Winters (three times exponential smoothing method) model to predict the price of agricultural products in the future. The data cleaned by spark SQL was imported and predicted by improved Holt-Winters into MYSQL database. The technologies of pringMVC, Ajax and Echarts were used to visualize the data.