Mathematical Models on the Hadoop Runtimes on Big Data

摘要

The problem of understanding runtime on big data processing has become key to solving the ever increasing volumes of data generated on machines. Nowadays big data is accessed through a searching system called Hadoop which uses the MapReduce algorithm. The effect of increasing machine clusters through which data is processed, the effect of machine failures on steady runtime, the effect of optimising runtime and machine cluster on the workflow process is analysed. The case in which the runtime and hours of data being processed differ is considered and the effect of the accumulation of data on runtime is analysed in detail. Mathematical models to analyse runtimes are proposed. The mathematical models proposed are borrowed from systems that process data in parallel processes. A simple runtime formula is adopted and numerical method is used to predict runtimes in the case where data is allowed to accumulate. Increasing the machine cluster reduce processing time. Increasing the overhead result in the increase in runtimes, A 15% machine failure result in the 261% increase on runtimes. The time to process one hour of data should be kept small. If one hour of data is processed in more than one hour the Hadoop system significantly slows down.