A container resource configuration method in Hadoop Transcoding cluster based on requirements of a sample split

摘要

Media transcoding is an important job in the streaming media processing. Huge amounts of media transcoding, a kind of data and computationally intensive work, are performed as a distributed way. In this paper, we model a distributed media transcoding system which is based on Hadoop and FFmpeg. To minimize the total completion time (i.e., makespan) and fully utilize resources, we propose a method to configure the container resources in different clusters based on the requirements of a sample split. The proposed method decreases the makespan by approximately 25% compared to Hadoop FIFO scheduler and could be applied to both virtual clusters and physical-machine clusters.