Method for creating efficient application on heterogeneous big data processing platform

摘要

This invention relates to a method and system for creating Big Data applications that can be executed on heterogeneous clusters. The applications can be executed on a particular platform, such as SPARK or UIMA-AS, but the method and system are able to translate the input to these targeted platforms without the developer needing to tailor the application specifically to the platform. The method and system are based on the use of an execution dependency graph, a cluster configuration, and a data size to create a stages table. The stages table is then optimized to increase the overall efficiency of the heterogeneous cluster. The stages table is then translated into a platform specific Big Data application.