Orchestrating big data analysis workflows in the cloud: research challenges, survey, and future directions

摘要

When processing different big data workflows, many new and (so far) unknown patterns and performance requirements are visible. We are forced to search new processing models and management techniques that can support the design of different aspects of big data workflows: infrastructure (hardware), platforms (software), and efficient methods for scheduling and deployment workflows. These big, serious, scientific, technological, organizational, and technical problems lead to at least three important challenges (research questions) that are expanded and developed in this paper: (1)A description of "the different models and fundamental requirements of big data workflow applications"; (2)The new challenges related to the cloud and edge data centers, and this type of workflow application; and (3)The known "approaches, techniques, tools, and technologies" for developing "a new big data orchestration system."