Machine learning based Oozie Workflow for Hive Query Schedule mechanism

摘要

Apache Oozie is a scheduler framework to run and oversee Hadoop jobs in a distributed environment. It enables various complex jobs, which is to be kept running in a sequential order to accomplish a greater task. In the task, at least two jobs can run parallel to each other without any ambiguity. There are two methods through which Oozie detects Completion of tasks as they are call back and pooling. In Oozie the workflow is a sequence of actions and these actions are arranged in a Directed acyclic graph (DAG) control dependency. These workflow actions can be hive action, pig action, shell action, java action etc, and by using decision trees one can decide how and on which condition a job should run.