Building a scientific workflow framework to enable real-time machine learning and visualization

摘要

Nowadays, we have entered the era of big data. In the area of high performance computing,large-scale simulations can generate huge amounts of data with potentially critical information.However, these data are usually saved in intermediate files and are not instantly visible untiladvanced data analytics techniques are applied after reading all simulation data from persistentstorages (eg, local disks or a parallel file system). This approach puts users in a situation wherethey spend long time on waiting for running simulations while not knowing the status of the runningjob. In this paper, we build a new computational framework to couple scientific simulationswith multi-step machine learning processes and in-situ data visualizations.We also design a newscalable simulation-time clustering algorithm to automatically detect fluid flow anomalies. Thiscomputational framework is built upon different software components and provides plug-in dataanalysis and visualization functions over complex scientific workflows.With this advanced framework,users can monitor and get real-time notifications of special patterns or anomalies fromongoing extreme-scale turbulent flow simulations.