Application of Jupyter Notebook interfaces and iLauncher to deep learning workflows on HPC systems

摘要

We present information on what users need in order to support complex deep learning workflows on DoD HPC systems with web interfaces using ipywidgets in Jupyter Notebooks and the Interface Launcher (iLauncher), a tool that automates the submission of HPC jobs and provides a mechanism for rapidly prototyping web interfaces from the user's desktop to powerful capabilities running on the HPC nodes. We detail a representative use case for a PyTorch deep learning workflow and show how to include the underlying software along with all dependencies in an all-inclusive software packaging technology called a Singularity container. We then show how to use ipywidgets in a Jupyter Notebook and convert it to a full-fledged web interface using the Voila server. Finally, we outline how to create the iLauncher plugin to run the web interface on DoD HPC system nodes to provide a complete user interface workflow solution that does not require special privileges to create, deploy, or use. Using Jupyter Notebooks, ipywidgets, iLauncher, and Singularity containers together provide an explosion of accessible capabilities that were previously inconceivable in the restrictive DoD environment.