New Lustre features to improve Lustre metadata and small-file performance

摘要

As HPC I/O evolves beyond the challenges of large I/O performance, several new Lustre features - Distributed Namespace (DNE) 2, Progressive File Layout (PFL), and Data on Metadata (DoM), when combined with flash-based Lustre targets, can provide metadata and small-file performance improvements to applications transparently. To improve the execution of single-directory operations and increase the scalability of overall metadata operations, Striped or Remote Directories with DNE2 can be configured with multiple MDTs. For small-file I/O, Progressive File Layout with flash and disks can be used to optimize small- and large-I/O by seamlessly storing files on flash-based MDTs or specific flash OSTs. This paper will share performance results that demonstrate the scalability benefits of metadata performance using striped or remote directories as well as small-file performance with DoM and Flash OSTs with PFL, both with and without Cray's block I/O accelerator (NXD) configured.