Get out of the Way! Applying Compression to Internal Data Structures

摘要

As the amount of memory per core decreases in post-petascale machines, the memory footprint of any libraries and middleware used by HPC applications must be reduced. While scientific data can contain a great deal of entropy and require specialized compression techniques, the descriptions of scientific data layouts, as opposed to contents, turn out to be highly compressible. In this paper we present two approaches to compressing scientific data layout descriptions. We also describe two data structures for managing the compressed data. We incorporated our approach into the ROMIO MPI-IO implementation to reduce the memory consumption, observing an 89× reduction in memory overhead with a 25% increase in CPU overhead.