Streaming File Transfer Optimization for Distributed Science Workflows

摘要

Driven by the advancements in computing and sensing technology, scientific applications started to generate a huge volume of data which needs to be streamed to highperformance computing clusters timely for real-time (or near-real time) processing, necessitating reliable network performance to operate seamlessly. However, existing data transfer applications are predominantly designed for batch workloads in a way that transfer configurations cannot be altered once they are set. This, in turn, severely limits streaming applications from adapting to changing dataset and network conditions therefore meeting stringent performance requirements. In this paper, we propose FStream to offer performance guarantees to time-sensitive streaming applications by dynamically adjusting transfer settings when system conditions deviate from initial assumptions to sustain high network performance throughout the runtime. We evaluate the performance of FStream by transferring several synthetic and real-world workloads in high-performance production networks and show that it offers up to 9x performance improvement over state-of-the-art data transfer solutions.