Today's Large-Scale Science depends on Network Data Movement based on Twenty Five Years of Technology Development

摘要

Today's large-scale science projects all involve world-wide collaborations that routinely move 10s of petabytes per year between international sites. The capabilities required to support this scale of data movement have involved hardware and software developments at all levels: Wire/fiber signal transport, layer 2 transport (e.g. Ethernet), data transport (TCP is still the norm), operating system evolution, data movement and management techniques and software, and in the science applications. With each generation of network transport technology (155 Mb/s was the norm for high speed networks in 1995, 100 Gb/s - 700 times greater - is the norm today) R&D groups, involving hardware engineers, computer scientists, and application specialists, worked to first demonstrate in a research environment that "filling the network pipe" was possible, as well as having the ability to carry out the development necessary for applications to make use of the new capabilities. This paper surveys the progress made over the past several decades toward building an infrastructure that enables large scale "data-intensive" science.