Back to StorNet Home

Introduction

DOE funded data intensive applications, such as astrophysics, earth science, high energy and nuclear physics (HENP), and nanoscale material science, will generate exabytes of data in the next five years. This expectation of explosive growth in stored data and globally distributed data processing underpinned by the maturing Grid and cloud computing has generated critical requirements for new data transfer technologies and automated tools. These data transfer tools need to intelligently assist scientists in replicating a great volume of data to any place where the scientists want and within the time frame when they want to expedite science discovery. Existing data transfer techniques face unprecedented challenges in handling not only this sheer volume of data, but also the heterogeneous environment where data are imported to and exported from. Furthermore, technology advancements give rise to performance improvement along with the complexity of resource management and provision. Networks, run by DOE ESnet and Internet 2, are enhanced with advanced dynamic circuit switch technologies along with the best effort packet switch and advanced network resource reservation systems to ensure on-demand bandwidth and quality of service. Data storage technologies have significant improvement as well, by using advanced parallel file systems to enhance I/O bandwidth, and solid state disks (SSD) that can provide as much as ten times faster read/write access than hard drives. BNL and LBNL jointly propose to address the end-to-end resource provisioning and management for automated data transfer by seamlessly integrating network resource provisioning to the existing storage resource management (SRM) product.