SDM
People
Publications
Projects
PI: Arie Shoshani ([email protected])
Co-PI: Don Petravick ([email protected])
Executive Summary
September 2001
Terascale computing often generate petascale data, where petabytes of data need to be collected and managed. Such data intensive applications already overwhelm scientists who spend much of their time managing data, rather than concentrate on scientific investigations. Because such data are now vital to large scientific collaborations dispersed over wide-area networks, there is a growing activity of developing a grid infrastructure to support such applications. Initially, the grid infrastructure mainly emphasized the computational aspect of supporting large distributed computational tasks, and optimizing the use of the network by using bandwidth reservation techniques. In this proposal we complement this with middleware components to manage the storage resources on the grid. Just as compute resources and network resources need to be carefully scheduled and managed, the management of storage resources is critical for data intensive applications since the access to data can become the main bottleneck. This situation stems from the fact that data files may be dispersed on multiple disk caches and much of the data are stored on tertiary (tape) storage system, which are slow because of their mechanical nature. Data replication in multiple disk caches is also limited due to the volume of data. Finally, transfer of large data files over wide area networks also needs to be minimized. Based on previous experience with several prototype storage management systems, we propose to enhance the features of storage resource managers (SRMs), to develop common application interfaces to them, and to deploy them in real grid applications. We target both tape-based and disk-based systems, and design their interfaces so that they inter-operate.
The technology developed will provide the following benefits:
In the past, storage management issues were avoided by pre-allocation of space, a great deal of data replication, and pre-partitioning of the data. For example, high energy physics data was pre-selected into subsets (referred to as “micro-DSTs” or “streams”), and those were stored in pre-selected sites where the scientists interested in this subset perform the analysis. Because the data are stored according to the access patterns anticipated when the experiment was implemented, analysis requiring new access patterns is made more difficult, since the time to get data other than the subset at hand was prohibitively long. Because of the volume of data being collected and simulated, this mode of operation is not effective any longer. Scientists accessing the data can be in multiple remote institutions, and the amount of data is only expected to grow. To improve access to the data in this mode, data grids are now being developed, and storage resource management is a needed integral part of the middleware of data grids.
The purpose of this project is to address the problem of managing the access to large amounts of data distributed over the sites of the network. It is necessary to have Storage Resource Managers (SRMs) associated with each storage resource on the grid in order to achieve the coordinated and optimized usage of these resources. The term “storage resource” refers to any storage system that can be shared by multiple clients. The goal is to use shared resources on the grid to minimize data staging from tape systems, as well as to minimize the amount of data transferred over the network. The main advantages of developing SRMs as part of the grid architecture are:
The major challenges are: