Investigators: Suren Byna, Quincey Koziol, Bin Dong, John Wu and Prabhat
Program Manager: Lucy Nowell
Efficient data movement in HPC storage with multiple levels of hierarchical and heterogeneous hardware is challenging, and software solutions to facilitate data movement in a deep storage hierarchy are still in infancy and are inefficient.
We developed Data Elevator to support low-contention data movement in hierarchical storage systems by offloading data movements to idle CPU cores.
Data Elevator supports efficient, transparent, and asynchronous data movement using HDF5 VOL without the need to modify application source code.
"Data Elevator abstraction is the right approach for optimizing our I/O on novel storage architectures, like Cori's burst buffers. It involved no code changes to our MPI + HDF5 approach on other platforms, and provides great performance without hurting our simulation throughput. We're looking forward to using this capability to improve our data pre-processing and indexing as well"
- Hans Johansen (Developer of CAMR - 1km high-res climate modeling code)