Berkeley Storage Manager (BeStMan)

What It Does

Manages underlying storage for files.
Supports multiple storage partitions.
Works on existing storages with posix-compatible file systems, and adaptable to special file systems and storages with customized plug-in.
Supports multiple transfer protocols and load balancing for multiple transfer servers
Optional Data Movement Broker mode.
Implements Storage Resource Management (SRM) interface v2.2, and compatible and interoperable with other 4 SRM implementations in WLCG.

Daily data transfer volume in OSG from 3/1/2015 to 4/15/2015. BeStMan is used for significant amount of transfer jobs in OSG.

BeStMan as Data Movement Broker manages long running storage-to-storage file replications, enabling recovery of transient failures. (318 file replications from BNL HPSS to NERSC HPSS for STAR experiment)

Accomplishments

RPM distribution through OSG software stack as well as a packaged download.
Highly customizable and pluggable for the storage needs.
Open source under BSD license.
Scalable performance with some file systems and storages, such as Xrootd and Hadoop.
Organized an international standard through OFG - GFD.129, 2008.
US Patent 8,705,342 B2, 2014. Co-scheduling of network resource provisioning and host-to-host bandwidth reservation on high-performance network and storage systems.

Impact

Enable scientific collaborations with uniform interface to the distributed storage resources.
Enable storage accessibility and improve science productivity.
43 BeStMan deployments worldwide and 5 backend deployments for CERN EOS system, as of 4/16/2015.
Being used in scientific collaborations such as ESGF, OSG, and WLCG.