Current Status and Milestones

October 04, 2010

version 1.2.1 on https://codeforge.lbl.gov/projects/bdm/
added support for -checkpoint <# of files>, periodically update the summary information after #of files completed.

September 30, 2010

version 1.2 on https://codeforge.lbl.gov/projects/bdm/
added support for concurrent browsing for better performance..
added support for concurrent threads to check the target file existence.
tuned up backend database for better performance.
added support for single file browsing.
added support for browsing directory with non-recursive.
added support for transfering directory with non-recursive.
added support for transfering files with spaces in it.
added support for browsing, transfering files from root director with limitations (currently supports only absolute URL).
modified outputs to appear in a better way.
added support to check network performance with -perf.
added support for add/remove connections dynamically for adaptive work.

July 1, 2010

version 0.12 on https://codeforge.lbl.gov/projects/bdm/
added -usepassive option for limited usage
added more error exception handling

Jan. 21, 2010

version 0.9 on https://codeforge.lbl.gov/projects/bdm/
added all promised features
enhancements and bug fixes will be done as time goes on.

Dec. 7, 2009

version 0.6 on https://codeforge.lbl.gov/projects/bdm/
same version that we had during Super Computing 2009

Oct. 21, 2009

version 0.3 release: https://codeforge.lbl.gov/projects/bdm/
version 0.2 release: https://codeforge.lbl.gov/projects/bdm/
From NERSC dtn01/02, 1000MB/sec is seen on average.
From LLNL GDO1 to NERSC DTN01, we had 250MB/sec.

Oct. 20, 2009

From NERSC DTN02 to NERSC DTN01, we had 771MB/sec.
From LLNL GDO2 to NERSC DTN01, we had 214MB/sec.

Oct. 12, 2009

codeforge created for all downloadables: https://codeforge.lbl.gov/projects/bdm/
From LLNL GDO2 to NERSC DTN01, we had 174MB/sec in 50 connections. Transfer connection management gets stabilized.

Oct. 10, 2009

Downloadable package v 0.1: https://codeforge.lbl.gov/projects/bdm/

Oct. 6, 2009

From LLNL GDO2 to NERSC DTN01, we had 165MB/sec which could be the maximum backend-storage bandwidth at LLNL through GDO2. We had 10 concurrent connections.

Oct 5, 2009

Initial working version deployed at NERSC DTN nodes for testing

Oct. 2, 2009

Initial testing from LLNL GDO2 to LBNL data1

Oct. 1, 2009

Initial testing from NERSC DTN01 to LBNL data1

Current Assumptions and Requirements

Java 1.5 or later support
Transfer is based on !GridFTP, and data source has !GridFTP server setup and grid-security in place.
User has a valid grid credential and local grid proxy, and is mapped at the source host.

Currently Implemented Features

Checksum calculation for md5 (default), adler32, crc32
Java API for matching command line options: limitation on having only one active request
XML input based on the format
XML output for publishing
Recursive source directory browsing and target directory hierarchy creation
Non recursive source directory browsing
Concurrent source directory browsing using transferservers
Concurrent file transfers
Transfer queue management for efficient concurrent transfers
measure performance with -perftest option.
browse root dir with recursive and non recursive option
transfer files from root dir with recursive and non recursive option
!GridFTP connection re-use and data channel caching
Test option: testing BDM capability and immediately remove the downloaded target files so that target disk space would not be used
Retry option for failed transfers: immediately after the transfer failure and after all files are tried for transfers at least once
New connection option: instead of re-using transfer connections, open new transfer connection for each file transfer.
Logging: INFO and DEBUG level logging, Format matches the recommendation from CEDPS troubleshooting team.
Request Token generation
Asynchronous request status checking with the request token
An option to overwrite already existing target files. Default is no-overwriting if destination path and file size match to the source file information.
An option for a request browsing source directory only without file transfers, and saving the source directory information for later use.
An option to resume file transfer from the previous request for browsing.
An option to resume file transfer to another target directory from the previous request for browsing.
Multiple transfer server support
In case of an interrupted network connections, re-open the failed/closed connections during the file transfers, and re-tries/resume the failed file transfers in the re-opened connection
Supported transfer protocol: !GridFTP

Plans

Enhancements on features for ESG
Documentation
Bug fixes

BDM Testing with ESG Framework

BDM has been tested with DKRZ successfully on Feb 09, 11. The following were identified during the testing:

DKRZ directory structure is found to be with several soft links. So to enable softlink transfer, the "-symlink" option needs to be used in the command for BDM.
Java API based BDM and java based globus-url-copy does not work with cmip-bdm1.badc node, but C version of globus-url-copy does. David suspects that it is because he has gsiftp server certificate and not a host certificate. David has applied for a host certificate, and we are waiting for that to arrive. Once it is ready, we will again test BDM against BADC.
A signing certificate (367b75c3) is missing in the PCMDI myproxy truststore and esg trusted certificates. DKRZ gsiftp connection did not complain about that certificate, but BADC gsiftp connection required that, which resulted in connection error.
Another signing certificate (1149214e) is missing in the PCMDI myproxy truststore and esg trusted certificates. DKRZ gsiftp connection required that. We still could not verify the same, with the BADC connection because of host certificate problem.
PCMDI myproxy trust store returns 3 signing policy files that has format error, and either signing policy files have to modified manually or re-deployed from the certificate tar file.
706 [Thread-11] WARN org.globus.gsi.TrustedCertificates - Signing policy ./certs/159117b6.signing_policy failed to load. Line format is incorrect, atleast three tokens are expected. Invalid line: "information"
714 [Thread-11] WARN org.globus.gsi.TrustedCertificates - Signing policy ./certs/ae9c66bf.signing_policy failed to load. Line format is incorrect, atleast three tokens are expected. Invalid line: "information"
743 [Thread-11] WARN org.globus.gsi.TrustedCertificates - Signing policy ./certs/0084963c.signing_policy failed to load. Line format is incorrect, atleast three tokens are expected. Invalid line: "information"