People Publications Projects

Back to BDM Home

BDM Implementation Details

The architectural diagram (see below/attachments) has 12 modules that fall into three categories: initialization, execution, and interactive functions. The functionality of each of these modules is described below.



Interactive functions

Timeline for Implementation Tasks

The implementation timeline has three phases. Each of the 12 modules described above describe tasks below according to these phases.

Phase I is designed to provide all the necessary basic functionality to perform bulk data movement, and set up several testbeds. Phase II will provide some enhanced capabilities and perform extensive scaling tests. Phase III will add all remaining advanced functionality. It is the intention of this implementation plan that at the end of phase I all the basic technical issues (such as communication with gateways and nodes, and well as performing concurrent transfers using GriFTP) will be resolved and implemented. Scalability testing and enhancements will follow.


The tasks described below and phases of their implementation are summarized in a table at the end of this section.

  1. Initialization
    1. Request submission
      • Phase 1: Support source URLs based on gsiftp servers
      • Phase 3: Support source URLs based on http and scp/sftp servers
    2. Verify source URLs and sizes
      • Phase 1: covers all
    3. Verify destination URLs and storage at Target
      • Phase 1: covers all
    4. Generate plan using statistics
      • Phase 1: Use assumptions on network performance, estimated bandwidth provided as parameters.
      • Phase 2: Use saved history of the past request performance
      • Phase 3: Use metrics service if available
    5. Initial request estimation
      • Phase 1: Simple estimation based on assumptions or given parameters on network performance
      • Phase 2: Estimation based on the saved history of the past request performance
      • Phase 3: Estimation based on the performance history from the metrics service in addition to the saved history of the past request performance
    6. Replicate directory structure at the destination
      • Phase 1: covers all
  2. Execution
    1. Multi-file request coordination
      • Phase 1: covers all
    2. File transfer client
      • Phase 1: Use gsiftp client api from globus 4.2.1
      • Phase 3: Support gsiftp, http, https and scp/sftp
    3. Monitor and generate statistics
      • Phase 1: covers all
    4. Logging
      • Phase 1: covers all
    5. Recovery and restart
      • Phase 1: Transient transfer failure recovery and restart
      • Phase 3: Recovery and restart from non-transient failures such machine or program crash
  3. Interactive functions
    1. On-demand status
      • Phase 1: covers all
    2. Suspend and resume
      • Phase 3: Support program suspend and resume
    3. Dynamic progress estimation
      • Phase 2: Use dynamic average transfer rate generated by monitoring
  4. Post transfer tasks
  5. Testbed setup
  6. Note: all scalability testing will depend on the hardware capabilities and network capacity between of sites involved.

    1. Initial sites in US
      • Phase 1:
      • Test setup on first candidate sites; LLNL, NCAR, ORNL and LBNL
      • Network performance measurement between sites and follow up with network people when necessary.
      • BDM client testing environment setup
    2. Additional sites in Europe
      • Phase 2:
      • Test for scalability of BDM client
      • Include other candidate sites
      • Continuous network performance measurement between sites and follow up with network people when necessary.
    3. Additional sites in other locations
      • Phase 3:
      • add sites in US and Europe
      • Add sites in Asia