SDM
 People Publications Projects

Back to BDM Home

BDM Frequently Asked Questions

Introduction

[[WebHome][BDM]] is a scalable data transfer management tool. The goal is to manage as much as 1+ PB with millions of files transfers reliably. Refer to NCAR page for more project description: https://wiki.ucar.edu/display/esgcet/Bulk+Data+Movement.

Client is behind a firewall, what needs to be done extra ?

Please try the following:
The gridftp port range should be properly set.

  1. Contact your network administrator to open up a range of ports (for GridFTP data channel connections) for the incomming connections. If the firewall blocks the outgoing connections, open up a range of ports for outgoing connections as well.
  2. Set the environment variable GLOBUS_TCP_PORT_RANGE
    3. edit ${BDM_HOME}/tester/bin/bdmclient

    add the following line in the script

    GLOBUS_TCP_PORT_RANGE=min,max

    where min,max specify the port range that you have opened for the incoming connections on the firewall. This restricts the listening ports of the GridFTP client to this range. Recommended range is 1000 (e.g., 50000-51000) but it really depends on how much use you expect.

  3. If you have a firewall blocking the outgoing connections and you have opened a range of ports, set the environment variable
    4. GLOBUS_TCP_SOURCE_RANGE: 

${BDM_HOME}/tester/bin/bdmclient

    add the following line in the script

    GLOBUS_TCP_SOURCE_RANGE=min,max

Run BDM with passive mode connection

If the site is strictly behind the firewall and gridftp port range cannot be set. Run BDM with passive mode connection, with -usepassive flag turned on. however there is a limitation, parallelism cannot be used and BDM opens new port connections for each file transfers.

Please don't set any gridftp port range.

If you are not sure, whether your machine is inside a firewall, please don't set any gridftp port range.

Does BDM provide API's to be called from another program?

Yes. Please see the sample programs, sample command and other details here.

Can we transfer multiple source dirs/multiple files with BDM?

Yes. BDM supports -f for transfering multiple source dirs/multiple files. Please see the format for the inputfiles and other details here.

Can we resume the transfer later for the same request?

Yes. BDM supports -resume option. When a transfer has to discontinue for some reason, the same request can be resumed later with -resume option.

How to check the status of a request, when a transfer is going on?

Use ${BDM_HOME}/tester/bin/bdmstatus script, it will show the current status of the latest request.

Does BDM only works with gsiftp protocol?

Yes. currently it only supports gsiftp protocol.

Can multiple transfer servers be used?

Yes. -transferserver where host1 and host2 have the same file structure, in this example, bdm transfers approx. 50% files from host1 and approx. 50% files from host2.

Can -usercert and -userkey pair be used instead of proxy?

Yes. -usercert -userkey can be used instead of proxy for BDM transfers.

How to report to the srm list in case if there is a fix needed?

Please locate the event log file, by default it write the event log file in ${BDM_HOME}/tester/bin location. Please also locate the server_debug.log in the ${BDM_HOME}/tester/bin location. Please provide the exact command you have used and the client console output, along with the attachments of the above file.

How to compute the checksum for the files in the request?

${BDM_HOME}/tester/bin/bdmclient -checksum will compare the checksum for the files if it is provided in the input file, if the checksum is not provided it calculates the checksum and writes in the database.

How to get the publish report for the request?

${BDM_HOME}/tester/bin/bdmstatus -publish ./test.xml will only generate report for the completed files.

${BDM_HOME}/tester/bin/bdmstatus -erroredoutput ./test.xml will only generate report for the error/failed files.

${BDM_HOME}/tester/bin/bdmstatus -alldetailedoutput ./test.xml will generate report for the all files (success and failed files).

Can any other relational db be used at the backend for BDM?

Yes. All relational db's which supports JDBC api's.

Do we need to download any other software in addition to BDM?

No. BDM comes with all necessary libs.

BDM hangs during callbacks. What do I do?

If you are running on a machine with DHCP enabled, the local IP address of the machine might be incorrectly detected by Java. Most commonly the IP address detected might be the local loop-back address - 127.0.0.1.

You can fix it by configure bdm with --with-globus-hostname option

example: --with-globus-hostname=64.34.58.128
or --with-globus-hostname=myhost.mydomain.com

What version of JDK does BDM work with?

Currently, BDM is tested with Sun/Oracle version of JDK. BDM is not tested with IBM version of JDK and OpenJDK.