People Publications Projects

Back to BDM Home

Sample commands to download IPCC AR4 CMIP-3 datasets

Support email : srmlbl.gov

Making the environment "faster"

Refer to the linux tuning: http://fasterdata.es.net/

Test commands before downloading from NERSC

NERSC has a specialized Data Transfer Nodes (DTN) for ESG dataset replication purpose at gsiftp://dtn02.nersc.gov:2821.
Access to the gridftp server and datasets should be explicit by request, and limited to the ESG collaboration and climate community.
1GB test file is located at gsiftp://dtn02.nersc.gov:2821//ipcc/1G.test.data
bdm-client or globus-url-copy can be used to test the network connection with this file path.

Sample BDM transfer plots from LLNL to NERSC, Aug 2010

Sample commands to download IPCC data from NERSC

NERSC data (/ipcc/cmip3) from LLNL is broken into following requests and stored in the database file.
To download the data, please [[/bdm/bdmDB-nersc.h2.db][download the database file]], store in your $BDM_HOME/bin location, ant use the following commands as it is, including the requesttoken.

Request Id	Sample command	Data from sub dirs.	Total Size in TeraBytes
dtn02--1287607846241	bdmclient -requesttoken dtn02--1287607846241 -transferserver gsiftp://dtn02.nersc.gov:2821 -concurrency 15 -td <localpath> -dbpath ./bdm-nersc.h2.db	data11 data1 data7	5.3
dtn02--1287608064263	bdmclient -requesttoken dtn02--1287608064263 -transferserver gsiftp://dtn02.nersc.gov:2821 -concurrency 15 -td <localpath> -dbpath ./bdm-nersc.h2.db	data16 data14	4.1
dtn02--1287608177888	bdmclient -requesttoken dtn02--1287608177888 -transferserver gsiftp://dtn02.nersc.gov:2821 -concurrency 15 -td <localpath> -dbpath ./bdm-nersc.h2.db	data2 data12 data8	5.4
dtn02--1287605147028	bdmclient -requesttoken dtn02--1287605147028 -transferserver gsiftp://dtn02.nersc.gov:2821 -concurrency 15 -td <localpath> -dbpath ./bdm-nersc.h2.db	data10 data18 data4	6.1
dtn02--1287608373841	bdmclient -requesttoken dtn02--1287608373841 -transferserver gsiftp://dtn02.nersc.gov:2821 -concurrency 15 -td <localpath> -dbpath ./bdm-nersc.h2.db	data13 commit	4.2
dtn02--1287608450984	bdmclient -requesttoken dtn02--1287608450984 -transferserver gsiftp://dtn02.nersc.gov:2821 -concurrency 15 -td <localpath> -dbpath ./bdm-nersc.h2.db	data5 data6 data15	5.0
dtn02--1287608615372	dmclient -requesttoken dtn02--1287608615372 -transferserver gsiftp://dtn02.nersc.gov:2821 -concurrency 15 -td <localpath> -dbpath ./bdm-nersc.h2.db	data17 1pctto2x	3.7
dtn02--1287608702625	bdmclient -requesttoken dtn02--1287608702625 -transferserver gsiftp://dtn02.nersc.gov:2821 -concurrency 15 -td <localpath> -dbpath ./bdm-nersc.h2.db	slabcntl	.3
dtn02--1287608764068	bdmclient -requesttoken dtn02--1287608764068 -transferserver gsiftp://dtn02.nersc.gov:2821 -concurrency 15 -td <localpath> -dbpath ./bdm-llnl.h2.db	data3 data9	3.6

Sample commands to download IPCC data from LLNL

IPCC data (/export/ftp/pub/dean/ipcc) from LLNL is broken into following requests and stored in the database file.
To download the data, please download the database file, store in your $BDM_HOME/bin location, ant use the following commands as it is, including the requesttoken.

Request Id	Sample command	Data from sub dirs.	Total Size in TeraBytes
dtn01--1285867142479	bdmclient -requesttoken dtn01--1285867142479 -transferserver gsiftp://gdo2.ucllnl.org,gsiftp://gdo126.ucllnl.org -concurrency 15 -td -dbpath ./bdm-llnl.h2.db	data11 data1 data7	5.3
dtn01--1285866979032	bdmclient -requesttoken dtn01--1285866979032 -transferserver gsiftp://gdo2.ucllnl.org,gsiftp://gdo126.ucllnl.org -concurrency 15 -td -dbpath ./bdm-llnl.h2.db	data16 data14	4.1
dtn01--1285866785956	bdmclient -requesttoken dtn01--1285866785956 -transferserver gsiftp://gdo2.ucllnl.org,gsiftp://gdo126.ucllnl.org -concurrency 15 -td -dbpath ./bdm-llnl.h2.db	data2 data12 data8	5.4
dtn01--1285866683991	bdmclient -requesttoken dtn01--1285866683991 -transferserver gsiftp://gdo2.ucllnl.org,gsiftp://gdo126.ucllnl.org -concurrency 15 -td -dbpath ./bdm-llnl.h2.db	data10 data18 data4	6.1
dtn01--1285867320945	bdmclient -requesttoken dtn01--1285867320945 -transferserver gsiftp://gdo2.ucllnl.org,gsiftp://gdo126.ucllnl.org -concurrency 15 -td -dbpath ./bdm-llnl.h2.db	data13 commit	4.2
dtn01--1285867427393	bdmclient -requesttoken dtn01--1285867427393 -transferserver gsiftp://gdo2.ucllnl.org,gsiftp://gdo126.ucllnl.org -concurrency 15 -td -dbpath ./bdm-llnl.h2.db	data5 data6 data15	5.0
dtn01--1285867550331	dmclient -requesttoken dtn01--1285867550331 -transferserver gsiftp://gdo2.ucllnl.org,gsiftp://gdo126.ucllnl.org -concurrency 15 -td -dbpath ./bdm-llnl.h2.db	data17 1pctto2x	3.7
dtn01--1285867609642	bdmclient -requesttoken dtn01--1285867609642 -transferserver gsiftp://gdo2.ucllnl.org,gsiftp://gdo126.ucllnl.org -concurrency 15 -td -dbpath ./bdm-llnl.h2.db	slabcntl	.3
dtn01--1285867658184	bdmclient -requesttoken dtn01--1285867658184 transferserver gsiftp://gdo2.ucllnl.org,gsiftp://gdo126.ucllnl.org -concurrency 15 -td -dbpath ./bdm-llnl.h2.db	data3 data9	3.6