SRM-COPY

BeStMan Copyright (c) 2007,2008,2009, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy).� All rights reserved.

If you have questions about your rights to use or distribute this software, please contact Berkeley Lab's Technology Transfer Department at [email protected] and [email protected].

NOTICE.� This software was developed under partial funding from the U.S. Department of Energy.� As such, the U.S. Government has been granted for itself and others acting on its behalf a paid-up, nonexclusive, irrevocable, worldwide license in the Software to reproduce, prepare derivative works, and perform publicly and display publicly.� Beginning five (5) years after the date permission to assert copyright is obtained from the U.S. Department of Energy, and subject to any subsequent five (5) year renewals, the U.S. Government is granted for itself and others acting on its behalf a paid-up, nonexclusive, irrevocable, worldwide license in the Software to reproduce, prepare derivative works, distribute copies to the public, perform publicly and display publicly, and to permit others to do so.

For the end user license agreement file for BeStMan for non-commercial research use, go to http://datagrid.lbl.gov/bestman/license-nc.html.

For the end user license agreement file for BeStMan for commercial research use, go to http://datagrid.lbl.gov/bestman/license-c.html.

1 Introduction

SRM client tools are full implementations of SRM v2.2 as generic SRM v2.2 clients, developed by Lawrence Berkeley National Laboratory. They have been tested for all current SRM v2.2 implementations such as BeStMan, CASTOR, dCache, DPM, SRM/iRODS-SRB and StoRM. They are continuously being tested for compatibility and interoperability.

It�s a java implementation that would run on any OS platforms, and currently supports gsiftp, http, https and ftp as its file transfer protocols.

It requires the minimal administrative efforts on the deployment and maintenance.

SRM v2.2 specification can be found on https://sdm.lbl.gov/srm-wg/doc/SRM.v2.2.html.
SRM client downloads and instructions can be found on http://datagrid.lbl.gov/bestman.

2 Requirements

� SUN Java 1.5.0_12 or IBM Java 1.5 or higher versions such as 1.6.0_07.

� Valid user grid proxy

3 Installation

� Refer to SRM Client User�s Guide.

4 Configuration, configuration options, configuration file and notes

� Refer to SRM Client User�s Guide.

5 SRM-COPY

srm-copy requests to copy files to and from SRM, between SRMs, between SRM and other storage repository. Operations include GET, PUT, BringOnline, and COPY.

5.1 Usage

� srm-copy <source_url> <target_url> �[command line options]

� srm-copy ��f <input_file> �[command line options]

5.2 Options

� Command line options take priority from the options from conf file.

� Options in the following table are in alphabetical order

-3partycopy	� Issues 3^rd party copy between source SRM and target SRM. � Client will request �get� to the source SRM, and retrieve the TURL from the source SRM. Client then requests �put� to the target SRM, and retrieves the TURL for the target SRM. Then client makes the 3^rd party gsiftp file transfer from source TURL to target TURL.
-accesslatency <online \| nearline>	� Specifies access latency � Default=online
-adjustturl	� Sometimes SRM-returned TURLs need few or more slashes to be a valid URL. When this option is on, it tries to adjust the number of slashes to be a valid URL. � Default=false
-authid <string>	� Authorization� ID to be used in SRM for the request
-bringonline	� Sets the request operation for srmBringOnline
-buffersize <int>	� Specifies gridftp buffer size in bytes � Default=1MB
-checkdiskspace	� Checks the availability of the local disk space when retrieving files. � Default=false
-concurrency <int>	� Specifies concurrent file transfers. � Default=1
-conf <path>	� Path to the configuration file. � Command line options will override the options from conf file
-connectiontimeout <int>	� Specifies time in seconds to wait for soap or gsi connection before timing out. � Default=1800
-copy	� Makes srmCopy requests explicitly with provided source url and target url
-dcau <true\|false>	� Specifies DCAU for gsiftp file transfers � Default=true
-debug	� Specifies debugging output � Default=false
-delegation <true\|false>	� Force proxy delegation. � When not provided, srm-copy makes no delegation by default, and automatically handles proxy delegation based on the source and target URLs. � When �delegation is provided, it overrides the automatic handling and forces the user choice.
-direct	� This option enables direct gsiftp file transfer from the gsiftp source or to the gsiftp target (default=false) � srm-copy does not go through SRM server.
-f <path>	� Path to the xml input file containing the source url, target url information for requests with more than one file � Refer to the format and an example in 6.3
-filelifetime <int>	� File (SURL) lifetime in seconds
-filestoragetype <p \| d \| v>	� Indicates filestorage type � p - Permanent, d - Durable, v - Volatile
-gucpath <path> -gurlcopy <path>	� Specifies path to the globus-url-copy (or its script)
-help	� Show the help message
-log <path>	� Specifies path to log file � Default=./srmclient-event-date-random.log
-maxfilesperrequest <int>	� maximum file requests per request. � It divides the files in multiple requests. � Default is to include all files in one request
-nodownload	� Not to download the requested files. � Files have to be retrieved with a direct file transfers. (e.g. globus-url-copy) � Default=false
-nooverwrite	� Enables no overwriting to the target. � Default=false to overwrite always
-noreleasefile	� Disable releasing file after the get request and file transfer � Default=release file
-numlevels <int>	� Specifies number of directory levels in recursive request. � Default=0 requesting all levels
-numretry <int>	� Specifies number of file transfer retries before giving up. � Default=3
-parallelism <int>	� Specifies gridftp parallel streams. � Default=1 � When parallelism is 1, passive file transfer is on by default.
-pinlifetime <int>	� Pin lifetime in seconds
-protocolslist <string>	� List of preferred file transfer protocol list separated by comma. � By default, gsiftp is used. � e.g. gsiftp,ftp,http,dcap
-proxyfile <path>	� Path to user grid proxy
-pushmode	� Issues push mode when srmCopy is done � Default=pull mode
-quiet	� Suppress output in the console. � This option writes the output to the log file.
-recursive	� Allows recursive GET from source url and/or PUT to target url
-releasespace	� Releases the space upon completion of the request � If -releasespace used in pair with -reservespace, then it releases the newly created space. � If used in pair with -spacetoken, then it releases the space associated with the spacetoken
-remotetransferinfo <string>	� Specifies extra file transfer parameters in srmCopy. � Formatted input separated by comma with the following keywords: buffersize:<int>,dcau:<true\|false>,parallelism:<int>, guc:<true\|false>,protection<true\|falase>
-renewproxy	� Enables automatic proxy renewal upon expiration � Works only when -usercert and -userkey are provided � Prompts for the pass phrase � Default=false
-report <path>	� Specifies path to full report output.
-reservespace	� Reserves a new space before the request, and use the space token with the request
-retentionpolicy <replica \| output \| custodial >	� Specifies retention quality � Default=replica
-retrydelay <int>	� Specifies number of seconds to sleep between file transfer retries. � Default=60
-s <source_url>	� Source URL
-serviceurl <ws_endpoint>	� Full web service endpoint � Required when source url or target url does not contain web service endpoint information
-spacelifetime <int>	� Specifies desired space lifetime in seconds for -reservespace
-spacegsize <int>	� Specifies desired guaranteed space size in bytes for -reservespace
-spacesize <int>	� Specifies desired total space size in bytes for -reservespace
-spacetoken <string>	� Space token to be used with the request
-statusmaxtime <int>	� Max time before status checks get timed out, in seconds � Default=600
-statuswaittime <int>	� Wait time delay between status check, in seconds � Default=30
-storageinfo	� Extra storage access information � Default=false � For BeStMan supporting MSS, a formatted input separated by comma is used with following keywords when necessary: for:<source\|target\|sourcetarget>,login:<string>,passwd:<string>, projectid:<string>,readpasswd:<string>,writepasswd<string>
-submitonly	� Exits after the request submission � Status must be checked separately. (e.g. srm-copy-status) � Default=false
-t <target_url>	� Target URL
-td <target_url>	� Target URL as directory path � Required for multiple �get� file requests if input file does not contain the target information. � When both target information in the input file and �td option are provided, command line option �td will take priority and information in the input file will be ignored.
-totalrequesttime <int>	� Total request time � TotalRequestTime means: All file transfers for the request must be completed within the totalRequestTime.� Otherwise, SRM will return SRM_REQUEST_TIMED_OUT as the request status code with individual file status of SRM_FAILURE with an appropriate explanation.
-usercert <path>	� Path to user grid certificate
-userdesc <string>	� Specifies user request description
-userkey <path>	� Path to user grid certificate key
-xmlreport <path>	� Specifies path to short output report in XML format.

5.3 Notes

� srm-copy can handle 3 slashes in the URL format.
e.g. file:///mydir/mypath

� srm-copy makes no proxy delegation by default, and automatically handles proxy delegation based on the source and target urls. A user can override the automatic handling by providing an option �-delegation� (or �-delegation true�) to force the delegation, and �-delegation false� to force no delegation. �-debug� option would show how the delegation is done on the output.

5.4 Input file format

<?xml version="1.0" encoding="UTF-8"?>

� <file>

�� <sourceurl>Source_URL</sourceurl>

�� <targeturl>Target_URL</targeturl>

� </file>

</request>

� Example 1 : PUT operation

<?xml version="1.0" encoding="UTF-8"?>

� <file>

�� <sourceurl>file:////tmp/my.source.data</sourceurl>

�� <targeturl>srm://bestman.lbl.gov:8443/srm/v2/server?SFN=/mydir/mypath/myfile</targeturl>

� </file>

</request>

� Example 2 : COPY operation from gsiftp to SRM

<?xml version="1.0" encoding="UTF-8"?>

� <file>

�� <sourceurl>gsiftp://gridftphost.lbl.gov//tmp/my.source.data</sourceurl>

�� <targeturl>srm://bestman.lbl.gov:8443/srm/v2/server?SFN=/mydir/mypath/myfile</targeturl>

� </file>

</request>

� Example 3 : COPY operation from SRM to SRM

<?xml version="1.0" encoding="UTF-8"?>

� <file>

�� <sourceurl>srm://bestman.lbl.gov:8443/srm/v2/server?SFN=/mydir/my.source.file</sourceurl>

�� <targeturl>srm://bestman2.lbl.gov:8443/srm/v2/server?SFN=/mydir/my.target.file</targeturl>

� </file>

</request>

� Example 4 : GET operation from SRM to local target directory (specified by command line option)

<?xml version="1.0" encoding="UTF-8"?>

� <file>

�� <sourceurl>srm://bestman.lbl.gov:8443/srm/v2/server?SFN=/mydir/my.source.file</sourceurl>

� </file>

</request>

5.5 Examples

5.5.1 Put a file from local disk to remote SRM cache

1. srm-copy� file:////tmp/source_filepath \
�� srm://host:port/wsept\?SFN=/target_filepath
This command puts the file into the SRM. Click here for the sample output.

1.1. srm-copy �file:////tmp/test.data \
�� "srm://bestman.lbl.gov:8443/srm/v2/server?SFN=/srmcache/guest/test.data"

2. srm-copy� file:////tmp/source_filepath \

�� srm:/ /host:port /wsept\?SFN=/target_filepath -reservespace

This command reserves a space at the beginning, and puts a file in the SRM cache using the dynamically reserved space, when dynamic space reservation is supported by the SRM server. Click here for the sample output.

2.1. srm-copy file:////tmp/test.data \
�� srm://bestman.lbl.gov:8443/srm/v2/server\?SFN=/srmcache/guest/test.data \
�� -reservespace

3. srm-copy� file:////tmp/source_filepath \

�� srm:/ /host:port /wsept\?SFN=/target_filepath \

�� -spacetoken <spaceToken>

This command puts a file in the SRM using the previously reserved space that is associated with the space token. Space token is given by the SRM. Click here for the sample output.

3.1. srm-copy file:////tmp/test.data \
�� srm://bestman.lbl.gov:8443/srm/v2/server\?SFN=/srmcache/guest/test.data \
�� -spacetoken V123
��

5.5.2 Get a file from remote site to local disk

1. srm-copy� srm://host:port/wsept\?SFN=/source_filepath \

�� file:////tmp/target_filepath

This command gets a file from SRM. Click here for the sample output.

1.1. srm-copy� srm://bestman.lbl.gov:8443/srm/v2/server\?SFN=/srmcache/guest/test.data� \
�� file:////tmp/ test.data.2

2. srm-copy� srm://host:port/wsept\?SFN=/source_filepath \

�� file:////tmp/target_filepath \

�� -spacetoken <spaceToken>

This command gets a file from SRM using the space already reserved space. Click here for the sample output.

2.1. srm-copy� srm://bestman.lbl.gov:8443/srm/v2/server\?SFN=/srmcache/guest/test.data� \
�� file:////tmp/ test.data.2 \
�� -spacetoken V123

3. srm-copy�� gsiftp://host:port//source_filepath \

�� file:////tmp/target_filepath \

�� -serviceurl srm://host:port/wsept

This command gets a file from gsiftp site via SRM. Click here for the sample output.

3.1. srm-copy �gsiftp://gridftphost.lbl.gov//tmp/test.data \
�� file:////tmp/tmp/test.data.3 \
�� -serviceurl srm://srm.lbl.gov:8443/srm/v2/server

4. srm-copy�� gsiftp://host:port//source_filepath \

�� file:////tmp/target_filepath \

�� -direct

This command gets a file from gsiftp site directly to local disk. Click here for the sample output.

4.1. srm-copy gsiftp://gridftphost.lbl.gov//tmp/test.data \
�� file:////tmp/tmp/test.data.3 \
�� -direct

5.5.3 BringOnline a file on an SRM

1. srm-copy� srm://host:port/wsept\?SFN=/source_filepath \
�� -bringonline

This command have a file available online on the SRM. Click here for the sample output.

1.1. srm-copy� srm://bestman.lbl.gov:8443/srm/v2/server\?SFN=/srmcache/guest/test.data� \
�� -bringonline

5.5.4 Copy a file from one SRM to another SRM

1. srm-copy� srm://host:port/wsept\?SFN=/source_copy_filepath \

�� srm://host:port/wsept\?SFN=/target_copy_filepath

This command copies a file from source SRM to target SRM. Click here for the sample output.

2. srm-copy� srm://host:port/wsept\?SFN=/source_copy_filepath \

�� srm://host:port/wsept\?SFN=/target_copy_filepath \

�� -spacetoken <spaceToken>

This command copies a file from source SRM to target SRM with the space already reserved in the target SRM. Click here for the sample output.

3. srm-copy� gsiftp://host:port/source_path \

�� srm://host:port/wsept\?SFN=/target_copy_filepath

This command copies a file from source gsiftp to target SRM. Click here for the sample output.

4. srm-copy� srm://host:port/wsept\?SFN=/source_copy_filepath \

�� srm://host:port/wsept\?SFN=/target_copy_filepath \

�� -pushmode

This command copies a file from source SRM to target SRM, having the source SRM push the file to the target SRM. Click here for the sample output.

5. srm-copy� srm://host:port/wsept\?SFN=/source_copy_filepath \

�� srm://host:port/wsept\?SFN=/target_copy_filepath \

�� -3partycopy

This command copies a file from source SRM to target SRM by the 3^rd party gsiftp into the target when supported. Click here for the sample output.

6 Exit codes

When SRM client commands exit, they have exit codes as following:

0	SRM_SUCCESS
51	SRM_FAILURE
52	SRM_AUTHENTICATION_FAILURE
53	SRM_AUTHORIZATION_FAILURE
54	SRM_INVALID_REQUEST
55	SRM_INVALID_PATH
56	SRM_FILE_LIFETIME_EXPIRED
57	SRM_SPACE_LIFETIME_EXPIRED
58	SRM_EXCEED_ALLOCATION
59	SRM_NO_USER_SPACE
60	SRM_NO_FREE_SPACE
61	SRM_DUPLICATION_ERROR
62	SRM_NON_EMPTY_DIRECTORY
63	SRM_TOO_MANY_RESULTS
64	SRM_INTERNAL_ERROR
65	SRM_FATAL_INTERNAL_ERROR
66	SRM_NOT_SUPPORTED
67	SRM_REQUEST_QUEUED
68	SRM_REQUEST_INPROGRESS
69	SRM_REQUEST_SUSPENDED
70	SRM_ABORTED
71	SRM_RELEASED
72	SRM_FILE_PINNED
73	SRM_FILE_IN_CACHE
74	SRM_SPACE_AVAILABLE
75	SRM_LOWER_SPACE_GRANTED
76	SRM_DONE
77	SRM_PARTIAL_SUCCESS
78	SRM_REQUEST_TIMED_OUT
79	SRM_LAST_COPY
80	SRM_FILE_BUSY
81	SRM_FILE_LOST
82	SRM_FILE_UNAVAILABLE
83	SRM_UNKNOWN_ERROR
90	Connection refused
91	GSI mapping not found
92	General unpredictable exception
93	Input error
94	Other error, not reached SRM yet
100	SRM returned no status, but it is a failure

7 Troubleshooting

� When a large request is submitted, the srm-copy may not fill all request information into the memory before submitting to the server. It is mostlikely because the memory size of the java virtual machine.� In such case, you can increase the size of the virtual machine in srm-copy by -Xmx###M, where # is the amount of memory which you want to allocate in megabytes. E.g. java �Xmx512M

� For memory conscious jobs, �-lite� option is provided to use less memory to handle the user request. The option the passes -Xms32M to the virtual machine.

8 See Also

srm-copy-status, srm-request