Berkeley Storage Manager

BeStMan Copyright (c) 2007,2008,2009, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy).� All rights reserved.

If you have questions about your rights to use or distribute this software, please contact Berkeley Lab's Technology Transfer Department at [email protected] and [email protected].

NOTICE.� This software was developed under partial funding from the U.S. Department of Energy.� As such, the U.S. Government has been granted for itself and others acting on its behalf a paid-up, nonexclusive, irrevocable, worldwide license in the Software to reproduce, prepare derivative works, and perform publicly and display publicly.� Beginning five (5) years after the date permission to assert copyright is obtained from the U.S. Department of Energy, and subject to any subsequent five (5) year renewals, the U.S. Government is granted for itself and others acting on its behalf a paid-up, nonexclusive, irrevocable, worldwide license in the Software to reproduce, prepare derivative works, distribute copies to the public, perform publicly and display publicly, and to permit others to do so.

For the end user license agreement file for BeStMan for non-commercial research use, go to http://datagrid.lbl.gov/bestman/license-nc.html.

For the end user license agreement file for BeStMan for commercial research use, go to http://datagrid.lbl.gov/bestman/license-c.html.

1 Introduction

SRM client tools are full implementations of SRM v2.2 as generic SRM v2.2 clients, developed by Lawrence Berkeley National Laboratory. They have been tested for all current SRM v2.2 implementations such as BeStMan, CASTOR, dCache, DPM, SRM/iRODS-SRB and StoRM. They are continuously being tested for compatibility and interoperability.

It�s a java implementation that would run on any OS platforms, and currently supports gsiftp, http, https and ftp as its file transfer protocols.

It requires the minimal administrative efforts on the deployment and maintenance.

SRM v2.2 specification can be found on https://sdm.lbl.gov/srm-wg/doc/SRM.v2.2.html.
SRM client downloads and instructions can be found on http://datagrid.lbl.gov/bestman.

2 Requirements

� SUN Java 1.5.0_12 or IBM Java 1.5 or higher versions such as 1.6.0_07.

� Valid user grid proxy

3 Installation

� Manual installation from a web downloadable file
After downloading from the web (http://datagrid.lbl.gov/bestman), expand from the gzipped tar file, and it creates srmclient2 directory.

� VDT packman installation
VDT (http://vdt.cs.wisc.edu) provides a package SRM-Client-LBNL for installing, using pacman.

3.1 Installation Preparation

� Java 1.5.0_x path
e.g. /sandbox/jdk1.5.0_12

� SRM Client installation directory
This directory may be created during the installation process,
e.g. /opt/srmclient2

� User Grid Certificate and grid mapping on the server side
The existing user certificate can be used, or a new user certificate can be obtained.
e.g. /DC=org/DC=doegrids/OU=People/CN=Arie Shoshani 123456
e.g. in /etc/grid-security/grid-mapfile,
��/DC=org/DC=doegrids/OU=People/CN=Arie Shoshani 123456� osg

� Valid grid CA certificates and CRLs
The directory /etc/grid-security/certificates needs to contain valid CA certificates and their valid CRLs.

� GLOBUS_TCP_PORT_RANGE
These ports have to be opened when there is a firewall on the system. If open ports cannot be configurable on the system, always use one parallel stream for gridftp file transfer for passive mode.
e.g. GLOBUS_TCP_PORT_RANGE=48001,48999

4 Configuration and notes

a. Change directory to the installed srmclient2/setup, and run configure with options.

b. Minimum options are --enable-clientonly when other options use default values.
configure --help to see options with default values.

c. Upon successful running of configure, srmclient2/bin would be created with all necessary client commands.

4.1 Sample Configuration options

Typical case where default values are used,
% configure \
--enable-clientonly
From the above value example,
% configure \
--enable-clientonly \
--with-globus-tcp-port-range=48001,48999
When CA certificate directory is not in usual place (/etc/grid-security/certificates),
% configure \
--enable-clientonly \
--with-cacert-path=/opt/osg/security/certificates \
--with-globus-tcp-port-range=48001,48999
Another example with an external globus location when available,
% configure \
--enable-clientonly \
--with-globus-tcp-port-range=48001,48999 \
--with-globus-location=/software/globus-4.0.3

4.2 Configure options

Upon successful configuration, srm client tools would be created in srmclient2/bin.

4.2.1 Required options

--enable-clientonly

Installation for SRM clients only (default=no).

4.2.2 Other options

--with-srm-home=<PATH>	Installation path for SRM clients. If not given, it will be guessed based on the current working directory.
--enable-verbose	Print output to the standard output during the configuration
--enable-backup	Enable backup before running a new configuration if there is a previous configuration (default=no)
--with-java-home=<PATH>	Specify the JAVA_HOME directory
--with-globus-tcp-port-range=<VALUES>	Specify the GLOBUS_TCP_PORT_RANGE when firewall is enabled. E.g. 6201,6299
--with-max-java-heap=<INT>	Specify the max java heap size in MB (default=512M)
--with-cacert-path=<PATH>	Specify the Grid CA Certificate directory path (default=/etc/grid-security/certificates)
--with-globus-location=<PATH>	Specify the GLOBUS_LOCATION path

4.3 Configuration file

� SRM client tools mayuse client configuration file to define default values for some command line options when manually created.

� Client configuration file is from srmclient2/conf/srmclient.conf or from a desginated path as an option -conf to srm client commands.��

� When a parameter is defined in the client configuration file, the corresponding values may not be provided in the command line. This is particularly useful when repeated commands are used.
e.g. If SRM service port number and service handle is defined in the client configuration file as
ServicePortNumber=8443
ServiceHandle=/srm/v2/server
�� Then, SURL can be srm://bestman.lbl.gov//mydir/myfile

� When the same parameter is provided in the command line with a different value, the command line value takes priority than one in the client configuration file.

� A sample client configuration file is in srmclient2/conf directory.

� Each entry in the configuration file has the following meaning.

ServicePortNumber	� Defines the default SRM service port number
	e.g. ServicePortNumber=8443
ServiceHandle	� Defines the default SRM service handle
	e.g. ServiceHandle=/srm/v2/server
ServiceURL	� Defines the default SRM service endpoint
	e.g. ServiceURL=srm://bestman.lbl.gov:8443/srm/v2/server

� e.g. When the full SURL is srm://bestman.lbl.gov:8443/srm/v2/server\?SFN=/mydir/mypath/myfile, it becomes srm://bestman.lbl.gov//mydir/mypath/myfile when ServicePortNumber and ServuceHandle are defined.

� e.g. A client command �srm-ls srm://bestman.lbl.gov:8443/srm/v2/server\?SFN=/mydir/mypath/myfile� can be �srm-ls srm://bestman.lbl.gov//mydir/mypath/myfile� when ServicePortNumber and ServiceHandle are defined.

� e.g. A client command �srm-ping srm://bestman.lbl.gov:8443/srm/v2/server� can be �srm-ping� when ServiceURL is defined.

BufferSize	� Defines the default GridFTP buffer size � Default=1048576
	e .g. BufferSize=1048576
Parallelism	� Defines the default number parallel streams for GridFTP transfers � Default=1
	e .g. Parallelism=2
dcau	� Defines the default DCAU setting � Default=true
	e.g. dcau=false
NumRetry	� Defines the default number of retries of file transfers when failed
	e.g. NumRetry=5
RetryDelay	� Defines the default number of seconds to sleep aftger a file transfer failure before next file transfer try
	e.g. RetryDelay=120

� Equivalent command line options are -buffersize, -parallelism, -dcau, -numretry and �retrydelay respectively.

StatusWaitTime	� Defines the default waiting time in seconds between checking status of a request
	e.g. StatusWaitTime=120
StatusMaxTime	� Defines the default timeout in seconds for checking status of a request
	e.g. StatusMaxTime=600

� Equivalent command line options are -statuswaittime and -statusmaxtime respectively.

4.3.4 Related to the space token

SpaceToken	� Defines the default space token to be used for all file requests
	e.g. SpaceToken=ATLASUSERDISK1

� Equivalent command line option is -spacetoken.

4.3.5 Related to the user GSI settings

ProxyFile	� Defines the default user proxy path
	e.g. ProxyFile=/tmp/x509up_u1234
UserCert	� Defines the default user certificate path
	e.g. UserCert=/home/users/mylogin/.globus/usercert.pem
UserKey	� Defines the default user certificate key path
	e.g. UserCert=/home/users/mylogin/.globus/userkey.pem

� When UserCert and UserKey are defined, user grid passphrase will be prompted on each request.

� When ProxyFile and UserCert/UserKey are defined, ProxyFile takes priority.

� Equivalent command line options are -proxyfile, -usercert and -userkey respectively.

5 Exit codes

When SRM client commands exit, they have exit codes as following:

0	SRM_SUCCESS
51	SRM_FAILURE
52	SRM_AUTHENTICATION_FAILURE
53	SRM_AUTHORIZATION_FAILURE
54	SRM_INVALID_REQUEST
55	SRM_INVALID_PATH
56	SRM_FILE_LIFETIME_EXPIRED
57	SRM_SPACE_LIFETIME_EXPIRED
58	SRM_EXCEED_ALLOCATION
59	SRM_NO_USER_SPACE
60	SRM_NO_FREE_SPACE
61	SRM_DUPLICATION_ERROR
62	SRM_NON_EMPTY_DIRECTORY
63	SRM_TOO_MANY_RESULTS
64	SRM_INTERNAL_ERROR
65	SRM_FATAL_INTERNAL_ERROR
66	SRM_NOT_SUPPORTED
67	SRM_REQUEST_QUEUED
68	SRM_REQUEST_INPROGRESS
69	SRM_REQUEST_SUSPENDED
70	SRM_ABORTED
71	SRM_RELEASED
72	SRM_FILE_PINNED
73	SRM_FILE_IN_CACHE
74	SRM_SPACE_AVAILABLE
75	SRM_LOWER_SPACE_GRANTED
76	SRM_DONE
77	SRM_PARTIAL_SUCCESS
78	SRM_REQUEST_TIMED_OUT
79	SRM_LAST_COPY
80	SRM_FILE_BUSY
81	SRM_FILE_LOST
82	SRM_FILE_UNAVAILABLE
83	SRM_UNKNOWN_ERROR
90	Connection refused
91	GSI mapping not found
92	General unpredictable exception
93	Input error
94	Other error, not reached SRM yet
100	SRM returned no status, but it is a failure

6 Client commands and samples

For the details and samples of each command, follow the link on each client command.

6.1 Client commands involving discovery functions

6.1.1 srm-ping

srm-ping checks the SRM server. In response to the call, SRM server returns the SRM version number as well as other backend information.

6.1.2 srm-tranasferprotocols

srm-transferprotocols retrieves a list of file transfer protocols that the SRM server supports.

6.1.3 srm-util

srm-util handles discovery functions and permission related functions.

6.2 Client commands involving data transfer functions

6.2.1 srm-copy

srm-copy requests to copy files to and from SRM, between SRMs, between SRM and other storage repository.

6.2.2 srm-copy-status

srm-copy-status checks status of the request that has been submitted with srm-copy.

6.3 Client commands involving directory functions

6.3.1 srm-dir

srm-dir supports directory management functionality.

6.3.2 srm-ls

srm-ls supports directory or file browsing functionality and returns meta data of the directories or files.

6.3.3 srm-ls-status

srm-ls-status checks the status of the srm-ls request when request token was assigned because there are too many results or because it takes more time than SRM server could return the results immediately. Upon successful request, SRM server returns meta data of the directories or files.

6.3.4 srm-mkdir

srm-mkdir supports directory creation functionality in the namespace that SRM manages or has access to.

6.3.5 srm-mv

srm-mv supports move functionality in the namespace that SRM manages.

6.3.6 srm-rm

srm-rm removes an SURL of a file from the namespace that SRM manages or has access to.

6.3.7 srm-rmdir

srm-rmdir removes an SURL of a directory from the namespace that SRM manages or has access to.

6.4 Client commands involving permission functions

6.4.1 srm-permission-check

srm-permission-check checks authorization information on an SURL.

6.4.2 srm-permission-get

srm-permission-get retrieves authorization information on an SURL.

6.4.3 srm-permission-set

srm-permission-set sets authorization information on an SURL.

6.5 Client commands involving request management functions

6.5.1 srm-extendfilelifetime

srm-extendfilelifetime supports lifetime extension functionality on SURLs or pinned files.

6.5.2 srm-putdone

srm-putdone finalizes the put request on SURLs that has been submitted with srm-copy.

6.5.3 srm-release

srm-release releases pinned files from a request that has been submitted with srm-copy.

6.5.4 srm-req-abort

srm-req-abort submits abort on a request to the SRM server.

6.5.5 srm-req-abortfiles

srm-req-abortfiles submits abort on files of a request to the SRM server.

6.5.6 srm-req-resume

srm-req-resume submits resume on a suspended request to the SRM server.

6.5.7 srm-req-summary

srm-req-summary retrieves a summary information on a request from the SRM server.

6.5.8 srm-req-suspend

srm-req-suspend submits suspend on an active request to the SRM server.

6.5.9 srm-req-tokens

srm-req-tokens retrieves request tokens that belong to the user from the SRM server.

6.5.10 srm-request

srm-request handles request management functions.

6.6 Client commands involving space management functions

6.6.1 srm-space

srm-space handles space management functions.

6.6.2 srm-sp-change

srm-sp-change submits a request to change space for SURLs to a new space with the space token.

6.6.3 srm-sp-info

srm-sp-info retrieves an information about a space that is associated with a space token.

6.6.4 srm-sp-purge

srm-sp-purge removes all files from a space that is associated with a space token, unless a file is the last copy.

6.6.5 srm-sp-release

srm-sp-release release a space associated with a space token.

6.6.6 srm-sp-reserve

srm-sp-reserve requests to reserve a space with desired size.

6.6.7 srm-sp-reserve-status

srm-sp-reserve-status checks the status of the space reservation request that has been submitted with srm-sp-reserve, when SRM server takes time to handle the request and returns a request token.

6.6.8 srm-sp-tokens

srm-sp-tokens retrieves space tokens that belong to the user.

6.6.9 srm-sp-update

srm-sp-udpate updates space information such as size and lifetime.

6.7 Notes

� srm client makes no proxy delegation by default, and automatically handles proxy delegation based on the source and target urls. A user can override the automatic handling by providing an option �-delegation� (or �-delegation true�) to force the delegation, and �-delegation false� to force no delegation. �-debug� option would show how the delegation is done on the output.

� When a large request is submitted, the srm client may not fill all request information into the memory before submitting to the server. It is mostlikely because the memory size of the java virtual machine.� In such case, you can increase the size of the virtual machine in srm client by -Xmx###M, where # is the amount of memory which you want to allocate in megabytes. E.g. java �Xmx512M

� For memory conscious jobs, �-lite� option is provided to use less memory to handle the user request. The option the passes -Xms32M to the virtual machine.