People Publications Projects

Back to ESG Home

ESG Data Node Deployment Experience at NERSC

Jump to:

This document explains our experience in ESG data node installation at NERSC.

Please note that there is an installation script provided at the distribution site in order to automate the process (visit The esg-node script prepares environment variables, downloads required software, installs and configures all necessary components, and also checks for updates automatically. However, it requires root privileges, and some of the installation directories and path information are embedded inside the script.

In order to complete installation without root privileges, we have edited the installation scripts (see esg-node-nerc and esg-globus-nersc ). Please take a look at those, you will need to change installation directory, log files, user passwords, script directory, etc. Besides, you might need to manually start/stop some services such as PostgreSQL. Here is the diff output for those modified scripts:

A Step-by-Step Installation Guide

The following is a step-by-step guide to install ESG Data node components based on the information given in the installation script (version 0.2.9).

Relevant info


Here is a list of components you need to have before starting installation. Also you need an account from ESG gateway portal (from association gateway, i.e.: - with publishing role). You will need this to authenticate with MyProxy client.

Note: Make sure your account has been added as a data publisher!

We assume that you already have necessary components installed (PostgreSQL, Apache Ant, JAVA, git, and curl), and PATH and LD_LIBRARY_PATH are set properly. If not, see NecessaryComponents first.

Determine an installation directory and set an environment variable for the installation directory, INSTALL_HOME (that will be referred in the next steps).

Initializing PostgreSQL database

Create data and log directories for PostgreSQL, and dont forget to set proper ownership/permission for those directories.

export PGDATA=$INSTALL_HOME/pgsql/data
mkdir -p $PGDATA
mkdir -p $INSTALL_HOME/pgsql/log
chmod 700

Initialize the database by running initdb;

$initdb -D $PGDATA

By default, "trust" authentication has been enabled for local connections. Change this by editing "pg_hba.conf". Or, give -A option while running initdb command. It is recommended to use "md5" since it sends encrypted passwords. In "trust" authentication, any local user can connect to the database.

Note: I would recommend to change it to "md5" after you run esgsetup --db (after CDAT installation) - had a problem in this when esgsetup is connecting to the database.

Start database:

pg_ctl -D $PGDATA  start
Set environment variables:
export PGUSER=dbsuper
export PGPORT=5432
export PGHOST=localhost

And, create a database user ("dbsuper") and set a password - this will be needed while setting up ESGCET later.

-P -s -e dbsuper

Edit "$PGDATA/postgresql.conf", to change port number (default is 5432) and other parameters such as logging options ( log directory and log filename).

Verify your installation by running:

psql -U dbsuper postgres

Installing CDAT (Python + CDMS)

Set an installation directory for CDAT (Climate Data Analysis Tool):


We will be using version eb8b668. Download the package and compile it...

git clone
cd cdat
git checkout eb8b668

If you have Python already installed, specify the path. Note that Python should have tk/tcl support, install Tkinter.

./configure --prefix=$CDAT_HOME --with-python=/usr/bin/python

Alternatively, if you dont give the Python path, CDAT installer will download and install Python itself.

./configure --prefix=$CDAT_HOME

Note: In Ubuntu; first install Tkinter packages and then specify the path for Python while running "configure". Python installed by CDAT (default) does not work somehow (saying missing tk/tcl support in Python).

Also update path information (make sure cdat/bin is before /usr/bin in your path):

export PATH=$CDAT_HOME/bin:$CDAT_HOME/Externals/lib:$PATH

Installing ESGCET (esgcet-2.4-py2.6.egg)

Dowload ESGCET package (that will be required scripts and packages for publishing)

chmod 755 esgcet-2.4-py2.6.egg
easy_install esgcet-2.4-py2.6.egg

Complete the setup by giving an organization ID (rootid in the following):

bin/esgsetup --config --rootid nersc

$HOME/.esgcet/esg.ini will be created and initial configuration will be saved in esg.ini file.

Before proceeding further, please make sure that PosgreSQL is up and running. Run esgsetup to create database entries. It will ask the database admin user (dbsuper), and will create esgcet database with owner esgcet (you also need to set a password for esgcet database user).

$CDAT_HOME/bin/esgsetup --db

Update environment by adding the organization ID as follows (advised);

export ESG_ROOT_ID=nersc

Note that you might need to edit ~/.esgcet/esg.ini to set the password for esgcet database user.

There is already a "test" project defined in esg.ini

Dowload a sample data file and scan this sample dataset and publish (ESG_ROOT_ID is nersc). Note that this sample file should be inside the Thredds root catalog directory! Also. you need to specify the full path of the directory while running esgscan_directory.

mkdir $INSTALL_HOME/data/testdir
cd testdir
cd ..
esgscan_directory --dataset pcdmi.nersc.test --project test $INSTALL_HOME/data/testdir > scan.out
esgpublish --map scan.out --project test

Tomcat Installation

Download and install tomcat:

tar xvf apache-tomcat-6.0.26.tar.gz -C $INSTALL_HOME
ln -s apache-tomcat-6.0.26 tomcat

Set TOMCAT_HOME environment variable:

tar xvf jsvc.tar.gz
cd jsvx-src
chmod 755 configure; ./configure --with-java=$JAVA_HOME
cp jsvc $TOMCAT_HOME/bin

Next step is to configure tomcat by editing $TOMCAT_HOME/conf/server.xml.

Make sure server.xml has appropriate permissions (chmod 600 server.xml). By default, port 8080 and 8443 will be used. If you want to change and use 80 and 443 instead, edit Connector port numbers in server.xml.

You may want to look at Tomcat documentation for Servlet/JSP and SSL configuration.

Now, we setup the keystore:

$JAVA_HOME/bin/keytool -genkey -alias tomcat -keyalg RSA -keystore $TOMCAT_HOME/conf/keystore-tomcat -validity 365

It will ask keystore password and key password for tomcat (default is "changeit").

Go to conf directory and download truststore file:

cd $TOMCAT_HOME/conf

Open server.xml and edit path for keystore and truststore. You can search keystore_file and truststore_file in this preconfigured sample server.xml file.

It is beneficial to create a "tomcat" user and start tomcat with this user's privileges. In that case, change the ownership of the tomcat directory

(chmod -R tomcat $TOMCAT_HOME).

It is useful to set CATALINA_HOME environment variable. You can start/stop tomcat using the script.

$CATALINA_HOME/bin/ stop
$CATALINA_HOME/bin/ start

In order to use jsvc, start tomcat (make sure JAVA_HOME is set) by running the following command (preparing a startup script will be helpful);

/bin/jsvc -Djava.endorsed.dirs=./endorsed -pidfile /tmp/ \
-cp $TOMCAT_HOME/bin/bootstrap.jar:$TOMCAT_HOME/bin/tomcat-juli.jar:$TOMCAT_HOME/bin/commons-daemon.jar \
-outfile ./logs/catalina.out -errfile ./logs/catalina.err -Xmx2048m -Xms1024m \ org.apache.catalina.startup.Bootstrap

Stop tomcat by running the following (jsvc);

./bin/jsvc -pidfile /tmp/ -stop org.apache.catalina.startup.Bootstrap

Thredds data server (v 4.1.6)

Download Thredds war file and put into the tomcat "webapps" directory:

cd $TOMCAT_HOME/webapps

Restart tomcat (after restart the war file will be extracted under webapps directory)

Edit $TOMCAT_HOME/conf/tomcat-user.xml. Search for user entry in tomcat-user.xml and add a user ( dnode_user) with administrative privileges. The entry should look like:

<role rolename="tdsConfig"/>
<role rolename="manager"/>
<role rolename="tdrAdmin"/>
<user username="dnode_user" password="digest_password_here" roles="tdrAdmin,tdsConfig"/>

First, generate a password hash by running

$TOMCAT/bin/ -a SHA <password for dnode_user>

Use this password hash and add line, shown below, to the tomcat-user.xml file. Then, restart the tomcat.

<user_entry='<user username="dnode_user" password="<PASSWORD_HASH_HERE>" roles="tdrAdmin,tdsConfig">

Configure tomcat for digest authentication. Create directory $TOMCAT_HOME/conf/Catalina/localhost if does not exists. Add or edit thredds.xml file in $TOMCAT_HOME/conf/Catalina/localhost. It should look like:

<?xml version="1.0" encoding="UTF-8"?>
<Context path="/thredds">
<Realm className="org.apache.catalina.realm.MemoryRealm" digest="SHA" />

A sample web.xml file is given here . Make sure SSL is enabled (this is used by the ESG-publisher to re-initialize Thredds Data server andcheck logs). It should look like:


Note: esg.ini is important. Make sure "thredds_url" "thredds_reinit_error_url" and "thredds_reinit_url" are correct (they should point to full host name - not localhost)

Security-Token-Filters and Certificate From Gateway

Here, we are using gateway node ESG-PCMDI( as myProxy end-point (default myProxy port 2119).

Download necessary classes into a temporary directory:

cd $TOMCAT_HOME/temp

End-point is, SSL port is 443, and default password for SSL end point is "changeit"

cd $TOMCAT_HOME/conf
cp jssecacerts jssecacerts.bak
$JAVA_HOME/bin/java -classpath .:$TOMCAT_HOME/temp InstallCert <password>

This will add certificate to keystore "jssecacerts". Change owner and permission of that file (chmod 644 jssecacerts; chown tomcat jssecacerts).

Note: Copy jssecacerts into $JAVA_HOME/jre/lib/security (Installation script does this but probably this is not necessary! Its path has been specified in server.xml already) cp -p $TOMCAT_HOME/conf/jssecacerts $JAVA_HOME/jre/lib/security

Add following into the environmen (optional)


Download ESG token validator filters

cd $TOMCAT_HOME/webapps/thredds/WEB-INF/lib

Now, you need to edit $TOMCAT_HOME/webapps/thredds/WEB-INF/web.xml and add the following filter specifications:

Add the following ESG security token filter and servlet entries into the web.xml:

A sample web.xml file is given here.

More information about ESG token validation filter can be found at ESG data node documentation

Restart Tomcat. Make sure PostgreSQL is running.

$CDAT_HOME/bin/esgsetup --thredds --publish --gateway

In this step, you need to specify Thredds content directory and ESG data path root directory. If they dont exist, create root directory (and replica directory).

mkdir $INSTALL_HOME/data
mkdir $INSTALL_HOME/data.replica

You may also need to edit esg.ini file and change the path for content directory. It should look like (give full path)

thredds_dataset_roots = esg_dataroot | /project/projectdirs/esg/datanode/data

Make sure thredds_username and thredds_password are set correctly.

Verify whether everything is configured properly (dont forget to restart Tomcat) by creating Tredds catalog for the data set we have scanned before. (ESG_ROOT_ID is nersc).

esgpublish --use-existing pcdmi.nersc.test --noscan --thredds

This step might take some time. It will reinitialize the Thredds Data Server, so make sure url's are set correctly in ~/.esgcet/esg.ini

Node Manager (0.0.2)

cd $TOMCAT_HOME/temp/
tar xzf esg-node.0.0.2.tar.gz
cd esg-node.0.0.2

Go to Tomcat webapp directory, and replace tokens in

mkdir -p $TOMCAT_HOME/webapps/esg-node
cd $TOMCAT_HOME/webapps/esg-node
jar xvf $TOMCAT_HOME/esg-node.0.0.2/esg-node.war
cd WEB-INF/classes

Edit Change the following options in file (in webapps/esg-node/WEB-INF/classes)

db.driver -> org.postgresql.Driver
db.protocol -> jdbc:postgresql -> localhost db.port -> 5432
db.database -> esgcet
db.user -> dbsuper
db.password -> <dbsuper password> -> <mail.admin.address>

Create esgcet database if not created yet

createdb esgcet

Configure PostgreSQL by running:

cd $TOMCAT_HOME/temp/esg-node.0.0.2/db
ant -buildfile database-tasks.ant.xml \$TOMCAT_HOME/webapps/esg-node/WEB-INF/classes/ \
-Dsql.jdbc.base.url=jdbc:postgresql://localhost:5432/ \
-Dsql.jdbc.database.user=dbsuper \ -Dsql.jdbc.database.password=<dbsuper_password>
-Dsql.jdbc.driver.jar=$TOMCAT_HOME/webapps/esg-node/WEB-INF/lib/postgresql-8.3-603.jdbc3.jar \

Restart Tomcat.

Globus Installation

See esg-globus-nersc.

Set Environmental Variables

Set installation directory (INSTALL_HOME) and create an environment file "$INSTALL_DIR/", so it can be used for sourcing the environment.


export PATH=$CDAT_HOME/bin:$CDAT_HOME/Externals/bin:$TOMCAT_HOME/bin:$GLOBUS_HOME/bin:$PATH

export PGDATA=$INSTALL_HOME/pgsql/data
export PGUSER=dbsuper
export PGPORT=5432
export PGHOST=localhost

export ESG_ROOT_ID=nersc

export X509_CERT_DIR=~/.globus/certificates


Test Publication

myproxy-logon -s -l <username_of_your_account_from gateway> -p 2119 -o ~/.globus/certificate-file -T

esglist_files pcmdi.nersc.test

esgpublish --use-existing pcmdi.nersc.test --noscan --publish

esgunpublish --skip-thredds pcmdi.nersc.test

Installing Necessary Components


tar xvzf curl-7.20.1.tar.gz
cd curl-7.20.1
./configure --prefix=$CURL_HOME
make all
make install
$CURL_HOME/bin/curl --version
export PATH=$CURL_HOME/bin:$PATH

GIT (with libcurl)

tar xvzf git-1.7.1.tar.gz
cd git-1.7.1
./configure --prefix=$GIT_HOME
make all
make install

export PATH=$GIT_HOME/bin:$PATH


tar xvfz jdk1.6.0_20-32.tar.gz -C $INSTALL_HOME
ln -s $INSTALL_HOME/jdk1.6.0_20
$JAVA_HOME/bin/java --version
export PATH=$JAVA_HOME/bin:$PATH

Apache ANT

tar xvfz apache-ant-1.8.1-bin.tar.gz -C $INSTALL_HOME
ln -s $INSTALL_HOME/apache-ant-1.8.1
$ANT_HOME/bin/ant -version
export PATH=$ANT_HOME/bin:$PATH


tar xvzf postgresql-8.4.3.tar.gz
cd postgresql-8.4.3
./configure --prefix=$PGHOME --enable-thread-safety
make install
cd contrib/tablefunc
make install
export PATH=$PGHOME/bin:$PATH