Notes for scalability

BeStMan is highly scalable to the number of request handling per unit time, depending on the underlying file system and storage. Configurable options need to be adjusted depending on the local storage and file system capacity, and also the operating system control needs to be adjusted.

OS: sysctl values

Adjust sysctl values on the node where bestman server runs. Put the values in /etc/sysctl.conf then run sysctl -p to apply them, or apply individually.

No need to reboot, and your OS kernel should be able to handle a lot more open connections.

Ref: http://fasterdata.es.net/tuning.html and http://fasterdata.es.net/TCP-tuning/linux.html

# General gigabit tuning:
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_syncookies = 1
# this gives the kernel more memory for tcp
# which you need with many (100k+) open socket connections
net.ipv4.tcp_mem = 50576   64768   98152
net.core.netdev_max_backlog = 2500
# I was also masquerading the port comet was on, you might not need this
net.ipv4.netfilter.ip_conntrack_max = 1048576

In addition to above, check and changes these:

/sbin/sysctl -w net.core.somaxconn=256
/sbin/sysctl -w net.core.netdev_max_backlog=2500

Also, check these values for long timeout values. If relatively short, keep them longer.

% sysctl \
net.ipv4.tcp_keepalive_time \
net.ipv4.tcp_keepalive_intvl \
net.ipv4.tcp_keepalive_probes
net.ipv4.tcp_keepalive_time = 1800
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9

Jetty web server control values

1. Increase the maximum container thread pool size for the jetty server

--with-max-container-threads

  e.g. configure --with-max-container-threads=1024

2. Have large number for jetty http connection acceptor threads for the server's channel connector

--with-connection-acceptor-thread-size

  e.g. configure --with-connection-acceptor-thread-size=8

3. Have large size for jetty http connection waiting queue

--with-connector-queue-size=

  e.g. configure --with-connector-queue-size=512

4. Increase the max java heap memory size

--with-max-java-heap=

  e.g. configure --with-max-java-heap=4096

5. Client side http connection timeout control - bestman clients (only)

  e.g. srm-copy -sethttptimeout 1800

6. Each connection we make requires an ephemeral port, and thus a file descriptor

By default this is limited to 1024. To avoid the Too many open files problem you'll need to modify the ulimit for your shell.

limit descriptors  65535

unlimit

7. This can be changed in /etc/security/limits.conf, but requires a logout/login.

For now you can just sudo and modify the current shell.

(su back to your non-priv'ed user after calling ulimit if you don't want to run as root):

$ sudo bash
# ulimit -n 999999
# erl