Need Help with Optimizing JBoss 7.1 HTTP Thread Pool

[sdsouza]

Hi,

We are currently using JBoss 7.1 as Application Server.

There is a Load Balancer that redirects traffic to the Application Servers (3 of them).

CPU is 5% on the Application Server & Memory is fine too.

We do receive spike in requests on the 0th & 30th minute of the hour and the Load Balancer starts rejecting the requests.

We could receive about 70,000 requests at a given point in time during the spikes.

As per Load Balancer team, the requests get rejected because the Application Servers starts taking time to provide a response.

Need some help in tweaking the HTTP Thread Pool to handle a large number of requests at a given point in time.

Currently, we have below configuration

        <subsystem xmlns="urn:jboss:domain:threads:1.1">

            <unbounded-queue-thread-pool name="http-executor">

                <max-threads count="512"/>

                <keepalive-time time="60" unit="seconds"/>

            </unbounded-queue-thread-pool>

        </subsystem>

As per this link: https://developer.jboss.org/wiki/ThreadPoolConfiguration

there are various Thread Pools that can be used for HTTP. But, i am not sure of the Pros & Cons of the Thread Pools for High Load, Spikes & Performance.

Can someone please help with providing any inputs to tweak the HTTP Thread Pool?

Please Note: Servers are Large. Hence, CPU & Memory should not be an issue.

Thanks,

Sheldon

[ptyagi_redhat.com]

You can use default web connector pool which is the best pool to use and this pool does not queue or reclaim idle threads. This is the pool you use when you do not define any exceutor.

[sdsouza]

Thanks for the reply Priyanka.

I did change standalone.xml for Jboss 7.1 to remove Thread Pooling and thus use the default one.

However, it was observed in repetitive tests that performance actually degraded and requests were rejected by the Load Balancer 2 times more.

On the other hand, it was observed that changing the Max Thread Count from 512 to a higher number like 2000 provided better performance & latency.

However, we are not sure if 2000 might be too high and cause JBoss to crash eventually because of too many sockets.

Below are some additional details of our server

1> Server Size: c4 4xlarge. Memory: 30 GB. This is an Amazon Cloud-based server. And we use an Amazon Cloud-based Load Balancer.

2> JBoss standalone.conf (We realized that Perm is not used too much as hence its set low)

JAVA_OPTS="-server -Xms25600M -Xmx25600M -XX:PermSize=1536M -XX:MaxPermSize=1536M -XX:NewSize=16384M -XX:+AggressiveHeap -XX:+UseParallelGC -XX:ReservedCodeCacheSize=256m -XX:+UseCodeCacheFlushing -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=1800000 -Dsun.rmi.dgc.server.gcInterval=1800000"

3> OS Level Settings: ulimit -a

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 118880

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 65535

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) 118880

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

Can you please suggest any other options available for JBoss to accept more requests per seconds (especially in our case we gets spikes at the 0th and 30th minute)?

Thanks,

Sheldon