Native thread creation and Stack Overflow Exceptions: Soluti

[clajoie]

This details the solution for the following two types exception on a Linux system:

java.lang.OutOfMemoryError: unable to create new native thread
This exception is caused by one of two things.

1. The most common is that you reached the max number of user processes for the account that you're running JBoss on. You can see the limit by using "ulimit -u". Raising the limit should resolve this problem. Please note that there seems to be a hard coded ceiling of 1024 user processes, this may be a 2.4 kernel issues or a libc issue (both have such values in them, but I haven't done enough testing yet to tell which is causing this).

2. Once you get number one solved you may see this error again, this time it does actually appear to be a memory issue. By default kernel 2.4 creates threads with a stack size of 2MB. There also seems to be some limit that only allows the use of 1GB of memory for stack allocations. So lots of threads could eat up lots of the 1GB the system has to work with. The fix for this one is use the -Xss jvm flag. The value of 1024k (so -Xss1024k) seems to work well. On the 2.4 kernel this allows some 300 threads to be created per vm.

Kernel 2.6 + NPTL Note: As many people may know Kernel 2.6 allows for a new threading model known as NPTL. This new model really helps with the above problem. First it allows more then 1024 user processes. I was able to create about 2400 threads with a stack size of 128k. This compared to the approximate 500 I could create with a similar setup under the 2.4 kernel. Plus NPTL was MUCH faster (a couple orders of magnitude).

StackOverflowException or
Attempt to ungaurd stack red zone failed
These errors are seen when you run out of stack space. Best fix for this one is to use the -Xss jvm flag and lower the amount of stack size each thread gets. Again, 1024k (-Xss1024k) seems to be a good number.

[camel]

1024K stack? Isn't that a bit large? I think the default stack size is around 64K or 96K, depending on your JVM version. I think that setting is the stack size per-thread (someone correct me if I'm wrong!).

See the volano report http://www.volano.com/report/. Here's an excerpt:

The main trick in getting these Java platforms to handle the required number of threads is to reduce the size of the thread stack with the -Xss option. I generally set the stack size to 32 kilobytes when permitted by the Java virtual machine on startup. Otherwise, I set the stack size to the minimum required by the virtual machine: 64 kilobytes for the HotSpot Server VM on Solaris and 96 kilobytes for the version 1.4.1 HotSpot Server VM on Linux and FreeBSD.

What happens if you try your setup with -Xss96K

[clajoie]

Yes, 1MB does seem like a lot, but the default is 2MB on a linux system. A stack size of 96k doesn't even allow the JVM to start, 128k and 256k and you encounter random stack overflow errors. Still testing 512 and 768, but 1024 has been reported by a number of people on here as the "magic" number. So we're using it to be on the safe side.

You are correct though, that is the stack size per thread.

[juha]

Hello Chad,

added this entry to Wiki where it will be more accessible to people.

Thanks,

[clajoie]

Some additional testing indicates that a stack size of 512k works fine for our apps. This may not be the case for all applications, but it does for us.