UNfortunately OOM Exception might have other reasons than memory.
In your case it might be that there are too many threads or the no. of filehandles is restricted.
Jboss is running on Window 2008 server and i had set the stack size as -xss128k. is there anywhere else where i can set stack space.
And I doubt as it is failing at RMI. Does it has to do anyhting with RMI and can i set thread pool for RMI.
The inability to create a new thread means that there is no more address space available to your application. I assume you are uysing a 32-bit JVM in which case you are restricted to 4GB of address space, 2GB of which is used by the OS. If you are using a 64-bit JVM than you have run out of physical memory. See this discussion on JVM memory usage:
If you are using a 32-bit JVM you could switch to 64-bit. If you are already using a 64-bit JVM you could decrease your heap size, add more RAM, or increase the pagefile size (that last is not recommended due to possible performance issues). Or you could find our how many threads you are actually running, and why, and see what you can do to reduce the thread count.
look like issue has not fixed for us, i did some JVM tunning, but eventually it again throws,
2012-11-09 03:23:27,949 WARNING [sun.rmi.transport.tcp] (RMI TCP Accept-11101) RMI TCP Accept-11101: accept loop for ServerSocket[addr=/0.0.0.0,port=0,localport=11101] throws
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
I am using Win2k8R2.
Below are my settings
set JAVA_OPTS=%JAVA_OPTS% -Xms512m -Xmx1024m -XX:MaxPermSize=512m -Xss128k -XX:-UseConcMarkSweepGC -XX:-UseGCOverheadLimit