Try reducing the "Heap size a bit" (So that you will have little more space for your jvms native area for the thread creation.
Try reducing the Thread Stack Size (-Xss) that is default to 1024k
Following blog might give some directions:
OutOfMemory Causes and First Aid Steps?
Regarding your query: "we increased memory and received OutOfMemoryError, It's strange"
>>> NOTE: You are not getting "OutOfMemory in Heap" Rather it is "Native OutOfMemory" so increasing the Heap will cause this issue to become more severe. Because that will cause more little space to be available for the Native Area of the JVM.
"...unable to create new native thread" indicates that OS exhausted limit for processes.
Can you check ulimits for your system? If you're on linux like machine you need to increase value for "max user processes" and also "open files".
I have same installation for both JBOSS servers, but problem happened in one server only
Yes, problem happened only on one instance because looks like you have set 4303m for Heap and 1024m for PermGen size but that will cause little less space to be available for the Native area. So either try any of the following suggestion to see what happens:
1. Try reducing the "Heap size a bit like -Xmx2048m " (So that you will have little more space for your jvms native area for the thread creation.
2. Try reducing the Thread Stack Size (-Xss) that is default to 1024k (like -Xss512k )
"...unable to create new native thread" is a thread leak. It's common problem with ulimits on linux like machines. Just try to set "max user processes" and also "open files" to higher values. Linux machines set this value too low.
If you see OOME after this change then use visualvm to see which threads exhausted the system.
I changed the Xss to be 128K, and now i am testing to see the results
How can I see OOME?
OOME = OutOfMemoryError
It's just if you can seee OOME afte adjusting ulimits.