[JBoss 5.0-CR2] OutOfMemory JVM Crash during JIT

[ceciiil]

We have a JEE application running with EJB2 that we have migrated to EJB3.

Before migration, this app runs perfectly on JBoss 423 compiled on JDK5 with runtime JRE5 and JRE6 (not at the same time though) up to 50 simulatneous users on a single JBoss instance.

After migration to EJB3, we are experiencing this error on both JRE5 and JRE6 as of 5 simultaneous users :

# java.lang.OutOfMemoryError: requested 2292728 bytes for Chunk::new. Out of swap space?

Current thread (0x54092800): JavaThread "CompilerThread0" daemon [_thread_in_native, id=4092, stack(0x54280000,0x542d0000)]

Stack: [0x54280000,0x542d0000]
[error occurred during error reporting (printing stack bounds), id 0x80000001]

Current CompileTask:
C2:4692 com.lectra.pdm.enterpriselayer.server.productmanagement ...

We don't have this problem if we force JVM options so that it only runs in interpreted mode (-Xint). We also have created a file with the list of methods we want to keep out of the JIT, gave this to the JVM as startup option and it works longer. But it eventualy crashes.

Any recommendation to fix this issue ? Needless to say this is a showstopper for us as far as upgrade to EJB3 / JBoss 5 is concerned.

We didn't want to use JBoss 423 as it is not fully JEE compliant and this is a strong constraint for our software solution.

Many thanks in advance for your help !

[ceciiil]

woops : forget to mention our JVM option at startup time :

jvm_args: -Dprogram.name=pdm.bat -Xms1024m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+DisableExplicitGC

Also : this is running on HP Server Quad Core / Windows 2003 SP2 / 4GB Memory. When the JVM crashes the GC output is :

{Heap before GC invocations=506 (full 0):
 PSYoungGen total 108032K, used 103148K [0x4c810000, 0x539d0000, 0x539d0000)
 eden space 99840K, 100% used [0x4c810000,0x52990000,0x52990000)
 from space 8192K, 40% used [0x52990000,0x52ccb3c0,0x53190000)
 to space 7808K, 0% used [0x53230000,0x53230000,0x539d0000)
 PSOldGen total 932096K, used 653196K [0x139d0000, 0x4c810000, 0x4c810000)
 object space 932096K, 70% used [0x139d0000,0x3b7b3240,0x4c810000)
 PSPermGen total 262144K, used 104506K [0x039d0000, 0x139d0000, 0x139d0000)
 object space 262144K, 39% used [0x039d0000,0x09fdeb50,0x139d0000)
823.775: [GC [PSYoungGen: 103148K->4727K(108736K)] 756345K->659744K(1040832K), 0.0289539 secs] [Times: user=0.06 sys=0.02, real=0.03 secs]
Heap after GC invocations=506 (full 0):
 PSYoungGen total 108736K, used 4727K [0x4c810000, 0x539d0000, 0x539d0000)
 eden space 100928K, 0% used [0x4c810000,0x4c810000,0x52aa0000)
 from space 7808K, 60% used [0x53230000,0x536cdec0,0x539d0000)
 to space 7744K, 0% used [0x52aa0000,0x52aa0000,0x53230000)
 PSOldGen total 932096K, used 655016K [0x139d0000, 0x4c810000, 0x4c810000)
 object space 932096K, 70% used [0x139d0000,0x3b97a250,0x4c810000)
 PSPermGen total 262144K, used 104506K [0x039d0000, 0x139d0000, 0x139d0000)
 object space 262144K, 39% used [0x039d0000,0x09fdeb50,0x139d0000)
}

[peterj]

This looks like a JVM crash. What JVM are you using ("java -version" should tell you, it is also listed in the beginning of the console log, and also in boot.log).

The "Out of swap space?" message suggests that there is a memory issue. Have you monitored Windows memory usage? What is the size of your paging file?

[ceciiil]

Hi Peter,

Thanks for your feedback. My Windows 2003 SP2 server running in 32 bits mode (with /3GB option in boot.ini) is set up with 4GB memory and 4GB paging.

The problem has been reproduced with both JRE-1.6.0_7 and JRE-1.5.0_15.

So I have tried to change my JVM options with increased Perm size as follow :

jvm_args: -Dprogram.name=pdm.bat -Xms1024m -Xmx1024m -XX:PermSize=512m -XX:MaxPermSize=512m -XX:+Dis
ableExplicitGC

Result : JBoss 5 crashes at startup time ! (with JRE6 - I didn't try this set up with JRE5). Again CompilerThread while this time compiling JBoss method :

#
# An unexpected error has been detected by Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 35608 bytes for Chunk::new. Out of swap space?
#
# Internal Error (allocation.cpp:218), pid=5756, tid=8176
# Error: Chunk::new
#
# Java VM: Java HotSpot(TM) Server VM (10.0-b23 mixed mode windows-x86)
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

--------------- T H R E A D ---------------

Current thread (0x64142800): JavaThread "CompilerThread0" daemon [_thread_in_native, id=8176, stack(0x64330000,0x64380000)]

Stack: [0x64330000,0x64380000]
[error occurred during error reporting (printing stack bounds), id 0x80000001]


Current CompileTask:
C2:1420 ! org.jboss.ejb3.Ejb3Deployment.deployElement(Ljava/io/InputStream;Lorg/jboss/ejb3/Ejb3HandlerFactory;Ljavax/naming/InitialContext;)V (66 bytes)

Heap
 PSYoungGen total 217472K, used 176224K [0x539d0000, 0x639d0000, 0x639d0000)
 eden space 170432K, 97% used [0x539d0000,0x5dbec148,0x5e040000)
 from space 47040K, 21% used [0x60be0000,0x615dc210,0x639d0000)
 to space 44672K, 0% used [0x5e040000,0x5e040000,0x60be0000)
 PSOldGen total 786432K, used 200494K [0x239d0000, 0x539d0000, 0x539d0000)
 object space 786432K, 25% used [0x239d0000,0x2fd9b9b0,0x539d0000)
 PSPermGen total 524288K, used 39978K [0x039d0000, 0x239d0000, 0x239d0000)
 object space 524288K, 7% used [0x039d0000,0x060da870,0x239d0000)

So i have fixed the issue decreasing the heap memory :

jvm_args: -Dprogram.name=pdm.bat -Xms768m -Xmx768m -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+Dis
ableExplicitGC

I can now run the test with the same level of users as in JBoss 4.2.3/EJB2 but lower perfromances as heap size is reduced.

To summarize what we have observed :

- Heap = 1024 / Perm = 256 : JVM crashes with 5 users (JRE5/JRE6)
- Heap = 1024 / Perm = 512 : JVM crashes at JBoss startup time (JRE6)
- Heap = 768 / Perm = 512 = OK (JRE6)

I suspect there is an issue in the way JBoss 5 handles Perm size and CompilerThreads in JIT. And I dont think we will migrate our solution to JBoss 5 as a result unless this issue is fixed.

If you would I can mail you the pic of our GC Viewer while the test is running and until the system crashes (I can't attach a file here and because of my company proxy I can't upload the picture on the www)

[ceciiil]

Please correct the above : the correct JVM setting that allows us to run our test is :

jvm_args: -Dprogram.name=pdm.bat -Xms768m -Xmx768m -XX:PermSize=512m -XX:MaxPermSize=512m -XX:+Dis

ableExplicitGC

However, if this works for 5, 10, 20 simulatenous users, it eventualy crashes when we reach 30 users. So this definitely is a showstopper for us.

[peterj]

You were using only 39% of your permgen when it was set to 256M, so upping it to 521M does no good.

Increasing or decreasing the total heap size should not matter. Based on the error message:

requested 35608 bytes for Chunk::new. Out of swap space?

This means that the operating system refused to allocate more memory for the JVM when requested. I suspect a problem with the box you are running on, either a physical problem (bad RAM), a corrupted DLL, or possibly a corrupted driver. Do you have another box you can try this on?

Also, the JVM should never crash regardless of what the Java application does. I doubt that there is anything that is done, or could be changed, in JBossAS 5.0, that would have any bearing on this issue. However, the JVM might crash if the operating system returns an unrecoverable error for a given operation, such as an error indicating a hardware or OS problem.

Finally, since this is a JVM crash, you should report the crash to Sun.

[ceciiil]

Hi Peter,

Thanks for that.

To be honest, I strongly doubt there is any problem on the physical server. I have been using it for a couple of months doing intensive testing on my app running on JBoss 4.2.3 with both JRE5 and JRE6 without encountering any JVM problem. Since this JVM crash on JBoss 5 I have switched back to JBoss 423 and again : no JVM problem.

As far as JVM settings are concerned, I am not saying I can explain, I have just observed and reproduced all the items below :

- Xms1024 Xmx1024 PermGen512 = JVM crash at JBoss startup time
- Xms1024 Xmx1024 PermGen256 = JVM crashes with 5 users after 10 mins
- Xms768 Xmx768 PermGen512 = System up to 20 users and then crashes

I have raised the issue to Sun already, waiting for their feedback. If I have anything meaningful back from them, I'll make sure I post it here.

[ceciiil]

found this : http://jroller.com/rreyelts/date/20040909


After running some tests with the JVM, I've come to the conclusion that Sun's JDK 1.4 allocates about 256K of address space per thread. I also ran some tests with JDK 1.5, which seem to indicate that it was allocating about 1M of address space per thread. Those numbers come from an assumption of a 2G address space per process under Windows, and the following statistics for the maximum number of threads I could create:

JDK1.4
-Xmx750 = 4580 threads.
-Xmx1000 = 3608 threads.
-Xmx1500M = 1663 threads

JDK1.5
-Xmx750M = 1129 threads
-Xmx1000M = 880 threads
-Xmx1500M = 384 threads

As you can see, the maximum number of threads I can create decreases as the heap size gets larger. This is because the JVM immediately reserves the address space specified for the maximum heap size - That's probably because it needs that memory to be contiguous.


Hence the reaon why when I reduce the heap I get more done ...

So I have restarted my JBoss 5 with 1024 Heap, 256 Permsize and limited the Thread size to 128K (-Xss128k). I have run my 5 users scenario (which didn't work previously) and I'm currently running the 20.

My 2 cents : it could just be that JBoss 5 creates far more threads than JBoss 4.2.3 used to.

[ceciiil]

I have used visualVM to count the active threads when I start the server and test with 10 users :

- JBoss 4.2.3 : 120
- JBoss 5.0RC2 = 350

Here is the reason why my server is running out of native memory on JBoss 5.0

[peterj]

Very interesting. You might want to ask about the increased number of threads in the JBossAS 5.0 forum. Include a reference back to this topic.

http://www.jboss.com/index.html?module=bb&op=viewforum&f=287

[higkoo]

Look at this blog:

http://hi.baidu.com/higkoo/blog/item/7276b539e5fa54f83b87cecf.html