Out of memory errors

[creynolds]

In production we are running 4.2.2ga clustered with 4 nodes. We are seeing this error ever few hours. The servers are w2k3 64bit 18gigs of ram with java 1.5.05_05 32bit.

INFO | jvm 1 | 2009/05/06 08:53:20 | 08:53:20,130 INFO [STDOUT] 2009-05-06 08:53:20,130 ERROR [System] Disabling Refresh for:/xxxxxxxxx request from x.x.x.x
INFO | jvm 1 | 2009/05/06 08:53:56 | 08:53:56,833 ERROR [[action]] Servlet.service() for servlet action threw exception
INFO | jvm 1 | 2009/05/06 08:53:56 | java.lang.OutOfMemoryError: unable to create new native thread
INFO | jvm 1 | 2009/05/06 08:53:56 | at java.lang.Thread.start0(Native Method)
INFO | jvm 1 | 2009/05/06 08:53:56 | at java.lang.Thread.start(Thread.java:574)
INFO | jvm 1 | 2009/05/06 08:53:56 | at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:833)
INFO | jvm 1 | 2009/05/06 08:53:56 | at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1025)
INFO | jvm 1 | 2009/05/06 08:53:56 | at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1038)
INFO | jvm 1 | 2009/05/06 08:53:56 | at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:405)
INFO | jvm 1 | 2009/05/06 08:53:56 | at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:170)
INFO | jvm 1 | 2009/05/06 08:53:56 | at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:836)
INFO | jvm 1 | 2009/05/06 08:53:56 | at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:230)
INFO | jvm 1 | 2009/05/06 08:53:56 | at com.sun.xml.rpc.client.http.HttpClientTransport.writeMessageToConnection(HttpClientTransport.java:382)
INFO | jvm 1 | 2009/05/06 08:53:56 | at com.sun.xml.rpc.client.http.HttpClientTransport.invoke(HttpClientTransport.java:86)
INFO | jvm 1 | 2009/05/06 08:53:56 | at com.sun.xml.rpc.client.StreamingSender._send(StreamingSender.java:92)


INFO | jvm 1 | 2009/05/06 08:54:02 | 08:54:02,364 ERROR [[action]] Servlet.service() for servlet action threw exception
INFO | jvm 1 | 2009/05/06 08:54:02 | java.lang.OutOfMemoryError: unable to create new native thread
INFO | jvm 1 | 2009/05/06 08:54:02 | at java.lang.Thread.start0(Native Method)
INFO | jvm 1 | 2009/05/06 08:54:02 | at java.lang.Thread.start(Thread.java:574)
INFO | jvm 1 | 2009/05/06 08:54:02 | at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:833)
INFO | jvm 1 | 2009/05/06 08:54:02 | at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1025)
INFO | jvm 1 | 2009/05/06 08:54:02 | at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1038)
INFO | jvm 1 | 2009/05/06 08:54:02 | at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:405)
INFO | jvm 1 | 2009/05/06 08:54:02 | at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:170)
INFO | jvm 1 | 2009/05/06 08:54:02 | at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:836)
INFO | jvm 1 | 2009/05/06 08:54:02 | at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:230)
INFO | jvm 1 | 2009/05/06 08:54:02 | at com.sun.xml.rpc.client.http.HttpClientTransport.writeMessageToConnection(HttpClientTransport.java:382)
INFO | jvm 1 | 2009/05/06 08:54:02 | at com.sun.xml.rpc.client.http.HttpClientTransport.invoke(HttpClientTransport.java:86)
INFO | jvm 1 | 2009/05/06 08:54:02 | at com.sun.xml.rpc.client.StreamingSender._send(StreamingSender.java:92)

INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] Exception in thread "RMI RenewClean-[XXX.XXX.XXX.XXX:1101]"
INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] java.lang.OutOfMemoryError: unable to create new native thread
INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] at java.lang.Thread.start0(Native Method)
INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] at java.lang.Thread.start(Thread.java:574)
INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] at sun.rmi.transport.tcp.TCPChannel.free(TCPChannel.java:322)
INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] at sun.rmi.server.UnicastRef.free(UnicastRef.java:395)
INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] at sun.rmi.server.UnicastRef.done(UnicastRef.java:412)
INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] at sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:328)
INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] at sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:144)
INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:539)
INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] at java.lang.Thread.run(Thread.java:595)



wrapper.java.additional.2=-Xss1m
wrapper.java.additional.3=-Xms1g
wrapper.java.additional.4=-Xmx1280m
wrapper.java.additional.9=-XX:+UseConcMarkSweepGC
wrapper.java.additional.10=-XX:+CMSPermGenSweepingEnabled
wrapper.java.additional.11=-XX:+CMSClassUnloadingEnabled
wrapper.java.additional.12=-XX:MaxPermSize=256m

any ideas? thanks in advanced!

[peterj]

This error: "java.lang.OutOfMemoryError: unable to create new native thread" usually means that the operating system has run out of memory with which to allocate the necessary memory for a new thread. Assuming you are running a 32-bit OS or a 32-bit JVM, There are several possibilities for this.

One possibility is the OS has run out of room in its memory area (usually the upper 1GB or 2GB address space of the application). I have seen this happen is, of example, I boot Windows with the /3GB option - this options leaves only 1GB for Windows memory area which is often not sufficient to hold the handles that Windows uses to manage threads.

Another possibility is that there is insufficient memory in the app's memory area to allocate the memory required for the new thread. Depending on you OS, each thread could require 1MB or memory (and it looks like you have it set for 1MB). Considering that you are allocating a total of 1.5GB memory to the heap and permgen, that leaves not that much extra room for many threads.

So you have a few options:

a) Reduce the memory allocated to each thread. Try using -Xss768k or even -Xss512k. Usually the lowest possible value without getting a stack overflow is what you want.

b) Decrease the heap size. For example, given your current thread stack memory size, setting -Xmx=1200m would allow 80 more threads to be allocated.

c) Reduce the maximum number of threads that can be allocated. This involves both the worker threads declared in jboss-service.xml and the htttp (and related) threads declared in the server.xml file. Though this option is not always guaranteed to be effective because there are other locations where threads are created that are not governed by these limits. For example, I'm not sure where, or if, there is a limit on RMI threads, which I think is the type of thread that could not be created.

[creynolds]

thanks, i'll give A & B a try, this is also now showing up in the logs:

INFO | jvm 1 | 2009/05/06 12:36:18 | Exception java.lang.OutOfMemoryError: requested 655360 bytes for GrET* in C:/BUILD_AREA/jdk1.5.0_05/hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap space?
ERROR | wrapper | 2009/05/06 12:36:19 | JVM exited unexpectedly.
STATUS | wrapper | 2009/05/06 12:36:23 | Launching a JVM...

[peterj]

requested 655360 bytes for GrET* in C:/BUILD_AREA/jdk1.5.0_05/hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap space?

Looks like not enough memory to support the thread stack (the second possibility I mentioned).

Actually, this points out another possibility that I overlooked - you don't have enough memory. That leads to another option to fix this - install more RAM and/or increase the swap space size (usually harder to do because it involves repartitioning your hard drive, or perhaps adding a new hard drive and creating another swap space there).

[creynolds]

The box has 18gigs of ram. Currently its only using around 2gigs with 16gigs free. Currently the page file is set for 4096-8096, a second drive isn't an issue.

[peterj]

Well, memory is not an issue. Also I'm not sure why I thought you were using Linux, your first post clearly calls out a 64-bit version of Windows Server 2003. So options a) and b) are still your best bet.

[creynolds]

Thanks I modified the wrapper.conf to use to following in hopes it will fix some issues:

wrapper.java.additional.2=-Xss768k
wrapper.java.additional.3=-Xms1200m
wrapper.java.additional.4=-Xmx1200m
wrapper.java.additional.9=-XX:+CMSPermGenSweepingEnabled
wrapper.java.additional.10=-XX:+CMSClassUnloadingEnabled
wrapper.java.additional.11=-XX:+UseParallelGC
wrapper.java.additional.12=-XX:MaxPermSize=256m
wrapper.java.additional.13=-XX:MaxNewSize=400m
wrapper.java.additional.14=-server
wrapper.java.additional.15=-Dsun.rmi.dgc.client.gcInterval=1800000
wrapper.java.additional.16=-Dsun.rmi.dgc.server.gcInterval=1800000

I'm going to restart the service in the middle of the night and see how things go tomorrow. I'll let you know, thanks for the help.