6 Replies Latest reply on May 6, 2009 3:41 PM by Chris R

    Out of memory errors

    Chris R Newbie

      In production we are running 4.2.2ga clustered with 4 nodes. We are seeing this error ever few hours. The servers are w2k3 64bit 18gigs of ram with java 1.5.05_05 32bit.


      INFO | jvm 1 | 2009/05/06 08:53:20 | 08:53:20,130 INFO [STDOUT] 2009-05-06 08:53:20,130 ERROR [System] Disabling Refresh for:/xxxxxxxxx request from x.x.x.x
      INFO | jvm 1 | 2009/05/06 08:53:56 | 08:53:56,833 ERROR [[action]] Servlet.service() for servlet action threw exception
      INFO | jvm 1 | 2009/05/06 08:53:56 | java.lang.OutOfMemoryError: unable to create new native thread
      INFO | jvm 1 | 2009/05/06 08:53:56 | at java.lang.Thread.start0(Native Method)
      INFO | jvm 1 | 2009/05/06 08:53:56 | at java.lang.Thread.start(Thread.java:574)
      INFO | jvm 1 | 2009/05/06 08:53:56 | at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:833)
      INFO | jvm 1 | 2009/05/06 08:53:56 | at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1025)
      INFO | jvm 1 | 2009/05/06 08:53:56 | at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1038)
      INFO | jvm 1 | 2009/05/06 08:53:56 | at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:405)
      INFO | jvm 1 | 2009/05/06 08:53:56 | at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:170)
      INFO | jvm 1 | 2009/05/06 08:53:56 | at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:836)
      INFO | jvm 1 | 2009/05/06 08:53:56 | at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:230)
      INFO | jvm 1 | 2009/05/06 08:53:56 | at com.sun.xml.rpc.client.http.HttpClientTransport.writeMessageToConnection(HttpClientTransport.java:382)
      INFO | jvm 1 | 2009/05/06 08:53:56 | at com.sun.xml.rpc.client.http.HttpClientTransport.invoke(HttpClientTransport.java:86)
      INFO | jvm 1 | 2009/05/06 08:53:56 | at com.sun.xml.rpc.client.StreamingSender._send(StreamingSender.java:92)
      
      
      INFO | jvm 1 | 2009/05/06 08:54:02 | 08:54:02,364 ERROR [[action]] Servlet.service() for servlet action threw exception
      INFO | jvm 1 | 2009/05/06 08:54:02 | java.lang.OutOfMemoryError: unable to create new native thread
      INFO | jvm 1 | 2009/05/06 08:54:02 | at java.lang.Thread.start0(Native Method)
      INFO | jvm 1 | 2009/05/06 08:54:02 | at java.lang.Thread.start(Thread.java:574)
      INFO | jvm 1 | 2009/05/06 08:54:02 | at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:833)
      INFO | jvm 1 | 2009/05/06 08:54:02 | at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1025)
      INFO | jvm 1 | 2009/05/06 08:54:02 | at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1038)
      INFO | jvm 1 | 2009/05/06 08:54:02 | at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:405)
      INFO | jvm 1 | 2009/05/06 08:54:02 | at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:170)
      INFO | jvm 1 | 2009/05/06 08:54:02 | at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:836)
      INFO | jvm 1 | 2009/05/06 08:54:02 | at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:230)
      INFO | jvm 1 | 2009/05/06 08:54:02 | at com.sun.xml.rpc.client.http.HttpClientTransport.writeMessageToConnection(HttpClientTransport.java:382)
      INFO | jvm 1 | 2009/05/06 08:54:02 | at com.sun.xml.rpc.client.http.HttpClientTransport.invoke(HttpClientTransport.java:86)
      INFO | jvm 1 | 2009/05/06 08:54:02 | at com.sun.xml.rpc.client.StreamingSender._send(StreamingSender.java:92)
      
      INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] Exception in thread "RMI RenewClean-[XXX.XXX.XXX.XXX:1101]"
      INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] java.lang.OutOfMemoryError: unable to create new native thread
      INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] at java.lang.Thread.start0(Native Method)
      INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] at java.lang.Thread.start(Thread.java:574)
      INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] at sun.rmi.transport.tcp.TCPChannel.free(TCPChannel.java:322)
      INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] at sun.rmi.server.UnicastRef.free(UnicastRef.java:395)
      INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] at sun.rmi.server.UnicastRef.done(UnicastRef.java:412)
      INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
      INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] at sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:328)
      INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] at sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:144)
      INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:539)
      INFO | jvm 1 | 2009/05/06 08:31:31 | 08:31:31,381 ERROR [STDERR] at java.lang.Thread.run(Thread.java:595)
      
      
      
      wrapper.java.additional.2=-Xss1m
      wrapper.java.additional.3=-Xms1g
      wrapper.java.additional.4=-Xmx1280m
      wrapper.java.additional.9=-XX:+UseConcMarkSweepGC
      wrapper.java.additional.10=-XX:+CMSPermGenSweepingEnabled
      wrapper.java.additional.11=-XX:+CMSClassUnloadingEnabled
      wrapper.java.additional.12=-XX:MaxPermSize=256m


      any ideas? thanks in advanced!

        • 1. Re: Out of memory errors
          Peter Johnson Master

          This error: "java.lang.OutOfMemoryError: unable to create new native thread" usually means that the operating system has run out of memory with which to allocate the necessary memory for a new thread. Assuming you are running a 32-bit OS or a 32-bit JVM, There are several possibilities for this.

          One possibility is the OS has run out of room in its memory area (usually the upper 1GB or 2GB address space of the application). I have seen this happen is, of example, I boot Windows with the /3GB option - this options leaves only 1GB for Windows memory area which is often not sufficient to hold the handles that Windows uses to manage threads.

          Another possibility is that there is insufficient memory in the app's memory area to allocate the memory required for the new thread. Depending on you OS, each thread could require 1MB or memory (and it looks like you have it set for 1MB). Considering that you are allocating a total of 1.5GB memory to the heap and permgen, that leaves not that much extra room for many threads.

          So you have a few options:

          a) Reduce the memory allocated to each thread. Try using -Xss768k or even -Xss512k. Usually the lowest possible value without getting a stack overflow is what you want.

          b) Decrease the heap size. For example, given your current thread stack memory size, setting -Xmx=1200m would allow 80 more threads to be allocated.

          c) Reduce the maximum number of threads that can be allocated. This involves both the worker threads declared in jboss-service.xml and the htttp (and related) threads declared in the server.xml file. Though this option is not always guaranteed to be effective because there are other locations where threads are created that are not governed by these limits. For example, I'm not sure where, or if, there is a limit on RMI threads, which I think is the type of thread that could not be created.

          • 2. Re: Out of memory errors
            Chris R Newbie

            thanks, i'll give A & B a try, this is also now showing up in the logs:

            INFO | jvm 1 | 2009/05/06 12:36:18 | Exception java.lang.OutOfMemoryError: requested 655360 bytes for GrET* in C:/BUILD_AREA/jdk1.5.0_05/hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap space?
            ERROR | wrapper | 2009/05/06 12:36:19 | JVM exited unexpectedly.
            STATUS | wrapper | 2009/05/06 12:36:23 | Launching a JVM...


            • 3. Re: Out of memory errors
              Peter Johnson Master

               

              requested 655360 bytes for GrET* in C:/BUILD_AREA/jdk1.5.0_05/hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap space?

              Looks like not enough memory to support the thread stack (the second possibility I mentioned).

              Actually, this points out another possibility that I overlooked - you don't have enough memory. That leads to another option to fix this - install more RAM and/or increase the swap space size (usually harder to do because it involves repartitioning your hard drive, or perhaps adding a new hard drive and creating another swap space there).

              • 4. Re: Out of memory errors
                Chris R Newbie

                The box has 18gigs of ram. Currently its only using around 2gigs with 16gigs free. Currently the page file is set for 4096-8096, a second drive isn't an issue.

                • 5. Re: Out of memory errors
                  Peter Johnson Master

                  Well, memory is not an issue. Also I'm not sure why I thought you were using Linux, your first post clearly calls out a 64-bit version of Windows Server 2003. So options a) and b) are still your best bet.

                  • 6. Re: Out of memory errors
                    Chris R Newbie

                    Thanks I modified the wrapper.conf to use to following in hopes it will fix some issues:

                    wrapper.java.additional.2=-Xss768k
                    wrapper.java.additional.3=-Xms1200m
                    wrapper.java.additional.4=-Xmx1200m
                    wrapper.java.additional.9=-XX:+CMSPermGenSweepingEnabled
                    wrapper.java.additional.10=-XX:+CMSClassUnloadingEnabled
                    wrapper.java.additional.11=-XX:+UseParallelGC
                    wrapper.java.additional.12=-XX:MaxPermSize=256m
                    wrapper.java.additional.13=-XX:MaxNewSize=400m
                    wrapper.java.additional.14=-server
                    wrapper.java.additional.15=-Dsun.rmi.dgc.client.gcInterval=1800000
                    wrapper.java.additional.16=-Dsun.rmi.dgc.server.gcInterval=1800000
                    


                    I'm going to restart the service in the middle of the night and see how things go tomorrow. I'll let you know, thanks for the help.