14 Replies Latest reply on Sep 4, 2008 4:19 AM by jhalliday

    Help JBoss JTS OutOfMemoryError happened in PROD env.

    wangliyu

      Hi,
      I'm using JBoss4.0.5 with JTS 4.2.2 in PROD env (Windows 2003sp1) , it happened twice the JTS OutOfMemoryError with XA transaction commit, here is the console log of JBoss:

      2008-08-27 13:29:25,714 WARN [com.arjuna.ats.jts.logging.loggerI18N] [com.arjuna.ats.internal.jts.orbspecific.coordinator.generror] ArjunaTransactionImple.destroyAction caught exception: java.lang.OutOfMemoryError: unable to create new native thread
      2008-08-27 13:31:50,321 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000
      2008-08-27 13:36:31,512 ERROR [jacorb.poa.controller] rid: 25187826 opname: commit invocation: throwable was thrown (java.lang.OutOfMemoryError)
      java.lang.OutOfMemoryError: unable to create new native thread
       at java.lang.Thread.start0(Native Method)
       at java.lang.Thread.start(Thread.java:597)
       at org.jacorb.poa.AOM.remove(AOM.java:365)
       at org.jacorb.poa.POA.deactivate_object(POA.java:895)
       at com.arjuna.orbportability.RootOA.shutdownObject(RootOA.java:265)
       at com.arjuna.ats.internal.jta.resources.jts.orbspecific.XAResourceRecord.removeConnection(XAResourceRecord.java:1395)
       at com.arjuna.ats.internal.jta.resources.jts.orbspecific.XAResourceRecord.commit(XAResourceRecord.java:560)
       at com.arjuna.ArjunaOTS.OTSAbstractRecordPOA._invoke(OTSAbstractRecordPOA.java:184)
       at org.jacorb.poa.RequestProcessor.invokeOperation(RequestProcessor.java:299)
       at org.jacorb.poa.RequestProcessor.process(RequestProcessor.java:593)
       at org.jacorb.poa.RequestProcessor.run(RequestProcessor.java:735)
      2008-08-27 13:36:31,512 ERROR [jacorb.poa.controller] rid: 25187900 opname: commit invocation: throwable was thrown (java.lang.OutOfMemoryError)
      java.lang.OutOfMemoryError: unable to create new native thread
       at java.lang.Thread.start0(Native Method)
       at java.lang.Thread.start(Thread.java:597)
       at org.jacorb.poa.AOM.remove(AOM.java:365)
       at org.jacorb.poa.POA.deactivate_object(POA.java:895)
       at com.arjuna.orbportability.RootOA.shutdownObject(RootOA.java:265)
       at com.arjuna.ats.internal.jta.resources.jts.orbspecific.XAResourceRecord.removeConnection(XAResourceRecord.java:1395)
       at com.arjuna.ats.internal.jta.resources.jts.orbspecific.XAResourceRecord.commit(XAResourceRecord.java:560)
       at com.arjuna.ArjunaOTS.OTSAbstractRecordPOA._invoke(OTSAbstractRecordPOA.java:184)
       at org.jacorb.poa.RequestProcessor.invokeOperation(RequestProcessor.java:299)
       at org.jacorb.poa.RequestProcessor.process(RequestProcessor.java:593)
       at org.jacorb.poa.RequestProcessor.run(RequestProcessor.java:735)
      2008-08-27 13:36:31,528 WARN [jacorb.poa] POA RootPOAoid:
      0B 39 29 36 15 39 05 22 11 4D 1A 0B 4D .9)6.9.".M..M
      cannot process request, because object is already in the deactivation process
      2008-08-27 13:36:31,528 WARN [jacorb.poa.controller] rid: 25187910 opname: shouldMerge request rejected with exception:
      2008-08-27 13:36:31,528 WARN [jacorb.poa] POA RootPOAoid:
      0B 39 29 36 15 39 05 22 11 4D 1A 0B 4D .9)6.9.".M..M
      cannot process request, because object is already in the deactivation process
      2008-08-27 13:36:31,528 WARN [jacorb.poa.controller] rid: 25187912 opname: shouldReplace request rejected with exception:
      2008-08-27 13:36:31,528 WARN [jacorb.poa] POA RootPOAoid:
      0B 39 29 36 15 39 05 22 11 4D 1A 0B 4D .9)6.9.".M..M
      cannot process request, because object is already in the deactivation process
      2008-08-27 13:36:31,528 WARN [jacorb.poa.controller] rid: 25187914 opname: shouldAdd request rejected with exception:
      2008-08-27 13:36:31,528 WARN [jacorb.poa] POA RootPOAoid:
      0B 39 29 36 15 39 05 22 11 4D 1A 0B 4D .9)6.9.".M..M
      cannot process request, because object is already in the deactivation process
      2008-08-27 13:36:31,528 WARN [jacorb.poa.controller] rid: 25187916 opname: shouldAlter request rejected with exception:
      2008-08-27 13:36:31,543 WARN [com.arjuna.ats.jts.logging.loggerI18N] [com.arjuna.ats.internal.jts.orbspecific.coordinator.generror] ArjunaTransactionImple.destroyAction caught exception: java.lang.OutOfMemoryError: unable to create new native thread
      2008-08-27 13:36:32,184 ERROR [jacorb.poa.controller] rid: 25187906 opname: commit invocation: throwable was thrown (java.lang.OutOfMemoryError)
      java.lang.OutOfMemoryError: unable to create new native thread
       at java.lang.Thread.start0(Native Method)
       at java.lang.Thread.start(Thread.java:597)
       at org.jacorb.poa.AOM.remove(AOM.java:365)
       at org.jacorb.poa.POA.deactivate_object(POA.java:895)
       at com.arjuna.orbportability.RootOA.shutdownObject(RootOA.java:265)
       at com.arjuna.ats.internal.jta.resources.jts.orbspecific.XAResourceRecord.removeConnection(XAResourceRecord.java:1395)
       at com.arjuna.ats.internal.jta.resources.jts.orbspecific.XAResourceRecord.commit(XAResourceRecord.java:560)
       at com.arjuna.ArjunaOTS.OTSAbstractRecordPOA._invoke(OTSAbstractRecordPOA.java:184)
       at org.jacorb.poa.RequestProcessor.invokeOperation(RequestProcessor.java:299)
       at org.jacorb.poa.RequestProcessor.process(RequestProcessor.java:593)
       at org.jacorb.poa.RequestProcessor.run(RequestProcessor.java:735)
      2008-08-27 13:36:32,294 WARN [com.arjuna.ats.jts.logging.loggerI18N] [com.arjuna.ats.internal.jts.orbspecific.coordinator.generror] ArjunaTransactionImple.destroyAction caught exception: java.lang.OutOfMemoryError: unable to create new native thread
      2008-08-27 13:36:33,185 ERROR [STDERR] Exception in thread "RequestController-1"
      2008-08-27 13:36:33,185 ERROR [STDERR] java.lang.OutOfMemoryError: unable to create new native thread
      2008-08-27 13:36:33,185 ERROR [STDERR] at java.lang.Thread.start0(Native Method)
      2008-08-27 13:36:33,185 ERROR [STDERR] at java.lang.Thread.start(Thread.java:597)
      2008-08-27 13:36:33,185 ERROR [STDERR] at org.jacorb.poa.RPPoolManager.addProcessor(RPPoolManager.java:99)
      2008-08-27 13:36:33,185 ERROR [STDERR] at org.jacorb.poa.RPPoolManager.getProcessor(RPPoolManager.java:181)
      2008-08-27 13:36:33,185 ERROR [STDERR] at org.jacorb.poa.RequestController.processRequest(RequestController.java:382)
      2008-08-27 13:36:33,185 ERROR [STDERR] at org.jacorb.poa.RequestController.run(RequestController.java:479)
      
      


      I killed the JBoss session, the JVM create the memory report in the error log file:

      Other Threads:
      =>0x2bd6e000 VMThread [stack: 0x2c1d0000,0x2c220000] [id=7364]
       0x2c00dc00 WatcherThread [stack: 0x2c450000,0x2c4a0000] [id=11848]
      
      VM state:at safepoint (normal execution)
      
      VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
      [0x00285a60/0x00000710] Threads_lock - owner thread: 0x2bd6e000
      [0x00285c00/0x000006d0] Heap_lock - owner thread: 0x45c2a400
      
      Heap
       PSYoungGen total 23872K, used 96K [0x280f0000, 0x29d60000, 0x2b9d0000)
       eden space 23744K, 0% used [0x280f0000,0x280f0000,0x29820000)
       from space 128K, 75% used [0x29820000,0x29838000,0x29840000)
       to space 2816K, 0% used [0x29aa0000,0x29aa0000,0x29d60000)
       PSOldGen total 233024K, used 54209K [0x0b9d0000, 0x19d60000, 0x280f0000)
       object space 233024K, 23% used [0x0b9d0000,0x0eec07c8,0x19d60000)
       PSPermGen total 39040K, used 38937K [0x039d0000, 0x05ff0000, 0x0b9d0000)
       object space 39040K, 99% used [0x039d0000,0x05fd6750,0x05ff0000)
      


      Looks the memory was good.
      The XA transaction involved with both Informix DB and a MainFrame calls.

      Both Informix Query timeout and Mainframe call's timeout are 60 sec.
      The XA transaction reaper time is 120 sec.

      And I looked the application log, all the transactions' timing are normal until suddenly it stop responding.

      Can anyone help or have same issue, please let me know?

      Thanks,
      Liyu

        • 1. Re: Help JBoss JTS OutOfMemoryError happened in PROD env.
          wangliyu

          The JBoss4.0.5 was running on JDK1.6.0u06 and the startup option is:
          -Xms256m -Xmx512m -XX:MaxPermSize=128m -XX:-HeapDumpOnOutOfMemoryError -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000

          the JVM complained:

          #
          # An unexpected error has been detected by Java Runtime Environment:
          #
          # java.lang.OutOfMemoryError: requested 128000 bytes for GrET in C:\BUILD_AREA\jdk6_06\hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap space?
          #
          # Internal Error (allocation.inline.hpp:42), pid=11136, tid=7364
          # Error: GrET in C:\BUILD_AREA\jdk6_06\hotspot\src\share\vm\utilities\growableArray.cpp
          #
          # Java VM: Java HotSpot(TM) Server VM (10.0-b22 mixed mode windows-x86)
          # If you would like to submit a bug report, please visit:
          # http://java.sun.com/webapps/bugreport/crash.jsp
          #
          
          --------------- T H R E A D ---------------
          
          Current thread (0x2bd6e000): VMThread [stack: 0x2c1d0000,0x2c220000] [id=7364]
          
          Stack: [0x2c1d0000,0x2c220000]
          [error occurred during error reporting (printing stack bounds), id 0x80000001]
          
          VM_Operation (0x322fe0b4): ParallelGCSystemGC, mode: safepoint, requested by thread 0x45c2a400
          
          
          


          • 2. Re: Help JBoss JTS OutOfMemoryError happened in PROD env.
            marklittle

            4.0.5 is not supported on JDK6, so this isn't really surprising.

            • 3. Re: Help JBoss JTS OutOfMemoryError happened in PROD env.
              wangliyu

              we'll move back to JDK1.5.0u07 this weekend, but I'm still not convinced that JDK cause the problem.

              • 4. Re: Help JBoss JTS OutOfMemoryError happened in PROD env.
                wangliyu

                also seems to me something in JTS cause the problem, the problem happened twice, first time it was up and running for 14 days before crash, the second time it up and running for 13 days.
                I used jmap to generate the heap dump at running time, and looks the application doesn't have any memroy leak.
                Both 2 times they were happened in the commit() method, the EJB is container managed transaction (requred).

                • 5. Re: Help JBoss JTS OutOfMemoryError happened in PROD env.
                  wangliyu

                   

                  "mark.little@jboss.com" wrote:
                  4.0.5 is not supported on JDK6, so this isn't really surprising.


                  So you are saying that JDK6 is cause the problem?

                  • 6. Re: Help JBoss JTS OutOfMemoryError happened in PROD env.
                    marklittle

                    I didn't say that JDK6 was the cause of your current error, only that I'm not surprised it isn't working.

                    You should consider moving to a later version of JBossAS. Even better, consider getting an EAP subscription. EAP 5.0 will have support for JTS in it too.

                    • 7. Re: Help JBoss JTS OutOfMemoryError happened in PROD env.
                      wangliyu

                       

                      "mark.little@jboss.com" wrote:
                      I didn't say that JDK6 was the cause of your current error, only that I'm not surprised it isn't working.

                      You should consider moving to a later version of JBossAS. Even better, consider getting an EAP subscription. EAP 5.0 will have support for JTS in it too.


                      The reason why we don't move to latest JBoss is because 2 reasons:
                      the client is some old system taht run with JDK1.3,
                      the JCA involved in the XA is only support JBoss4.0 or 4.2 and 4.0 is strongly recommend.

                      and the JBoss4.0.5 + JTS 4.2.2 + JDK6 is running fine, even the heaviest load day, it still works fine, only some normal day that during the afternoon (which is very light trafic time), it hung, and it report OOME in the destroyAction() (last step of the transaction), I took a look, in the code, JTS catch OOME, and if it does, JTS will do System.gc() and log a WARN. but after that, the following Transaction will also hung.

                      and JDK create the error report, in there you can see the heap memory was OK.

                      • 8. Re: Help JBoss JTS OutOfMemoryError happened in PROD env.
                        jhalliday

                        That looks like it's the request for a block of native (rather than java heap) memory to hold thread stack that is failing. Is the system short on virtual memory or running other process that may reserve but not fully allocate large chunks of it? This problem occurs if the system's free memory is small or heavily fragmented, even where space remains free in the JVM's heap. It may also be hitting per-user limits in the O/S.

                        • 9. Re: Help JBoss JTS OutOfMemoryError happened in PROD env.
                          skajotde
                          • 10. Re: Help JBoss JTS OutOfMemoryError happened in PROD env.
                            marklittle

                             

                            "liyu2008" wrote:

                            and the JBoss4.0.5 + JTS 4.2.2 + JDK6 is running fine, even the heaviest load day, it still works fine, only some normal day that during the afternoon (which is very light trafic time), it hung, and it report OOME in the destroyAction() (last step of the transaction), I took a look, in the code, JTS catch OOME, and if it does, JTS will do System.gc() and log a WARN. but after that, the following Transaction will also hung.


                            You're caught between a couple of competing problems. The first is that we don't support JDK 6 with JBossTS at this time. The second is that we won't support JTS from JBossTS on anything other than JBossAS 5.0.

                            • 11. Re: Help JBoss JTS OutOfMemoryError happened in PROD env.
                              wangliyu

                               

                              "jhalliday" wrote:
                              That looks like it's the request for a block of native (rather than java heap) memory to hold thread stack that is failing. Is the system short on virtual memory or running other process that may reserve but not fully allocate large chunks of it? This problem occurs if the system's free memory is small or heavily fragmented, even where space remains free in the JVM's heap. It may also be hitting per-user limits in the O/S.


                              Hi, Thanks for responding this post, due to the historical reason, the application do a lot of String manipulation, it could be the memory heavily fragmented. I'll try to reduce the string, stringbuffer stuff.

                              • 12. Re: Help JBoss JTS OutOfMemoryError happened in PROD env.
                                wangliyu

                                 

                                "mark.little@jboss.com" wrote:
                                "liyu2008" wrote:

                                and the JBoss4.0.5 + JTS 4.2.2 + JDK6 is running fine, even the heaviest load day, it still works fine, only some normal day that during the afternoon (which is very light trafic time), it hung, and it report OOME in the destroyAction() (last step of the transaction), I took a look, in the code, JTS catch OOME, and if it does, JTS will do System.gc() and log a WARN. but after that, the following Transaction will also hung.


                                You're caught between a couple of competing problems. The first is that we don't support JDK 6 with JBossTS at this time. The second is that we won't support JTS from JBossTS on anything other than JBossAS 5.0.


                                Is JBossAS 5.0 GA or still Beta?

                                • 13. Re: Help JBoss JTS OutOfMemoryError happened in PROD env.
                                  wangliyu

                                   

                                  "skajotde" wrote:
                                  Maybe this helps: http://wiki.jboss.org/wiki/OutOfMemoryExceptionWhenCannotCreateThread


                                  Thanks for reply the post, unfortunately , I'm running JBoss on Windows 2003 SP1, not sure what's the default threadstacksize on Windows 2003?
                                  and how do I check the max number of user processes ?

                                  • 14. Re: Help JBoss JTS OutOfMemoryError happened in PROD env.
                                    jhalliday

                                    > not sure what's the default threadstacksize on Windows 2003?
                                    > and how do I check the max number of user processes ?

                                    please learn to use google. This is not a microsoft o/s support forum and neither of those questions is transactions specific.

                                    P.S. what makes you think the String/StringBuffer use will fragment the system's native memory?