4 Replies Latest reply on Jun 8, 2004 2:43 PM by cgonzalez

    JBoss JMS server hangs after running for an extended period

      We are using JBoss 3.2.3 on Windows 2000 Server runnning Sun JDK 1.4.2, with heap size = 256M
      Here is the beginning of our boot log:
      11:25:36,621 INFO [ServerInfo] Java version: 1.4.2,Sun Microsystems Inc.
      11:25:36,621 INFO [ServerInfo] Java VM: Java HotSpot(TM) Client VM 1.4.2-b28,Sun Microsystems Inc.
      11:25:36,621 INFO [ServerInfo] OS-System: Windows 2000 5.0,x86
      11:25:36,621 DEBUG [ServerInfo] Full System Properties Dump
      11:25:36,637 DEBUG [ServerInfo] java.runtime.name: Java(TM) 2 Runtime Environment, Standard Edition
      11:25:36,637 DEBUG [ServerInfo] jboss.server.base.dir: D:\jboss-3.2.3\server

      The JMS application has one Queue and 3 pools of MDBs using message selectors resident on the server. Also (this is the interesting part) we have 2 pools of MDBs that are resident on two other external servers that are bound to the same JBoss Queue.

      We have two different scenarios.
      Scenario 1:
      When we let the system run for extended periods of time, approx 12 hours , we notice that the JBoss server completely hangs up.
      - the JBoss console is no longer accessible
      - the server.log is no longer written to
      - the server doesn't grant any more connections of any kind (JNDI,JMS,HTTP)

      Scenario 2:
      After the server has been running for extended periods often times time we note the following error keeps getting posted to our server.log:
      org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.IOException: ping timeout.)
      at org.jboss.mq.Connection.asynchFailure(Connection.java:718)
      at org.jboss.mq.Connection$PingTask.run(Connection.java:1311)
      at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
      at java.lang.Thread.run(Thread.java:534)
      Caused by: java.io.IOException: ping timeout.
      at org.jboss.mq.Connection$PingTask.run(Connection.java:1303)
      ... 2 more

      When this happens, the server also stops responding to all requests as above, with the only difference being , the ping error listed above is still constantly written to the log.

      I've noted that many people suggest that we implement the javax.jms.Connection.setExceptionListener() interface in their clients, but I believe that I CANT do this from an MDB since the container is supposed to be managing my JMS connections. Is this true?

      Also, we suspect that the remote MDBs (the MDBs that are resident on remote servers) may be the cause. Do we need to implement any reconnection logic for these remote MDB's?

      Thanks in advance.

        • 1. Re: JBoss JMS server hangs after running for an extended per

          MDBs automatically reconnect if the connection fails.

          Does JBoss respond to a thread dump request see "READ THIS FIRST"
          If not your JVM has locked up.

          You should also use search, where you will discover that Windows
          default networking configuration isn't necessarily well configured.

          You would also have discovered my first two points had you used search
          and I could get on with implementing interesting JBoss features rather
          than wasting time here.

          • 2. Re: JBoss JMS server hangs after running for an extended per

            Also do you know about "Quick Edit"?

            • 3. Re: JBoss JMS server hangs after running for an extended per

              Just to close this issue:
              It looks like it was OptimizeIt. As an FYI to everyone, when JBoss crashes it writes a file Hs_err_pidXXX to your <jboss-server-home> directory. Looking at the file, it was obvious what the cause was. I removed the Optimize it libraries from our classpath and it seems to have fixed the lockup problem.

              Here is our log:

              An unexpected exception has been detected in native code outside the VM.
              Unexpected Signal : EXCEPTION_ACCESS_VIOLATION (0xc0000005) occurred at PC=0x1C5F9A64
              Function=oiiGetToolID+0x9904
              Library=d:\Borland\JavaStudio\OptimizeitEntSuite\lib\pri.dll

              Current Java thread:
              at intuitive.audit.profiler.OIContext.stringBufferAllocatorEntered0(Native Method)
              at intuitive.audit.profiler.OIContext.stringBufferAllocatorEntered(OIContext.java:257)
              at java.util.LinkedList$ListItr.(LinkedList.java:468)
              at java.util.LinkedList.listIterator(LinkedList.java:459)
              at org.opennms.protocols.snmp.SnmpTimer$Scheduler.run(SnmpTimer.java:159)
              - locked <0x11060bd8> (a java.lang.Object)
              at java.lang.Thread.run(Thread.java:534)

              Dynamic libraries:
              0x00400000 - 0x00406000 d:\j2sdk1.4.2\bin\java.exe
              0x77F80000 - 0x77FFD000 C:\WINNT\system32\ntdll.dll
              0x7C2D0000 - 0x7C332000 C:\WINNT\system32\ADVAPI32.dll
              0x7C570000 - 0x7C628000 C:\WINNT\system32\KERNEL32.DLL
              0x77D30000 - 0x77DA1000 C:\WINNT\system32\RPCRT4.DLL
              0x78000000 - 0x78045000 C:\WINNT\system32\MSVCRT.dll
              0x08000000 - 0x08136000 d:\j2sdk1.4.2\jre\bin\client\jvm.dll
              0x77E10000 - 0x77E75000 C:\WINNT\system32\USER32.dll
              0x77F40000 - 0x77F7E000 C:\WINNT\system32\GDI32.DLL
              0x77570000 - 0x775A0000 C:\WINNT\system32\WINMM.dll
              0x10000000 - 0x10007000 d:\j2sdk1.4.2\jre\bin\hpi.dll
              0x007C0000 - 0x007CE000 d:\j2sdk1.4.2\jre\bin\verify.dll
              0x007D0000 - 0x007E8000 d:\j2sdk1.4.2\jre\bin\java.dll
              0x007F0000 - 0x007FD000 d:\j2sdk1.4.2\jre\bin\zip.dll
              0x1C1B0000 - 0x1C1CB000 d:\Borland\JavaStudio\OptimizeitEntSuite\lib\oii.dll
              0x1C5E0000 - 0x1C613000 d:\Borland\JavaStudio\OptimizeitEntSuite\lib\pri.dll
              0x1CB30000 - 0x1CB4C000 d:\j2sdk1.4.2\jre\bin\jdwp.dll
              0x6D1D0000 - 0x6D1D5000 C:\Program Files\Oracle\jre\1.3.1\bin\dt_socket.dll
              0x75030000 - 0x75044000 C:\WINNT\system32\ws2_32.dll
              0x75020000 - 0x75028000 C:\WINNT\system32\WS2HELP.DLL
              0x74FD0000 - 0x74FEE000 C:\WINNT\system32\msafd.dll
              0x75010000 - 0x75017000 C:\WINNT\System32\wshtcpip.dll
              0x20E60000 - 0x20F6A000 D:\j2sdk1.4.2\jre\bin\awt.dll
              0x77800000 - 0x7781E000 C:\WINNT\system32\WINSPOOL.DRV
              0x76620000 - 0x76630000 C:\WINNT\system32\MPR.DLL
              0x75E60000 - 0x75E7A000 C:\WINNT\system32\IMM32.dll
              0x77A50000 - 0x77B3F000 C:\WINNT\system32\ole32.dll
              0x20F70000 - 0x20FC0000 D:\j2sdk1.4.2\jre\bin\fontmanager.dll
              0x20FC0000 - 0x20FCF000 D:\j2sdk1.4.2\jre\bin\net.dll
              0x782C0000 - 0x782CC000 C:\WINNT\System32\rnr20.dll
              0x77980000 - 0x779A4000 C:\WINNT\system32\DNSAPI.DLL
              0x75050000 - 0x75058000 C:\WINNT\system32\WSOCK32.DLL
              0x77340000 - 0x77353000 C:\WINNT\system32\iphlpapi.dll
              0x77520000 - 0x77525000 C:\WINNT\system32\ICMP.DLL
              0x77320000 - 0x77337000 C:\WINNT\system32\MPRAPI.DLL
              0x75150000 - 0x7515F000 C:\WINNT\system32\SAMLIB.DLL
              0x75170000 - 0x751BF000 C:\WINNT\system32\NETAPI32.DLL
              0x7C340000 - 0x7C34F000 C:\WINNT\system32\SECUR32.DLL
              0x751C0000 - 0x751C6000 C:\WINNT\system32\NETRAP.DLL
              0x77950000 - 0x7797A000 C:\WINNT\system32\WLDAP32.DLL
              0x779B0000 - 0x77A4B000 C:\WINNT\system32\OLEAUT32.DLL
              0x773B0000 - 0x773DF000 C:\WINNT\system32\ACTIVEDS.DLL
              0x77380000 - 0x773A3000 C:\WINNT\system32\ADSLDPC.DLL
              0x77830000 - 0x7783E000 C:\WINNT\system32\RTUTILS.DLL
              0x77880000 - 0x7790E000 C:\WINNT\system32\SETUPAPI.DLL
              0x7C0F0000 - 0x7C151000 C:\WINNT\system32\USERENV.DLL
              0x774E0000 - 0x77513000 C:\WINNT\system32\RASAPI32.DLL
              0x774C0000 - 0x774D1000 C:\WINNT\system32\RASMAN.DLL
              0x77530000 - 0x77552000 C:\WINNT\system32\TAPI32.DLL
              0x77B50000 - 0x77BD9000 C:\WINNT\system32\COMCTL32.DLL
              0x63180000 - 0x631C8000 C:\WINNT\system32\SHLWAPI.DLL
              0x77360000 - 0x77379000 C:\WINNT\system32\DHCPCSVC.DLL
              0x777E0000 - 0x777E8000 C:\WINNT\System32\winrnr.dll
              0x777F0000 - 0x777F5000 C:\WINNT\system32\rasadhlp.dll
              0x21480000 - 0x214C3000 D:\Borland\JavaStudio\OptimizeitEntSuite\lib\auditjni.dll
              0x2FC70000 - 0x2FC75000 D:\j2sdk1.4.2\jre\bin\rmi.dll
              0x62F00000 - 0x62F0F000 D:\oracle\ora92\bin\ocijdbc9.dll
              0x610A0000 - 0x6113F000 D:\oracle\ora92\bin\oracore9.dll
              0x612A0000 - 0x61319000 D:\oracle\ora92\bin\oranls9.dll
              0x613A0000 - 0x613B1000 D:\oracle\ora92\bin\oraunls9.dll
              0x60600000 - 0x60784000 D:\oracle\ora92\bin\oraclient9.dll
              0x60800000 - 0x60806000 D:\oracle\ora92\bin\oravsn9.dll
              0x60500000 - 0x60590000 D:\oracle\ora92\bin\oracommon9.dll
              0x30300000 - 0x30557000 D:\oracle\ora92\bin\orageneric9.dll
              0x62FC0000 - 0x6303D000 D:\oracle\ora92\bin\oraxml9.dll
              0x780A0000 - 0x780B2000 C:\WINNT\system32\MSVCIRT.dll
              0x630F0000 - 0x63110000 D:\oracle\ora92\bin\oraxsd9.dll
              0x615A0000 - 0x61629000 D:\oracle\ora92\bin\orannzsbb9.dll
              0x61480000 - 0x61534000 D:\oracle\ora92\bin\oran9.dll
              0x61400000 - 0x6142C000 D:\oracle\ora92\bin\oranl9.dll
              0x61960000 - 0x61971000 D:\oracle\ora92\bin\oranldap9.dll
              0x62000000 - 0x62024000 D:\oracle\ora92\bin\oraldapclnt9.dll
              0x616B0000 - 0x616C7000 D:\oracle\ora92\bin\orancrypt9.dll
              0x62300000 - 0x6233E000 D:\oracle\ora92\bin\ORATRACE9.dll
              0x61730000 - 0x61766000 D:\oracle\ora92\bin\oranro9.dll
              0x617C0000 - 0x617C6000 D:\oracle\ora92\bin\oranhost9.dll
              0x617D0000 - 0x617D6000 D:\oracle\ora92\bin\oranoname9.dll
              0x616A0000 - 0x616A6000 D:\oracle\ora92\bin\orancds9.dll
              0x61820000 - 0x61827000 D:\oracle\ora92\bin\orantns9.dll
              0x64000000 - 0x64007000 D:\oracle\ora92\bin\oranms.dll
              0x64020000 - 0x64030000 D:\oracle\ora92\bin\oranmsp.dll
              0x60A00000 - 0x60D13000 D:\oracle\ora92\bin\orapls9.dll
              0x62500000 - 0x62507000 D:\oracle\ora92\bin\oraslax9.dll
              0x61350000 - 0x61360000 D:\oracle\ora92\bin\orasnls9.dll
              0x60810000 - 0x60816000 D:\oracle\ora92\bin\orawtc9.dll
              0x62600000 - 0x62675000 D:\oracle\ora92\bin\orasql9.dll
              0x61910000 - 0x6191C000 D:\oracle\ora92\bin\orantcp9.dll
              0x61890000 - 0x6189D000 D:\oracle\ora92\bin\orannts9.dll
              0x75500000 - 0x75504000 C:\WINNT\system32\security.dll
              0x78280000 - 0x782B6000 C:\WINNT\system32\kerberos.dll
              0x76670000 - 0x7667E000 C:\WINNT\system32\CRYPTDLL.DLL
              0x77430000 - 0x77440000 C:\WINNT\system32\MSASN1.DLL
              0x77920000 - 0x77943000 C:\WINNT\system32\imagehlp.dll
              0x72A00000 - 0x72A2D000 C:\WINNT\system32\DBGHELP.dll
              0x690A0000 - 0x690AB000 C:\WINNT\system32\PSAPI.DLL

              Heap at VM Abort:
              Heap
              def new generation total 3136K, used 3136K [0x10010000, 0x10370000, 0x109e0000)
              eden space 2816K, 100% used [0x10010000, 0x102d0000, 0x102d0000)
              from space 320K, 100% used [0x102d0000, 0x10320000, 0x10320000)
              to space 320K, 0% used [0x10320000, 0x10320000, 0x10370000)
              tenured generation total 40280K, used 20475K [0x109e0000, 0x13136000, 0x18010000)
              the space 40280K, 50% used [0x109e0000, 0x11ddeda8, 0x11ddee00, 0x13136000)
              compacting perm gen total 24320K, used 24200K [0x18010000, 0x197d0000, 0x1c010000)
              the space 24320K, 99% used [0x18010000, 0x197b2360, 0x197b2400, 0x197d0000)

              Local Time = Wed Apr 21 01:00:39 2004
              Elapsed Time = 45296
              #
              # The exception above was detected in native code outside the VM
              #
              # Java VM: Java HotSpot(TM) Client VM (1.4.2-b28 mixed mode)
              #

              • 4. Re: JBoss JMS server hangs after running for an extended per
                cgonzalez

                Hi Paul,

                I have the same problem using JBoss, and I want to know if you resolved it and if you resolved please, help me because i have not idea.

                "Paul_ADP" wrote:
                We are using JBoss 3.2.3 on Windows 2000 Server runnning Sun JDK 1.4.2, with heap size = 256M
                Here is the beginning of our boot log:
                11:25:36,621 INFO [ServerInfo] Java version: 1.4.2,Sun Microsystems Inc.
                11:25:36,621 INFO [ServerInfo] Java VM: Java HotSpot(TM) Client VM 1.4.2-b28,Sun Microsystems Inc.
                11:25:36,621 INFO [ServerInfo] OS-System: Windows 2000 5.0,x86
                11:25:36,621 DEBUG [ServerInfo] Full System Properties Dump
                11:25:36,637 DEBUG [ServerInfo] java.runtime.name: Java(TM) 2 Runtime Environment, Standard Edition
                11:25:36,637 DEBUG [ServerInfo] jboss.server.base.dir: D:\jboss-3.2.3\server

                The JMS application has one Queue and 3 pools of MDBs using message selectors resident on the server. Also (this is the interesting part) we have 2 pools of MDBs that are resident on two other external servers that are bound to the same JBoss Queue.

                We have two different scenarios.
                Scenario 1:
                When we let the system run for extended periods of time, approx 12 hours , we notice that the JBoss server completely hangs up.
                - the JBoss console is no longer accessible
                - the server.log is no longer written to
                - the server doesn't grant any more connections of any kind (JNDI,JMS,HTTP)

                Scenario 2:
                After the server has been running for extended periods often times time we note the following error keeps getting posted to our server.log:
                org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.IOException: ping timeout.)
                at org.jboss.mq.Connection.asynchFailure(Connection.java:718)
                at org.jboss.mq.Connection$PingTask.run(Connection.java:1311)
                at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
                at java.lang.Thread.run(Thread.java:534)
                Caused by: java.io.IOException: ping timeout.
                at org.jboss.mq.Connection$PingTask.run(Connection.java:1303)
                ... 2 more

                When this happens, the server also stops responding to all requests as above, with the only difference being , the ping error listed above is still constantly written to the log.

                I've noted that many people suggest that we implement the javax.jms.Connection.setExceptionListener() interface in their clients, but I believe that I CANT do this from an MDB since the container is supposed to be managing my JMS connections. Is this true?

                Also, we suspect that the remote MDBs (the MDBs that are resident on remote servers) may be the cause. Do we need to implement any reconnection logic for these remote MDB's?

                Thanks in advance.