7 Replies Latest reply on Feb 21, 2008 10:22 AM by brian.stansberry

    Potential issue with JGroups using JBoss 3.2.7 and Java 1.4.

    ross.nelson
    ross.nelson Feb 20, 2008 10:38 AM

      Hi there, first post on the site, I currently have a live application (GT-X7 running on JBoss 3.2.7) which makes use of JGroups for clustering.

      We have 5 "kernels" involved in the group and both today and yesterday we have experienced a "hang" on the 1st kernel which is the originator of the group – several hours after startup.

      This hang renders the kernel completely inaccessible, all logging is halted (Garbage Collection halted in the middle of spooling a row), it's as if the kernel has paused itself waiting for something to happen. It would appear in the meantime that the other 4 kernels continue working away.

      On the first occasion we did not manage to the get the kill -QUIT to output anything before we restarted.

      Fortunately today on the 2nd occassion we managed to get the kill -QUIT to go through and I will enclose the output below.

      The reason I come to JGroups is the item at "waiting" on the dump appears to be part of the JGroups setup. - "MessageDispatcher up processing thread" (from google searching)

      On both occasions I have observed that around 1 minute before the "hung" kernel occurs, another member of the cluster is suspected, removed and then quickly re-introduced to the cluster, from google searching I did see a bit about "simultaneous" kernel exclusion causing a problem, however this doesn't appear "simultaneous".

      Thanks in advance for any assistance or even advice on areas to focus on – I’m a bit between JVM (Garbage Collection) and JGroups on this one, will be posting questions both here and on the sun site.

      I’ll post the “dump� first and then snippets from the STD Out Logs which contain the JGroups comings and goings…..

      Ross

      • 1736 Views
      • Tags:
        • 1. Re: Potential issue with JGroups using JBoss 3.2.7 and Java
          ross.nelson
          ross.nelson Feb 20, 2008 10:43 AM (in response to ross.nelson)

          Apologies for the strange text that appeared above, hopefully this notepad version works better.

          Java version is 1.4.2_09b05 (cut off from title)

          Hi there, first post on the site, I currently have a live application (GT-X7 running on JBoss 3.2.7) which makes use of JGroups for clustering.

          We have 5 "kernels" involved in the group and both today and yesterday we have experienced a "hang" on the 1st kernel which is the originator of the group several hours after startup.

          This hang renders the kernel completely inaccessible, all logging is halted (Garbage Collection halted in the middle of spooling a row), it's as if the kernel has paused itself waiting for something to happen. It would appear in the meantime that the other 4 kernels continue working away.

          On the first occasion we did not manage to the get the kill -QUIT to output anything before we restarted.

          Fortunately today we managed to get the kill -QUIT to go through and I will enclose the output below.

          The reason I come to JGroups is the item at "waiting" on the dump appears to be part of the JGroups setup. - "MessageDispatcher up processing thread" (from google searching)

          On both occasions I have observed that around 1 minute before the "hung" kernel occurs, another member of the cluster is suspected, removed and then quickly re-introduced to the cluster, from google searching I did see a bit about "simultaneous" kernel exclusion causing a problem, however this doesn't appear "simultaneous".

          Thanks in advance for any assistance or even advice on areas to focus on - I'm a bit between JVM (Garbage Collection) and JGroups on this one, will be posting questions both here and on the sun site.

          I'll post the dump first and then snippets from the STD Out Logs which contain the JGroups comings and goings ....

          Ross

            • Actions
          • 2. Re: Potential issue with JGroups using JBoss 3.2.7 and Java
            ross.nelson
            ross.nelson Feb 20, 2008 11:03 AM (in response to ross.nelson)

            Any help much appreciated.

            Kernels running on IP's labelled 10.9.181.* and 10.9.118.* (also known as cathlbgts*, kernel which failed is cathlbgts01)

            STD Out Logs

            <PROBLEMS APPEAR TO START 11:25>
            11:25:02,892 INFO [STDOUT] C: DELE 1
            11:25:02,898 INFO [STDOUT] S: +OK
            11:25:02,898 INFO [STDOUT] C: DELE 2
            11:25:02,898 INFO [STDOUT] S: +OK
            11:25:02,898 INFO [STDOUT] C: QUIT
            11:25:02,900 INFO [STDOUT] S: +OK Microsoft Exchange Server 2003 POP3 server version 6.5.7638.1 signing off.
            [Unloading class sun.reflect.GeneratedConstructorAccessor365]
            [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor218]
            <GARBAGE COLLECTION ENDS SEVERAL LINES LATER AND NOTHING IN LOGS UNTIL 12:15 when we run kill -QUIT, the app doesn't initially respond to the command but does 10 minutes later and resumes its work>

            Full thread dump Java HotSpot(TM) Server VM (1.4.2_09-b05 mixed mode):

            "RMI TCP Connection(1861)-10.9.181.50" daemon prio=1 tid=0x4d23aad8 nid=0x7771 runnable [0..0]

            "RMI TCP Connection(1860)-10.9.181.51" daemon prio=1 tid=0x4cd8fad8 nid=0x7771 runnable [4c6ff000..4c6ff868]
            at java.io.OutputStream.(OutputStream.java:29)
            at java.io.FileOutputStream.(FileOutputStream.java:198)
            at java.net.SocketOutputStream.(SocketOutputStream.java:41)
            at java.net.PlainSocketImpl.getOutputStream(PlainSocketImpl.java:389)
            - locked <0x8db870e8> (a java.net.PlainSocketImpl)
            at java.net.Socket$3.run(Socket.java:746)
            at java.security.AccessController.doPrivileged(Native Method)
            at java.net.Socket.getOutputStream(Socket.java:743)
            at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:635)
            at java.lang.Thread.run(Thread.java:534)

            "RMI TCP Connection(1859)-10.9.181.48" daemon prio=1 tid=0x4d2b5208 nid=0x7771 runnable [4af7d000..4af7e868]
            at java.lang.Object.hashCode(Native Method)
            at java.util.HashMap.hash(HashMap.java:261)
            at java.util.HashMap.put(HashMap.java:379)
            at java.util.HashSet.add(HashSet.java:192)
            at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.getConnection(BaseWrapperManagedConnection.java:252)
            - locked <0x73f671b8> (a java.util.HashSet)
            at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:515)
            at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:888)
            at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:102)
            at com.gtnet.workflow.entityBeans.GTW_WORKBean.selectAllByRoles(GTW_WORKBean.java:1050)
            at com.gtnet.workflow.entityBeans.GTW_WORKBean.ejbFindByRoles(GTW_WORKBean.java:293)
            at sun.reflect.GeneratedMethodAccessor266.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.ejb.plugins.BMPPersistenceManager.callFinderMethod(BMPPersistenceManager.java:560)
            at org.jboss.ejb.plugins.BMPPersistenceManager.findEntity(BMPPersistenceManager.java:289)
            at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.findEntity(CachedConnectionInterceptor.java:302)
            at org.jboss.ejb.EntityContainer.findSingleObject(EntityContainer.java:1039)
            at org.jboss.ejb.EntityContainer.find(EntityContainer.java:675)
            at sun.reflect.GeneratedMethodAccessor140.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.ejb.EntityContainer$ContainerInterceptor.invokeHome(EntityContainer.java:1066)
            at org.jboss.ejb.plugins.EntitySynchronizationInterceptor.invokeHome(EntitySynchronizationInterceptor.java:204)
            at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invokeHome(CachedConnectionInterceptor.java:215)
            at org.jboss.ejb.plugins.AbstractInterceptor.invokeHome(AbstractInterceptor.java:88)
            at org.jboss.ejb.plugins.EntityInstanceInterceptor.invokeHome(EntityInstanceInterceptor.java:97)
            at org.jboss.ejb.plugins.EntityLockInterceptor.invokeHome(EntityLockInterceptor.java:61)
            at org.jboss.ejb.plugins.EntityCreationInterceptor.invokeHome(EntityCreationInterceptor.java:28)
            at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:88)
            at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:317)
            at org.jboss.ejb.plugins.TxInterceptorCMT.invokeHome(TxInterceptorCMT.java:130)
            at org.jboss.ejb.plugins.SecurityInterceptor.invokeHome(SecurityInterceptor.java:85)
            at org.jboss.ejb.plugins.LogInterceptor.invokeHome(LogInterceptor.java:121)
            at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invokeHome(ProxyFactoryFinderInterceptor.java:93)
            at org.jboss.ejb.EntityContainer.internalInvokeHome(EntityContainer.java:478)
            at org.jboss.ejb.Container.invoke(Container.java:729)
            at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:198)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
            at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:97)
            at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:115)
            at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:101)
            at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
            at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:55)
            at org.jboss.proxy.ejb.HomeInterceptor.invoke(HomeInterceptor.java:173)
            at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:86)
            at $Proxy146.findByRoles(Unknown Source)
            at com.gtnet.workflow.enactmentService.ejb.ClientAPIBean.getNextWorkflowMessageForRoles(ClientAPIBean.java:1347)
            at com.gtnet.workflow.enactmentService.ejb.ClientAPIBean.getNextWorkflowMessageForRoles(ClientAPIBean.java:391)
            at com.gtnet.workflow.enactmentService.ejb.ClientAPIBean.getWorkForRoles(ClientAPIBean.java:311)
            at sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:683)
            at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:186)
            at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:72)
            at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:84)
            at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:317)
            at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:150)
            at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:111)
            at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
            at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
            at org.jboss.ejb.StatelessSessionContainer.internalInvoke(StatelessSessionContainer.java:331)
            at org.jboss.ejb.Container.invoke(Container.java:709)
            at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:198)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
            at org.jboss.invocation.jrmp.server.JRMPInvoker.invoke(JRMPInvoker.java:360)
            at sun.reflect.GeneratedMethodAccessor264.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261)
            at sun.rmi.transport.Transport$1.run(Transport.java:148)
            at java.security.AccessController.doPrivileged(Native Method)
            at sun.rmi.transport.Transport.serviceCall(Transport.java:144)
            at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460)
            at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701)
            at java.lang.Thread.run(Thread.java:534)

            "RMI TCP Connection(1858)-10.9.118.2" daemon prio=1 tid=0x3ee29ea8 nid=0x7771 runnable [53f77000..53f78868]
            at java.lang.Class.isAssignableFrom(Native Method)
            at java.io.ObjectStreamClass.getClassDataLayout0(ObjectStreamClass.java:984)
            at java.io.ObjectStreamClass.getClassDataLayout(ObjectStreamClass.java:972)
            at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1728)
            at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1646)
            at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1274)
            at java.io.ObjectInputStream.readObject(ObjectInputStream.java:324)
            at org.jboss.invocation.MarshalledInvocation.readExternal(MarshalledInvocation.java:530)
            at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1686)
            at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1644)
            at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1274)
            at java.io.ObjectInputStream.readObject(ObjectInputStream.java:324)
            at sun.rmi.server.UnicastRef.unmarshalValue(UnicastRef.java:297)
            at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:246)
            at sun.rmi.transport.Transport$1.run(Transport.java:148)
            at java.security.AccessController.doPrivileged(Native Method)
            at sun.rmi.transport.Transport.serviceCall(Transport.java:144)
            at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460)
            at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701)
            at java.lang.Thread.run(Thread.java:534)

            "MessageDispatcher up processing thread" daemon prio=1 tid=0x4bee8948 nid=0x7771 in Object.wait() [53c72000..53c72868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x611dd008> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at org.jgroups.util.Queue.remove(Queue.java:221)
            - locked <0x611dd008> (a java.lang.Object)
            at org.jgroups.blocks.MessageDispatcher$1.run(MessageDispatcher.java:689)
            at java.lang.Thread.run(Thread.java:534)

            "UDP mcast receiver" daemon prio=1 tid=0x095f93f0 nid=0x7771 runnable [4a2fd000..4a2fd868]
            at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:2701)
            at java.io.ObjectInputStream.readUTF(ObjectInputStream.java:999)
            at org.jgroups.Message.readExternal(Message.java:490)
            at org.jgroups.protocols.UDP.handleIncomingUdpPacket(UDP.java:686)
            at org.jgroups.protocols.UDP.run(UDP.java:249)
            at java.lang.Thread.run(Thread.java:534)

            "UDP.UcastReceiverThread" daemon prio=1 tid=0x08ad2188 nid=0x7771 runnable [4a37e000..4a37e868]
            at java.net.PlainDatagramSocketImpl.receive(Native Method)
            - locked <0x80833930> (a java.net.PlainDatagramSocketImpl)
            at java.net.DatagramSocket.receive(DatagramSocket.java:711)
            - locked <0x7fd91fc8> (a java.net.DatagramPacket)
            - locked <0x7fd91fe8> (a java.net.DatagramSocket)
            at org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:1264)
            at java.lang.Thread.run(Thread.java:534)

            "TimeScheduler.Thread" daemon prio=1 tid=0x4c1bb3d8 nid=0x7771 in Object.wait() [538eb000..538eb868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x7ebb1cf8> (a org.jgroups.util.TimeScheduler$TaskQueue)
            at org.jgroups.util.TimeScheduler._run(TimeScheduler.java:391)
            - locked <0x7ebb1cf8> (a org.jgroups.util.TimeScheduler$TaskQueue)
            at org.jgroups.util.TimeScheduler.access$000(TimeScheduler.java:46)
            at org.jgroups.util.TimeScheduler$Loop.run(TimeScheduler.java:135)
            at java.lang.Thread.run(Thread.java:534)

            "UpHandler (STATE_TRANSFER)" daemon prio=1 tid=0x4c8f8e68 nid=0x7771 in Object.wait() [4a3ff000..4a3ff868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x7ed5c108> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at org.jgroups.util.Queue.remove(Queue.java:221)
            - locked <0x7ed5c108> (a java.lang.Object)
            at org.jgroups.stack.UpHandler.run(Protocol.java:48)

            "DownHandler (STATE_TRANSFER)" daemon prio=1 tid=0x4ca695b0 nid=0x7771 in Object.wait() [4e26e000..4e26e868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x7ed5c118> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at org.jgroups.util.Queue.remove(Queue.java:221)
            - locked <0x7ed5c118> (a java.lang.Object)
            at org.jgroups.stack.DownHandler.run(Protocol.java:105)

            "UpHandler (GMS)" daemon prio=1 tid=0x4c1632f8 nid=0x7771 in Object.wait() [53a6e000..53a6e868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x7ed5c128> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at org.jgroups.util.Queue.remove(Queue.java:221)
            - locked <0x7ed5c128> (a java.lang.Object)
            at org.jgroups.stack.UpHandler.run(Protocol.java:48)

            "DownHandler (GMS)" daemon prio=1 tid=0x4c1b7d50 nid=0x7771 in Object.wait() [4b7ff000..4b7ff868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x7ed5c638> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at org.jgroups.util.Queue.remove(Queue.java:221)
            - locked <0x7ed5c638> (a java.lang.Object)
            at org.jgroups.stack.DownHandler.run(Protocol.java:105)

            "UpHandler (FRAG)" daemon prio=1 tid=0x4c1b9020 nid=0x7771 in Object.wait() [4bdff000..4bdff868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x7ed5fe50> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at org.jgroups.util.Queue.remove(Queue.java:221)
            - locked <0x7ed5fe50> (a java.lang.Object)
            at org.jgroups.stack.UpHandler.run(Protocol.java:48)

            "DownHandler (FRAG)" daemon prio=1 tid=0x4c8f6a40 nid=0x7771 in Object.wait() [53aef000..53aef868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x7ed5fe60> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at org.jgroups.util.Queue.remove(Queue.java:221)
            - locked <0x7ed5fe60> (a java.lang.Object)
            at org.jgroups.stack.DownHandler.run(Protocol.java:105)

            "UpHandler (STABLE)" daemon prio=1 tid=0x4c4e73e8 nid=0x7771 in Object.wait() [4e3f1000..4e3f1868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x7ed5ffc0> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at org.jgroups.util.Queue.remove(Queue.java:221)
            - locked <0x7ed5ffc0> (a java.lang.Object)
            at org.jgroups.stack.UpHandler.run(Protocol.java:48)

            "DownHandler (STABLE)" daemon prio=1 tid=0x4c8fed30 nid=0x7771 in Object.wait() [4e6f7000..4e6f7868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x7ed5ffd0> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at org.jgroups.util.Queue.remove(Queue.java:221)
            - locked <0x7ed5ffd0> (a java.lang.Object)
            at org.jgroups.stack.DownHandler.run(Protocol.java:105)

            "UpHandler (UNICAST)" daemon prio=1 tid=0x4bcf60d8 nid=0x7771 in Object.wait() [5450a000..5450a868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x7ed60080> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at org.jgroups.util.Queue.remove(Queue.java:221)
            - locked <0x7ed60080> (a java.lang.Object)
            at org.jgroups.stack.UpHandler.run(Protocol.java:48)

            "DownHandler (UNICAST)" daemon prio=1 tid=0x4bc8f248 nid=0x7771 in Object.wait() [5417c000..5417c868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x7ed60100> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at org.jgroups.util.Queue.remove(Queue.java:221)
            - locked <0x7ed60100> (a java.lang.Object)
            at org.jgroups.stack.DownHandler.run(Protocol.java:105)

            "UpHandler (NAKACK)" daemon prio=1 tid=0x4c077710 nid=0x7771 in Object.wait() [5427e000..5427e868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x7ed3f970> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at org.jgroups.util.Queue.remove(Queue.java:221)
            - locked <0x7ed3f970> (a java.lang.Object)
            at org.jgroups.stack.UpHandler.run(Protocol.java:48)

            "DownHandler (NAKACK)" daemon prio=1 tid=0x4d08cd60 nid=0x7771 in Object.wait() [540fb000..540fb868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x7ed42798> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at org.jgroups.util.Queue.remove(Queue.java:221)
            - locked <0x7ed42798> (a java.lang.Object)
            at org.jgroups.stack.DownHandler.run(Protocol.java:105)

            "UpHandler (VERIFY_SUSPECT)" daemon prio=1 tid=0x085f3368 nid=0x7771 runnable [53e76000..53e76868]
            at java.lang.Thread.setPriority0(Native Method)
            at java.lang.Thread.setPriority(Thread.java:878)
            at java.lang.Thread.init(Thread.java:293)
            at java.lang.Thread.(Thread.java:388)
            at org.jgroups.protocols.VERIFY_SUSPECT.startTimer(VERIFY_SUSPECT.java:203)
            at org.jgroups.protocols.VERIFY_SUSPECT.suspect(VERIFY_SUSPECT.java:185)
            at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:88)
            at org.jgroups.stack.UpHandler.run(Protocol.java:59)

            "DownHandler (VERIFY_SUSPECT)" daemon prio=1 tid=0x08858310 nid=0x7771 in Object.wait() [4b4ff000..4b4ff868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x7ed428e8> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at org.jgroups.util.Queue.remove(Queue.java:221)
            - locked <0x7ed428e8> (a java.lang.Object)
            at org.jgroups.stack.DownHandler.run(Protocol.java:105)

            "UpHandler (FD)" daemon prio=1 tid=0x08a29e18 nid=0x7771 in Object.wait() [53cf3000..53cf3868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x7ed43ad8> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at org.jgroups.util.Queue.remove(Queue.java:221)
            - locked <0x7ed43ad8> (a java.lang.Object)
            at org.jgroups.stack.UpHandler.run(Protocol.java:48)

            "DownHandler (FD)" daemon prio=1 tid=0x09515268 nid=0x7771 in Object.wait() [4e1ed000..4e1ed868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x7ed43ac8> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at org.jgroups.util.Queue.remove(Queue.java:221)
            - locked <0x7ed43ac8> (a java.lang.Object)
            at org.jgroups.stack.DownHandler.run(Protocol.java:105)

            "UpHandler (PING)" daemon prio=1 tid=0x08857610 nid=0x7771 in Object.wait() [54489000..54489868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x7eda5268> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at org.jgroups.util.Queue.remove(Queue.java:221)
            - locked <0x7eda5268> (a java.lang.Object)
            at org.jgroups.stack.UpHandler.run(Protocol.java:48)

            "DownHandler (PING)" daemon prio=1 tid=0x08ad1488 nid=0x7771 in Object.wait() [53b70000..53b70868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x7eda5258> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at org.jgroups.util.Queue.remove(Queue.java:221)
            - locked <0x7eda5258> (a java.lang.Object)
            at org.jgroups.stack.DownHandler.run(Protocol.java:105)

            "DownHandler (UDP)" daemon prio=1 tid=0x08859010 nid=0x7771 runnable [45ba4000..45ba4868]
            at java.io.ObjectOutputStream$ReplaceTable.(ObjectOutputStream.java:2141)
            at java.io.ObjectOutputStream.(ObjectOutputStream.java:181)
            at org.jgroups.protocols.UDP.messageToBuffer(UDP.java:835)
            at org.jgroups.protocols.UDP.send(UDP.java:791)
            at org.jgroups.protocols.UDP.sendUdpMessage(UDP.java:784)
            at org.jgroups.protocols.UDP.down(UDP.java:624)
            at org.jgroups.stack.DownHandler.run(Protocol.java:122)

            "RMI TCP Connection(1536)-10.9.181.50" daemon prio=1 tid=0x4cc207f8 nid=0x7771 waiting on condition [4cfff000..4cfff868]
            at java.net.SocketInputStream.read(SocketInputStream.java:133)
            at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
            at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
            - locked <0x7fb283d8> (a java.io.BufferedInputStream)
            at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:397)
            at java.io.DataInputStream.readUTF(DataInputStream.java:633)
            at java.io.DataInputStream.readUTF(DataInputStream.java:610)
            at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:685)
            at java.lang.Thread.run(Thread.java:534)

            "RMI TCP Connection(1533)-10.9.118.2" daemon prio=1 tid=0x3ee1fd20 nid=0x7771 waiting on condition [541fd000..541fd868]
            at java.net.SocketInputStream.read(SocketInputStream.java:133)
            at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
            at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
            - locked <0x7fb28658> (a java.io.BufferedInputStream)
            at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:397)
            at java.io.DataInputStream.readUTF(DataInputStream.java:633)
            at java.io.DataInputStream.readUTF(DataInputStream.java:610)
            at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:685)
            at java.lang.Thread.run(Thread.java:534)

            "RMI TCP Connection(1529)-10.9.181.50" daemon prio=1 tid=0x4b217e48 nid=0x7771 waiting on condition [4e06a000..4e06a868]
            at java.net.SocketInputStream.read(SocketInputStream.java:133)
            at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
            at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
            - locked <0x7fb29b40> (a java.io.BufferedInputStream)
            at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:397)
            at java.io.DataInputStream.readUTF(DataInputStream.java:633)
            at java.io.DataInputStream.readUTF(DataInputStream.java:610)
            at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:685)
            at java.lang.Thread.run(Thread.java:534)

            "RMI TCP Connection(1527)-10.9.118.2" daemon prio=1 tid=0x45c37e60 nid=0x7771 runnable [0..4ef5b970]

            "RMI TCP Connection(1516)-10.9.181.50" daemon prio=1 tid=0x4d24e340 nid=0x7771 waiting on condition [58907000..58907868]
            at java.net.SocketInputStream.read(SocketInputStream.java:133)
            at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
            at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
            - locked <0x7fb29f20> (a java.io.BufferedInputStream)
            at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:397)
            at java.io.DataInputStream.readUTF(DataInputStream.java:633)
            at java.io.DataInputStream.readUTF(DataInputStream.java:610)
            at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:685)
            at java.lang.Thread.run(Thread.java:534)

            "RMI RenewClean-[10.9.181.51:49801]" daemon prio=1 tid=0x085fed58 nid=0x7771 runnable [4e7f9000..4e7f9868]
            at java.lang.Thread.start(Native Method)
            - waiting to lock <0x5e97a518> (a java.lang.Thread)
            at sun.rmi.transport.tcp.TCPChannel.free(TCPChannel.java:322)
            - locked <0x75048f00> (a java.util.ArrayList)
            at sun.rmi.server.UnicastRef.free(UnicastRef.java:402)
            at sun.rmi.server.UnicastRef.done(UnicastRef.java:419)
            at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
            at sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:328)
            at sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:144)
            at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:539)
            at java.lang.Thread.run(Thread.java:534)

            "GT-X UniversalController_1B8594071C7C8AE1981C590F4FC2ADB2" daemon prio=1 tid=0x08ac4848 nid=0x7771 in Object.wait() [4c67e000..4c67e868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x7403d500> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at com.gtnet.j2ee.comms.UniversalController.mainLoop(UniversalController.java:1030)
            - locked <0x7403d500> (a java.lang.Object)
            at com.gtnet.j2ee.comms.UniversalController.run(UniversalController.java:1014)
            at java.lang.Thread.run(Thread.java:534)

            "GT-X UniversalController_D8A6507C9DC061068CC29722F921C0D9" daemon prio=1 tid=0x085dfd08 nid=0x7771 in Object.wait() [4e778000..4e778868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x700d7a70> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at com.gtnet.j2ee.comms.UniversalController.mainLoop(UniversalController.java:1030)
            - locked <0x700d7a70> (a java.lang.Object)
            at com.gtnet.j2ee.comms.UniversalController.run(UniversalController.java:1014)
            at java.lang.Thread.run(Thread.java:534)

            "GT-X UniversalController_A3892207C47D1A642D374A2A02AD2D46" daemon prio=1 tid=0x08ee4230 nid=0x7771 in Object.wait() [4e16c000..4e16c868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x6fcba740> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at com.gtnet.j2ee.comms.UniversalController.mainLoop(UniversalController.java:1030)
            - locked <0x6fcba740> (a java.lang.Object)
            at com.gtnet.j2ee.comms.UniversalController.run(UniversalController.java:1014)
            at java.lang.Thread.run(Thread.java:534)

            "GT-X UniversalController_1BF5DC48697971F913914DDF69AD6BFE" daemon prio=1 tid=0x08b4bf48 nid=0x7771 in Object.wait() [4d3b7000..4d3b7868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x7071d8b8> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at com.gtnet.j2ee.comms.UniversalController.mainLoop(UniversalController.java:1030)
            - locked <0x7071d8b8> (a java.lang.Object)
            at com.gtnet.j2ee.comms.UniversalController.run(UniversalController.java:1014)
            at java.lang.Thread.run(Thread.java:534)

            "GT-X UniversalController_F2DC2A6E69546D609B20B058D7D2B40C" daemon prio=1 tid=0x505a6d50 nid=0x7771 in Object.wait() [4e472000..4e472868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x7fecf1d8> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at com.gtnet.j2ee.comms.UniversalController.mainLoop(UniversalController.java:1030)
            - locked <0x7fecf1d8> (a java.lang.Object)
            at com.gtnet.j2ee.comms.UniversalController.run(UniversalController.java:1014)
            at java.lang.Thread.run(Thread.java:534)

            "GT-X UniversalController_659FD505E0548067B3629F5FA435F04E" daemon prio=1 tid=0x4eb1af48 nid=0x7771 in Object.wait() [4c9ff000..4c9ff868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x7687f3f0> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:429)
            at com.gtnet.j2ee.comms.UniversalController.mainLoop(UniversalController.java:1030)
            - locked <0x7687f3f0> (a java.lang.Object)
            at com.gtnet.j2ee.comms.UniversalController.run(UniversalController.java:1014)
            at java.lang.Thread.run(Thread.java:534)

            "RMI RenewClean-[10.9.181.48:39885]" daemon prio=1 tid=0x0928a040 nid=0x7771 runnable [4e9fd000..4e9fd868]
            at java.lang.Thread.setPriority0(Native Method)
            at java.lang.Thread.setPriority(Thread.java:878)
            at java.lang.Thread.init(Thread.java:293)
            at java.lang.Thread.(Thread.java:438)
            at sun.rmi.runtime.NewThreadAction.run(NewThreadAction.java:110)
            at java.security.AccessController.doPrivileged(Native Method)
            at sun.rmi.transport.tcp.TCPChannel.free(TCPChannel.java:317)
            - locked <0x6e69e180> (a java.util.ArrayList)
            at sun.rmi.server.UnicastRef.free(UnicastRef.java:402)
            at sun.rmi.server.UnicastRef.done(UnicastRef.java:419)
            at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
            at sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:328)
            at sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:144)
            at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:539)
            at java.lang.Thread.run(Thread.java:534)

            "HAJNDI-AutomaticDiscovery" prio=1 tid=0x0898a110 nid=0x7771 runnable [4e87a000..4e87a868]
            at java.net.PlainDatagramSocketImpl.receive(Native Method)
            - locked <0x611c2310> (a java.net.PlainDatagramSocketImpl)
            at java.net.DatagramSocket.receive(DatagramSocket.java:711)
            - locked <0x6e686f58> (a java.net.DatagramPacket)
            - locked <0x611c2340> (a java.net.MulticastSocket)
            at org.jboss.ha.jndi.DetachedHANamingService$AutomaticDiscovery.run(DetachedHANamingService.java:489)
            at java.lang.Thread.run(Thread.java:534)

            "RMI RenewClean-[10.9.181.50:36409]" daemon prio=1 tid=0x08aa8088 nid=0x7771 runnable [4cbff000..4cbff868]
            at java.lang.Thread.setPriority0(Native Method)
            at java.lang.Thread.setPriority(Thread.java:878)
            at java.lang.Thread.init(Thread.java:293)
            at java.lang.Thread.(Thread.java:438)
            at sun.rmi.runtime.NewThreadAction.run(NewThreadAction.java:110)
            at java.security.AccessController.doPrivileged(Native Method)
            at sun.rmi.transport.tcp.TCPChannel.free(TCPChannel.java:317)
            - locked <0x6e663430> (a java.util.ArrayList)
            at sun.rmi.server.UnicastRef.free(UnicastRef.java:402)
            at sun.rmi.server.UnicastRef.done(UnicastRef.java:419)
            at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
            at sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:328)
            at sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:144)
            at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:539)
            at java.lang.Thread.run(Thread.java:534)

            "HAJNDI-Listener" prio=1 tid=0x08b38df8 nid=0x7771 runnable [4dfe9000..4dfe9868]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
            - locked <0x611b5950> (a java.net.PlainSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:448)
            at java.net.ServerSocket.accept(ServerSocket.java:419)
            at org.jboss.ha.jndi.DetachedHANamingService.run(DetachedHANamingService.java:320)
            at java.lang.Thread.run(Thread.java:534)

            "RMI RenewClean-[10.9.118.3:35956]" daemon prio=1 tid=0x088325c8 nid=0x7771 runnable [4e574000..4e574868]
            at java.lang.Thread.setPriority0(Native Method)
            at java.lang.Thread.setPriority(Thread.java:878)
            at java.lang.Thread.init(Thread.java:293)
            at java.lang.Thread.(Thread.java:438)
            at sun.rmi.runtime.NewThreadAction.run(NewThreadAction.java:110)
            at java.security.AccessController.doPrivileged(Native Method)
            at sun.rmi.transport.tcp.TCPChannel.free(TCPChannel.java:317)
            - locked <0x6585f3e8> (a java.util.ArrayList)
            at sun.rmi.server.UnicastRef.free(UnicastRef.java:402)
            at sun.rmi.server.UnicastRef.done(UnicastRef.java:419)
            at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
            at sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:328)
            at sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:144)
            at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:539)
            at java.lang.Thread.run(Thread.java:534)

            "TimeoutFactory" daemon prio=1 tid=0x08753480 nid=0x7771 runnable [4df68000..4df68868]
            at java.lang.Thread.setPriority0(Native Method)
            at java.lang.Thread.setPriority(Thread.java:878)
            at java.lang.Thread.init(Thread.java:293)
            at java.lang.Thread.(Thread.java:438)
            at org.jboss.util.threadpool.BasicThreadPool$ThreadPoolThreadFactory.newThread(BasicThreadPool.java:424)
            at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.addThread(PooledExecutor.java:517)
            at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.execute(PooledExecutor.java:880)
            - locked <0x677897d8> (a org.jboss.util.threadpool.MinPooledExecutor)
            at org.jboss.util.threadpool.BasicThreadPool.execute(BasicThreadPool.java:380)
            at org.jboss.util.threadpool.BasicThreadPool.runTaskWrapper(BasicThreadPool.java:165)
            at org.jboss.util.threadpool.BasicThreadPool.run(BasicThreadPool.java:184)
            at org.jboss.util.threadpool.BasicThreadPool.run(BasicThreadPool.java:179)
            at org.jboss.util.timeout.TimeoutFactory.doWork(TimeoutFactory.java:521)
            at org.jboss.util.timeout.TimeoutFactory.access$300(TimeoutFactory.java:28)
            at org.jboss.util.timeout.TimeoutFactory$1.run(TimeoutFactory.java:549)

            "OracleTimeoutPollingThread" daemon prio=1 tid=0x08531640 nid=0x7771 sleeping[4f0de000..4f0de868]
            at java.lang.Thread.sleep(Native Method)
            at oracle.jdbc.driver.OracleTimeoutPollingThread.run(OracleTimeoutPollingThread.java:161)

            "RMI TCP Accept-0" daemon prio=1 tid=0x08670108 nid=0x7771 runnable [4f15f000..4f15f868]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
            - locked <0x629213b8> (a java.net.PlainSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:448)
            at java.net.ServerSocket.accept(ServerSocket.java:419)
            at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
            at java.lang.Thread.run(Thread.java:534)

            "TP-Monitor" daemon prio=1 tid=0x56387440 nid=0x7771 in Object.wait() [4f1e0000..4f1e0868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x62780a98> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
            at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:559)
            - locked <0x62780a98> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
            at java.lang.Thread.run(Thread.java:534)

            "TP-Processor4" daemon prio=1 tid=0x552ee330 nid=0x7771 runnable [4f261000..4f261868]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
            - locked <0x6277b270> (a java.net.PlainSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:448)
            at java.net.ServerSocket.accept(ServerSocket.java:419)
            at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:295)
            at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:638)
            at org.apache.jk.common.SocketAcceptor.runIt(ChannelSocket.java:847)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
            at java.lang.Thread.run(Thread.java:534)

            "TP-Processor3" daemon prio=1 tid=0x552ee148 nid=0x7771 in Object.wait() [4f2e2000..4f2e2868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x6277f930> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:429)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
            - locked <0x6277f930> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:534)

            "TP-Processor2" daemon prio=1 tid=0x5551fc20 nid=0x7771 in Object.wait() [4f363000..4f363868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x62786870> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:429)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
            - locked <0x62786870> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:534)

            "TP-Processor1" daemon prio=1 tid=0x555fc7f0 nid=0x7771 in Object.wait() [4f3e4000..4f3e4868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x62597120> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:429)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
            - locked <0x62597120> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:534)

            "http-0.0.0.0-7280-Monitor" prio=1 tid=0x557e3ab8 nid=0x7771 in Object.wait() [4f465000..4f465868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x62787fd8> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
            at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:559)
            - locked <0x62787fd8> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
            at java.lang.Thread.run(Thread.java:534)

            "http-0.0.0.0-7280-Processor25" daemon prio=1 tid=0x54c8fe50 nid=0x7771 runnable [4f4e4000..4f4e6868]
            at java.lang.Throwable.fillInStackTrace(Native Method)
            - waiting to lock <0x5ea8e240> (a java.lang.Exception)
            at java.lang.Throwable.(Throwable.java:195)
            at java.lang.Exception.(Exception.java:41)
            at org.jboss.resource.adapter.jdbc.WrappedStatement.registerResultSet(WrappedStatement.java:852)
            - locked <0x5e9a97a0> (a org.jboss.resource.adapter.jdbc.WrappedPreparedStatement)
            at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:315)
            at com.gtnet.workflow.entityBeans.GTW_WORKBean.getExcludeMessageIds(GTW_WORKBean.java:1386)
            at com.gtnet.workflow.entityBeans.GTW_WORKBean.selectAllByClientLogin(GTW_WORKBean.java:967)
            at com.gtnet.workflow.entityBeans.GTW_WORKBean.ejbFindByClientLogin(GTW_WORKBean.java:324)
            at sun.reflect.GeneratedMethodAccessor242.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.ejb.plugins.BMPPersistenceManager.callFinderMethod(BMPPersistenceManager.java:560)
            at org.jboss.ejb.plugins.BMPPersistenceManager.findEntity(BMPPersistenceManager.java:289)
            at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.findEntity(CachedConnectionInterceptor.java:302)
            at org.jboss.ejb.EntityContainer.findSingleObject(EntityContainer.java:1039)
            at org.jboss.ejb.EntityContainer.find(EntityContainer.java:675)
            at sun.reflect.GeneratedMethodAccessor140.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.ejb.EntityContainer$ContainerInterceptor.invokeHome(EntityContainer.java:1066)
            at org.jboss.ejb.plugins.EntitySynchronizationInterceptor.invokeHome(EntitySynchronizationInterceptor.java:204)
            at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invokeHome(CachedConnectionInterceptor.java:215)
            at org.jboss.ejb.plugins.AbstractInterceptor.invokeHome(AbstractInterceptor.java:88)
            at org.jboss.ejb.plugins.EntityInstanceInterceptor.invokeHome(EntityInstanceInterceptor.java:97)
            at org.jboss.ejb.plugins.EntityLockInterceptor.invokeHome(EntityLockInterceptor.java:61)
            at org.jboss.ejb.plugins.EntityCreationInterceptor.invokeHome(EntityCreationInterceptor.java:28)
            at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:88)
            at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:317)
            at org.jboss.ejb.plugins.TxInterceptorCMT.invokeHome(TxInterceptorCMT.java:130)
            at org.jboss.ejb.plugins.SecurityInterceptor.invokeHome(SecurityInterceptor.java:85)
            at org.jboss.ejb.plugins.LogInterceptor.invokeHome(LogInterceptor.java:121)
            at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invokeHome(ProxyFactoryFinderInterceptor.java:93)
            at org.jboss.ejb.EntityContainer.internalInvokeHome(EntityContainer.java:478)
            at org.jboss.ejb.Container.invoke(Container.java:729)
            at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:198)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
            at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:97)
            at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:115)
            at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:101)
            at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
            at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:55)
            at org.jboss.proxy.ejb.HomeInterceptor.invoke(HomeInterceptor.java:173)
            at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:86)
            at $Proxy146.findByClientLogin(Unknown Source)
            at com.gtnet.workflow.enactmentService.ejb.ClientAPIBean.getNextWorkflowMessageForClientLogin(ClientAPIBean.java:1319)
            at com.gtnet.workflow.enactmentService.ejb.ClientAPIBean.getNextWorkflowMessageForClientLogin(ClientAPIBean.java:328)
            at com.gtnet.workflow.enactmentService.ejb.ClientAPIBean.getNextWorkflowMessage(ClientAPIBean.java:301)
            at sun.reflect.GeneratedMethodAccessor241.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:683)
            at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:186)
            at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:72)
            at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:84)
            at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:317)
            at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:150)
            at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:111)
            at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
            at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
            at org.jboss.ejb.StatelessSessionContainer.internalInvoke(StatelessSessionContainer.java:331)
            at org.jboss.ejb.Container.invoke(Container.java:709)
            at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:198)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
            at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:97)
            at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:115)
            at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:101)
            at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
            at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:55)
            at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:100)
            at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:86)
            at $Proxy121.getNextWorkflowMessage(Unknown Source)
            at com.gtnet.pk.ecmas.extensions.workflow.GTWClientObject.getNextWorkflowMessage(GTWClientObject.java:361)
            at com.gtnet.pk.ecmas.extensions.workflow.InteractiveProcessRunnerPrototype.callConstructor(InteractiveProcessRunnerPrototype.java:112)
            at com.gtnet.pk.ecmas.extensions.ProcessDefinitionPrototype.onEntry(ProcessDefinitionPrototype.java:396)
            at com.gtnet.pk.nav.ChildReference.enteringTree(ChildReference.java:629)
            at com.gtnet.pk.nav.ProcessTree.enterTreeNode(ProcessTree.java:695)
            at com.gtnet.pk.nav.KernelSession.runUntilGuarded(KernelSession.java:1565)
            at com.gtnet.pk.nav.KernelSession.runOnEvent(KernelSession.java:1240)
            at com.gtnet.pk.nav.KernelSession.handleEventImpl(KernelSession.java:1104)
            at com.gtnet.pk.nav.KernelSession.handleEvent(KernelSession.java:1064)
            at com.gtnet.pk.nav.KernelSession.start(KernelSession.java:2532)
            at com.gtnet.pk.nav.KernelSession.start(KernelSession.java:2517)
            at com.gtnet.j2ee.components.ejb.ProcessEngineBean.executeProcess(ProcessEngineBean.java:182)
            at com.gtnet.j2ee.components.ejb.ProcessEngineBean.startBatchSession(ProcessEngineBean.java:80)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.ejb.StatefulSessionContainer$ContainerInterceptor.invoke(StatefulSessionContainer.java:982)
            at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:111)
            at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:186)
            at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:84)
            at org.jboss.ejb.plugins.AbstractTxInterceptorBMT.invokeNext(AbstractTxInterceptorBMT.java:144)
            at org.jboss.ejb.plugins.TxInterceptorBMT.invoke(TxInterceptorBMT.java:62)
            at org.jboss.ejb.plugins.StatefulSessionInstanceInterceptor.invoke(StatefulSessionInstanceInterceptor.java:273)
            at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
            at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
            at org.jboss.ejb.StatefulSessionContainer.internalInvoke(StatefulSessionContainer.java:449)
            at org.jboss.ejb.Container.invoke(Container.java:709)
            at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:198)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
            at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:97)
            at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:115)
            at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:101)
            at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
            at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:55)
            at org.jboss.proxy.ejb.StatefulSessionInterceptor.invoke(StatefulSessionInterceptor.java:105)
            at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:86)
            at $Proxy171.startBatchSession(Unknown Source)
            at com.gtnet.j2ee.comms.UniversalController.startBatchSession(UniversalController.java:535)
            at com.gtnet.batch.BatchAcceptor.doGet(BatchAcceptor.java:69)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:697)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
            at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:75)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:214)
            at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
            at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
            at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198)
            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152)
            at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
            at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:66)
            at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
            at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:162)
            at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
            at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137)
            at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118)
            at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
            at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:535)
            at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
            at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
            at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
            at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
            at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
            at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160)
            at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799)
            at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705)
            at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
            at java.lang.Thread.run(Thread.java:534)

            "http-0.0.0.0-7280-Processor24" daemon prio=1 tid=0x54c8fcd0 nid=0x7771 runnable [4f565000..4f567868]
            at java.lang.Exception.(Exception.java:41)
            at org.jboss.resource.adapter.jdbc.WrappedConnection.registerStatement(WrappedConnection.java:819)
            - locked <0x5e9ab588> (a org.jboss.resource.adapter.jdbc.WrappedConnection)
            at org.jboss.resource.adapter.jdbc.WrappedStatement.(WrappedStatement.java:60)
            at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.(WrappedPreparedStatement.java:45)
            at org.jboss.resource.adapter.jdbc.WrappedConnection.prepareStatement(WrappedConnection.java:217)
            at com.gtnet.workflow.entityBeans.GTW_WORKBean.selectAllByClientLogin(GTW_WORKBean.java:983)
            at com.gtnet.workflow.entityBeans.GTW_WORKBean.ejbFindByClientLogin(GTW_WORKBean.java:324)
            at sun.reflect.GeneratedMethodAccessor242.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.ejb.plugins.BMPPersistenceManager.callFinderMethod(BMPPersistenceManager.java:560)
            at org.jboss.ejb.plugins.BMPPersistenceManager.findEntity(BMPPersistenceManager.java:289)
            at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.findEntity(CachedConnectionInterceptor.java:302)
            at org.jboss.ejb.EntityContainer.findSingleObject(EntityContainer.java:1039)
            at org.jboss.ejb.EntityContainer.find(EntityContainer.java:675)
            at sun.reflect.GeneratedMethodAccessor140.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.ejb.EntityContainer$ContainerInterceptor.invokeHome(EntityContainer.java:1066)
            at org.jboss.ejb.plugins.EntitySynchronizationInterceptor.invokeHome(EntitySynchronizationInterceptor.java:204)
            at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invokeHome(CachedConnectionInterceptor.java:215)
            at org.jboss.ejb.plugins.AbstractInterceptor.invokeHome(AbstractInterceptor.java:88)
            at org.jboss.ejb.plugins.EntityInstanceInterceptor.invokeHome(EntityInstanceInterceptor.java:97)
            at org.jboss.ejb.plugins.EntityLockInterceptor.invokeHome(EntityLockInterceptor.java:61)
            at org.jboss.ejb.plugins.EntityCreationInterceptor.invokeHome(EntityCreationInterceptor.java:28)
            at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:88)
            at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:317)
            at org.jboss.ejb.plugins.TxInterceptorCMT.invokeHome(TxInterceptorCMT.java:130)
            at org.jboss.ejb.plugins.SecurityInterceptor.invokeHome(SecurityInterceptor.java:85)
            at org.jboss.ejb.plugins.LogInterceptor.invokeHome(LogInterceptor.java:121)
            at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invokeHome(ProxyFactoryFinderInterceptor.java:93)
            at org.jboss.ejb.EntityContainer.internalInvokeHome(EntityContainer.java:478)
            at org.jboss.ejb.Container.invoke(Container.java:729)
            at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:198)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
            at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:97)
            at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:115)
            at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:101)
            at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
            at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:55)
            at org.jboss.proxy.ejb.HomeInterceptor.invoke(HomeInterceptor.java:173)
            at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:86)
            at $Proxy146.findByClientLogin(Unknown Source)
            at com.gtnet.workflow.enactmentService.ejb.ClientAPIBean.getNextWorkflowMessageForClientLogin(ClientAPIBean.java:1319)
            at com.gtnet.workflow.enactmentService.ejb.ClientAPIBean.getNextWorkflowMessageForClientLogin(ClientAPIBean.java:328)
            at com.gtnet.workflow.enactmentService.ejb.ClientAPIBean.getNextWorkflowMessage(ClientAPIBean.java:301)
            at sun.reflect.GeneratedMethodAccessor241.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:683)
            at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:186)
            at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:72)
            at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:84)
            at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:317)
            at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:150)
            at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:111)
            at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
            at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
            at org.jboss.ejb.StatelessSessionContainer.internalInvoke(StatelessSessionContainer.java:331)
            at org.jboss.ejb.Container.invoke(Container.java:709)
            at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:198)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
            at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:97)
            at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:115)
            at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:101)
            at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
            at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:55)
            at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:100)
            at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:86)
            at $Proxy121.getNextWorkflowMessage(Unknown Source)
            at com.gtnet.pk.ecmas.extensions.workflow.GTWClientObject.getNextWorkflowMessage(GTWClientObject.java:361)
            at com.gtnet.pk.ecmas.extensions.workflow.InteractiveProcessRunnerPrototype.callConstructor(InteractiveProcessRunnerPrototype.java:112)
            at com.gtnet.pk.ecmas.extensions.ProcessDefinitionPrototype.onEntry(ProcessDefinitionPrototype.java:396)
            at com.gtnet.pk.nav.ChildReference.enteringTree(ChildReference.java:629)
            at com.gtnet.pk.nav.ProcessTree.enterTreeNode(ProcessTree.java:695)
            at com.gtnet.pk.nav.KernelSession.runUntilGuarded(KernelSession.java:1565)
            at com.gtnet.pk.nav.KernelSession.runOnEvent(KernelSession.java:1240)
            at com.gtnet.pk.nav.KernelSession.handleEventImpl(KernelSession.java:1104)
            at com.gtnet.pk.nav.KernelSession.handleEvent(KernelSession.java:1064)
            at com.gtnet.pk.nav.KernelSession.start(KernelSession.java:2532)
            at com.gtnet.pk.nav.KernelSession.start(KernelSession.java:2517)
            at com.gtnet.j2ee.components.ejb.ProcessEngineBean.executeProcess(ProcessEngineBean.java:182)
            at com.gtnet.j2ee.components.ejb.ProcessEngineBean.startBatchSession(ProcessEngineBean.java:80)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.ejb.StatefulSessionContainer$ContainerInterceptor.invoke(StatefulSessionContainer.java:982)
            at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:111)
            at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:186)
            at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:84)
            at org.jboss.ejb.plugins.AbstractTxInterceptorBMT.invokeNext(AbstractTxInterceptorBMT.java:144)
            at org.jboss.ejb.plugins.TxInterceptorBMT.invoke(TxInterceptorBMT.java:62)
            at org.jboss.ejb.plugins.StatefulSessionInstanceInterceptor.invoke(StatefulSessionInstanceInterceptor.java:273)
            at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
            at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
            at org.jboss.ejb.StatefulSessionContainer.internalInvoke(StatefulSessionContainer.java:449)
            at org.jboss.ejb.Container.invoke(Container.java:709)
            at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:198)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
            at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:97)
            at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:115)
            at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:101)
            at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
            at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:55)
            at org.jboss.proxy.ejb.StatefulSessionInterceptor.invoke(StatefulSessionInterceptor.java:105)
            at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:86)
            at $Proxy171.startBatchSession(Unknown Source)
            at com.gtnet.j2ee.comms.UniversalController.startBatchSession(UniversalController.java:535)
            at com.gtnet.batch.BatchAcceptor.doGet(BatchAcceptor.java:69)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:697)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
            at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:75)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:214)
            at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
            at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
            at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198)
            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152)
            at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
            at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:66)
            at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
            at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:162)
            at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
            at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137)
            at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118)
            at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
            at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:535)
            at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
            at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
            at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
            at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
            at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
            at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160)
            at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799)
            at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705)
            at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
            at java.lang.Thread.run(Thread.java:534)

            "http-0.0.0.0-7280-Processor23" daemon prio=1 tid=0x5521fe38 nid=0x7771 in Object.wait() [4f5e8000..4f5e8868]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x627868b0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:429)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
            - locked <0x627868b0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:534)

            "http-0.0.0.0-7280-Processor22" daemon prio=1 tid=0x555aaa08 nid=0x7771 sleeping[4f667000..4f669868]
            at java.lang.Thread.sleep(Native Method)
            at com.gtnet.workflow.enactmentService.ejb.ClientAPIBean.getNextWorkflowMessageForClientLogin(ClientAPIBean.java:337)
            at com.gtnet.workflow.enactmentService.ejb.ClientAPIBean.getNextWorkflowMessage(ClientAPIBean.java:301)
            at sun.reflect.GeneratedMethodAccessor241.invoke(Unknown Source)
            at sun.reflect.DelegatingMethod

              • Actions
            • 3. Re: Potential issue with JGroups using JBoss 3.2.7 and Java
              ross.nelson
              ross.nelson Feb 20, 2008 11:13 AM (in response to ross.nelson)

              Doesn't look like all the dump can fit on here, posting the end of the dump and the sudden restart of the Kernel at 12:15

              "RMI TCP Accept-1987" daemon prio=1 tid=0x5635d058 nid=0x7771 runnable [56048000..56048868]
              at java.net.PlainSocketImpl.socketAccept(Native Method)
              at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
              - locked <0x5fca6688> (a java.net.PlainSocketImpl)
              at java.net.ServerSocket.implAccept(ServerSocket.java:448)
              at java.net.ServerSocket.accept(ServerSocket.java:419)
              at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
              at java.lang.Thread.run(Thread.java:534)

              "ScannerThread" daemon prio=1 tid=0x5635b050 nid=0x7771 runnable [560c9000..560c9868]
              at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
              at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228)
              at java.io.File.isDirectory(File.java:701)
              at org.jboss.net.protocol.file.FileURLLister.doesDirTriggersRecursiveSearch(FileURLLister.java:101)
              at org.jboss.net.protocol.file.FileURLLister.listFiles(FileURLLister.java:75)
              at org.jboss.net.protocol.file.FileURLLister.listMembers(FileURLLister.java:40)
              at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:376)
              - locked <0x5fcb7420> (a java.util.Collections$SynchronizedRandomAccessList)
              - locked <0x5fcb9598> (a org.jboss.deployment.scanner.URLDeploymentScanner)
              at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:201)
              at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:212)
              at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:191)

              "Thread-0" daemon prio=1 tid=0x5634aa48 nid=0x7771 in Object.wait() [5614a000..5614a868]
              at java.lang.Object.wait(Native Method)
              - waiting on <0x5fb54ee0> (a java.util.TaskQueue)
              at java.util.TimerThread.mainLoop(Timer.java:429)
              - locked <0x5fb54ee0> (a java.util.TaskQueue)
              at java.util.TimerThread.run(Timer.java:382)

              "DestroyJavaVM" prio=1 tid=0x59217710 nid=0x7771 waiting on condition [0..bfffb6d4]

              "Signal Dispatcher" daemon prio=1 tid=0x59204000 nid=0x7771 waiting on condition [0..0]

              "Surrogate Locker Thread (CMS)" daemon prio=1 tid=0x59202b88 nid=0x7771 waiting on condition [0..591ff9e8]

              "Finalizer" daemon prio=1 tid=0x59200690 nid=0x7771 runnable [593c4000..593c4868]
              at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
              at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83)
              at java.lang.ref.Finalizer.access$100(Finalizer.java:14)
              at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)

              "Reference Handler" daemon prio=1 tid=0x080f9d08 nid=0x7771 in Object.wait() [59445000..59445868]
              at java.lang.Object.wait(Native Method)
              - waiting on <0x5f9655d0> (a java.lang.ref.Reference$Lock)
              at java.lang.Object.wait(Object.java:429)
              at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115)
              - locked <0x5f9655d0> (a java.lang.ref.Reference$Lock)

              "VM Thread" prio=1 tid=0x080f92b0 nid=0x7771 runnable

              "VM Periodic Task Thread" prio=1 tid=0x59208378 nid=0x7771 waiting on condition
              "Suspend Checker Thread" prio=1 tid=0x592035c8 nid=0x7771 runnable
              12:15:20,433 ERROR [Log4jService$ThrowableListenerLoggingAdapter] Unhandled Throwable
              java.lang.IllegalStateException: Transaction has terminated
              at org.jboss.tm.TransactionImpl.lock(TransactionImpl.java:983)
              at org.jboss.tm.TransactionImpl.timedOut(TransactionImpl.java:233)
              at org.jboss.util.timeout.TimeoutFactory$TimeoutWorker.run(TimeoutFactory.java:105)
              at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:134)
              at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
              at java.lang.Thread.run(Thread.java:534)
              12:15:21,492 WARN [TransactionImpl] Lock contention, tx=TransactionImpl:XidImpl [FormatId=257, GlobalId=cathlbgts01//258801, BranchQual=]
              12:15:21,904 WARN [TransactionImpl] Lock contention, tx=TransactionImpl:XidImpl [FormatId=257, GlobalId=cathlbgts01//258828, BranchQual=]
              12:15:21,901 WARN [TransactionImpl] Lock contention, tx=TransactionImpl:XidImpl [FormatId=257, GlobalId=cathlbgts01//258823, BranchQual=]
              12:15:21,250 WARN [GMS] checkSelfInclusion() failed, cathlbgts01:35704 (additional data: 15 bytes) is not a member of view [cathlbgts04:37194 (additional data: 16 bytes)|21] [cathlbgts04:37194 (additional data: 16 bytes), cathlbgts07:38148 (additional data: 16 bytes), cathlbgts02:37356 (additional data: 15 bytes), cathlbgts06:39566 (additional data: 16 bytes)]; discarding view
              12:15:21,491 WARN [TransactionImpl] Transaction TransactionImpl:XidImpl [FormatId=257, GlobalId=cathlbgts01//258801, BranchQual=] timed out. status=STATUS_ACTIVE
              12:15:22,524 WARN [TransactionImpl] Transaction TransactionImpl:XidImpl [FormatId=257, GlobalId=cathlbgts01//258828, BranchQual=] timed out. status=STATUS_ACTIVE
              12:15:21,731 WARN [TransactionImpl] Transaction TransactionImpl:XidImpl [FormatId=257, GlobalId=cathlbgts01//258805, BranchQual=] timed out. status=STATUS_ACTIVE
              12:15:21,703 WARN [TransactionImpl] Transaction TransactionImpl:XidImpl [FormatId=257, GlobalId=cathlbgts01//258823, BranchQual=] timed out. status=STATUS_ACTIVE
              12:15:23,245 WARN [TransactionImpl] Lock contention, tx=TransactionImpl:XidImpl [FormatId=257, GlobalId=cathlbgts01//258805, BranchQual=]
              12:15:23,434 WARN [GMS] I (cathlbgts01:35704 (additional data: 15 bytes)) am being shunned, will leave and rejoin group (prev_members are [cathlbgts04:37194 (additional data: 16 bytes) cathlbgts07:38148 (additional data: 16 bytes) cathlbgts02:37356 (additional data: 15 bytes) cathlbgts01:35704 (additional data: 15 bytes) cathlbgts06:39562 (additional data: 16 bytes) cathlbgts06:39566 (additional data: 16 bytes) ])
              12:15:24,813 INFO [JChannel] received an EXIT event, will leave the channel
              12:15:25,022 INFO [JChannel] closing the channel
              12:15:25,055 INFO [DefaultPartition] Suspected member: cathlbgts01:35704 (additional data: 15 bytes)
              12:15:26,359 WARN [TxConnectionManager] Prepare called on a local tx. Use of local transactions on a jta transaction with more than one branch may result in inconsistent data in some cases of failure.
              12:15:29,007 INFO [JChannel] reconnecting to group DefaultPartition
              12:15:36,698 INFO [UDP] unicast sockets will use interface 10.9.118.2
              12:15:37,231 INFO [UDP] socket information:
              local_addr=cathlbgts01:35708 (additional data: 15 bytes), mcast_addr=228.1.2.4:45567, bind_addr=/10.9.118.2, ttl=32
              sock: bound to 10.9.118.2:35708, receive buffer size=131071, send buffer size=131071
              mcast_recv_sock: bound to 10.9.118.2:45567, send buffer size=131071, receive buffer size=131071
              mcast_send_sock: bound to 10.9.118.2:35709, send buffer size=131071, receive buffer size=131071
              12:15:37,863 INFO [STDOUT]
              -------------------------------------------------------
              GMS: address is cathlbgts01:35708 (additional data: 15 bytes)
              -------------------------------------------------------
              12:15:40,546 WARN [TxConnectionManager] Prepare called on a local tx. Use of local transactions on a jta transaction with more than one branch may result in inconsistent data in some cases of failure.
              12:15:40,642 WARN [TxConnectionManager] Prepare called on a local tx. Use of local transactions on a jta transaction with more than one branch may result in inconsistent data in some cases of failure.

                • Actions
              • 4. Re: Potential issue with JGroups using JBoss 3.2.7 and Java
                ross.nelson
                ross.nelson Feb 20, 2008 11:26 AM (in response to ross.nelson)

                Finally I thought it worthwhile posting a view of what happened around the time of the "hang" from the STD Out logs of one of the unaffected kernels...

                1 kernel leaves and rejoins then around 2 minutes later the problem kernel leaves the cluster.

                11:23:45,111 INFO [DefaultPartition] Suspected member: cathlbgts06:39562 (additional data: 16 bytes)
                11:23:45,111 INFO [DefaultPartition] New cluster view for partition DefaultPartition (id: 19, delta: -1) : [10.9.181.48:1104, 10.9.181.51:1104, 10.9.118.3:1104, 10.9.118.2:1104]
                11:23:45,112 INFO [DefaultPartition] I am (10.9.181.48:1104) received membershipChanged event:
                11:23:45,112 INFO [DefaultPartition] Dead members: 1 ([10.9.181.50:1104])
                11:23:45,112 INFO [DefaultPartition] New Members : 0 ([])
                11:23:45,112 INFO [DefaultPartition] All Members : 4 ([10.9.181.48:1104, 10.9.181.51:1104, 10.9.118.3:1104, 10.9.118.2:1104])
                11:23:46,394 INFO [DefaultPartition] New cluster view for partition DefaultPartition (id: 20, delta: 1) : [10.9.181.48:1104, 10.9.181.51:1104, 10.9.118.3:1104, 10.9.118.2:1104, 10.9.181.50:1104]
                11:23:46,394 INFO [DefaultPartition] I am (10.9.181.48:1104) received membershipChanged event:
                11:23:46,394 INFO [DefaultPartition] Dead members: 0 ([])
                11:23:46,395 INFO [DefaultPartition] New Members : 1 ([10.9.181.50:1104])
                11:23:46,395 INFO [DefaultPartition] All Members : 5 ([10.9.181.48:1104, 10.9.181.51:1104, 10.9.118.3:1104, 10.9.118.2:1104, 10.9.181.50:1104])
                26651.767: [Rescan (parallel) , 0.0052600 secs]26726.655: [Rescan (parallel) , 0.0429900 secs]11:25:42,333 INFO [DefaultPartition] Suspected member: cathlbgts01:35704 (additional data: 15 bytes)
                11:25:42,335 INFO [DefaultPartition] New cluster view for partition DefaultPartition (id: 21, delta: -1) : [10.9.181.48:1104, 10.9.181.51:1104, 10.9.118.3:1104, 10.9.181.50:1104]
                11:25:42,335 INFO [DefaultPartition] I am (10.9.181.48:1104) received membershipChanged event:
                11:25:42,335 INFO [DefaultPartition] Dead members: 1 ([10.9.118.2:1104])
                11:25:42,335 INFO [DefaultPartition] New Members : 0 ([])
                11:25:42,335 INFO [DefaultPartition] All Members : 4 ([10.9.181.48:1104, 10.9.181.51:1104, 10.9.118.3:1104, 10.9.181.50:1104])



                Thanks in advance for any assistance.

                  • Actions
                • 5. Re: Potential issue with JGroups using JBoss 3.2.7 and Java
                  belaban
                  belaban Feb 21, 2008 3:48 AM (in response to ross.nelson)

                  I don't see anything suspicious going on in the JGroups threads. The MessageDispatcher up thread dump looks okay, it simply blocks, waiting for work.

                    • Actions
                  • 6. Re: Potential issue with JGroups using JBoss 3.2.7 and Java
                    ross.nelson
                    ross.nelson Feb 21, 2008 3:52 AM (in response to ross.nelson)

                    Thanks for the reply Bela, I'll certainly pursue the JVM / Garbage Collection angle as well.

                      • Actions
                    • 7. Re: Potential issue with JGroups using JBoss 3.2.7 and Java
                      brian.stansberry
                      brian.stansberry Feb 21, 2008 10:22 AM (in response to ross.nelson)

                      If this sort of thing happens, take several thread dumps and look for threads that aren't progressing. As with the MessageDispatcherThread here though, a thread that's "not progressing" because it's sitting in wait() is usually not an issue. Look for threads that aren't progressing while they wait to acquire a lock; then find the thread the holds the lock. No guarantee that you'll find your problem, but it's an approach that often helps.

                      Here I *suspect* there was something going on with one of the transactions referred to in the log messages at the end. No idea what though.

                        • Actions