1 Reply Latest reply on Oct 30, 2008 5:46 AM by Jeremy Stone

    Deadlock when receiving messages via https connector

    Jeremy Stone Newbie

      Topic subscriptions that use https connection factory can result in deadlocks when BlockingCallbackStore is used.

      See stack traces below. But here is a summary of what appears to be happening...

      On the client
      Timer-3 thread:
      - Is polling for messages

      On the application server
      http-0.0.0.0-4459-5 thread (the server-side continuation of Client
      Timer-3):
      - Is waiting to lock the ArrayList of callbacks held by
      the eventListenerContainer-10 thread.

      eventListenerContainer-10 thread:
      - Because the ServerInvokerCallbackHandler has decided that it needs to
      persist the callback and is using a BlockingCallbackStore,
      eventListenerContainer-10 is also waiting on an object that will be release
      when someone calls BlockingCallbackStore getNext() (i.e. its waiting for
      http-0.0.0.0-4459-5)

      Hence deadlock.

      This also results in other appication server threads locking up when they attempt to commit messages posted to the destination.

      We are using JBM 1.4.0.SP3 with JBR 2.2.2.SP4.

      (Not sure if its a JB remoting or a JB messaging issue but it manifests in messaging so I've posted it here for now.)

      Do you have any workaround suggestions? (Thought about using the non-blocking CallbackStore for the connector maybe but the remoting-https-service.xml says not to change these attributes.)

      Client stack (for Timer-3 only):

      "Timer-3" daemon prio=6 tid=0x2392d800 nid=0x1050 runnable [0x2677e000..0x2677fb94]
       java.lang.Thread.State: RUNNABLE
       at java.net.SocketInputStream.socketRead0(Native Method)
       at java.net.SocketInputStream.read(Unknown Source)
       at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
       at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
       at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
       - locked <0x0500b5f0> (a java.lang.Object)
       at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
       at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
       - locked <0x0500b6c8> (a com.sun.net.ssl.internal.ssl.AppInputStream)
       at java.io.BufferedInputStream.fill(Unknown Source)
       at java.io.BufferedInputStream.read1(Unknown Source)
       at java.io.BufferedInputStream.read(Unknown Source)
       - locked <0x05023b88> (a java.io.BufferedInputStream)
       at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
       at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
       at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
       - locked <0x05023bc0> (a sun.net.www.protocol.https.DelegateHttpsURLConnection)
       at java.net.HttpURLConnection.getResponseCode(Unknown Source)
       at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
       at org.jboss.remoting.transport.http.HTTPClientInvoker.useHttpURLConnection(HTTPClientInvoker.java:281)
       at org.jboss.remoting.transport.http.HTTPClientInvoker.transport(HTTPClientInvoker.java:136)
       at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
       at org.jboss.remoting.Client.invoke(Client.java:1634)
       at org.jboss.remoting.Client.invoke(Client.java:548)
       at org.jboss.remoting.Client.getCallbacks(Client.java:1168)
       at org.jboss.remoting.callback.CallbackPoller.run(CallbackPoller.java:188)
       - locked <0x046304f0> (a org.jboss.remoting.callback.CallbackPoller)
       at java.util.TimerThread.mainLoop(Unknown Source)
       at java.util.TimerThread.run(Unknown Source)
      


      Application server stack (uninteresting threads elided for clarity - can send full stack if reqd):

      2008-09-25 12:09:44
      Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.6.0_03-b05 mixed mode):
      
      "Attach Listener" daemon prio=10 tid=0x0000002af06fcc00 nid=0x36bf waiting on condition [0x0000000000000000..0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
      
       Locked ownable synchronizers:
       - None
      
      "http-0.0.0.0-4459-5" daemon prio=10 tid=0x0000002af2a9c400 nid=0x2bca waiting for monitor entry [0x000000004cff7000..0x000000004cff7a30]
       java.lang.Thread.State: BLOCKED (on object monitor)
       at org.jboss.remoting.callback.ServerInvokerCallbackHandler.getCallbacks(ServerInvokerCallbackHandler.java:600)
       - waiting to lock <0x0000002ac1765dd0> (a java.util.ArrayList)
       at org.jboss.remoting.ServerInvoker.handleInternalInvocation(ServerInvoker.java:1513)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:782)
       at org.jboss.remoting.transport.coyote.CoyoteInvoker.service(CoyoteInvoker.java:310)
       at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
       at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
       at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
       at java.lang.Thread.run(Thread.java:619)
      
       Locked ownable synchronizers:
       - None
      
      "http-0.0.0.0-4459-3" daemon prio=10 tid=0x0000002aee182400 nid=0x2bc7 runnable [0x000000004b8e0000..0x000000004b8e0bb0]
       java.lang.Thread.State: RUNNABLE
       at java.net.SocketInputStream.socketRead0(Native Method)
       at java.net.SocketInputStream.read(SocketInputStream.java:129)
       at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
       at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
       at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
       - locked <0x0000002aa3a1e688> (a java.lang.Object)
       at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
       at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
       - locked <0x0000002aa3a1e740> (a com.sun.net.ssl.internal.ssl.AppInputStream)
       at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:700)
       at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
       at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:805)
       at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
       at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
       at java.lang.Thread.run(Thread.java:619)
      
       Locked ownable synchronizers:
       - None
      
      "http-0.0.0.0-4459-1" daemon prio=10 tid=0x0000002ae8ef5c00 nid=0x2bb7 runnable [0x0000000050a31000..0x0000000050a31c30]
       java.lang.Thread.State: RUNNABLE
       at java.net.SocketInputStream.socketRead0(Native Method)
       at java.net.SocketInputStream.read(SocketInputStream.java:129)
       at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
       at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
       at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
       - locked <0x0000002aa787c578> (a java.lang.Object)
       at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
       at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
       - locked <0x0000002aa78848c0> (a com.sun.net.ssl.internal.ssl.AppInputStream)
       at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:700)
       at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
       at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:805)
       at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
       at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
       at java.lang.Thread.run(Thread.java:619)
      
       Locked ownable synchronizers:
       - None
      
      
      "http-0.0.0.0-443-15" daemon prio=10 tid=0x0000002ae6ea7800 nid=0x2703 waiting for monitor entry [0x000000004f316000..0x000000004f31aa30]
       java.lang.Thread.State: BLOCKED (on object monitor)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1299)
       - waiting to lock <0x0000002ac174ba68> (a org.jboss.jms.server.endpoint.ServerSessionEndpoint)
       at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:322)
       - locked <0x0000002ac1810058> (a java.lang.Object)
       at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
       at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:582)
       at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
       at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:606)
       at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:505)
       at org.jboss.messaging.core.impl.ChannelSupport$InMemoryCallback.afterCommit(ChannelSupport.java:935)
       - locked <0x0000002ac1764c68> (a java.lang.Object)
       at org.jboss.messaging.core.impl.tx.Transaction.commit(Transaction.java:228)
       - locked <0x0000002ad2852870> (a org.jboss.messaging.core.impl.tx.Transaction)
       at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendTransaction(ServerConnectionEndpoint.java:474)
       at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.org$jboss$jms$server$endpoint$advised$ConnectionAdvised$sendTransaction$aop(ConnectionAdvised.java:101)
       at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$sendTransaction_N3268650789275322226.invokeNext(ConnectionAdvised$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.server.container.SecurityAspect.handleSendTransaction(SecurityAspect.java:195)
       at sun.reflect.GeneratedMethodAccessor463.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:119)
       at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$sendTransaction_N3268650789275322226.invokeNext(ConnectionAdvised$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
       at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$sendTransaction_N3268650789275322226.invokeNext(ConnectionAdvised$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.sendTransaction(ConnectionAdvised.java)
       at org.jboss.jms.wireformat.ConnectionSendTransactionRequest.serverInvoke(ConnectionSendTransactionRequest.java:82)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:795)
       at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
       at org.jboss.remoting.Client.invoke(Client.java:1634)
       at org.jboss.remoting.Client.invoke(Client.java:548)
       at org.jboss.remoting.Client.invoke(Client.java:536)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate.org$jboss$jms$client$delegate$ClientConnectionDelegate$sendTransaction$aop(ClientConnectionDelegate.java:221)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N3268650789275322226.invokeNext(ClientConnectionDelegate$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:106)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N3268650789275322226.invokeNext(ClientConnectionDelegate$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:106)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N3268650789275322226.invokeNext(ClientConnectionDelegate$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate.sendTransaction(ClientConnectionDelegate.java)
       at org.jboss.jms.tx.ResourceManager.sendTransactionXA(ResourceManager.java:637)
       at org.jboss.jms.tx.ResourceManager.commit(ResourceManager.java:370)
       at org.jboss.jms.tx.MessagingXAResource.commit(MessagingXAResource.java:238)
       at org.jboss.resource.connectionmanager.xa.JcaXAResourceWrapper.commit(JcaXAResourceWrapper.java:53)
       at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:636)
       at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2619)
       at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1779)
       - locked <0x0000002ad2852ba8> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction)
       at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:88)
       at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:177)
       at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1389)
       at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:135)
       at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:87)
       at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.commit(ServerVMClientUserTransaction.java:140)
       at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1028)
       at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:709)
       at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:678)
       at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:321)
       at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)
       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
       at $Proxy86.startServer(Unknown Source)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at com.cyclone.services.AdaptorProxyFactoryBean$_A.invoke(np:197)
       at $Proxy112.startServer(Unknown Source)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
       at $Proxy113.startServer(Unknown Source)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.springframework.remoting.support.RemoteInvocation.invoke(RemoteInvocation.java:205)
       at org.springframework.remoting.support.DefaultRemoteInvocationExecutor.invoke(DefaultRemoteInvocationExecutor.java:38)
       at org.springframework.remoting.support.RemoteInvocationBasedExporter.invoke(RemoteInvocationBasedExporter.java:78)
       at com.cyclone.services.cluster.ClusteredHttpInvokerServiceExporter.invokeAndCreateResult(np:50)
       at org.springframework.remoting.httpinvoker.HttpInvokerServiceExporter.handleRequest(HttpInvokerServiceExporter.java:74)
       at org.springframework.web.servlet.mvc.HttpRequestHandlerAdapter.handle(HttpRequestHandlerAdapter.java:49)
       at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)
       at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:809)
       at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
       at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
       at com.cyclone.netPrefect.webservices.initialization.ApplicationRunningFilter.doFilter(ApplicationRunningFilter.java:80)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
       at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
       at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
       at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:524)
       at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
       at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
       at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
       at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
       at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
       at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
       at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
       at java.lang.Thread.run(Thread.java:619)
      
       Locked ownable synchronizers:
       - None
      
      "http-0.0.0.0-443-14" daemon prio=10 tid=0x0000002ae9746400 nid=0x2702 waiting for monitor entry [0x000000004f214000..0x000000004f219ab0]
       java.lang.Thread.State: BLOCKED (on object monitor)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1299)
       - waiting to lock <0x0000002ac174ba68> (a org.jboss.jms.server.endpoint.ServerSessionEndpoint)
       at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:322)
       - locked <0x0000002ac1942870> (a java.lang.Object)
       at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
       at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:582)
       at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
       at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:606)
       at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:505)
       at org.jboss.messaging.core.impl.ChannelSupport$InMemoryCallback.afterCommit(ChannelSupport.java:935)
       - locked <0x0000002ac17650a8> (a java.lang.Object)
       at org.jboss.messaging.core.impl.tx.Transaction.commit(Transaction.java:228)
       - locked <0x0000002ad2853440> (a org.jboss.messaging.core.impl.tx.Transaction)
       at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendTransaction(ServerConnectionEndpoint.java:490)
       at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.org$jboss$jms$server$endpoint$advised$ConnectionAdvised$sendTransaction$aop(ConnectionAdvised.java:101)
       at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$sendTransaction_N3268650789275322226.invokeNext(ConnectionAdvised$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.server.container.SecurityAspect.handleSendTransaction(SecurityAspect.java:195)
       at sun.reflect.GeneratedMethodAccessor463.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:119)
       at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$sendTransaction_N3268650789275322226.invokeNext(ConnectionAdvised$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
       at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$sendTransaction_N3268650789275322226.invokeNext(ConnectionAdvised$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.sendTransaction(ConnectionAdvised.java)
       at org.jboss.jms.wireformat.ConnectionSendTransactionRequest.serverInvoke(ConnectionSendTransactionRequest.java:82)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:795)
       at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
       at org.jboss.remoting.Client.invoke(Client.java:1634)
       at org.jboss.remoting.Client.invoke(Client.java:548)
       at org.jboss.remoting.Client.invoke(Client.java:536)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate.org$jboss$jms$client$delegate$ClientConnectionDelegate$sendTransaction$aop(ClientConnectionDelegate.java:221)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N3268650789275322226.invokeNext(ClientConnectionDelegate$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:106)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N3268650789275322226.invokeNext(ClientConnectionDelegate$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:106)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N3268650789275322226.invokeNext(ClientConnectionDelegate$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate.sendTransaction(ClientConnectionDelegate.java)
       at org.jboss.jms.tx.ResourceManager.sendTransactionXA(ResourceManager.java:637)
       at org.jboss.jms.tx.ResourceManager.commit(ResourceManager.java:393)
       at org.jboss.jms.tx.MessagingXAResource.commit(MessagingXAResource.java:238)
       at org.jboss.resource.connectionmanager.xa.JcaXAResourceWrapper.commit(JcaXAResourceWrapper.java:53)
       at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelCommit(XAResourceRecord.java:487)
       at com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:3107)
       at com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:3022)
       at com.arjuna.ats.arjuna.coordinator.BasicAction.phase2Commit(BasicAction.java:2126)
       - locked <0x0000002ad2853778> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction)
       at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1819)
       - locked <0x0000002ad2853778> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction)
       at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:88)
       at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:177)
       at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1389)
       at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:135)
       at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:87)
       at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.commit(ServerVMClientUserTransaction.java:140)
       at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1028)
       at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:709)
       at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:678)
       at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:321)
       at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)
       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
       at $Proxy104.createOrUpdateAutoDiscoveredTransport(Unknown Source)
       at sun.reflect.GeneratedMethodAccessor886.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
       at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
       at org.springframework.aop.framework.adapter.ThrowsAdviceInterceptor.invoke(ThrowsAdviceInterceptor.java:126)
       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
       at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:50)
       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
       at $Proxy105.createOrUpdateAutoDiscoveredTransport(Unknown Source)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at com.cyclone.services.AdaptorProxyFactoryBean$_A.invoke(np:197)
       at $Proxy110.createOrUpdateAutoDiscoveredTransport(Unknown Source)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
       at $Proxy111.createOrUpdateAutoDiscoveredTransport(Unknown Source)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.springframework.remoting.support.RemoteInvocation.invoke(RemoteInvocation.java:205)
       at org.springframework.remoting.support.DefaultRemoteInvocationExecutor.invoke(DefaultRemoteInvocationExecutor.java:38)
       at org.springframework.remoting.support.RemoteInvocationBasedExporter.invoke(RemoteInvocationBasedExporter.java:78)
       at com.cyclone.services.cluster.ClusteredHttpInvokerServiceExporter.invokeAndCreateResult(np:50)
       at org.springframework.remoting.httpinvoker.HttpInvokerServiceExporter.handleRequest(HttpInvokerServiceExporter.java:74)
       at org.springframework.web.servlet.mvc.HttpRequestHandlerAdapter.handle(HttpRequestHandlerAdapter.java:49)
       at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)
       at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:809)
       at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
       at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
       at com.cyclone.netPrefect.webservices.initialization.ApplicationRunningFilter.doFilter(ApplicationRunningFilter.java:80)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
       at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
       at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
       at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:524)
       at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
       at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
       at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
       at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
       at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
       at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
       at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
       at java.lang.Thread.run(Thread.java:619)
      
       Locked ownable synchronizers:
       - None
      
      "eventListenerContainer-10" prio=10 tid=0x0000002ae697f800 nid=0x268c in Object.wait() [0x000000004b3d9000..0x000000004b3dbab0]
       java.lang.Thread.State: WAITING (on object monitor)
       at java.lang.Object.wait(Native Method)
       - waiting on <0x0000002ac17cadc8> (a java.lang.Object)
       at java.lang.Object.wait(Object.java:485)
       at org.jboss.remoting.callback.BlockingCallbackStore.add(BlockingCallbackStore.java:102)
       - locked <0x0000002ac17cadc8> (a java.lang.Object)
       at org.jboss.remoting.callback.ServerInvokerCallbackHandler.persistCallback(ServerInvokerCallbackHandler.java:863)
       at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:751)
       - locked <0x0000002ac1765dd0> (a java.util.ArrayList)
       at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:697)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1432)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1344)
       - locked <0x0000002ac174ba68> (a org.jboss.jms.server.endpoint.ServerSessionEndpoint)
       at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:322)
       - locked <0x0000002ac180fe30> (a java.lang.Object)
       at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
       at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:582)
       at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
       at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:606)
       at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:505)
       at org.jboss.messaging.core.impl.ChannelSupport$InMemoryCallback.afterCommit(ChannelSupport.java:935)
       - locked <0x0000002ac1764a48> (a java.lang.Object)
       at org.jboss.messaging.core.impl.tx.Transaction.commit(Transaction.java:228)
       - locked <0x0000002ac9974110> (a org.jboss.messaging.core.impl.tx.Transaction)
       at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendTransaction(ServerConnectionEndpoint.java:490)
       at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.org$jboss$jms$server$endpoint$advised$ConnectionAdvised$sendTransaction$aop(ConnectionAdvised.java:101)
       at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$sendTransaction_N3268650789275322226.invokeNext(ConnectionAdvised$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.server.container.SecurityAspect.handleSendTransaction(SecurityAspect.java:195)
       at sun.reflect.GeneratedMethodAccessor463.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:119)
       at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$sendTransaction_N3268650789275322226.invokeNext(ConnectionAdvised$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
       at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$sendTransaction_N3268650789275322226.invokeNext(ConnectionAdvised$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.sendTransaction(ConnectionAdvised.java)
       at org.jboss.jms.wireformat.ConnectionSendTransactionRequest.serverInvoke(ConnectionSendTransactionRequest.java:82)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:795)
       at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
       at org.jboss.remoting.Client.invoke(Client.java:1634)
       at org.jboss.remoting.Client.invoke(Client.java:548)
       at org.jboss.remoting.Client.invoke(Client.java:536)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate.org$jboss$jms$client$delegate$ClientConnectionDelegate$sendTransaction$aop(ClientConnectionDelegate.java:221)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N3268650789275322226.invokeNext(ClientConnectionDelegate$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:106)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N3268650789275322226.invokeNext(ClientConnectionDelegate$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:106)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N3268650789275322226.invokeNext(ClientConnectionDelegate$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate.sendTransaction(ClientConnectionDelegate.java)
       at org.jboss.jms.tx.ResourceManager.sendTransactionXA(ResourceManager.java:637)
       at org.jboss.jms.tx.ResourceManager.commit(ResourceManager.java:393)
       at org.jboss.jms.tx.MessagingXAResource.commit(MessagingXAResource.java:238)
       at org.jboss.resource.connectionmanager.xa.JcaXAResourceWrapper.commit(JcaXAResourceWrapper.java:53)
       at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelCommit(XAResourceRecord.java:487)
       at com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:3107)
       at com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:3022)
       at com.arjuna.ats.arjuna.coordinator.BasicAction.phase2Commit(BasicAction.java:2126)
       - locked <0x0000002ac9974448> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction)
       at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1819)
       - locked <0x0000002ac9974448> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction)
       at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:88)
       at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:177)
       at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1389)
       at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:135)
       at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:87)
       at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.commit(ServerVMClientUserTransaction.java:140)
       at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1028)
       at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:709)
       at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:678)
       at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:321)
       at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)
       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
       at $Proxy100.onMessage(Unknown Source)
       at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:531)
       at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:466)
       at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:435)
       at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:322)
       at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:260)
       at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:944)
       at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:868)
       at java.lang.Thread.run(Thread.java:619)
      
       Locked ownable synchronizers:
       - None
      
      Lots more threads like this...
      "eventListenerContainer-9" prio=10 tid=0x0000002ae6604000 nid=0x2689 waiting for monitor entry [0x000000004b0d6000..0x000000004b0d8c30]
       java.lang.Thread.State: BLOCKED (on object monitor)
       at org.jboss.messaging.core.impl.ChannelSupport$InMemoryCallback.afterCommit(ChannelSupport.java:901)
       - waiting to lock <0x0000002ac1764a48> (a java.lang.Object)
       at org.jboss.messaging.core.impl.tx.Transaction.commit(Transaction.java:228)
       - locked <0x0000002ac9d1eaf0> (a org.jboss.messaging.core.impl.tx.Transaction)
       at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendTransaction(ServerConnectionEndpoint.java:490)
       at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.org$jboss$jms$server$endpoint$advised$ConnectionAdvised$sendTransaction$aop(ConnectionAdvised.java:101)
       at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$sendTransaction_N3268650789275322226.invokeNext(ConnectionAdvised$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.server.container.SecurityAspect.handleSendTransaction(SecurityAspect.java:195)
       at sun.reflect.GeneratedMethodAccessor463.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:119)
       at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$sendTransaction_N3268650789275322226.invokeNext(ConnectionAdvised$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
       at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$sendTransaction_N3268650789275322226.invokeNext(ConnectionAdvised$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.sendTransaction(ConnectionAdvised.java)
       at org.jboss.jms.wireformat.ConnectionSendTransactionRequest.serverInvoke(ConnectionSendTransactionRequest.java:82)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:795)
       at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
       at org.jboss.remoting.Client.invoke(Client.java:1634)
       at org.jboss.remoting.Client.invoke(Client.java:548)
       at org.jboss.remoting.Client.invoke(Client.java:536)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate.org$jboss$jms$client$delegate$ClientConnectionDelegate$sendTransaction$aop(ClientConnectionDelegate.java:221)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N3268650789275322226.invokeNext(ClientConnectionDelegate$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:106)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N3268650789275322226.invokeNext(ClientConnectionDelegate$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:106)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N3268650789275322226.invokeNext(ClientConnectionDelegate$sendTransaction_N3268650789275322226.java)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate.sendTransaction(ClientConnectionDelegate.java)
       at org.jboss.jms.tx.ResourceManager.sendTransactionXA(ResourceManager.java:637)
       at org.jboss.jms.tx.ResourceManager.commit(ResourceManager.java:393)
       at org.jboss.jms.tx.MessagingXAResource.commit(MessagingXAResource.java:238)
       at org.jboss.resource.connectionmanager.xa.JcaXAResourceWrapper.commit(JcaXAResourceWrapper.java:53)
       at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelCommit(XAResourceRecord.java:487)
       at com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:3107)
       at com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:3022)
       at com.arjuna.ats.arjuna.coordinator.BasicAction.phase2Commit(BasicAction.java:2126)
       - locked <0x0000002ac99746c0> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction)
       at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1819)
       - locked <0x0000002ac99746c0> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction)
       at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:88)
       at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:177)
       at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1389)
       at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:135)
       at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:87)
       at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.commit(ServerVMClientUserTransaction.java:140)
       at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1028)
       at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:709)
       at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:678)
       at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:321)
       at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)
       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
       at $Proxy100.onMessage(Unknown Source)
       at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:531)
       at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:466)
       at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:435)
       at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:322)
       at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:260)
       at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:944)
       at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:868)
       at java.lang.Thread.run(Thread.java:619)
      
       Locked ownable synchronizers:
       - None