4 Replies Latest reply on Mar 24, 2011 1:30 PM by ataylor

    HornetQ hanging in setTransactionTimeout

    jeremystone

      Very occassionally we have had strange behaviour where on sending a message to a topic, the thread hangs while setting the transaction timeout (see trace below). We're using HornetQ 2.1.2 and the client is in the local VM and is using the InVMConnector.

       

      Basically no response is received to the send of the SessionXASetTimeoutMessage, so the thread waits until it eventually times out.

       

      On receipt of the message the 'server' end of the session's xa timeout is set and the response is sent back using an IOAsyncTask passed to OperationContextImpl's executeOnCompletion(...). Given that some response is send back even when an exception is thrown, my only thought is that perhaps the OperationContextImpl instance is not processing tasks any more??

       

      Have tried to debug but not got to the bottom of why it might happen (I'm not sure how to reproduce it). I notice that OperationContextImpl has had a finally block added to the Runnable in the execute method. Was this added to fix similar behaviour?

       

      Any idead? Has anyone else seen this?

       

      Stack dump from hanging thread:

      sun.misc.Unsafe.park(Native Method)

          java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

          java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2116)

          org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:257)

          org.hornetq.core.client.impl.ClientSessionImpl.setTransactionTimeout(ClientSessionImpl.java:1411)

          org.hornetq.ra.HornetQRAXAResource.setTransactionTimeout(HornetQRAXAResource.java:241)

          org.jboss.resource.connectionmanager.xa.JcaXAResourceWrapper.setTransactionTimeout(JcaXAResourceWrapper.java:108)

          com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:750)

          com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:497)

          org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener$TransactionSynchronization.enlist(TxConnectionManager.java:830)

          org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:584)

          org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:352)

          org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:525)

          org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:406)

          org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:850)

          org.hornetq.ra.HornetQRASessionFactoryImpl.allocateConnection(HornetQRASessionFactoryImpl.java:837)

          org.hornetq.ra.HornetQRASessionFactoryImpl.createSession(HornetQRASessionFactoryImpl.java:470)

          org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196)

          org.springframework.jms.core.JmsTemplate.access$1(JmsTemplate.java:1)

          org.springframework.jms.core.JmsTemplate$JmsTemplateResourceFactory.createSession(JmsTemplate.java:1035)

          org.springframework.jms.connection.ConnectionFactoryUtils.doGetTransactionalSession(ConnectionFactoryUtils.java:300)

          org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:453)

          org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:543)

        • 1. HornetQ hanging in setTransactionTimeout
          ataylor

          its hard to say, i see your using spring which always muddies the waters when it comes to debugging. your not doing anything weird like using the same session from multiple threads are you?

          • 2. HornetQ hanging in setTransactionTimeout
            ataylor

            I alsso see your using the managed connection pool too, this is usually controlled by the JCA so not sure how that would effect things

            • 3. HornetQ hanging in setTransactionTimeout
              jeremystone

              Yes, the message we're sending is a notification to remote clients (subscribers to the topic) that some configuration setting has been changed in our database (hence our use of XA to co-ordinate the jdbc and jms bits of the transaction).

               

              I did manage to reproduce it (but only once - have since tried but failed) by temporarily setting a breakpoint in OperationContextImpl done() [figuring that this might cause non-immediate task execution which I suspected could be related] for a while. When I resumed the thread, and forced a message to be sent to the topic (by changing a config setting using our client) it hung up with the same thread stack. [Can send full stack dump if you might need it]

              • 4. HornetQ hanging in setTransactionTimeout
                ataylor
                Yes, the message we're sending is a notification to remote clients (subscribers to the topic) that some configuration setting has been changed in our database (hence our use of XA to co-ordinate the jdbc and jms bits of the transaction).

                would expect that

                Ok, thats fine, I was just wondering whether spring was closing its jms objects properly, this could screw things up as its a pool, usually in an MDB the JCA layer does it iirc

                 

                I did manage to reproduce it (but only once - have since tried but failed) by temporarily setting a breakpoint in OperationContextImpl done() [figuring that this might cause non-immediate task execution which I suspected could be related] for a while. When I resumed the thread, and forced a message to be sent to the topic (by changing a config setting using our client) it hung up with the same thread stack. [Can send full stack dump if you might need it]

                I would expect that to happen, you are simulating a server call hang. If the response is being sent like you say i can only see 2 way this could happen, either the connection is somehow being destroyed or Spring is being naughty and doing illegal stuff on the session, i.e. multiple threads making calls on the same session etc.

                 

                 

                Id be glad to take a look at this if you could provide an easily runnable test case, ideally without spring.