Hornetq - Producers blocked in a bean managed transaction

[manu29585]

Hello All,

But. we have a bad situation in our production system.

We are using 2.1.2.Final version of Hornetq in JBoss 6.1.0 final.

Currently, we are reusing the connection and session and are created as shown below:

cachedQueueConnection = cachedConnectionFactory.createConnection();

// KEY - register for exception callbacks

cachedQueueConnection.setExceptionListener(new ExceptionJMSListenerImpl());

session = cachedQueueConnection.createSession(false, Session.AUTO_ACKNOWLEDGE);

producer = session.createProducer(cachedQueue);

producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT);

We are pushing data to the queue as shown below:

ObjectMessage contextMessage = session.createObjectMessage();

1. contextMessage.setObject(context);
2. producer.send(contextMessage);

Application runs fine for few days and we have a situation sometimes that, "producer.send(contextMessage);"

is getting blocked. Its returning only after sometime (sometimes ~15s to 15 min).

This pushing to the queue happens within a bean managed transaction. (This transaction includes database operation such as update, insert and delete as well and as a final action, this data of collected information on what is being done is pushed to the queue )

This inturn makes the transaction inactive since the locks held by the transaction will not be committed.

There are 2 typical scenarios out of this.

It is observed sometimes that, eventhough producer call is not returned (Thread goes to the waiting state and returns later), consumer on the other end which is a MDB listening to the queue and responding to onMessage:

MDBean implements MessageListener{

public void onMessage(Message msg) {

ObjectMessage objMsg = (ObjectMessage) msg;  

• objMsg.getObject();

}

}

Sometimes, eventhough, producer.send() has not returned, on the other end, message is being already processed.

But, on the other scenario, message is not consumed.

Is there anything wrong in the code? We are failing to understand why producer is getting blocked!

Also, since this problem is appearing only on the production system, it would be great, if any of you can give a hint on generating additional logs which will enable debugging this problem.

Any hints would really be helpful.

Thanks a lot in advance!

[jbertram]

My first guess is that you're running into producer flow control.  To continue investigating I'd need to see your HornetQ broker's configuration as well as 2-3 thread dumps captured on the producer when the block occurs.

[manu29585]

Hello Justin,

Sorry for the delayed response, it took some time for the issue to appear in the production and after generating thread dump, we could see that, we are running into same problem as we see in:

https://developer.jboss.org/message/791396#791396

From the thread dump, we were able to see that, threads are in waiting state:

Thread: pool-70-thread-25 : priority:5, demon:false, threadId:616, threadState:WAITING

  - waiting on <0x6263a14b> (a java.util.concurrent.Semaphore$NonfairSync)
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
java.util.concurrent.Semaphore.acquire(Semaphore.java:441)
org.hornetq.core.client.impl.ClientProducerCreditsImpl.acquireCredits(ClientProducerCreditsImpl.java:74)
org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:305)
org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:142)
org.hornetq.jms.client.HornetQMessageProducer.doSend(HornetQMessageProducer.java:451)
org.hornetq.jms.client.HornetQMessageProducer.send(HornetQMessageProducer.java:199)

Also, we could see that, after we ran multiple iterations of performance tests, we were able to simulate this blocking by changing max-size-bytes to very small number:

<address-settings>

      <!--default for catch all-->

      <address-setting match="#">

         <dead-letter-address>jms.queue.DLQ</dead-letter-address>

         <expiry-address>jms.queue.ExpiryQueue</expiry-address>

         <!-- redeliver JMS message on transaction rollback after 10s delay -->

         <redelivery-delay>10000</redelivery-delay>

         <!-- redeliver JMS message on transaction rollback 20 times before giving up -->

         <max-delivery-attempts>20</max-delivery-attempts>

         <max-size-bytes>100</max-size-bytes>

         <message-counter-history-day-limit>10</message-counter-history-day-limit>

        <address-full-policy>BLOCK</address-full-policy>

      </address-setting>

   </address-settings>

/usr/local/jboss-6.1.0.Final/server/default/deploy/hornetq/hornetq-configuration.xml

Then we changed it to:

<address-settings>

      <!--default for catch all-->

      <address-setting match="#">

         <dead-letter-address>jms.queue.DLQ</dead-letter-address>

         <expiry-address>jms.queue.ExpiryQueue</expiry-address>

         <!-- redeliver JMS message on transaction rollback after 10s delay -->

         <redelivery-delay>10000</redelivery-delay>

         <!-- redeliver JMS message on transaction rollback 20 times before giving up -->

         <max-delivery-attempts>20</max-delivery-attempts>

         <max-size-bytes>10485760</max-size-bytes>

          <page-size-bytes>1048576</page-size-bytes>

         <message-counter-history-day-limit>10</message-counter-history-day-limit>

         <address-full-policy>PAGE</address-full-policy>

      </address-setting>

   </address-settings>

We were not able to see the threads hanging over "send".

We've recommended this setting on our production system as well to see if we still get into same problem or else we should synchronize the send.

Thanks for the support so far.

Rgds

Manohar

[manu29585]

jbertram : Did you get a chance to have a look into the above problem?

[jbertram]

Your last message indicated to me you found and fixed the issue.  Please elaborate on any further problems.

[manu29585]

jbertram

Inspite of changing the configurations, we still run into same problem that the thread gets struck in acquireCredits.

We had increased max-size-bytes to 200 MB and also changed address-full-policy to PAGE. Each of our message will be ~500 KB.

You wrote initially that, we might be running into producer flow control. Do you see a gap in the implemented code?

With the AUTO_ACK mode, when exactly, will there be an ack sent? Does it happen the moment message is written to the server or ?

Could observe in couple of scenarios though send has not returned (got struck in acquireCredits), clients have consumed this message on the other end.

Why is this behavior? Can you please help me to understand this?

Thanks in advance.

[jbertram]

You wrote initially that, we might be running into producer flow control.

Yes, I assumed you were hitting flow control and it turns out you were.  Apparently you are still hitting flow control for some unknown reason.

Do you see a gap in the implemented code?

What implemented code?

With the AUTO_ACK mode, when exactly, will there be an ack sent? Does it happen the moment message is written to the server or ?

Message acknowledgement certainly doesn't happen when the message is written to the server.  It happens when a message is consumed.

When using AUTO_ACK mode a consumer will physically send the acknowledgement essentially when the message is received by the consumer.

[manu29585]

As mentioned earlier, implementation is as below:

Do you see a gap or problem in the way this is being implemented?

Currently, we are reusing the connection and session and are created as shown below (This is being done after we encountered: [HORNETQ-724] Small memory leak in ServerLocator.factories? - JBoss Issue Tracker )

cachedQueueConnection = cachedConnectionFactory.createConnection();

// KEY - register for exception callbacks

cachedQueueConnection.setExceptionListener(new ExceptionJMSListenerImpl());

session = cachedQueueConnection.createSession(false, Session.AUTO_ACKNOWLEDGE);

producer = session.createProducer(cachedQueue);

producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT);

ObjectMessage contextMessage = session.createObjectMessage();

contextMessage.setObject(context);

producer.send(contextMessage);

Should the "send" be synchronized? Because, it would eventually be a performance bottleneck.

[jbertram]

Re-using connections is almost always a good idea.  Sessions can certainly be re-used as well, but it's important to make sure they are not used concurrently (i.e. by multiple threads simultaneously).  If you're using the same session in multiple threads that's the only time you'd need to synchronize access.

I strongly recommend you upgrade.  Lots of work has been done in the last few years since HornetQ 2.1.2.Final was released.  If you weren't aware, the HornetQ project itself is actually no longer under active development.  The HornetQ code-base was donated to the Apache ActiveMQ community and has become ActiveMQ Artemis.  This new ActiveMQ Artemis broker is now shipped in the latest versions of the Wildfly application server.

If you still hit the issue after the upgrade then follow-up here with an reproducible test-case or concise steps to reproduce the issue.

[manu29585]

Thank you Justin. I'm trying to reproduce the problem with standalone application. I'll let you know the results.

We are almost at the end of release cycle and we are planning to upgrade to Wildfly in next version and this upgrade is not feasible in the current version.