JBoss 6 stops receiving messages from HornetQ 2.2.5 topic

[florian.kunz]

Hi,

I've got a single standalone HornetQ 2.2.5 server delivering messages to a single JBoss 6 host. Messages come in at a rate of about 5/s. Messages are consumed in an MDB. Everything works fine at first. After a while messages are not received anymore and start piling up on the HornetQ host. It seems that restarting the consumer solvers the problem. So maybe the problem lies on the JBoss side, but still it's the HornetQ client libraries that's involved here, so I hope that's the right forum ^^

Anyway here's my config. I just included the lines that I think are important. If anything else is needed, I'm happy to provide.

The name of the topic is "updatenotification".

HornetQ side:

hornetq-configuration.xml

   <address-settings>

      <address-setting match="jms.topic.updatenotification">

         <dead-letter-address>jms.queue.DLQ</dead-letter-address>

         <expiry-address>jms.queue.ExpiryQueue</expiry-address>

         <max-size-bytes>10485760</max-size-bytes>

         <page-size-bytes>512000</page-size-bytes>

         <message-counter-history-day-limit>10</message-counter-history-day-limit>

         <address-full-policy>PAGE</address-full-policy>

      </address-setting>

      <address-setting match="#">

         <dead-letter-address>jms.queue.DLQ</dead-letter-address>

         <expiry-address>jms.queue.ExpiryQueue</expiry-address>

         <max-delivery-attempts>5</max-delivery-attempts>

         <redelivery-delay>0</redelivery-delay>

         <max-size-bytes>10485760</max-size-bytes>      

         <message-counter-history-day-limit>10</message-counter-history-day-limit>

         <address-full-policy>DROP</address-full-policy>

      </address-setting>

JBoss side:

MDB config

@MessageDriven(activationConfig = { @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Topic"),

        @ActivationConfigProperty(propertyName = "destination", propertyValue = "topic/updatenotification"),

        @ActivationConfigProperty(propertyName = "subscriptionDurability", propertyValue ="Durable"),

        @ActivationConfigProperty(propertyName = "clientID", propertyValue = "MyClientId"),

        @ActivationConfigProperty(propertyName = "subscriptionName", propertyValue = "MySubscritionName"),

        @ActivationConfigProperty(propertyName = "user", propertyValue = "---"),

        @ActivationConfigProperty(propertyName = "password", propertyValue = "---"),

        @ActivationConfigProperty(propertyName = "maxSession", propertyValue = "1") })

@Pool(value=PoolDefaults.POOL_IMPLEMENTATION_STRICTMAX, maxSize=1, timeout=PLayerMessageConsumer.JMS_POOL_TIMEOUT)

One note to the pool size: As the order of messages is important, we restricted the pool size to 1, to make sure that messages that are sent from one client, are processed in that order.

I would really apprecaite any help here.

Thank you.

[gaohoward]

Did you notice any exceptions or error messages when the message pile-up happened?

Also if you need ordering you may want to take a look at the "Message Group" feature. It's described in chapter 28 and also there are some examples.

Howard

[florian.kunz]

Unfortunately there are no exceptions in the logs. We've had issues with long running transactions in the message consumer that eventually led to a transaction time out, which of course blocked message consumption for a while. It's hard to say if that has anything to do with our issue. At least it should not.

Thanks for pointing out the message group feature. Problem is that we'll be having a second consumer for the topic soon, which should get the same messages in the same order. If I understood the group feature correctly, it would mean that only one consumer gets the messages (although in the correct order ).

[florian.kunz]

Another note:

I have encountered another manifestion of said issue, this time a pool size of 20 was used. We were not able to get the system working again by restarting either the consumer or the HornetQ server. The only option I see now is to stop both systems, delete the hornet data directory and start everything up again.

We had problems like this a lot when using version 2.1.0 but I think back then this was because of corrupted journal files. I thought it has been fixed with one of the later versions. Anyway, if anyone else has an idea how to get to the bottom of this, please don't hesitate. I don't want to clutter up this post by dumping all sorts of config files in here, but if you need anything I'll add it.

Thanks in advance.

[clebert.suconic]

There are a few fixed on Branch_2_2_AS7 and EAP...

We are working towards a release for AS7, and it may be useful to get it.

[florian.kunz]

I just checked out the branches but I'm not able to build at the moment. Getting compile errors, but thats not the point of my reply. Thing is I'm now able to reproduce the behaviour stated in my earlier posts. The following happens:

1. A message is consumed by JBoss 6.1. The message is sent via a standalone HornetQ 2.2.5 server.

2. onMessage method of MDB is called

3, MDB calls a stateless session bean and passes the message payload

4. The session bean executes a database operation which causes a JDBC/SQL exception

-> from there on no new messages are consumed anymore

This all runs in one XA transaction.

So what do you think? Is this an issue of the hornetq resource adapter? Or is it a JBoss thing?

Additional note: executing the DB operation in an own transactions seems to be a viable workaround. Still not convinced that this is the intended behaviour ^^