Messaging gets stuck

[hguser]

Hi all,

we have two JBoss Application Servers (jboss-4.2.3.GA, JDK 1.5/1.6) in a non-clustered-environment. Both are configured with JBoss-Messaging (1.4.0.SP3), one server sends messges to queues and topics the other consumes (remote). Everything works fine unless occasionally messages in a topic get stuck and are no longer consumed by the client. The message counter in the view

listDurableSubscriptionsAsHTML

is increasing. No exceptions or errors are shown in the log files and snooping the corresponding ports show a continuous flow of TCP packets in both directions. Hence the connection is fine.

Messaging can be restarted successfully using the stop/start buttons of the corresponding message driven bean in the jmx-console.

This problem appears usually after many hours of service.

Any ideas what could be the cause of the problem? Should we upgrade to Messaging-1.4.2?

Regards

[clebert.suconic]

Should we upgrade to Messaging-1.4.2?

Yes.. I would suggest that... Followed by the upgrade on remoting as required by the installation section on 1.4.2 docs.

[hguser]

Unfortunately upgrading to messaging-1.4.2 and newest remoting didn't resolve our problem. Messages still get stuck after a while.
Perhaps I should mention that we use transactions on both sides, producer and consumer and persistence:

Producer:

queueSession = queueConnection.createQueueSession(true, Session.AUTO_ACKNOWLEDGE);
...
sender.send(objectMessage, DeliveryMode.PERSISTENT, 4, 0);

Consumer is a MDB with the following annotation:

@ActivationConfigProperty(
 propertyName="sessionTransacted",
 propertyValue="true"
 )

Any ideas?

[timfox]

For a chance of someone being able to help you, can you please define what you mean by "stuck", describe your use case a bit more and provide a test program that demonstrates the issue.

Right now you haven't really provided enough information for anyone to help.

[hguser]

With the following code messages are sent to a queue (using an Apache connection pool for JMS connections):

 private void sendMessage(MASMessage message, String destinationNameQueue) throws JMSException{

 if (connectionFactory != null) {
 QueueSession queueSession = null;
 QueueConnection queueConnection = null;
 try{
 queueConnection = (QueueConnection)queueConnectionPool.borrowObject();
 queueSession = queueConnection.createQueueSession(true, Session.AUTO_ACKNOWLEDGE);
 QueueSender sender = queueSession.createSender((Queue)getDestination(destinationNameQueue));
 ObjectMessage objectMessage = queueSession.createObjectMessage(message);

 sender.send(objectMessage, DeliveryMode.PERSISTENT, 4, 0);
 Log.debug("Message sent of type: " + message.getClass().getName() + " to queue " + destinationNameQueue, JmsUtil.class);
 queueConnection.start();
 queueSession.commit();
 } catch (Exception jmsexc){
 // exception handling
 } finally {
 // ...
 }
 } else {
 // ...
 }
 }

A message driven bean acts as listener and receives these messages:

@MessageDriven(
 name="MCSAccountingMessageBean",
 activationConfig = {
 @ActivationConfigProperty(
 propertyName="destinationType",
 propertyValue="javax.jms.Queue"
 ),
 @ActivationConfigProperty(
 propertyName="sessionTransacted",
 propertyValue="true"
 ),
 @ActivationConfigProperty(
 propertyName="providerAdapterJNDI",
 propertyValue="java:/JMSSecureProvider"
 )
 /* see additional dynamic properties in deployment descriptor */
 }
)
public class MCSAccountingMessageBean implements MessageListener {

 public void onMessage(Message message) {
 Log.debug("Received message of type: " + message.getClass().getName() + " at " + new Date(), MCSAccountingMessageBean.class);
 if (message instanceof ObjectMessage) {
 ObjectMessage om = (ObjectMessage) message;
 if (om != null) {
 try {
 if (om.getObject() instanceof MASWalletTransactionMessage) {
 MASWalletTransactionMessage transactionMessage = (MASWalletTransactionMessage) om.getObject();
 // dispatch message now
 }else{
 Log.error("Cannot process object message with object of unexpected type: " + om.getObject().getClass().getName(), MCSAccountingMessageBean.class);
 }
 } catch (JMSException e) {
 Log.error("Cannot receive JMS message: \n" + e.toString(), MCSAccountingMessageBean.class);
 throw new RuntimeException("Unable to receive or process wallet transaction message!");
 }
 }
 } else {
 Log.error("Cannot process message of unexpected type: " + message.getClass().getName(), MCSAccountingMessageBean.class);
 }
 Log.debug("Finished processing message at " + new Date(), MCSAccountingMessageBean.class);
 }
}

After starting both application servers sending and receiving messages works fine for a while. After several hours of service the MDB stops receiving messages. We can see the messages in the messaging tables JBM_MSG and JBM_MSG_REF of the database (hence sending works), but the MDB shows no reaction. No error messages in the log files on both sides. We must stop and restart the MDB using JMX console. Sometimes we must restart the queue in order to get messaging working.

Hope to be more clearly now.

[bonio100]

Any luck with this. I appear to have a similar problem and was going to create a new post with some details.

[gaohoward]

yes, please give as much details as possible. That'll help find the problem.

[bonio100]

First I will move from 1.4.0.SP3 to 1.4.2.SP1. If the problem still persists, I will try to post the details.

My symptoms are very similar. during sustained high throughput (~120 object msg/sec), normally after 2-3 days JBM will apear to stop consuming messages from the queue. Can post fine, queue count goes up but nothing reads off it, even though it looks like the listener thread is still listening.

When it happens again, will get a stack trace and try to isolate the problem as a unit test.

[timfox]

*a lot* of things have been fixed since 1.4.0.SP3 (I think we've done 6 releases since then?) so definitely try upgrading.

Also, make sure you use the correct remoting version on both client and server sides!

[bonio100]

Hi,

I can confirm the upgrade fixed the problem - This has been running flawless for weeks and faster too.

I suggest anyone with the same problem do the same - The upgrade wont take as long as you think it will and the config files are not that different.