client window size and the JMS specification

[andreas_back]

Hello HornetQ-Team!

HornetQ is now used as the messaging server of our application that has been migrated from JBoss 3.2.4 and JBossMQ.

The application requires to set the client size window to 0 since the queue delivers jobs to a cluster of service units and the processing of such a job may take several hours. (JBoss Messaging's inability to set the prefechSize to 0 forced us to migrate to HornetQ.)

- Observation:

We have seen that the value of the parameter client-size-window influences the number messages the QueueBrowser reports.

- Test-Case:

A simple test class with a main method is attached.

- Motivation:

For administration purposes we have a (Pojo-) graphical tool to display the messages in the queue.

This tool uses just the JMS interface and this should have eased the migration from JBossMQ/JBoss Messaging to finally HornetQ.

But during our tests we have seen that only one or two messages have been found by the QueueBrower if client-size-window has been set to 0

while the queue must have contained many more messages. We have not seen such a dependency while using JBossMQ or JBoss Messaging.

This makes it impossible to use the QueueBrowser for an estimation of the current workload of the system.

- Workaround:

As a work around we used the following snipplet and set the consumerWindowSize to for example 2 000 000.

            Queue queue = (Queue) initialContext.lookup(QUEUEZDSJOBQUEUE);

            // Step 3. Perform a lookup on the Connection Factory
            HornetQConnectionFactory cf = (HornetQConnectionFactory) initialContext.lookup("/ConnectionFactory");
            cf.setConsumerWindowSize(consumerWindowSize());

Then the whole content of the queue is displayed. But now the code depends on HornetQ!

JMS specification:

The JMS specification version 1.1 April 12, 2001 states on page 78:

"The browse methods return a java.util.Enumeration that is used to scan the
queue’s messages. It may be an enumeration of the entire content of a queue,
or it may contain only the messages matching a message selector."

I would say that if no message arrives or expires when the scan is done then the result of the scanning

of the QueueBrowser shall reflect the status of the queue and the number of the messages in it.

Or to express it in another way. I cannot find a phrase like

"and the messages reported by the QueueBrowser may depend of the client-window-size" in this context of the specification ;-)

- Questions:

1.     Does HornetQ violate the JMS specification if the number of messages reported by the QueueBrowser depends on the client-size-window?

2.     Does HornetQ 2.1.1.GA allow to remove the usage of the HornetQConnectionFactory within the Java code?

Greeitings

Andreas

[timfox]

Any of the connection factory properties can be set in the server side xml, you don't have to set them programmatically on the client side.

[timfox]

I don't understand why you are setting consumer-window-size to zero for browsing the queue. Why not just keep the default value?

[andreas_back]

Hello Tim,

thank you for your interest into this point.

The consumer-window-size is set in the hornetq-jms.xml file:

<connection-factory name="NettyConnectionFactory">
      <connectors>
         <connector-ref connector-name="netty"/>
      </connectors>
      <entries>
         <entry name="/ConnectionFactory"/>
         <entry name="/XAConnectionFactory"/>
      </entries>
       <consumer-window-size>0</consumer-window-size>
   </connection-factory>

And the QueueBrowser just uses the (Netty) ConnectionFactory. Therefore the consumer-window-size

for the QueueBrowser is zero.

But then the admin-tool and the QueueBrowser just showed one or two messages. Therefore the consumer-window-size has been set in the Java-Code in the special case of the admin-tool.

Your argument is that we could use two server configured ConnectionFactories, one for consumer-window-size = 0 and one with the default value?

Greetings

Andreas

[timfox]

Yes, you could just deploy another connection factory into JNDI.

[andreas_back]

Hello Tim,

thank you for your advice.

At the moment we are doing some stress tests for the migrated application where we have a parallel processing of the productive work load on the test system. (The systems are coupled just by copying input files, there is no interference between the queueing systems.)

When the client-window-size was zero for the QueueBrowser of the admin-tool, we saw just the one or two messages in the queue, but the messages were processed by the queue and the service-units.

The present version where we set the client-window-size programatically to high value larger than 0 (see above) has 2 times led to a situation where the message in the job queue became "stale":

(1)     The messages are seen in the queue with the admin-tool, but the service-units are idle and do not consume further messages.

(2)     After the JBoss AS 5.1.0 has been stopped and started again - and with it the HornetQ-Server - the messages have been processed again!

I'm sorry to say that I have not been able to reproduce this case by a simple test programm.

Questions:      What might be the reason for this?

                      What are adequate strategies to cope with this?

Greetings

Andreas

[andreas_back]

Hello HornetQ team!

Two further hints to the problem of the stale messages can be added:

1.        Stopping and starting the POJO-Consumer

The service-units that consume the jobs are simple POJO-Consumers that use a MessageListener with an onMessage-method to consume the messages.

If the queue still containes messages and the consumer does not consume further this can be cured by closing the session and the connection to the queue and opening them and set the MessageListener again.

2.          Stopping and starting the Message Driven Bean

The service-units send messages to a controller via the (same) JMS-Queue. The messages dircetions are - for historic reasons - distinguished by a message selelctor via a 'destination' property. These messages also have become stale. After the MDB has been stopped and started again via the JMX-Console the MDB again has started to consume those messages as well.

Perhaps these two observations may inspire someone to give a further hint to solve this point ?!

Greetings

Andreas

[andreas_back]

Hello HornetQ team!

According to Tim's hint two connection factories configured on the server and a connection factory with the default consumer-window-size is used for browsing the queue.

But the clients still sometimes stop to consume messages and start to consume them if the connection/session is closed and reopenend.

Greetings

Andreas

[black_ice]

Taking the consumer window size as the size of 100messages (just an example)

I guess thats because one client could be "buffering" 100 messages, and they are "Delivering" (and locked) until that session gets closed. When you close it, 100 messages (maybe less if client already processed some) which were on delivering will get "back" into the queue (they will be unlocked), other clients will notice those messages and consume them.

But if you let the client be alive enough, it should process every message (unless it stops/gets locked by some reason).

All your other problems seems like this one. What restarting JBOSS AS 5.1 (or killing client which makes session timeout) does its "unlocking" all those "stuck-on-delivering" messages and thats why clients will consume them later. <-- Its not hornetq bug, at least i didnt get it (was getting it 24/7 on jboss messaging which comes with JBOSS as 5.1) and im using your same JBoss AS 5.1 + HornetQ 2.1.1 combo.

Im not expert, and i dont know how MessageListeners works, since i do all my consuming/producing work manually via senders and receivers, so if you can get help from someother it may be better :P.

Use the jmx-console on the JBOSS as administration and watch to delivering-consumer-message count. It'll help a lot with all those problems.

[andreas_back]

Hello Asdsad,

thank you for spending your time and sharing your experience!

1.     The HornetQ is mainly used with the default parameters.

*      The client-window-size is set to 0 for the ConnectionFactory that is not used for browsing and

*      the max-delivery-attempts has been set to 1.

2.     The QueueBrowser is used by the following code snipplet:

        QueueConnection qc=null;
        QueueSession session=null;
        QueueBrowser browser = null;
        try {
            qc = getJMSBrowsingConnection();
            javax.jms.Queue jmsq = getJMSQueue(queueName);
            session = qc.createQueueSession(false, QueueSession.AUTO_ACKNOWLEDGE);

            DefaultListModel model = new DefaultListModel();
            browser = session.createBrowser(jmsq);
            for (Enumeration e = browser.getEnumeration(); e.hasMoreElements();) {
                Message msg = (Message) e.nextElement();
                model.addElement(msg);
            }
            this.queueEntries.setModel(model);

           
        } catch (JMSException e) {
            log4j.error("...", e);
        } catch (NamingException e) {
            log4j.error("...", e);
        } finally {
                try {
                    if (browser != null) {
                        browser.close();
                    }
                } catch (JMSException e) {
                    log4j.error("...", e);
                }

                try {
                    if (session != null) {
                        session.close();
                    }
                } catch (JMSException e) {
                    log4j.error("...", e);
                }
                try {
                    if (qc!=null) {
                        qc.close();
                    }
                } catch (JMSException e) {
                    log4j.error("...", e);
                }
        }

3.     The application has a EJB2-based Controller and 12 Pojo-based service units and they communicate via Hornetq.

4.     The messages are sent to the queue for example by the following method of a stateless session bean within a transaction - the calling method has transaction type "Required":

    private void publish(SortimentJob job) {
         
        QueueSession queueSession=null;
        QueueSender zdsJobSender=null;
       
        try {
        
            queueSession = this.queueConnection.createQueueSession(true,Session.SESSION_TRANSACTED);
       
            zdsJobSender= queueSession.createSender(this.zdsJobQueue);
            zdsJobSender.send(job.createMessage(queueSession));
        
        } catch(JMSException e) {
            throw new ZDSFatalException("...",e);
        } finally{

            if(zdsJobSender!=null) { try { zdsJobSender.close(); }catch(JMSException e) { log4j.warn("...", e); }}
            if(queueSession!=null) { try { queueSession.close(); }catch(JMSException e) { log4j.warn("...", e); }}
        }
    }

5.          In one case the controller sent - as a reaction to an incoming mesage - 15 new messages to the queue by calling the method publish 15 times. 11 of the 15 messages have been processed, 4 of them became stale and have been not processed for more than one day. And they only were processed after the HornetQ-Server together with the JBoss has been stopped and started.

Obviously some ressources have been released by the restart. But the publish method closes the session. And we have not noticed this problem the last 6 years under JBossMQ and not during our intermediate tests with JBoss Messaging.

And the TRACE, that we have enabled now, reports that the session is closed!

At the moment we have not been able to reproduce this problem with a test program and especially not when sending the messages with a Pojo-sender.

Greetings

Andreas

[andreas_back]

Hello!

The stale messages may be related to

          https://jira.jboss.org/browse/HORNETQ-469

as has been discussed in the tread

          https://community.jboss.org/message/556104#556104

Best regards,

Andreas