Server side load balancing

[johncprout]

I'm using HornetQ in two JBoss 4.2.3 servers, HornetQ is configured as from the install, the "all-with-hornetq" (clustered) configuration.

I am posting a bunch of messages to a queue that's configured on both of the servers; but all of the messages are posted by one of the servers. I expected to see the messages being distributed to both the cluster servers, but what I'm seeing is that all of the messages are delivered to one server, the one where they were posted.

Is this what I should expect to see or is there some configuration I'm missing?

The relevant part of hornetq-configuration.xml is as follows:

   <broadcast-groups>
      <broadcast-group name="bg-group1">
         <group-address>231.7.7.7</group-address>
         <group-port>9876</group-port>
         <broadcast-period>5000</broadcast-period>
         <connector-ref connector-name="netty"/>
      </broadcast-group>
   </broadcast-groups>
 
   <discovery-groups>
      <discovery-group name="dg-group1">
         <group-address>231.7.7.7</group-address>
         <group-port>9876</group-port>
         <refresh-timeout>10000</refresh-timeout>
      </discovery-group>
   </discovery-groups>
 
   <cluster-connections>
      <cluster-connection name="my-cluster">
         <address>jms</address>
         <forward-when-no-consumers>true</forward-when-no-consumers>
              <discovery-group-ref discovery-group-name="dg-group1"/>
      </cluster-connection>
   </cluster-connections>
 
   <security-settings>
      <security-setting match="#">
         <permission type="createTempQueue" roles="guest"/>
         <permission type="deleteTempQueue" roles="guest"/>
         <permission type="consume" roles="guest"/>
         <permission type="send" roles="guest"/>
      </security-setting>
   </security-settings>
 
   <address-settings>
      <!--default for catch all-->
      <address-setting match="#">
         <dead-letter-address>jms.queue.DLQ</dead-letter-address>
         <expiry-address>jms.queue.ExpiryQueue</expiry-address>
         <redelivery-delay>0</redelivery-delay>
         <max-size-bytes>-1</max-size-bytes>
         <page-size-bytes>10485760</page-size-bytes>
         <message-counter-history-day-limit>10</message-counter-history-day-limit>
      </address-setting>
   </address-settings>
 

Thanks

John

[timfox]

It's pretty much impossible to say, unless you provide your test code.

[johncprout]

Try again...

Test code is as simple as it can be, straight from a JMS sample:

private void sendJMSMessage(String message)
{
Connection connection = null;
InitialContext initialContext = null;
 
 
// Step 1. Create an initial context to perform the JNDI lookup.
initialContext = new InitialContext();
 
// Step 2. Perfom a lookup on the queue
Queue queue = (Queue) initialContext.lookup("/queue/testQueue");
 
// Step 3. Perform a lookup on the Connection Factory
ConnectionFactory cf = (ConnectionFactory) initialContext
   .lookup("/ConnectionFactory");
 
// Step 4.Create a JMS Connection
connection = cf.createConnection();
 
// Step 5. Create a JMS Session
Session session = connection.createSession(false,
   Session.AUTO_ACKNOWLEDGE);
 
// Step 6. Create a JMS Message Producer
MessageProducer producer = session.createProducer(queue);
 
// Step 7. Create a Text Message
TextMessage textMessage = session.createTextMessage(message);
 
// Step 8. Send the Message
producer.send(textMessage);
 
 
 
}

 

Exception handling and cleanup omitted, to keep it short. This method is called from a web service. No transactions being used yet.

Consumer is the Spring DefaultMessageListenerContainer, which calls MessageConsumer.receive() (JMS). Consumer appears to be working fine.

John

[timfox]

That just sends one message.

Clearly one message can only be on one node.

We'd need to see code that demonstrates sending more than one message, and also a description of

a) What results you expected to see

b) What you actually saw

[timfox]

Also, please post your consuming code, without use of Spring code.

Please see http://community.jboss.org/wiki/Howtoreportabugissue

[johncprout]

The code I posted is called from a web service. I have an external test utility and use that to invoke the service 10 times, posting 10 messages to the queue (I verified the messages are there, using jmx-console).

Consumer code is again very simple (if you ignore the Spring class). MessageListener does nothing but log a message, wait 5s and then log another message:

public class SendSimpleMessageListener implements MessageListener
{
static final      Logger logger = LoggerFactory.getLogger(SendSimpleMessageListener.class);
 
@Override
public void onMessage(Message message)
{
if(message instanceof TextMessage)
{
try
{
TextMessage textMessage = (TextMessage)message;
logger.info("TextMessage received: " + textMessage.getText());
synchronized (textMessage)
        {
try
       {
       textMessage.wait(5000L);
       }
       catch (InterruptedException e)
       {
       // TODO Auto-generated catch block
       e.printStackTrace();
       }
        }
logger.info("TextMessage processed after waiting: " + textMessage.getText());
}catch(JMSException jmsException)
{
logger.error("Error processing TextMessage: " + jmsException.getMessage(), jmsException);
}
}
 
}
}
 

The Spring class is started from an MBean, packaged in an sar archive.

What I'm seeing is that log messages, for processing all JMS messages appear on one server. What I expect is to see some messages being processed on each of the two cluster servers

I've no reason to think this is a bug; I expect it's a configuration problem

Thanks for your help

John

[timfox]

I'd suggest forgetting the spring consumer for now.

Just send a bunch of messages, then using jconsole point it at each node and count the messages in each queue (see management chapter in user manual). This will tell you where your messages really are.

[johncprout]

I've written a simple Consumer class, using consumer.receive; code is attached.

I've also altered my web service so that rather than queuing a single message when called, it queues 100, numbered messages.

Upon testing, the result is the same as before - all messages are consumed on the same server as they were produced (queued). Not a single message was distributed to the other server in the cluster (easy to see from the log file, because the messages are numbered and processed in order). The "messagesAdded" property from the queue MBean confirms these results on both servers.

Looking at the jms/clustered-queue sample, I don't see any significant differences to my configuration, but my next step is to make sure my configuration is identical to the sample.

One possibility is that UDP is not being allowed on my network (I think it is); Is there any way to tell from the log or from the MBeans that the two servers are operating as a hornetq cluster?

Thanks

John

[johncprout]

I corrected one inconsistency - the netty connector and acceptor in hornetq-configuration.xml both had the same name. Unfortunately, changing the names didn't make any difference - all messages are still consumed on the same server as they're queued.

John

[timfox]

One observation - both your producer and consumerr programs are exhibiting a classic anti-pattern - your're doing a JNDI lookup, creating a connection, session etc for *every* message you send/consume.

This will be extremely slow and is not a proper use of JMS resources.

[timfox]

Your code is bizarre in other ways too:

MessageListener listener = new SendSimpleMessageListener();
                    
                    while(!mustStop)
                    {
                        jmsMessage = consumer.receive(1000);
                        if(jmsMessage != null)
                        {
                            listener.onMessage(jmsMessage);
                        }
                    }

You're receiving a message sychronously then manually sending it to a MessageListener?? .. I won't ask, but that's not how async delivery is supposed to work.

Like you say, it's a possibility that UDP is not working on your network, you can see if the nodes formed a cluster by calling:

http://hornetq.sourceforge.net/docs/hornetq-2.0.0.GA/api/org/hornetq/api/core/management/ClusterConnectionControl.html#getNodes%28%29

Via JMX

[johncprout]

I agree - this is just a little test program written to test the hornetq cluster; it will never be used in any production application. Performance isn't bad - the web service returns in about 50ms, which includes queuing the message, so the JCA caching must be helping out.

John

[timfox]

50 ms is slow, also your test program is not using JCA caching since it's not using the java:/JmsXA JCA connection factory.

And, even if it was using JCA you'd still have a network round trip for the JNDI lookup.

[johncprout]

Code in this consumer is extracted from the Spring DefaultMessageListenerContainer. It uses this structure to make it easy to vary the number of consumer threads at run time, which is a key requirement for my application (I need to lower the rate of message processing if load on the database gets too high).

Looking at the results from the ClusterConnection MBean, The two servers may not be forming a cluster, but I don't know how to interpret these results;

Results from getNodes on the two servers are as follows:

server1: {6ffde098-477c-11df-a7be-00065bfd2945=localhost/127.0.0.1:5445}

server2: {9d3dd93a-47f5-11df-8362-00065bfd1c12=localhost/127.0.0.1:5445}

Results from getNodeID are as follows

server1: 9d3dd93a-47f5-11df-8362-00065bfd1c12

server2: 6ffde098-477c-11df-a7be-00065bfd2945

Can you interpret these?

Thanks

John

[johncprout]

Good to know that the java:/JmsXA Connection Factory is needed to use JCA caching. Both sides of the exchange will certainly use all possible caching before they go to production.