1 2 Previous Next 18 Replies Latest reply on Apr 17, 2010 1:11 PM by Tim Fox

    Server side load balancing

    John Prout Newbie

      I'm using HornetQ in two JBoss 4.2.3 servers, HornetQ is configured as from the install, the "all-with-hornetq" (clustered) configuration.

       

      I am posting a bunch of messages to a queue that's configured on both of the servers; but all of the messages are posted by one of the servers. I expected to see the messages being distributed to both the cluster servers, but what I'm seeing is that all of the messages are delivered to one server, the one where they were posted.

       

      Is this what I should expect to see or is there some configuration I'm missing?

       

      The relevant part of hornetq-configuration.xml is as follows:

       

       

         <broadcast-groups>
            <broadcast-group name="bg-group1">
               <group-address>231.7.7.7</group-address>
               <group-port>9876</group-port>
               <broadcast-period>5000</broadcast-period>
               <connector-ref connector-name="netty"/>
            </broadcast-group>
         </broadcast-groups>
       
         <discovery-groups>
            <discovery-group name="dg-group1">
               <group-address>231.7.7.7</group-address>
               <group-port>9876</group-port>
               <refresh-timeout>10000</refresh-timeout>
            </discovery-group>
         </discovery-groups>
       
         <cluster-connections>
            <cluster-connection name="my-cluster">
               <address>jms</address>
               <forward-when-no-consumers>true</forward-when-no-consumers>
                    <discovery-group-ref discovery-group-name="dg-group1"/>
            </cluster-connection>
         </cluster-connections>
       
         <security-settings>
            <security-setting match="#">
               <permission type="createTempQueue" roles="guest"/>
               <permission type="deleteTempQueue" roles="guest"/>
               <permission type="consume" roles="guest"/>
               <permission type="send" roles="guest"/>
            </security-setting>
         </security-settings>
       
         <address-settings>
            <!--default for catch all-->
            <address-setting match="#">
               <dead-letter-address>jms.queue.DLQ</dead-letter-address>
               <expiry-address>jms.queue.ExpiryQueue</expiry-address>
               <redelivery-delay>0</redelivery-delay>
               <max-size-bytes>-1</max-size-bytes>
               <page-size-bytes>10485760</page-size-bytes>
               <message-counter-history-day-limit>10</message-counter-history-day-limit>
            </address-setting>
         </address-settings>
       
      
      

       

       

      Thanks

       

      John

        • 1. Re: Server side load balancing
          Tim Fox Master

          It's pretty much impossible to say, unless you provide your test code.

          • 2. Re: Server side load balancing
            John Prout Newbie

            Try again...

             

            Test code is as simple as it can be, straight from a JMS sample:

             

             

            private void sendJMSMessage(String message)
            {
            Connection connection = null;
            InitialContext initialContext = null;
             
             
            // Step 1. Create an initial context to perform the JNDI lookup.
            initialContext = new InitialContext();
             
            // Step 2. Perfom a lookup on the queue
            Queue queue = (Queue) initialContext.lookup("/queue/testQueue");
             
            // Step 3. Perform a lookup on the Connection Factory
            ConnectionFactory cf = (ConnectionFactory) initialContext
               .lookup("/ConnectionFactory");
             
            // Step 4.Create a JMS Connection
            connection = cf.createConnection();
             
            // Step 5. Create a JMS Session
            Session session = connection.createSession(false,
               Session.AUTO_ACKNOWLEDGE);
             
            // Step 6. Create a JMS Message Producer
            MessageProducer producer = session.createProducer(queue);
             
            // Step 7. Create a Text Message
            TextMessage textMessage = session.createTextMessage(message);
             
            // Step 8. Send the Message
            producer.send(textMessage);
             
             
             
            }
            
             
            
            

            Exception handling and cleanup omitted, to keep it short. This method is called from a web service. No transactions being used yet.

             

            Consumer is the Spring DefaultMessageListenerContainer, which calls MessageConsumer.receive() (JMS). Consumer appears to be working fine.

             

            John

            • 3. Re: Server side load balancing
              Tim Fox Master

              That just sends one message.

               

              Clearly one message can only be on one node.

               

              We'd need to see code that demonstrates sending more than one message, and also a description of

               

              a) What results you expected to see

              b) What you actually saw

              • 4. Re: Server side load balancing
                Tim Fox Master

                Also, please post your consuming code, without use of Spring code.

                 

                Please see http://community.jboss.org/wiki/Howtoreportabugissue

                • 5. Re: Server side load balancing
                  John Prout Newbie

                  The code I posted is called from a web service. I have an external test utility and use that to invoke the service 10 times, posting 10 messages to the queue (I verified the messages are there, using jmx-console).

                  Consumer code is again very simple (if you ignore the Spring class). MessageListener does nothing but log a message, wait 5s and then log another message:

                   

                  public class SendSimpleMessageListener implements MessageListener
                  {
                  static final      Logger logger = LoggerFactory.getLogger(SendSimpleMessageListener.class);
                   
                  @Override
                  public void onMessage(Message message)
                  {
                  if(message instanceof TextMessage)
                  {
                  try
                  {
                  TextMessage textMessage = (TextMessage)message;
                  logger.info("TextMessage received: " + textMessage.getText());
                  synchronized (textMessage)
                          {
                  try
                         {
                         textMessage.wait(5000L);
                         }
                         catch (InterruptedException e)
                         {
                         // TODO Auto-generated catch block
                         e.printStackTrace();
                         }
                          }
                  logger.info("TextMessage processed after waiting: " + textMessage.getText());
                  }catch(JMSException jmsException)
                  {
                  logger.error("Error processing TextMessage: " + jmsException.getMessage(), jmsException);
                  }
                  }
                   
                  }
                  }
                   
                  
                  

                   

                  The Spring class is started from an MBean, packaged in an sar archive.

                   

                  What I'm seeing is that log messages, for processing all JMS messages appear on one server. What I expect is to see some messages being processed on each of the two cluster servers

                   

                  I've no reason to think this is a bug; I expect it's a configuration problem

                   

                  Thanks for your help

                   

                  John

                  • 6. Re: Server side load balancing
                    Tim Fox Master

                    I'd suggest forgetting the spring consumer for now.

                     

                    Just send a bunch of messages, then using jconsole point it at each node and count the messages in each queue (see management chapter in user manual). This will tell you where your messages really are.

                    • 7. Re: Server side load balancing
                      John Prout Newbie

                      I've written a simple Consumer class, using consumer.receive; code is attached.

                       

                      I've also altered my web service so that rather than queuing a single message when called, it queues 100, numbered messages.

                       

                      Upon testing, the result is the same as before - all messages are consumed on the same server as they were produced (queued). Not a single message was distributed to the other server in the cluster (easy to see from the log file, because the messages are numbered and processed in order). The "messagesAdded" property from the queue MBean confirms these results on both servers.

                       

                      Looking at the jms/clustered-queue sample, I don't see any significant differences to my configuration, but my next step is to make sure my configuration is identical to the sample.

                       

                      One possibility is that UDP is not being allowed on my network (I think it is); Is there any way to tell from the log or from the MBeans that the two servers are operating as a hornetq cluster?

                       

                      Thanks

                       

                      John

                      • 8. Re: Server side load balancing
                        John Prout Newbie

                        I corrected one inconsistency - the netty connector and acceptor in hornetq-configuration.xml both had the same name. Unfortunately, changing the names didn't make any difference - all messages are still consumed on the same server as they're queued.

                         

                        John

                        • 9. Re: Server side load balancing
                          Tim Fox Master

                          One observation - both your producer and consumerr programs are exhibiting a classic anti-pattern - your're doing a JNDI lookup, creating a connection, session etc for *every* message you send/consume.

                           

                          This will be extremely slow and is not a proper use of JMS resources.

                          • 10. Re: Server side load balancing
                            Tim Fox Master

                            Your code is bizarre in other ways too:

                             

                            MessageListener listener = new SendSimpleMessageListener();
                                                
                                                while(!mustStop)
                                                {
                                                    jmsMessage = consumer.receive(1000);
                                                    if(jmsMessage != null)
                                                    {
                                                        listener.onMessage(jmsMessage);
                                                    }
                                                }

                             

                            You're receiving a message sychronously then manually sending it to a MessageListener?? .. I won't ask, but that's not how async delivery is supposed to work.

                             

                            Like you say, it's a possibility that UDP is not working on your network, you can see if the nodes formed a cluster by calling:

                             

                            http://hornetq.sourceforge.net/docs/hornetq-2.0.0.GA/api/org/hornetq/api/core/management/ClusterConnectionControl.html#getNodes%28%29

                             

                            Via JMX

                            • 11. Re: Server side load balancing
                              John Prout Newbie

                              I agree - this is just a little test program written to test the hornetq cluster; it will never be used in any production application. Performance isn't bad - the web service returns in about 50ms, which includes queuing the message, so the JCA caching must be helping out.

                               

                              John

                              • 12. Re: Server side load balancing
                                Tim Fox Master

                                50 ms is slow, also your test program is not using JCA caching since it's not using the java:/JmsXA JCA connection factory.

                                 

                                And, even if it was using JCA you'd still have a network round trip for the JNDI lookup.

                                • 13. Re: Server side load balancing
                                  John Prout Newbie

                                  Code in this consumer is extracted from the Spring DefaultMessageListenerContainer. It uses this structure to make it easy to vary the number of consumer threads at run time, which is a key requirement for my application (I need to lower the rate of message processing if load on the database gets too high).

                                   

                                  Looking at the results from the ClusterConnection MBean, The two servers may not be forming a cluster, but I don't know how to interpret these results;

                                   

                                  Results from getNodes on the two servers are as follows:

                                   

                                  server1: {6ffde098-477c-11df-a7be-00065bfd2945=localhost/127.0.0.1:5445}

                                  server2: {9d3dd93a-47f5-11df-8362-00065bfd1c12=localhost/127.0.0.1:5445}

                                   

                                  Results from getNodeID are as follows

                                  server1: 9d3dd93a-47f5-11df-8362-00065bfd1c12

                                  server2: 6ffde098-477c-11df-a7be-00065bfd2945

                                   

                                  Can you interpret these?

                                   

                                  Thanks

                                   

                                  John

                                  • 14. Re: Server side load balancing
                                    John Prout Newbie

                                    Good to know that the java:/JmsXA Connection Factory is needed to use JCA caching. Both sides of the exchange will certainly use all possible caching before they go to production.

                                    1 2 Previous Next