1 2 Previous Next 26 Replies Latest reply on May 29, 2012 2:03 AM by Andy Taylor

    Nodes Failover in HornetQ

    vineet badoni Newbie

      Hi All,

       

      We have been using Jboss 4.2.3 from a long time. Now we want to migrate from 4.2.3 to the latest jboss. I started with 6.0.0. Final. I am able to deploy our application in Jboss 6.0.0 final. Our product heavily depends on Queues. Basically our business logic is coupled with queues .

      In order to get performance gain we created cluster of jboss. In Jboss 6.0.0 final we now have HornetQ and we used to use JbossMQ earlier. HornetQ seems to achieve whatever we needed except the following

       

      Scenario:

       

      (1) Node1 has queue name Q.

      (2) Node2 also has queue name Q.

       

      Here Node1 and Node2 are the cluster nodes. When I did set this up I was able to send messages to both of the queues in a Round Robin fashion but when failover happened the messages in the queue in Node1 didnot deliver to Node2. When I made Node1 up the those messages got delivered to Node1 again which is against our requirement.

       

      My question here is there any way to achieve this in HornetQ. I saw there is something called backup node but backup configuration expects static configuration which I dont know before hand.

       

      I would really appreciate any help on this. Please ask me if there is something unclear about the problem.

       

      Thanks and Regards,

      Vineet Badoni

        • 1. Re: Nodes Failover in HornetQ
          Andy Taylor Master

          Failover is obtained by configuring backup servers, since in HornetQ each server uses its own data store a backup server needs to be configured. Take a look at the user manual its exp[lained fully in there.

          • 2. Re: Nodes Failover in HornetQ
            Andy Taylor Master

            My question here is there any way to achieve this in HornetQ. I saw there is something called backup node but backup configuration expects static configuration which I dont know before hand.

            You can configure UDP for this

            • 3. Re: Nodes Failover in HornetQ
              vineet badoni Newbie

              Hi Andy,

               

              Thanks for  your quick reply. But I am not sure that I am clear on the point which you are trying to point me to. Do you mean that I can use UDP for dynamic values for message synchronisation. Can you point me to some examples somewhere?

              Can you please explain to me how does queue data replicate over to the nodes in cluster?

              Thanks again to your quick response.

              • 4. Re: Nodes Failover in HornetQ
                Andy Taylor Master

                vineet badoni wrote:

                 

                Hi Andy,

                 

                Thanks for  your quick reply. But I am not sure that I am clear on the point which you are trying to point me to. Do you mean that I can use UDP for dynamic values for message synchronisation. Can you point me to some examples somewhere?

                Can you please explain to me how does queue data replicate over to the nodes in cluster?

                Thanks again to your quick response.

                You can use UDP via discovery so that nodes can discover each other.

                 

                there are examples and a full section on the user manual.

                 

                I'll be happy to answer any specific questions you have.

                • 5. Re: Nodes Failover in HornetQ
                  vineet badoni Newbie

                  Andy, below is my configuration for UDP for discovery

                  <broadcast-groups>

                        <broadcast-group name="bg-group1">

                           <group-address>${jboss.partition.udpgroup}</group-address>

                           <group-port>${jboss.hornetq.group.port}</group-port>

                           <broadcast-period>5000</broadcast-period>

                           <connector-ref connector-name="netty"/>

                        </broadcast-group>

                     </broadcast-groups>

                   

                     <discovery-groups>

                        <discovery-group name="dg-group1">

                           <group-address>${jboss.partition.udpgroup}</group-address>

                           <group-port>${jboss.hornetq.group.port}</group-port>

                           <refresh-timeout>10000</refresh-timeout>

                        </discovery-group>

                     </discovery-groups>

                   

                  Here ${jboss.partition.udpgroup} is substituted by a class D address which is same for node1 and node2. After this my loadbalancing of queues started wroking in Round Robin fashion but i have no luck on failover.

                   

                  Please tell me what I am missing for failover.

                  • 6. Re: Nodes Failover in HornetQ
                    Andy Taylor Master

                    its hard to say, do you actually have any backup server configured or running.

                    • 7. Re: Nodes Failover in HornetQ
                      vineet badoni Newbie

                      No and I can't have backup servers in my case because the configrations for backup server is kind of static. Static means you need to know the backup servers beforehand when you run the server which is not possible in our case. Our case is somewhat like when we install first node of the cluster and we call it primary server. You can install rest of the cluster nodes when Primary is up and running so I really have no opportunity to get the configurations of rest of the nodes in the cluster.

                      Can I use mulitcasting address in backup server settings? because if this is possible I can mandate all cluster nodes to use the same multicast address.

                       

                      Please provide me your opinion on this.

                       

                      Thanks,

                      Vineet

                      • 8. Re: Nodes Failover in HornetQ
                        Andy Taylor Master

                        Can I use mulitcasting address in backup server settings? because if this is possible I can mandate all cluster nodes to use the same multicast address.

                        Yes using Discovery, all this is explaied in the user manual if you take a look, however you will still need backup servers configured

                        • 9. Re: Nodes Failover in HornetQ
                          vineet badoni Newbie

                          Which section in user manual??

                          • 12. Re: Nodes Failover in HornetQ
                            Yong Hao Gao Master

                            Hi Vineet,

                             

                            First I'd suggest you to try latest release of HornetQ. Second I suggest you take a look at some of the failover examples (like non-transaction-failover and transaction-failover).

                             

                            Howard

                            • 13. Re: Nodes Failover in HornetQ
                              vineet badoni Newbie

                              Hi Yong,

                               

                               

                              Let me put my problem in simple words which could help you pointing out the exact thing I should follow.

                               

                               

                              Environment:

                               

                              Jboss 6.0.0 final with HornetQ version 2.1.2.Final.

                               

                               

                              Setup:

                               

                              (1) Jboss cluster setup which are identical in configuration. For starting I have 2 cluster nodes node1 and node2.

                               

                              (2) "node1" and "node2" has a queue called "ResponseQueue".

                               

                              (3) I have servlet which sends a number of messages to the queue in a loop.

                               

                              (4) Queues are clustered.

                               

                              (5) I have queue("ResponseQueue") receiver end points as MDBs on both the nodes , call it "ResponseListener".

                               

                              (6) When I invoke the servlet it used to lookup the connectionfactory and queues and sends the messages to the queue in loop.

                               

                              (7) The messages were received by the MDBs on both the nodes in "Round Robin fashion" which means if there are 10 messages 5 are going to node1 of the cluster and rest of the 5 are going to node2 and because the messages distribution to the queues in "Round Robin" I was able to see 1,3,5,7,9 were one node of the cluster and 2,4,6,8,10 were going to other node of the cluster.

                               

                               

                              which is for me is perfect.

                               

                               

                              My problem starts in case of failover. I increased the number of messages to posted to 60 from the servlet. Because I posted 60 messages and the message scheduling is round robin that means 30-30 will go to each node.

                               

                              (1) When the messages were being posted in the queues in round robin fashion. I shutdown one of the cluster nodes (node1) and I made sure that there were messages in the queue which were not delivered.

                               

                              Below is what happened

                               

                              Message numbers in node1

                              5,11,13,1,3,7,9,15,17,19,21,23,25,27,29,41,49,47,45,43,57,55,59,53,51,39,37,7

                              • I shutdown the node1 now.
                              • There three more messages to be delivered to this queue but I shut node1 down.

                               

                              Message numbers in node2

                              2,8,4,6,10,30,28,26,24,22,20,18,16,14,12,32,38,34,36,46,52,48,50,40,54,56,58,60,44,42

                              • I kept node2 up all the time so all 30 delivered to the node2 queue.

                               

                              Now I restarted node 1 the messages I received now

                              1  ==== redelivered to primary.

                              3  ==== redelivered to primary.

                              5  ==== redelivered to primary.

                              7  ==== redelivered to primary.

                              37 ==== redelivered to primary.

                               

                              31 ==== this was not posted to the receiver.

                              33 ==== this was not posted to the receiver.

                              35 ==== this was not posted to the receiver.

                               

                              The messages 31,33,35 were not received  because I shut down the node1 which means that the messages are still in queue in node1.

                               

                              The whole point of trouble for us is that the messages delivery waited to come node1 up instead of getting delivered to the other cluster node(node2).

                               

                              I really don't understand why this such a critical functionality is missed in HornetQ and the worst part of it is that you cannot deploy Jboss messaging in Jboss 6.0.0 final which was present in Jboss 6 M2. I don't know how come one can stop supporting which was there in one of the milestone releases but not it final.

                               

                              If you remember we were talking in some other post. I don't know what happened to that post. Had it been deleted??

                              • 14. Re: Nodes Failover in HornetQ
                                Yong Hao Gao Master

                                Hi Vineet,

                                 

                                Thanks for the explanation. I think I understand your issue now. As I know that Hornetq's server side failover happens between a node and one of  its backup servers. This is different from JBM cluster failover model, in which a node and it's backup (buddy) are all cluster nodes.

                                 

                                Chapter 39 of Hornetq's user guide give some details on failover.

                                 

                                So to get expected failover, you need to configure a backup for node1. When you shutdown node1, its backup takes over and continue the delivering.

                                 

                                So you need a backup for node1 and node2 in order to get the failover behavior. (4 nodes, or 2 pair of nodes).

                                 

                                What do you think?

                                 

                                Howard

                                1 2 Previous Next