9 Replies Latest reply on Jun 27, 2012 3:08 AM by saifeldeen

    How to set up JMS (hornetq) failover in JBoss 6.0

    rhinmass

      I am having trouble figuring out how to implement jms failover in JBoss 6.0.  I have a simple ear that is composed of a war and a jar.  The war listens over http and enqueues a message for each request it gets.  The jar contains one MDB that listens on the queue and acts upon each message it receives.  (For  now, just prints out the message ID.)

       

      I have been able to set this up in a 2-node cluster and observe that messages going to either server are split equally across both nodes.

       

      However, I have a requirement that if either node goes down, no messages will be lost.  The way it is configured currently, the messages that are sitting in the queue of the failed node are not processed until that node comes back up.  So while the current configuration achieves load-balancing, it does not achieve failover.

       

      The documented method for setting up failover in hornetq seems to apply to standalone hornetq, rather than hornetq running inside JBoss.

       

      What is the correct topology to achieve HA load-balancing and failover?  And how would I set it up?

       

      Thanks for any help.

        • 1. How to set up JMS (hornetq) failover in JBoss 6.0
          jaikiran

          Robin, welcome to the forums!

           

          I've moved your thread to the HornetQ forum where someone more experienced with HornetQ might be able to help you.

          • 2. How to set up JMS (hornetq) failover in JBoss 6.0
            clebert.suconic

            Are you talking about prepared transactions on the node?

             

            If that's the case, the TM has to come back live to process these prepared TXs.

             

             

            BTW: We have been doing a lot of work on the integration of EAP, fixing issues with TM integration.. etc, what will be now integrated on JBoss 7.

             

             

            We will have 2.2 out shortly, and then we will integrate these changes/fixes on AS7.

            • 3. How to set up JMS (hornetq) failover in JBoss 6.0
              rhinmass

              Thank you so much for replying.  I'm not familiar with prepared transactions - just normal messages.

               

              Currently, in my 2-node jbossas6 cluster using hajndi, my message producers put messages equally on both queues (HQ1 and HQ2).  If 100 messages came in, 50 would be sitting in HQ1 and 50 in HQ2.  If the node containing HQ1 went down and had to be taken out of service, those 50 messages would be lost.

               

              The desired behavior is that HQ1 would replicate its state to HQ2 and that both nodes' applications would only interact with HQ1.  If the node containing HQ1 went down, the surviving node(s) would automatically begin interacting with HQ2.

               

              An alternative would be that as before HQ1 replicates to HQ2.  Node1 would interact with HQ1 and node2 would interact with HQ2.   Duplication would need to be prevented by the two HQs sharing their state, so that as messages are removed from HQ2, they also dissappear from HQ1 (therefore not getting consumed on HQ1).   If either node went out of service, nothing would be lost because both queues contain all the messages.

               

              What do you think?  Are either of these scenarios possible or coming in the next version?

              • 4. How to set up JMS (hornetq) failover in JBoss 6.0
                clebert.suconic

                You need a backup node.

                • 5. How to set up JMS (hornetq) failover in JBoss 6.0
                  rhinmass

                  Ok. I thought that HQ2 was the backup node.  Do you think I need a third node?  Can you describe the setup in a little more detail?  Thank you so much.  I have scoured all of the documentation and searched forums, but have not found any information about how to set this up in jboss.  If it's already described somewhere, a pointer would be great. 

                   

                  thanks!

                  • 6. Re: How to set up JMS (hornetq) failover in JBoss 6.0
                    fivalo

                    Hi,

                     

                    I have de same question, did you find any solution?

                     

                    I have a 4-nodes cluster of Jboss6.1 and when I put messages on the queue they are distributed into the nodes of the cluster, but how do I do for prevent lost messages when a node fails?

                     

                    I have read much documentation and I found that hornetQ works with persistence in files named journals, they could be save in a SAN and the backup could dispatch the the slope message, but I would need 1 HornetQ backups for every node of my cluster??? is there any other solution?? could I to have persistence in a database of the journal?

                     

                    HornetQ talks about replication in the next release.

                     

                    Thanks if you can answer

                     

                    Fido

                    • 7. Re: How to set up JMS (hornetq) failover in JBoss 6.0
                      ataylor

                      you have to have backup servers configured with shared store.

                      replication will be released very soon

                      • 8. Re: How to set up JMS (hornetq) failover in JBoss 6.0
                        fivalo

                        Thanks for Answer

                         

                        I undestood the topic.

                         

                        Now I´m having other problem, I´m testing with two-node from a JBoss 6.1 cluster, I send messages to the queue and I can see in the logs the round robin distribution between the nodes that is great, but if one node is shutdown the other node doesn´t continue processing the new messages that arrive to the queue, it looks like locked waiting the other node. I think this must proccess the new messages from de queue and  not to wait the other node start again.

                         

                        The excepción in the node that didnt shutdown is this:

                         

                        09:26:35,325 ERROR [org.hornetq.core.server.cluster.impl.ClusterConnectionImpl] Failed to handle message: java.lang.IllegalStateException: Cannot find binding for jms.queue.core5Queued5fabfab-5329-11e1-821c-005056b30030

                                at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.doConsumerClosed(ClusterConnectionImpl.java:967) [:6.1.0.Final]

                                at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.onMessage(ClusterConnectionImpl.java:707) [:6.1.0.Final]

                                at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:866) [:6.1.0.Final]

                                at org.hornetq.core.client.impl.ClientConsumerImpl.access$100(ClientConsumerImpl.java:44) [:6.1.0.Final]

                                at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:983) [:6.1.0.Final]

                                at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:6.1.0.Final]

                                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_30]

                                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_30]

                                at java.lang.Thread.run(Thread.java:662) [:1.6.0_30]

                         

                        09:26:36,633 WARN  [org.hornetq.core.server.cluster.impl.BridgeImpl] sf.sicep-cluster.015c3455-5329-11e1-afd3-005056b30030::Connection failed before reconnect : HornetQException[errorCode=4 message=The connection was disconnected because of server shutdown]

                                at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1262) [:6.1.0.Final]

                                at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:6.1.0.Final]

                                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_30]

                                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_30]

                                at java.lang.Thread.run(Thread.java:662) [:1.6.0_30]

                         

                         

                        My question is: this is correct? is there any configuration for the hornetq node active doesn´t locked when the other node falls o is shutdown?

                         

                        Thanks for your help

                        • 9. Re: How to set up JMS (hornetq) failover in JBoss 6.0
                          saifeldeen

                          We also faced a similar problem as described by Fidel (see above), i.e. running 2 nodes in a cluster using JAS 6.1, we also had 2 failover hornetq servers running as well.  When stopping a JBoss instance and then restarting, the following error was seen in the logs:

                          Failed to handle message: java.lang.IllegalStateException: Cannot find binding for jms.queue.

                           

                          We realised that the jnp.port, jnp.rmiPort, hornetq.remoting.netty.port and hornetq.remoting.netty.batch.port settings were somehow conflicting (as a result of JBoss using the same port numbers).  When changing these values to unique values, the queues were failed over and failed back correctly.

                           

                          HTH

                           

                          Regards,

                          Saifeldeen