7 Replies Latest reply on Sep 7, 2015 6:00 PM by jbertram

    Hornetqcluster setup

    kunu

      Hi Team,

       

      I am working on hornetq-2.4 setup with Jboss EAP6.4 on 2 servers as Failover mode.

      server setup is as :

       

      server-1 jboss
      server-2 jboss
      server-1 hornetqPrimary
      server-2 hornetqsecondary

       

      on server2, using NFS referrign to data directory on server1 as below:

      server1:/apps/claimcheck_hornetQ_QA

                             30G   11G   18G  39% /apps/claimcheck_hornetQ_QA

       

       

      When i am starting each node separately they work fine, but when i start hornetq on 01 as primary and 02 after that as secondary....

       

       

       

      i get below error on both nodes:

       

       

      09-03 00:00:20,378 WARN  [org.hornetq.core.client] HQ212034: There are more than one servers on the network broadcasting the same node id. You will see this message exactly once (per node) if a node is restarted, in which case it can be safely ignored. But if it is logged continuously it means you really do have more than one node on the same network active concurrently with the same node id. This could occur if you have a backup node active at the same time as its live node. nodeID=4ae1e73b-5209-11e5-91a9-df8f21ccca6b

       

       

      Also on Failover node, this hornetq starts as live node, though still primary is up and running. Please find below log.

      Could you please advise.

       

      Thanks in advance

       

       

      09-03 12:31:16,155 INFO  [org.hornetq.core.server] HQ221032: Waiting to become backup node

      09-03 12:31:16,165 INFO  [org.hornetq.core.server] HQ221033: ** got backup lock

      09-03 12:31:16,310 INFO  [org.hornetq.core.server] HQ221012: Using AIO Journal

      09-03 12:31:16,598 WARN  [org.hornetq.core.server] HQ222007: Security risk! HornetQ is running with the default cluster admin user and default password. Please see the HornetQ user guide, cluster chapter, for instructions on how to change this.

      09-03 12:31:16,686 INFO  [org.hornetq.core.server] HQ221043: Adding protocol support CORE

      09-03 12:31:16,692 INFO  [org.hornetq.core.server] HQ221043: Adding protocol support STOMP

      09-03 12:31:16,699 INFO  [org.hornetq.core.server] HQ221043: Adding protocol support AMQP

      09-03 12:31:16,913 INFO  [org.hornetq.core.server] HQ221109: HornetQ Backup Server version 2.5.0.SNAPSHOT (Wild Hornet, 124) [eeb7a4cb-5242-11e5-85fb-b513f90d63cf] started, waiting live to fail before it gets active

      09-03 12:31:19,495 INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.queue.avmsFCRAQ

      09-03 12:31:19,569 INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.queue.avmsFCRADLQ

      09-03 12:31:19,577 INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.queue.HTMLPublisherQ

      09-03 12:31:19,583 INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.queue.HTMLPublisherDLQ

      09-03 12:31:19,588 INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.queue.PDFPublisherQ

      09-03 12:31:19,592 INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.queue.PDFPublisherDLQ

      09-03 12:31:19,597 INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.queue.ACIServiceQ

      09-03 12:31:19,602 INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.queue.ACIServiceDLQ

      09-03 12:31:19,606 INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.queue.AvmsWebQ

      09-03 12:31:19,611 INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.queue.AvmsWebDLQ

      09-03 12:31:19,615 INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.queue.AvmsBatchQ

      09-03 12:31:19,619 INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.queue.AvmsBatchDLQ

      09-03 12:31:19,623 INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.queue.DLQ

      09-03 12:31:19,626 INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.queue.ExpiryQueue

      09-03 12:31:20,245 INFO  [org.hornetq.core.server] HQ221020: Started Netty Acceptor version 4.0.13.Final 0.0.0.0:5455

      09-03 12:31:20,256 INFO  [org.hornetq.core.server] HQ221020: Started Netty Acceptor version 4.0.13.Final 0.0.0.0:5445

      09-03 12:31:20,277 INFO  [org.hornetq.core.server] HQ221010: Backup Server is now live

        • 1. Re: Hornetqcluster setup
          jbertram

          First let's confirm that your configuration is correct.  Can you attach the HornetQ configuration from both your live and your backup?

           

          Aside from a configuration issue my first guess would be that your NFS mount doesn't have locking properly configured so that when the backup starts it is able to get a lock on the journal and therefore thinks the live is dead.

          • 2. Re: Hornetqcluster setup
            kunu

            Hi Justin,

             

            Thank you for checking this, PFA details.

            Could you please advise how can i setup NFS locking .

             

            Thanks.

            • 3. Re: Hornetqcluster setup
              kunu

              Hi All,

               

              Could you please advise me, i am really stuck..

               

              Thanks

              • 4. Re: Hornetqcluster setup
                jbertram

                Are you passing a value for "data.dir" on the command line to each server?  If so, what value are you using on each?

                 

                As for the NFS setup goes, that's something you'll need to discuss with your storage administrator.

                • 5. Re: Hornetqcluster setup
                  kunu

                  Hi Justin,

                   

                  NFS setup looks good, as when we start nodes individually it works fine.

                  Also, We are setting this back up parameter in hornetq startup file. FYI, This is standalone hornetq, and they share a common NFS folder for data sharing.

                  You can see java_opts= "-Ddata.dir=/apps/claimcheck_hornetQ_QA" in attached files, this is the data sharing folder between Hornetq1 and Hornetq2. on Server1, this folder(/apps/claimcheck_hornetQ_QA) is a local folder and on Server2, data folder is referring server1 folder through NFS setup as below:


                  on server2, using NFS referrign to data directory on server1 as below:

                  server1:/apps/claimcheck_hornetQ_QA 30G 11G 18G 39% /apps/claimcheck_hornetQ_QA


                  PFA hornetq startup files.

                  run.sh=>this is for primary hornetq startup file.

                  run-ha.sh =>this is for backup Hornetq startup file.


                  Thanks.

                  • 6. Re: Hornetqcluster setup
                    kunu

                    PFA files.

                    Thanks

                    • 7. Re: Hornetqcluster setup
                      jbertram

                      I would expect each node to work properly if it was started and run independently of the other.  The real question here is whether or not they will function when they are running at the same time which is required if you want failover functionality.  If the file locking on the NFS mount isn't configured properly then the nodes won't be able to function properly at the same time.  I've seen this before from a handful of users.

                       

                      That said, I'm not an NFS expert and don't know what specific configuration options needs to be set to configure locking properly.  All I know is that people get it wrong sometimes and this is what it looks like when they do.

                       

                      One thing you could try is running both the live and backup instances on the same physical server to validate that all the other configuration works properly.  If the live/backup pair works as expected when running on the same physical server that would pretty much prove the problem was with the NFS configuration. 

                       

                      Good luck.