8 Replies Latest reply on Mar 25, 2013 4:38 AM by ataylor

    Live and Backup in Symmetrical cluster with shared store

    joker1981

      Hello,

      i am trying to setup following scenario as described here:

      http://docs.jboss.org/hornetq/2.2.2.Final/user-manual/en/html/clusters.html

      at point 38.1.2

       

      and getting the whole time this WARN message on Server A:

      15:10:46,341 WARN  [org.hornetq.core.client] HQ212050: There are more than one servers on the network broadcasting the same node id. You will see this message exactly once (per node) if a node is restarted, in which case it can be safely ignored. But if it is logged continuously it means you really do have more than one node on the same network active concurrently with the same node id. This could occur if you have a backup node active at the same time as its live node. nodeID=c9053b21-7b66-11e2-a5b7-eb840aef0229

       

      and thise WARN and ERROR message on Server B:

      15:09:47,189 WARN  [org.hornetq.core.server] HQ222189: Unable to announce backup, retrying

      15:09:47,692 ERROR [org.hornetq.core.server] Error announcing backup: backupServerLocator is null. org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$2@154d659d

       

      The following path is shared on both server:

      /usr/java/hornetq/data

       

      Config Server A:

      hornetq-configuration-A.xml

      hornetq-jms-A.xml

      start-A.sh

       

      Config Server B:

      hornetq-configuration-B.xml

      hornetq-jms-B.xml

      start-B.sh

       

       

      What have I done wrong at my configuration? Why I am getting these exceptions?

       

      I hope somebody can help!

        • 1. Re: Live and Backup in Symmetrical cluster with shared store
          ataylor

          if you keep seeing the same message then both servers must be live or you have more than one live server using the same journal (maybe you copied it). check that they are actually pointing at the same shared store.and any live servers arent using the same copied journal.

           

          Also you seem to have raised this discussion twice

          • 2. Re: Live and Backup in Symmetrical cluster with shared store
            joker1981

            I have deleted the second discussion.

             

            Yes, both server are running and pointing to the shared path:

            /usr/java/hornetq/data

            and it looks like that they are using the same journal.

            How can i make shure that every server is using a different journal?

             

            • 3. Re: Live and Backup in Symmetrical cluster with shared store
              joker1981

              Let me clarify something: in the hornetq-configuration.xml I have these data storages defined

               

              <paging-directory>${data.dir:../data}/paging</paging-directory>
              <bindings-directory>${data.dir:../data}/bindings</bindings-directory>

               

              <journal-type>NIO</journal-type>

              <journal-directory>${data.dir:../data}/journal</journal-directory>

               

              <journal-min-files>10</journal-min-files>

               

              <large-messages-directory>${data.dir:../data}/large-messages</large-messages-directory>

               

               

              Which are the ones to share among the cluster node? It sounds like:

               

               

              Not Shared

              - Journal

               

               

              Shared:

              - Rest shared?

              • 4. Re: Live and Backup in Symmetrical cluster with shared store
                ataylor

                How can i make shure that every server is using a different journal?

                simple configure different directories in the config files

                Which are the ones to share among the cluster node?

                I'm not really sure what you are asking here, but just make sure that only live/backup pairs share the same journal

                • 5. Re: Live and Backup in Symmetrical cluster with shared store
                  joker1981

                  Hello,

                  i am a litte bit further with my problem.

                  The first problem was i did a wrong setup. I wanted an active-active Cluster with a shared filesystem. Ok this is not possible.

                  Now I decided an active-passive Cluster with a shared file system.

                  First I started with one virtual Server (192.168.206.132) on which i installed two hornetq instances. Following setup:

                  Instance1

                  start.sh

                  ...

                  ###############################################################################################

                  # JVM Settings

                  export JVM_ARGS="-XX:+UseParallelGC -XX:+UseFastAccessorMethods"

                  export JVM_ARGS="$JVM_ARGS -Xms1024M -Xmx1024M"

                  export JVM_ARGS="$JVM_ARGS -Djava.library.path=."

                  export JVM_ARGS="$JVM_ARGS -Djava.net.preferIPv4Stack=true"

                  export JVM_ARGS="$JVM_ARGS -Dfile.encoding=utf-8"

                  export JVM_ARGS="$JVM_ARGS -Dcom.sun.management.jmxremote"

                  export JVM_ARGS="$JVM_ARGS -Dcom.sun.management.jmxremote.port=13000"

                  export JVM_ARGS="$JVM_ARGS -Dcom.sun.management.jmxremote.ssl=false"

                  export JVM_ARGS="$JVM_ARGS -Dcom.sun.management.jmxremote.authenticate=false"

                   

                  ###############################################################################################

                  # HornetQ Config

                  #export JVM_ARGS="$JVM_ARGS -Djava.util.logging.manager=org.jboss.logmanager.LogManager"

                  export JVM_ARGS="$JVM_ARGS -Djava.util.logging.config.file=$CONFIG_DIR/logging.properties"

                   

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.config.dir=$CONFIG_DIR"

                  export JVM_ARGS="$JVM_ARGS -Ddata.dir=/usr/java/hornetq/dev/data/test1"

                  export JVM_ARGS="$JVM_ARGS -Dcluster.user=password"

                  export JVM_ARGS="$JVM_ARGS -Dcluster.password=user"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.connector.host1=192.168.206.132"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.connector.host1.port=15445"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.connector.host2=192.168.206.132"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.connector.host2.port=15446"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.acceptor.host=192.168.206.132"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.acceptor.host.port=15445"

                  export JVM_ARGS="$JVM_ARGS -Djnp.port=11099"

                  export JVM_ARGS="$JVM_ARGS -Djnp.host=192.168.206.132"

                  export JVM_ARGS="$JVM_ARGS -Djnp.rmiPort=11098"

                  export JVM_ARGS="$JVM_ARGS -Djava.naming.factory.initial=org.jnp.interfaces.NamingContextFactory"

                  export JVM_ARGS="$JVM_ARGS -Djava.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces"

                  ...

                  hornetq-configuration.xml

                  <configuration xmlns="urn:hornetq"

                                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                                 xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">

                   

                      <clustered>true</clustered> 

                      <backup>false</backup>                                              # second instance true

                      <shared-store>true</shared-store>

                      <allow-failback>true</allow-failback>

                      <failover-on-shutdown>true</failover-on-shutdown>

                      <journal-type>ASYNCIO</journal-type>

                      <paging-directory>${data.dir:../data}/paging</paging-directory>

                      <bindings-directory>${data.dir:../data}/bindings</bindings-directory>

                      <journal-directory>${data.dir:../data}/journal</journal-directory>

                      <journal-min-files>10</journal-min-files>

                      <large-messages-directory>${data.dir:../data}/large-messages</large-messages-directory>

                         <cluster-user>${cluster.user:clusteruser}</cluster-user>

                      <cluster-password>${cluster.password:password}</cluster-password>

                      <!-- Connectors -->

                      <connectors>

                          <connector name="netty-connector-host1">

                              <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

                              <param key="host" value="${hornetq.remoting.netty.connector.host1:server1}"/>

                              <param key="port" value="${hornetq.remoting.netty.connector.host1.port:5445}"/>

                          </connector>

                   

                          <!-- Static connector to the HornetQ Server 2 -->

                          <connector name="netty-connector-host2">

                              <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

                              <param key="host" value="${hornetq.remoting.netty.connector.host2:server2}"/>

                              <param key="port" value="${hornetq.remoting.netty.connector.host2.port:5445}"/>

                          </connector>

                   

                      </connectors>

                      <!-- Acceptors -->

                      <acceptors>

                          <acceptor name="netty-acceptor">

                              <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>

                              <param key="host" value="${hornetq.remoting.netty.acceptor.host:server1}"/>

                              <param key="port" value="${hornetq.remoting.netty.acceptor.host.port:5455}"/>

                          </acceptor>

                      </acceptors>

                   

                      <cluster-connections>

                          <cluster-connection name="DevCluster">

                              <address>jms</address>

                              <connector-ref>netty-connector-host1</connector-ref>

                              <retry-interval>500</retry-interval>

                              <use-duplicate-detection>true</use-duplicate-detection>

                              <forward-when-no-consumers>true</forward-when-no-consumers>

                              <max-hops>1</max-hops>

                              <static-connectors>

                                  <connector-ref>netty-connector-host2</connector-ref>               

                              </static-connectors>

                          </cluster-connection>

                      </cluster-connections>

                   

                  Instance2

                  start.sh

                  ...

                  ###############################################################################################

                  # JVM Settings

                  export JVM_ARGS="-XX:+UseParallelGC -XX:+UseFastAccessorMethods"

                  export JVM_ARGS="$JVM_ARGS -Xms1024M -Xmx1024M"

                  export JVM_ARGS="$JVM_ARGS -Djava.library.path=."

                  export JVM_ARGS="$JVM_ARGS -Djava.net.preferIPv4Stack=true"

                  export JVM_ARGS="$JVM_ARGS -Dfile.encoding=utf-8"

                  export JVM_ARGS="$JVM_ARGS -Dcom.sun.management.jmxremote"

                  export JVM_ARGS="$JVM_ARGS -Dcom.sun.management.jmxremote.port=23000"

                  export JVM_ARGS="$JVM_ARGS -Dcom.sun.management.jmxremote.ssl=false"

                  export JVM_ARGS="$JVM_ARGS -Dcom.sun.management.jmxremote.authenticate=false"

                   

                  ###############################################################################################

                  # HornetQ Config

                  #export JVM_ARGS="$JVM_ARGS -Djava.util.logging.manager=org.jboss.logmanager.LogManager"

                  export JVM_ARGS="$JVM_ARGS -Djava.util.logging.config.file=$CONFIG_DIR/logging.properties"

                   

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.config.dir=$CONFIG_DIR"

                  export JVM_ARGS="$JVM_ARGS -Ddata.dir=/usr/java/hornetq/dev/data/test1"

                  export JVM_ARGS="$JVM_ARGS -Dcluster.user=password"

                  export JVM_ARGS="$JVM_ARGS -Dcluster.password=user"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.connector.host1=192.168.206.132"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.connector.host1.port=25445"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.connector.host2=192.168.206.132"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.connector.host2.port=25446"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.acceptor.host=192.168.206.132"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.acceptor.host.port=25445"

                  export JVM_ARGS="$JVM_ARGS -Djnp.port=21099"

                  export JVM_ARGS="$JVM_ARGS -Djnp.host=192.168.206.132"

                  export JVM_ARGS="$JVM_ARGS -Djnp.rmiPort=21098"

                  export JVM_ARGS="$JVM_ARGS -Djava.naming.factory.initial=org.jnp.interfaces.NamingContextFactory"

                  export JVM_ARGS="$JVM_ARGS -Djava.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces"

                  ...

                   

                  EVERYTHING WORKS FINE!

                   


                  Then I tried with two virtual Server (192.168.206.132 and 192.168.206.133) on which i installed one hornetq instance. Following setup:

                   

                  Server 1

                  start.sh

                  ...

                  ###############################################################################################

                  # JVM Settings

                  export JVM_ARGS="-XX:+UseParallelGC -XX:+UseFastAccessorMethods"

                  export JVM_ARGS="$JVM_ARGS -Xms1024M -Xmx1024M"

                  export JVM_ARGS="$JVM_ARGS -Djava.library.path=."

                  export JVM_ARGS="$JVM_ARGS -Djava.net.preferIPv4Stack=true"

                  export JVM_ARGS="$JVM_ARGS -Dfile.encoding=utf-8"

                  export JVM_ARGS="$JVM_ARGS -Dcom.sun.management.jmxremote"

                  export JVM_ARGS="$JVM_ARGS -Dcom.sun.management.jmxremote.port=3000"

                  export JVM_ARGS="$JVM_ARGS -Dcom.sun.management.jmxremote.ssl=false"

                  export JVM_ARGS="$JVM_ARGS -Dcom.sun.management.jmxremote.authenticate=false"

                   

                  ###############################################################################################

                  # HornetQ Config

                  #export JVM_ARGS="$JVM_ARGS -Djava.util.logging.manager=org.jboss.logmanager.LogManager"

                  export JVM_ARGS="$JVM_ARGS -Djava.util.logging.config.file=$CONFIG_DIR/logging.properties"

                   

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.config.dir=$CONFIG_DIR"

                  export JVM_ARGS="$JVM_ARGS -Ddata.dir=/usr/java/hornetq/dev/data/test2"

                  export JVM_ARGS="$JVM_ARGS -Dcluster.user=password"

                  export JVM_ARGS="$JVM_ARGS -Dcluster.password=user"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.connector.host1=192.168.206.132"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.connector.host1.port=15445"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.connector.host2=192.168.206.133"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.connector.host2.port=15446"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.acceptor.host=192.168.206.132"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.acceptor.host.port=15445"

                  export JVM_ARGS="$JVM_ARGS -Djnp.port=11099"

                  export JVM_ARGS="$JVM_ARGS -Djnp.host=192.168.206.132"

                  export JVM_ARGS="$JVM_ARGS -Djnp.rmiPort=11098"

                  export JVM_ARGS="$JVM_ARGS -Djava.naming.factory.initial=org.jnp.interfaces.NamingContextFactory"

                  export JVM_ARGS="$JVM_ARGS -Djava.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces"

                  ...

                  hornetq-configuration.xml

                  <configuration xmlns="urn:hornetq"

                                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                                 xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">

                   

                      <clustered>true</clustered> 

                      <backup>false</backup>                                              # second server true

                      <shared-store>true</shared-store>

                      <allow-failback>true</allow-failback>

                      <failover-on-shutdown>true</failover-on-shutdown>

                      <journal-type>ASYNCIO</journal-type>

                      <paging-directory>${data.dir:../data}/paging</paging-directory>

                      <bindings-directory>${data.dir:../data}/bindings</bindings-directory>

                      <journal-directory>${data.dir:../data}/journal</journal-directory>

                      <journal-min-files>10</journal-min-files>

                      <large-messages-directory>${data.dir:../data}/large-messages</large-messages-directory>

                         <cluster-user>${cluster.user:clusteruser}</cluster-user>

                      <cluster-password>${cluster.password:password}</cluster-password>

                      <!-- Connectors -->

                      <connectors>

                          <connector name="netty-connector-host1">

                              <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

                              <param key="host" value="${hornetq.remoting.netty.connector.host1:server1}"/>

                              <param key="port" value="${hornetq.remoting.netty.connector.host1.port:5445}"/>

                          </connector>

                   

                          <!-- Static connector to the HornetQ Server 2 -->

                          <connector name="netty-connector-host2">

                              <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

                              <param key="host" value="${hornetq.remoting.netty.connector.host2:server2}"/>

                              <param key="port" value="${hornetq.remoting.netty.connector.host2.port:5445}"/>

                          </connector>

                   

                      </connectors>

                      <!-- Acceptors -->

                      <acceptors>

                          <acceptor name="netty-acceptor">

                              <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>

                              <param key="host" value="${hornetq.remoting.netty.acceptor.host:server1}"/>

                              <param key="port" value="${hornetq.remoting.netty.acceptor.host.port:5455}"/>

                          </acceptor>

                      </acceptors>

                   

                      <cluster-connections>

                          <cluster-connection name="DevCluster">

                              <address>jms</address>

                              <connector-ref>netty-connector-host1</connector-ref>

                              <retry-interval>500</retry-interval>

                              <use-duplicate-detection>true</use-duplicate-detection>

                              <forward-when-no-consumers>true</forward-when-no-consumers>

                              <max-hops>1</max-hops>

                              <static-connectors>

                                  <connector-ref>netty-connector-host2</connector-ref>               

                              </static-connectors>

                          </cluster-connection>

                      </cluster-connections>

                   

                  Server2

                  start.sh

                  ...

                  ###############################################################################################

                  # JVM Settings

                  export JVM_ARGS="-XX:+UseParallelGC -XX:+UseFastAccessorMethods"

                  export JVM_ARGS="$JVM_ARGS -Xms1024M -Xmx1024M"

                  export JVM_ARGS="$JVM_ARGS -Djava.library.path=."

                  export JVM_ARGS="$JVM_ARGS -Djava.net.preferIPv4Stack=true"

                  export JVM_ARGS="$JVM_ARGS -Dfile.encoding=utf-8"

                  export JVM_ARGS="$JVM_ARGS -Dcom.sun.management.jmxremote"

                  export JVM_ARGS="$JVM_ARGS -Dcom.sun.management.jmxremote.port=3000"

                  export JVM_ARGS="$JVM_ARGS -Dcom.sun.management.jmxremote.ssl=false"

                  export JVM_ARGS="$JVM_ARGS -Dcom.sun.management.jmxremote.authenticate=false"

                   

                  ###############################################################################################

                  # HornetQ Config

                  #export JVM_ARGS="$JVM_ARGS -Djava.util.logging.manager=org.jboss.logmanager.LogManager"

                  export JVM_ARGS="$JVM_ARGS -Djava.util.logging.config.file=$CONFIG_DIR/logging.properties"

                   

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.config.dir=$CONFIG_DIR"

                  export JVM_ARGS="$JVM_ARGS -Ddata.dir=/usr/java/hornetq/dev/data/test2"

                  export JVM_ARGS="$JVM_ARGS -Dcluster.user=password"

                  export JVM_ARGS="$JVM_ARGS -Dcluster.password=user"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.connector.host1=192.168.206.133"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.connector.host1.port=15445"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.connector.host2=192.168.206.132"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.connector.host2.port=15446"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.acceptor.host=192.168.206.133"

                  export JVM_ARGS="$JVM_ARGS -Dhornetq.remoting.netty.acceptor.host.port=15445"

                  export JVM_ARGS="$JVM_ARGS -Djnp.port=11099"

                  export JVM_ARGS="$JVM_ARGS -Djnp.host=192.168.206.133"

                  export JVM_ARGS="$JVM_ARGS -Djnp.rmiPort=11098"

                  export JVM_ARGS="$JVM_ARGS -Djava.naming.factory.initial=org.jnp.interfaces.NamingContextFactory"

                  export JVM_ARGS="$JVM_ARGS -Djava.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces"

                  ...

                   

                  PROBLEM

                  Both server are starting at the same time, the second server is not waiting for an event (crash or shutdown) on the first one.

                  Does someone have an idea why this happens?

                  • 6. Re: Live and Backup in Symmetrical cluster with shared store
                    jbertram

                    If both servers are active simultaneously (i.e. the backup server is not starting up and staying in backup mode while the live server is active) then my guess is that the two servers are not actually sharing the same journal.

                    • 7. Re: Live and Backup in Symmetrical cluster with shared store
                      joker1981

                      Or is the requirement to have a cluster file system for this setup?

                      • 8. Re: Live and Backup in Symmetrical cluster with shared store
                        ataylor

                        for shared store clustering you need to use some sort of shared file system. see http://docs.jboss.org/hornetq/2.3.0.CR1/docs/user-manual/html_single/index.html#ha.mode.shared