1 2 Previous Next 16 Replies Latest reply on May 17, 2007 6:19 PM by Brian Stansberry

    Sever Failover

    Terry Boyington Newbie

      Initially I want to use clustering to basically have a node backup. The primary node would have several threads collecting data from remote systems via JMS. The secondary would be idle. If the primary server is
      stopped or becomes unresponsive, I would like the secondary to take over.
      I thought this would be the cleanest since servers in cluster discover each other and keep track. Is there a good reference for this, I have looked at the docs, maybe I'm missing this?

        • 1. Re: Sever Failover
          Brian Stansberry Master

          Deploy your service as an HASingleton. Easiest way to do this is discussed at http://wiki.jboss.org/wiki/Wiki.jsp?page=HASingletonDeployer.

          • 2. Re: Sever Failover
            Terry Boyington Newbie

            I have started trying to configure clustering by copying the all/deploy/cluster-service.xml file. I am running with a slimmed downed server based off of "default".



            <!-- ==================================================================== -->
            <!-- Cluster Partition: defines cluster -->
            <!-- ==================================================================== -->



            <!-- Name of the partition being built -->
            ${jboss.partition.name:DefaultPartition}

            <!-- The address used to determine the node name -->
            ${jboss.bind.address}

            <!-- Determine if deadlock detection is enabled -->
            False

            <!-- Max time (in ms) to wait for state transfer to complete. Increase for large states -->
            30000

            <!-- The JGroups protocol configuration -->

            <!--
            The default UDP stack:
            - If you have a multihomed machine, set the UDP protocol's bind_addr attribute to the
            appropriate NIC IP address, e.g bind_addr="192.168.0.2".
            - On Windows machines, because of the media sense feature being broken with multicast
            (even after disabling media sense) set the UDP protocol's loopback attribute to true
            -->

            <UDP mcast_addr="${jboss.partition.udpGroup:228.1.2.3}" mcast_port="45566"
            ip_ttl="8" ip_mcast="true"
            mcast_send_buf_size="800000" mcast_recv_buf_size="150000"
            ucast_send_buf_size="800000" ucast_recv_buf_size="150000"
            loopback="false"/>
            <PING timeout="2000" num_initial_members="3"
            up_thread="true" down_thread="true"/>
            <MERGE2 min_interval="10000" max_interval="20000"/>
            <FD shun="true" up_thread="true" down_thread="true"
            timeout="2500" max_tries="5"/>
            <VERIFY_SUSPECT timeout="3000" num_msgs="3"
            up_thread="true" down_thread="true"/>
            <pbcast.NAKACK gc_lag="50" retransmit_timeout="300,600,1200,2400,4800"
            max_xmit_size="8192"
            up_thread="true" down_thread="true"/>
            <UNICAST timeout="300,600,1200,2400,4800" window_size="100" min_threshold="10"
            down_thread="true"/>
            <pbcast.STABLE desired_avg_gossip="20000"
            up_thread="true" down_thread="true"/>
            <FRAG frag_size="8192"
            down_thread="true" up_thread="true"/>
            <pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
            shun="true" print_local_addr="true"/>
            <pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>


            jboss:service=Naming




            Now I get the following exception on startup?

            2007-05-10 13:54:37,768 DEBUG [org.jboss.system.ServiceCreator] About to create bean: jboss:service=DefaultPartition with code: org.
            jboss.ha.framework.server.ClusterPartition
            2007-05-10 13:54:37,788 DEBUG [org.jboss.util.NestedThrowable] org.jboss.util.NestedThrowable.parentTraceEnabled=true
            2007-05-10 13:54:37,788 DEBUG [org.jboss.util.NestedThrowable] org.jboss.util.NestedThrowable.nestedTraceEnabled=false
            2007-05-10 13:54:37,789 DEBUG [org.jboss.util.NestedThrowable] org.jboss.util.NestedThrowable.detectDuplicateNesting=true
            2007-05-10 13:54:37,790 DEBUG [org.jboss.deployment.SARDeployer] create operation failed for package file:/cts/jboss-4.0.4.GA/server
            /sla-bkup/deploy/cluster-service.xml
            org.jboss.deployment.DeploymentException: No ClassLoaders found for: org.jboss.ha.framework.server.ClusterPartition; - nested throwa
            ble: (java.lang.ClassNotFoundException: No ClassLoaders found for: org.jboss.ha.framework.server.ClusterPartition)
            at org.jboss.system.ServiceConfigurator.install(ServiceConfigurator.java:196)
            at org.jboss.system.ServiceController.install(ServiceController.java:226)
            at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
            at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
            at $Proxy4.install(Unknown Source)
            at org.jboss.deployment.SARDeployer.create(SARDeployer.java:249)
            at org.jboss.deployment.MainDeployer.create(MainDeployer.java:953)
            at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:807)
            at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:771)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
            at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
            at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
            at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
            at $Proxy8.deploy(Unknown Source)
            at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421)
            at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:634)
            at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263)
            at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:336)
            at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
            at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
            at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
            at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
            at $Proxy0.start(Unknown Source)
            at org.jboss.system.ServiceController.start(ServiceController.java:417)
            at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
            at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
            at $Proxy4.start(Unknown Source)
            at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
            at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1007)
            at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:808)
            at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:771)
            at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:755)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
            at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
            at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
            at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
            at $Proxy5.deploy(Unknown Source)
            at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:482)
            at org.jboss.system.server.ServerImpl.start(ServerImpl.java:362)
            at org.jboss.Main.boot(Main.java:200)
            at org.jboss.Main$1.run(Main.java:464)
            at java.lang.Thread.run(Thread.java:619)
            Caused by: java.lang.ClassNotFoundException: No ClassLoaders found for: org.jboss.ha.framework.server.ClusterPartition
            at org.jboss.mx.loading.LoadMgr3.beginLoadTask(LoadMgr3.java:212)
            at org.jboss.mx.loading.RepositoryClassLoader.loadClassImpl(RepositoryClassLoader.java:511)
            at org.jboss.mx.loading.RepositoryClassLoader.loadClass(RepositoryClassLoader.java:405)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
            at org.jboss.mx.server.MBeanServerImpl.instantiate(MBeanServerImpl.java:1204)
            at org.jboss.mx.server.MBeanServerImpl.instantiate(MBeanServerImpl.java:286)
            at org.jboss.mx.server.MBeanServerImpl.createMBean(MBeanServerImpl.java:344)
            at org.jboss.system.ServiceCreator.install(ServiceCreator.java:157)
            at org.jboss.system.ServiceConfigurator.internalInstall(ServiceConfigurator.java:449)
            at org.jboss.system.ServiceConfigurator.install(ServiceConfigurator.java:171)

            • 3. Re: Sever Failover
              Cheenu M Newbie

              Seems like it is just a ClassNotFoundException -
              Caused by: java.lang.ClassNotFoundException: No ClassLoaders found for: org.jboss.ha.framework.server.ClusterPartition

              You need [server]/lib/jbossha.jar and probably other jars if you get other ClassNotFoundException after adding this.

              You can get them from [server]/all/lib

              Cheenu

              • 4. Re: Sever Failover
                Terry Boyington Newbie

                Yes that was it, copied jbossha.jar from the "all" server config. Now the cluster service show in JMX console. Thank you!
                (-:}

                • 5. Re: Sever Failover
                  Brian Stansberry Master

                  You need to copy over jbossha.jar and jgroups.jar from all/lib to default/lib. If you want to use JBoss Cache you need jboss-cache.jar as well.

                  For more semi-related info see http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossASTuningSliming. Towards the bottom there's a section on removing clustering features from 'all'. You can use that to get a better understanding of what stuff in 'all' does what.

                  • 6. Re: Sever Failover
                    Brian Stansberry Master

                    Oops, didn't see cheenu2 had already replied. Thanks, cheenu2. :)

                    • 7. Re: Sever Failover
                      Terry Boyington Newbie

                      OK, so I want to use the "HASingleton Deployments using a Barrier".

                      1- setup cluster on 2 nodes inititally with "DefaultPartition"
                      2- copy deploy-hasingleton-service.xml from the server/all/deploy to server/ofmychoice/deploy for both nodes.
                      3- add jboss.ha:service=HASingletonDeployer,type=Barrier to my mBean service definition with jboss-service.xml
                      4 - same SAR file is deployed to /server/ofmychoice/deploy on both clustered nodes.

                      Is this correct, what determines the master node?

                      • 8. Re: Sever Failover
                        Terry Boyington Newbie

                        step three is a ""

                        • 9. Re: Sever Failover
                          Terry Boyington Newbie

                          step 3 is a depends, can embed XML tags

                          • 10. Re: Sever Failover
                            Brian Stansberry Master

                            The master node is generally (but not always) the member of the group that has been running the longest.

                            Your description of what you did sounds OK presuming in your step #1 your "ofmychoice" config has a cluster-service.xml.

                            • 11. Re: Sever Failover
                              Terry Boyington Newbie

                              Thanks Brian. Yes, the cluster-service.xml will be in the /server/ofmychoice/deploy on both nodes. If one wants a particular node to always be the master by default, what is the best method?

                              • 12. Re: Sever Failover
                                Brian Stansberry Master

                                Not easy to do reliably. :(

                                In 4.0.x, the master node is always the JGroups coordinator, which as I said is usually the node that's been running longest. Usually but not always. So, starting the node you want as master first is a quasi-solution

                                But, that's not very nice and isn't bulletproof, since if there is a disruption on the network, another server can become coordinator. Plus if you need to restart the preferred master, it won't be coordinator again unless you restart all the nodes.

                                In 4.2 we introduced the ability to configure a pluggable policy for determining the master. Pluggable = implement an interface and specify in the service.xml for your HASingleton the class of the impl. There's no implementation yet of a policy that does what you want, though. There's a JIRA for it: http://jira.jboss.com/jira/browse/JBAS-4228 and I'd be quite happy if someone from the community wanted to implement that.

                                • 13. Re: Sever Failover
                                  Brian Stansberry Master

                                  Slight qualification to the above: in 4.0.x the master for an HASingleton service is the cluster member that:

                                  1) Has the service deployed.
                                  2) Holds the lowest position in the cluster view, which is usually ordered by when nodes joined the cluster.

                                  If the typical case where all the nodes in the cluster have the service deployed, the JGroups coordinator is the master. The JGroups coordinator always has the lowest (i.e first) position in the view.

                                  This is also the default behavior in 4.2.x, although the pluggable policies I mentioned let you change that.

                                  • 14. Re: Sever Failover
                                    Terry Boyington Newbie

                                    Is this just a matter of putting the "preferred server" definition in the jboss-service.xml and then implementing an interface to determine what server you are on? When I initially started down this path, I had written a small amount of code to compare the current server with a configured LDAP value to see if one is the preferred:

                                    * PrimaryServer.java
                                    *
                                    */
                                    package com.eds.sea.common.jmx.service.config;

                                    import org.apache.log4j.Logger;
                                    import java.io.*;

                                    import com.eds.sea.common.jmx.service.config.*;

                                    public class PrimaryServer
                                    {
                                    public PrimaryServer() {}

                                    public static boolean IsPrimary(String complex)
                                    {
                                    try
                                    {
                                    String currServer = null;
                                    String outErrStr = null;
                                    boolean rCode = false;
                                    LDAPConfig lConfig = new LDAPConfig();
                                    String priServer = lConfig.getPrimaryServer(complex);
                                    String cmd = "/usr/bin/uname -n";
                                    Logger.getLogger(PrimaryServer.class.getName()).info(cmd);
                                    Process p = Runtime.getRuntime().exec(cmd);
                                    BufferedReader stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));
                                    BufferedReader stdError = new BufferedReader(new InputStreamReader(p.getErrorStream()));

                                    while ((currServer = stdInput.readLine()) != null)
                                    System.out.println(currServer);

                                    while ((outErrStr = stdError.readLine()) != null)
                                    System.out.println(outErrStr);

                                    rCode = currServer.equalsIgnoreCase(priServer) ? true : false;
                                    }
                                    catch ( Exception e )
                                    {
                                    rCode = false;
                                    Logger.getLogger(PrimaryServer.class.getName()).error(e);
                                    }

                                    return rCode;
                                    }
                                    }

                                    1 2 Previous Next