Sever Failover

[sboying]

Initially I want to use clustering to basically have a node backup. The primary node would have several threads collecting data from remote systems via JMS. The secondary would be idle. If the primary server is
stopped or becomes unresponsive, I would like the secondary to take over.
I thought this would be the cleanest since servers in cluster discover each other and keep track. Is there a good reference for this, I have looked at the docs, maybe I'm missing this?

[brian.stansberry]

Deploy your service as an HASingleton. Easiest way to do this is discussed at http://wiki.jboss.org/wiki/Wiki.jsp?page=HASingletonDeployer.

[sboying]

I have started trying to configure clustering by copying the all/deploy/cluster-service.xml file. I am running with a slimmed downed server based off of "default".



<!-- ==================================================================== -->
<!-- Cluster Partition: defines cluster -->
<!-- ==================================================================== -->



<!-- Name of the partition being built -->
${jboss.partition.name:DefaultPartition}

<!-- The address used to determine the node name -->
${jboss.bind.address}

<!-- Determine if deadlock detection is enabled -->
False

<!-- Max time (in ms) to wait for state transfer to complete. Increase for large states -->
30000

<!-- The JGroups protocol configuration -->

<!--
The default UDP stack:
- If you have a multihomed machine, set the UDP protocol's bind_addr attribute to the
appropriate NIC IP address, e.g bind_addr="192.168.0.2".
- On Windows machines, because of the media sense feature being broken with multicast
(even after disabling media sense) set the UDP protocol's loopback attribute to true
-->

<UDP mcast_addr="${jboss.partition.udpGroup:228.1.2.3}" mcast_port="45566"
ip_ttl="8" ip_mcast="true"
mcast_send_buf_size="800000" mcast_recv_buf_size="150000"
ucast_send_buf_size="800000" ucast_recv_buf_size="150000"
loopback="false"/>
<PING timeout="2000" num_initial_members="3"
up_thread="true" down_thread="true"/>
<MERGE2 min_interval="10000" max_interval="20000"/>
<FD shun="true" up_thread="true" down_thread="true"
timeout="2500" max_tries="5"/>
<VERIFY_SUSPECT timeout="3000" num_msgs="3"
up_thread="true" down_thread="true"/>
<pbcast.NAKACK gc_lag="50" retransmit_timeout="300,600,1200,2400,4800"
max_xmit_size="8192"
up_thread="true" down_thread="true"/>
<UNICAST timeout="300,600,1200,2400,4800" window_size="100" min_threshold="10"
down_thread="true"/>
<pbcast.STABLE desired_avg_gossip="20000"
up_thread="true" down_thread="true"/>
<FRAG frag_size="8192"
down_thread="true" up_thread="true"/>
<pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
shun="true" print_local_addr="true"/>
<pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>


jboss:service=Naming




Now I get the following exception on startup?

2007-05-10 13:54:37,768 DEBUG [org.jboss.system.ServiceCreator] About to create bean: jboss:service=DefaultPartition with code: org.
jboss.ha.framework.server.ClusterPartition
2007-05-10 13:54:37,788 DEBUG [org.jboss.util.NestedThrowable] org.jboss.util.NestedThrowable.parentTraceEnabled=true
2007-05-10 13:54:37,788 DEBUG [org.jboss.util.NestedThrowable] org.jboss.util.NestedThrowable.nestedTraceEnabled=false
2007-05-10 13:54:37,789 DEBUG [org.jboss.util.NestedThrowable] org.jboss.util.NestedThrowable.detectDuplicateNesting=true
2007-05-10 13:54:37,790 DEBUG [org.jboss.deployment.SARDeployer] create operation failed for package file:/cts/jboss-4.0.4.GA/server
/sla-bkup/deploy/cluster-service.xml
org.jboss.deployment.DeploymentException: No ClassLoaders found for: org.jboss.ha.framework.server.ClusterPartition; - nested throwa
ble: (java.lang.ClassNotFoundException: No ClassLoaders found for: org.jboss.ha.framework.server.ClusterPartition)
at org.jboss.system.ServiceConfigurator.install(ServiceConfigurator.java:196)
at org.jboss.system.ServiceController.install(ServiceController.java:226)
at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
at $Proxy4.install(Unknown Source)
at org.jboss.deployment.SARDeployer.create(SARDeployer.java:249)
at org.jboss.deployment.MainDeployer.create(MainDeployer.java:953)
at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:807)
at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:771)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
at $Proxy8.deploy(Unknown Source)
at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421)
at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:634)
at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263)
at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:336)
at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
at $Proxy0.start(Unknown Source)
at org.jboss.system.ServiceController.start(ServiceController.java:417)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
at $Proxy4.start(Unknown Source)
at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1007)
at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:808)
at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:771)
at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:755)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
at $Proxy5.deploy(Unknown Source)
at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:482)
at org.jboss.system.server.ServerImpl.start(ServerImpl.java:362)
at org.jboss.Main.boot(Main.java:200)
at org.jboss.Main$1.run(Main.java:464)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.ClassNotFoundException: No ClassLoaders found for: org.jboss.ha.framework.server.ClusterPartition
at org.jboss.mx.loading.LoadMgr3.beginLoadTask(LoadMgr3.java:212)
at org.jboss.mx.loading.RepositoryClassLoader.loadClassImpl(RepositoryClassLoader.java:511)
at org.jboss.mx.loading.RepositoryClassLoader.loadClass(RepositoryClassLoader.java:405)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at org.jboss.mx.server.MBeanServerImpl.instantiate(MBeanServerImpl.java:1204)
at org.jboss.mx.server.MBeanServerImpl.instantiate(MBeanServerImpl.java:286)
at org.jboss.mx.server.MBeanServerImpl.createMBean(MBeanServerImpl.java:344)
at org.jboss.system.ServiceCreator.install(ServiceCreator.java:157)
at org.jboss.system.ServiceConfigurator.internalInstall(ServiceConfigurator.java:449)
at org.jboss.system.ServiceConfigurator.install(ServiceConfigurator.java:171)

[cheenu2]

Seems like it is just a ClassNotFoundException -
Caused by: java.lang.ClassNotFoundException: No ClassLoaders found for: org.jboss.ha.framework.server.ClusterPartition

You need [server]/lib/jbossha.jar and probably other jars if you get other ClassNotFoundException after adding this.

You can get them from [server]/all/lib

Cheenu

[sboying]

Yes that was it, copied jbossha.jar from the "all" server config. Now the cluster service show in JMX console. Thank you!
(-:}

[brian.stansberry]

You need to copy over jbossha.jar and jgroups.jar from all/lib to default/lib. If you want to use JBoss Cache you need jboss-cache.jar as well.

For more semi-related info see http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossASTuningSliming. Towards the bottom there's a section on removing clustering features from 'all'. You can use that to get a better understanding of what stuff in 'all' does what.

[brian.stansberry]

Oops, didn't see cheenu2 had already replied. Thanks, cheenu2. :)

[sboying]

OK, so I want to use the "HASingleton Deployments using a Barrier".

1- setup cluster on 2 nodes inititally with "DefaultPartition"
2- copy deploy-hasingleton-service.xml from the server/all/deploy to server/ofmychoice/deploy for both nodes.
3- add jboss.ha:service=HASingletonDeployer,type=Barrier to my mBean service definition with jboss-service.xml
4 - same SAR file is deployed to /server/ofmychoice/deploy on both clustered nodes.

Is this correct, what determines the master node?

[sboying]

step three is a ""

[sboying]

step 3 is a depends, can embed XML tags

[brian.stansberry]

The master node is generally (but not always) the member of the group that has been running the longest.

Your description of what you did sounds OK presuming in your step #1 your "ofmychoice" config has a cluster-service.xml.

[sboying]

Thanks Brian. Yes, the cluster-service.xml will be in the /server/ofmychoice/deploy on both nodes. If one wants a particular node to always be the master by default, what is the best method?

[brian.stansberry]

Not easy to do reliably. :(

In 4.0.x, the master node is always the JGroups coordinator, which as I said is usually the node that's been running longest. Usually but not always. So, starting the node you want as master first is a quasi-solution

But, that's not very nice and isn't bulletproof, since if there is a disruption on the network, another server can become coordinator. Plus if you need to restart the preferred master, it won't be coordinator again unless you restart all the nodes.

In 4.2 we introduced the ability to configure a pluggable policy for determining the master. Pluggable = implement an interface and specify in the service.xml for your HASingleton the class of the impl. There's no implementation yet of a policy that does what you want, though. There's a JIRA for it: http://jira.jboss.com/jira/browse/JBAS-4228 and I'd be quite happy if someone from the community wanted to implement that.

[brian.stansberry]

Slight qualification to the above: in 4.0.x the master for an HASingleton service is the cluster member that:

1) Has the service deployed.
2) Holds the lowest position in the cluster view, which is usually ordered by when nodes joined the cluster.

If the typical case where all the nodes in the cluster have the service deployed, the JGroups coordinator is the master. The JGroups coordinator always has the lowest (i.e first) position in the view.

This is also the default behavior in 4.2.x, although the pluggable policies I mentioned let you change that.

[sboying]

Is this just a matter of putting the "preferred server" definition in the jboss-service.xml and then implementing an interface to determine what server you are on? When I initially started down this path, I had written a small amount of code to compare the current server with a configured LDAP value to see if one is the preferred:

* PrimaryServer.java
*
*/
package com.eds.sea.common.jmx.service.config;

import org.apache.log4j.Logger;
import java.io.*;

import com.eds.sea.common.jmx.service.config.*;

public class PrimaryServer
{
public PrimaryServer() {}

public static boolean IsPrimary(String complex)
{
try
{
String currServer = null;
String outErrStr = null;
boolean rCode = false;
LDAPConfig lConfig = new LDAPConfig();
String priServer = lConfig.getPrimaryServer(complex);
String cmd = "/usr/bin/uname -n";
Logger.getLogger(PrimaryServer.class.getName()).info(cmd);
Process p = Runtime.getRuntime().exec(cmd);
BufferedReader stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));
BufferedReader stdError = new BufferedReader(new InputStreamReader(p.getErrorStream()));

while ((currServer = stdInput.readLine()) != null)
System.out.println(currServer);

while ((outErrStr = stdError.readLine()) != null)
System.out.println(outErrStr);

rCode = currServer.equalsIgnoreCase(priServer) ? true : false;
}
catch ( Exception e )
{
rCode = false;
Logger.getLogger(PrimaryServer.class.getName()).error(e);
}

return rCode;
}
}