Recovering from a Cluster Shutdown

The default HA proxies can only fail over as long as one node in the cluster exists. If there is a complete cluster shutdown, the proxy becomes orphaned and loses knowledge of the available nodes in the cluster. There is no way for the proxy to recover from this. The proxy needs to be looked up out of JNDI/HAJNDI after the nodes are restarted.

The 3.2.7/4.0.2 releases contain a RetryInterceptor that can be added to the proxy client side interceptor stack to allow for a transparent recovery from such a restart failure. To enable it for an ejb, set up an invoker-proxy-binding that includes the RetryInterceptor:

jboss.xml:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE jboss PUBLIC
      "-//JBoss//DTD JBOSS 3.2//EN"
      "http://www.jboss.org/j2ee/dtd/jboss_3_2.dtd">

<jboss>
      <session>
         <ejb-name>nextgen_RetryInterceptorStatelessSession</ejb-name>
         <invoker-bindings>
            <invoker>
               <invoker-proxy-binding-name>clustered-retry-stateless-rmi-invoker</invoker-proxy-binding-name>
               <jndi-name>nextgen_RetryInterceptorStatelessSession</jndi-name>
            </invoker>
         </invoker-bindings>
         <clustered>true</clustered>
      </session>

      <invoker-proxy-binding>
         <name>clustered-retry-stateless-rmi-invoker</name>
         <invoker-mbean>jboss:service=invoker,type=jrmpha</invoker-mbean>
         <proxy-factory>org.jboss.proxy.ejb.ProxyFactoryHA</proxy-factory>
         <proxy-factory-config>
            <client-interceptors>
               <home>
                  <interceptor>org.jboss.proxy.ejb.HomeInterceptor</interceptor>
                  <interceptor>org.jboss.proxy.SecurityInterceptor</interceptor>
                  <interceptor>org.jboss.proxy.TransactionInterceptor</interceptor>
                  <interceptor>org.jboss.proxy.ejb.RetryInterceptor</interceptor>
                  <interceptor>org.jboss.invocation.InvokerInterceptor</interceptor>
               </home>
               <bean>
                  <interceptor>org.jboss.proxy.ejb.StatelessSessionInterceptor</interceptor>
                  <interceptor>org.jboss.proxy.SecurityInterceptor</interceptor>
                  <interceptor>org.jboss.proxy.TransactionInterceptor</interceptor>
                  <interceptor>org.jboss.proxy.ejb.RetryInterceptor</interceptor>
                  <interceptor>org.jboss.invocation.InvokerInterceptor</interceptor>
               </bean>
            </client-interceptors>
         </proxy-factory-config>
      </invoker-proxy-binding>

Example client fragment:

import java.util.Properties;

import javax.naming.Context;
import javax.naming.InitialContext;

import org.jboss.proxy.ejb.RetryInterceptor;
...

   public void testStatelessRetryInterceptor()
      throws Exception
   {
      getLog().debug("+++ testStatelessRetryInterceptor");

      // Connect to the server0 JNDI
      String[] urls = getNamingURLs();
      Properties env1 = new Properties();
      env1.setProperty(Context.INITIAL_CONTEXT_FACTORY,
         "org.jnp.interfaces.NamingContextFactory");
      env1.setProperty(Context.PROVIDER_URL, urls[0]);
      InitialContext ctx = new InitialContext(env1);

      // Set the InitialContext env to recover from
      RetryInterceptor.setRetryEnv(env1);

      getLog().debug("Looking up nextgen_RetryInterceptorStatelessSession");
      StatelessSessionHome statelessSessionHome =
         (StatelessSessionHome) ctx.lookup("nextgen_RetryInterceptorStatelessSession");
      StatelessSession statelessSession = statelessSessionHome.create();

      statelessSession.makeCountedCall();
      System.err.println("Sleeping 60 seconds, restart servers");
      Thread.sleep(60 * 1000);

      // Make a call after the server restart to test recovery
      statelessSession.makeCountedCall();
      getLog().debug("Called makeCountedCall");
      statelessSession.remove();
      getLog().debug("Called remove");
   }

JNDI Lookup Process

In order to recover the HA proxy, the RetryInterceptor does a lookup in JNDI.  This means that internally it creates a new InitialContext and does a JNDI lookup.  But, for that lookup to succeed, the InitialContext needs to be configured properly to find your naming server.  The RetryInterceptor will go through the following steps in attempting to determine the proper naming environment properties:

1. It will check its own static retryEnv field.  This field can be set by client code via a call to RetryInterceptor.setRetryEnv(Properties).  The client code example above uses this method.  This approach to configuration has two downsides: first, it reduces portability by introducing JBoss-specific calls to the client code; and second, since a static field is used only a single configuration per JVM is possible.

2. If the retryEnv field is null, it will check for any environment properties bound to a ThreadLocal by the org.jboss.naming.NamingContextFactory class.  See NamingContextFactory for more on how to use this class as the factory for your naming context.  The advantage of this approach is use of org.jboss.naming.NamingContextFactory is simply a configuration option in your jndi.properties file, and thus your java code is unaffected.  The downside is the naming properties are stored in a ThreadLocal and thus are only visible to the thread that originally created an InitialContext.

3. If neither of the above approaches yield a set of naming environment properties, a default InitialContext is used.  In this case, an attempt will be made to contact a naming server at localhost:1099.

If the attempt to contact a naming server is unsuccessful, by default the InitialContext will attempt to fall back on multicast discovery to find an HA-JNDI naming server.  See NamingContextFactory and the JBoss AS Clustering Guide for more on multicast discovery of HA-JNDI.

SingleRetryInterceptor

The RetryInterceptor is useful in many use cases, but a disadvantage it has is that it will continue attempting to re-lookup the HA proxy in JNDI until it succeeds.  If for some reason it cannot succeed, this process could go on forever, and thus the EJB call that triggered the RetryInterceptor will never return. For many client applications, this possibility is unacceptable.  As a result, JBoss doesn't make the RetryInterceptor part of its default client interceptor stacks for clustered EJBs.

In the 4.0.4.RC1 release, a new flavor of retry interceptor was introduced, the org.jboss.proxy.ejb.SingleRetryInterceptor.  This version works like the RetryInterceptor, but only makes a single attempt to re-lookup the HA proxy in JNDI.  If this attempt fails, the EJB call will fail just as if no retry interceptor was used.  Beginning with 4.0.4.CR2, the SingleRetryInterceptor is part of the default client interceptor stacks for clustered EJBs.

The downside of the SingleRetryInterceptor is that if the retry attempt is made during a portion of a cluster restart where no servers are available, the retry will fail and no further attempts will be made.

Writing your own flavor of RetryInterceptor

As discussed above, the RetryInterceptor and the SingleRetryInterceptor are useful, but have limitations.  The RetryInterceptor may go on trying to re-lookup the HA proxy forever; the SingleRetryInterceptor will only try once and throw an exception if unsuccessful.  Client applications may want a middle-ground approach; e.g. "keep retrying for 30 seconds and then fail if unsuccessful."

Beginning with release 4.0.4.RC1, a protected constructor was added to the RetryInterceptor making it trivial to write your own version of the retry interceptor with the desired behavior.  Such a custom retry interceptor must:

1. Subclass org.jboss.proxy.ejb.RetryInterceptor

2. Expose a public default (no-arg) constructor

3. In that constructor, call the superclass constructor, passing an int param telling how many retry attempts to make and a long param telling how many milliseconds to wait between retry attempts.

For example:

package com.xyz.proxy.ejb;

import org.jboss.proxy.ejb.RetryInterceptor;

public class ThirtySecondRetryInterceptor extends RetryInterceptor
{
   private static final long serialVersionUID = 1L;
   
   public ThirtySecondRetryInterceptor()
   {
      super(30, 1000);
   }
}