3 Replies Latest reply on Apr 29, 2004 6:30 PM by jason1

    Solution for load-balancing MDBs in 3.2.4RC2 HA-JMS

      I'm using 3.2.4RC2 (April 13th drop), which according to Adrian doesn't have load-balancing capabilities for MDBs across a cluster. So, I thought I'd post my solution for load-balanced, fault-tolerant MDBs that use the new HA-JMS in a cluster.

      My application looks like this:

      1. MDBs deployed in /farm directory across several JBoss servers in a cluster. MDBs in remote JBoss servers connect to the singleton, remote HA-JMS resources while the locally deployed MDBs connect locally.
      2. Oracle DB configured as a shared resource for JMS tables
      3. custom JMS Provider used to connect to the singleton HA-JMS from the other clustered JBoss servers via HA-JNDI.

      When the singleton HA-JMS server fails and another is selected, the custom JMS Provider re-connects (DLQHandlers, JMSContainerInvoker, etc.) to the new master node's resources. The locally deployed MDBs process more messages than the others because of their proximity to the JMS destinations.

      I've tested it and it seems to process messages fine (none lost, load-balancing really works, fail-over of JMSContainerInvoker and its DLQHandlers is seamless except for a few annoying but harmless error messages that get logged when trying to stop an already disconnected JMS SpyConnection, etc.).

      Steps to get it setup (see listings below):
      1. I turned logging in log4j.xml for the org.jnp.NamingContext to ERROR, so it doesn't log each attempt in HA-JNDI that fails as a WARN. Keeps logs cleaner.
      2. You have to build and deploy the custom JMS Provider listed below into a Jar and put it into the /deploy/jms directory.
      3. jms-ds.xml must be changed to reference the new Jar (cdot-jbossx.jar in my case)
      4. jms-ds.xml must point to the new, custom JMS Provider class.

      This solution is only for clustered (e.g., /all). It is not meant to work in standalone-mode, so don't try to use the custom JMS Provider in the default server.


      Custom JMS Provider code (modified from original JBossMQProvider):

      /*
       * Copyright (c) 2000 Peter Antman DN <peter.antman@dn.se>
       *
       * This library is free software; you can redistribute it and/or
       * modify it under the terms of the GNU Lesser General Public
       * License as published by the Free Software Foundation; either
       * version 2 of the License, or (at your option) any later version
       *
       * This library is distributed in the hope that it will be useful,
       * but WITHOUT ANY WARRANTY; without even the implied warranty of
       * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
       * Lesser General Public License for more details.
       *
       * You should have received a copy of the GNU Lesser General Public
       * License along with this library; if not, write to the Free Software
       * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
       */
      package cdot.jboss;
      
      import java.util.Hashtable;
      
      import javax.naming.Context;
      import javax.naming.InitialContext;
      import javax.naming.NamingException;
      
      import org.apache.commons.logging.Log;
      import org.apache.commons.logging.LogFactory;
      import org.jboss.jms.jndi.AbstractJMSProviderAdapter;
      
      /**
       * A JMS provider adapter for <em>JBossMQ</em> that can fail-over to
       * another JMS server automatically using the HA-JNDI lookup mechanism.
       *
       * Created: Fri Dec 22 09:34:04 2000
       * 6/22/01 - hchirino - The queue/topic jndi references are now configed via JMX
       * 4/27/04 - jwestra - Uses HA-JNDI settings to make InitialContext
       *
       * @version <pre>$Revision: 1.9 $</pre>
       * @author <a href="mailto:peter.antman@dn.se">Peter Antman</a>
       * @author <a href="mailto:cojonudo14@hotmail.com">Hiram Chirino</a>
       * @author <a href="mailto:jason@planet57.com">Jason Dillon</a>
       * @author <a href="mailto:jwestra@csc.com">Jason Westra</a>
       */
      public class JBossMQProvider
       extends AbstractJMSProviderAdapter
      {
       /** The initial context factory to use. */
       public static final String INITIAL_CONTEXT_FACTORY =
       "org.jnp.interfaces.NamingContextFactory";
      
       /** The url package prefixes. */
       public static final String URL_PKG_PREFIXES =
       "org.jboss.naming:org.jnp.interfaces";
      
       /** The security manager to use. */
       private static final String SECURITY_MANAGER =
       "java.naming.rmi.security.manager";
      
       /** Instance logger. */
       private transient Log log = LogFactory.getLog("cdot.jboss.JBossMQProvider");
      
       /** Flag to enable JNDI security manager. */
       private String hasJndiSecurityManager = "yes";
      
       /**
       * Default no-argument constructor.
       */
       public JBossMQProvider() {
       // empty
       log.debug("initializing");
       }
      
       /** Override of standard de-serialization to re-create logger. */
       private void readObject(java.io.ObjectInputStream in)
       throws java.io.IOException, ClassNotFoundException
       {
       in.defaultReadObject();
       this.log = LogFactory.getLog("cdot.jboss.JBossMQProvider");
       }
      
       /**
       * Create a new InitialContext suitable for this JMS provider.
       *
       * @return An InitialContext suitable for this JMS provider.
       *
       * @throws NamingException Failed to construct context.
       */
       public Context getInitialContext() throws NamingException {
       Context ctx = null;
       boolean debug = log.isDebugEnabled();
      
       try {
       // uses HA-JNDI settings to provide lookup of InitialContext
       Hashtable props = new Hashtable();
       props.put(Context.INITIAL_CONTEXT_FACTORY,
       INITIAL_CONTEXT_FACTORY);
       props.put(Context.PROVIDER_URL, providerURL);
       props.put(SECURITY_MANAGER, hasJndiSecurityManager);
       props.put(Context.URL_PKG_PREFIXES, URL_PKG_PREFIXES);
      
       if (debug)
       log.debug("connecting to remote JNDI with props: " + props);
       ctx = new InitialContext(props);
      
       // sanity check... can remove later
       String connectionFactoryRef = this.getQueueFactoryRef();
       ctx.lookup(connectionFactoryRef);
       // end sanity check
      
       if (debug) {
       log.debug("created context: " + ctx.getEnvironment());
       }
      
       } catch(Throwable t) {
       NamingException ex = new NamingException(t.getMessage());
       ex.setRootCause(t);
       throw ex;
       }
      
       return ctx;
       }
      }


      jms-ds.xml:

      <connection-factories>
      
       <!-- ==================================================================== -->
       <!-- JMS Stuff -->
       <!-- 1.) added classpath setting for custom JMSProvider -->
       <!-- 2.) added provider URLs for clustered JMS -->
       <!-- ==================================================================== -->
      
      
       <classpath codebase="deploy/jms" archives="cdot-jbossx.jar"/>
      
       <!-- The JMS provider loader -->
       <mbean code="org.jboss.jms.jndi.JMSProviderLoader"
       name="jboss.mq:service=JMSProviderLoader,name=JBossMQProvider">
       <attribute name="ProviderName">DefaultJMSProvider</attribute>
       <attribute name="ProviderAdapterClass">cdot.jboss.JBossMQProvider</attribute>
       <attribute name="ProviderUrl">localhost:1100,10.82.70.15:1100,10.82.70.16:1100</attribute>
       <attribute name="QueueFactoryRef">XAConnectionFactory</attribute>
       <attribute name="TopicFactoryRef">XAConnectionFactory</attribute>
       </mbean>
      
       <!-- The server session pool for Message Driven Beans -->
       <mbean code="org.jboss.jms.asf.ServerSessionPoolLoader"
       name="jboss.mq:service=ServerSessionPoolMBean,name=StdJMSPool">
       <depends optional-attribute-name="XidFactory">jboss:service=XidFactory</depends>
       <attribute name="PoolName">StdJMSPool</attribute>
       <attribute name="PoolFactoryClass">
       org.jboss.jms.asf.StdServerSessionPoolFactory
       </attribute>
       </mbean>
      
      
       <!-- JMS XA Resource adapter, use this to get transacted JMS in beans -->
      
       <tx-connection-factory>
      
       <jndi-name>JmsXA</jndi-name>
      
       <xa-transaction/>
      
       <adapter-display-name>JMS Adapter</adapter-display-name>
      
       <config-property name="SessionDefaultType" type="java.lang.String">javax.jms.Topic</config-property>
      
       <security-domain-and-application>JmsXARealm</security-domain-and-application>
      
       </tx-connection-factory>
      
      </connection-factories>


      log4j.xml:

      <!-- Limit JNP categories to ERROR. This avoids pesky WARNs about fails to connect to HA-JNDI -->
       <category name="org.jnp.interfaces.NamingContext">
       <priority value="ERROR"/>
       </category>


        • 1. Re: Solution for load-balancing MDBs in 3.2.4RC2 HA-JMS

          Great post, you seem to have understood it.

          One question, one clarification and one request. :-)

          Question:
          Why do you need a custom provider adapter? I don't see any difference given
          your configuration, other than the sanity check.
          I do see a need to be able to specify partition name or discovery address,
          but you haven't added this?
          All you seem to have done is disable the local jndi optimization which
          isn't used if you specify a provider url.

          Clarification:
          This is not load balancing as I would define it. Load balancing would require
          multiple jms servers serving requests to an MDB.

          Request:
          Could you add this to the WIKI

          • 2. Re: Solution for load-balancing MDBs in 3.2.4RC2 HA-JMS

            Answer:
            The difference is simply URL_PKG_PREFIXES which includes 'org.jnp.interfaces'. The whole new class just for a minor change allows us to stay of the way of new JBoss releases without getting clobbered.

            Clarification:
            Load-balancing as you'd define it is tough. Tibco doesn't even do it with their E4JMS product. But, if anyone could, it would be JBoss...

            My view on load-balancing w/ MDBs is to balance the processing load (e.g., the MDBs which process the messages) not the dispatching load (e.g., load-balancing the JMS server). Using MDBs to balance load of async requests is great because it processes the message on whichever node in the cluster is least busy, providing a nice even distribution compared to round-robin.

            Wiki:
            I can post to the Wiki.


            • 3. Re: Solution for load-balancing MDBs in 3.2.4RC2 HA-JMS