LoadBalancing only works with standalone client but not with Resource Adaptor

[mlange]

We are executing functional and performance using a 3-node cluster based on EAP6.0.1 and noticed a strange phenomena. We use JMS

When connecting with a standalone client (only the first node is in the connection map), the consumers are distributed equally among all 3 nodes:

Map<String, Object> configMap = new HashMap<String, Object>();

String hornetQDestinationHost = System.getProperty("hornetq.destination.host", "localhost");

configMap.put("host", hornetQDestinationHost);

String hornetQDestinationPort = System.getProperty("hornetq.destination.port", "5445");

configMap.put("port", hornetQDestinationPort);

TransportConfiguration hornetQTC = new TransportConfiguration("org.hornetq.core.remoting.impl.netty.NettyConnectorFactory", configMap);

HornetQJMSConnectionFactory hornetQJMSConnectionFactory = new HornetQJMSConnectionFactory(true, hornetQTC);

When connecting using the Resource Adaptor by looking up the connection factory all consumers are always and only bound to the first node:

Context naming;

try {

    naming = new InitialContext();

    jmsConnectionFactory = (ConnectionFactory) naming.lookup(jmsConnectionFactoryName);

} catch (NamingException e) {

    throw new EventSystemException("Could not lookup JMS Connection Factory", e);

}

whereas the configuration looks good:

hornetq-ds.xml:

<connection-factories>

  <tx-connection-factory>

    <jndi-name>HornetqPooledConnectionFactory</jndi-name>

    <xa-transaction/>

    <rar-name>hornetq-ra.rar</rar-name>

    <connection-definition>org.hornetq.ra.HornetQRAConnectionFactory</connection-definition>

    <config-property name="SessionDefaultType" type="java.lang.String">javax.jms.Topic</config-property>

    <config-property name="ConnectionParameters" type="java.lang.String">host=app96;port=11830</config-property>

    <config-property name="ConnectorClassName" type="java.lang.String">org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</config-property>

    <config-property name="Strict" type="java.lang.Boolean">false</config-property>

    <config-property>

        <description>Does we support HA</description>

        <config-property-name>HA</config-property-name>

        <config-property-type>java.lang.Boolean</config-property-type>

        <config-property-value>true</config-property-value>

    </config-property>

    <min-pool-size>0</min-pool-size>

    <max-pool-size>100</max-pool-size>

  </tx-connection-factory>

</connection-factories>

hornetq-rar.rar/META-INF/ra.xml:

<resourceadapter>

      <resourceadapter-class>org.hornetq.ra.HornetQResourceAdapter</resourceadapter-class>

      <config-property>

         <description>

            The transport type. Multiple connectors can be configured by using a comma separated list,

            i.e. org.hornetq.core.remoting.impl.invm.InVMConnectorFactory,org.hornetq.core.remoting.impl.invm.InVMConnectorFactory.

         </description>

         <config-property-name>ConnectorClassName</config-property-name>

         <config-property-type>java.lang.String</config-property-type>

         <config-property-value>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</config-property-value>

      </config-property>

      <config-property>

         <description>The transport configuration. These values must be in the form of key=val;key=val;,

            if multiple connectors are used then each set must be separated by a comma i.e. host=host1;port=5445,host=host2;port=5446.

            Each set of params maps to the connector classname specified.

         </description>

         <config-property-name>ConnectionParameters</config-property-name>

         <config-property-type>java.lang.String</config-property-type>

         <config-property-value>host=app96;port=11830</config-property-value>

      </config-property>

      <config-property>

        <description>The user name used to login to the JMS server</description>

        <config-property-name>UserName</config-property-name>

        <config-property-type>java.lang.String</config-property-type>

        <config-property-value>myUser</config-property-value>

      </config-property>

      <config-property>

        <description>The password used to login to the JMS server</description>

        <config-property-name>Password</config-property-name>

        <config-property-type>java.lang.String</config-property-type>

        <config-property-value>myPassword</config-property-value>

      </config-property>

      <config-property>

        <description>Does we support HA</description>

        <config-property-name>HA</config-property-name>

        <config-property-type>java.lang.Boolean</config-property-type>

        <config-property-value>true</config-property-value>

      </config-property>

When debugging I can see that both run through the same lines of code:

HornetQJMSClient.java:

public static HornetQConnectionFactory createConnectionFactoryWithHA(final DiscoveryGroupConfiguration groupConfiguration, JMSFactoryType jmsFactoryType)

   {

      HornetQConnectionFactory factory = null;

      if (jmsFactoryType.equals(JMSFactoryType.CF))

      {

         // STANDALONE CLIENT

         factory = new HornetQJMSConnectionFactory(true, groupConfiguration);

      }     

      else if (jmsFactoryType.equals(JMSFactoryType.XA_CF))

      {

         // RESOURCE ADAPTOR CLIENT

         factory = new HornetQXAConnectionFactory(true, groupConfiguration);

      }

calling both HornetQConnectionFactory.java:

public HornetQConnectionFactory(final boolean ha, final DiscoveryGroupConfiguration groupConfiguration)

   {

      if (ha)

      {

         serverLocator = HornetQClient.createServerLocatorWithHA(groupConfiguration);

      }

I am not able to find out why the load balancing works with the standalone client but not with the Resource Adaptor although everything looks correctly configured. Do you have any idea?

Thanks!

Marek

[ataylor]

The resource adapter currently doesnt support load balancing, this is becasue each MDB has its own connection and chooses the first server every time, there is a jira, https://issues.jboss.org/browse/JBPAPP-5792, to add this.

[mlange]

Hi Andy,

I also recognized that there is always a dedicated ServerLocatorImpl for each call when creating a consumer using the connection factory from the RA (which is not the case with a core client). The cluster topology updates are never received in these instances.

I cannot view the JBPAPP-5792 issue (due to permissions). Is there any way for a workaround to make the load balancing work? It does not seem to be such a big deal, or?

Thanks,

Marek

[ataylor]

you could deploy 2 MDB's each with there own static connector to the both nodes in the cluster or rely on server side load balancing.

As for whether it is a big deal or not we havent had much call for it as the main use case is hornetq in the same server as the MDB, altho I will be implementing it post 2.3.0

[mlange]

In our use case we do not have MDBs but JMS clients deployed within the appserver retrieving the connection factory via JNDI. How could your MDB proposal be abstracted to this? What exactly do you mean with server-side loadbalancing? A separate piece of hardware in front of the HornetQ instances?

Thanks for your input!

Marek

[ataylor]

In our use case we do not have MDBs but JMS clients deployed within the appserver retrieving the connection factory via JNDI.

The same principle applies, you could create 2 connection factories each configured to connect to connect to each node, you would however have to code this into your clients as to which factory to use.

What exactly do you mean with server-side loadbalancing?

HornetQ has both client side and server side load balancing. server side load balancing means that messages will be distriobuted around the nodes in the cluster, so if only one node has consumers messages received at the other node will be distributed to the node with the consumers.

[mlange]

The same principle applies, you could create 2 connection factories each configured to connect to connect to each node, you would however have to code this into your clients as to which factory to use.

But this would also require to implement failover on the client or? We have configured a backup server for each master node running on standby. When one node crashes or goes down for planned maintenance the backup server should transparently handle the traffic.

[ataylor]

But this would also require to implement failover on the client or? We have configured a backup server for each master node running on standby. When one node crashes or goes down for planned maintenance the backup server should transparently handle the traffic.

thats fine, that will still work, just add the backup server to the static list of servers and makes ure you are using HA