8 Replies Latest reply on Jul 12, 2011 5:17 AM by nstronics

Problem in Load Balancing in a Jboss 5.1 Cluster

rajsoni Mar 12, 2010 11:54 AM

Hello Everyone!

I'm having a problem with load balancing a call to the two nodes that I have in a Jboss cluster.Would really appreciate any help on this!

Am running a local Java client that calls a Stateless EJB deployed on the two nodes in a cluster.

Here's how the jboss.xml looks for the stateless EJB:

<?xml version="1.0"?>
<jboss xmlns:xs="http://www.jboss.org/j2ee/schema"
       xs:schemaLocation="http://www.jboss.org/j2ee/schema jboss_5_0.xsd"
       version="5.0">
    <enterprise-beans>
          <session>
               <ejb-name>MktSearch</ejb-name>
               <jndi-name>MktSearchEJB</jndi-name>
               <local-jndi-name>LocalMktSearchEJB</local-jndi-name>
               <clustered>True</clustered>
              <cluster-config>
                  <partition-name>DefaultPartition</partition-name>
                  <home-load-balance-policy>
                      org.jboss.ha.framework.interfaces.RoundRobin
                  </home-load-balance-policy>
                  <bean-load-balance-policy>
                      org.jboss.ha.framework.interfaces.RoundRobin
                  </bean-load-balance-policy>
              </cluster-config>
            </session>
      </enterprise-beans>

    <container-configurations>
        <container-configuration>
          <container-name>Standard Stateless SessionBean</container-name>
          <call-logging>false</call-logging>
          <invoker-proxy-binding-name>stateless-unified-invoker</invoker-proxy-binding-name>
          <container-interceptors>
            <interceptor>org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor</interceptor>
            <interceptor>org.jboss.ejb.plugins.LogInterceptor</interceptor>
            <interceptor>org.jboss.ejb.plugins.security.PreSecurityInterceptor</interceptor>
            <interceptor>org.jboss.ejb.plugins.SecurityInterceptor</interceptor>
            
            <interceptor transaction="Container">org.jboss.ejb.plugins.TxInterceptorCMT</interceptor>
            <interceptor transaction="Container">org.jboss.ejb.plugins.CallValidationInterceptor</interceptor>
            <interceptor transaction="Container">org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor</interceptor>
            
            <interceptor transaction="Bean">org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor</interceptor>
            <interceptor transaction="Bean">org.jboss.ejb.plugins.TxInterceptorBMT</interceptor>
            <interceptor transaction="Bean">org.jboss.ejb.plugins.CallValidationInterceptor</interceptor>
            <interceptor>org.jboss.resource.connectionmanager.CachedConnectionInterceptor</interceptor>
          </container-interceptors>
          <instance-pool>org.jboss.ejb.plugins.StatelessSessionInstancePool</instance-pool>
          <instance-cache></instance-cache>
          <persistence-manager></persistence-manager>
          <container-pool-conf>
            <MaximumSize>100</MaximumSize>
          </container-pool-conf>
        </container-configuration>
         
     </container-configurations>

</jboss>

I see that for an MDB which is deployed on the cluster,load balancing is working fine.In this case I see messages in logs,some messages get processed on node1 and some on node2.

For my local Java client,if I shut down node1, and restart my client,all calls go to node2 and vice-versa.But I need to restart the client for all calls to get rerouted.

If load balancing was working fine,I would expect that I would not need to restart the client.

Am starting the two nodes like this :

./run.sh -c node1 -b 10.9.10.164 -g DefaultPartition -u 239.255.100.102 -Djboss.messaging.ServerPeerID=1 -Djboss.service.binding.set=ports-01

./run.sh -c node2 -b 10.9.10.164 -g DefaultPartition -u 239.255.100.102 -Djboss.messaging.ServerPeerID=2 -Djboss.service.binding.set=ports-02

Would really appreciate and help/pointers for this!!

thanks!

1. Re: Problem in Load Balancing in a Jboss 5.1 Cluster

galder.zamarreno Mar 16, 2010 6:03 AM (in response to rajsoni)

Can you verify that the AS nodes are joining correctly in the cluster?

There's an FAQ on EJB load balancing in http://community.jboss.org/docs/DOC-9306 but I don't think it fits your situation since your client does not appear to be running within AS. Can you confirm?
Actions
2. Re: Problem in Load Balancing in a Jboss 5.1 Cluster

rajsoni Mar 16, 2010 8:30 AM (in response to galder.zamarreno)

yes..the AS nodes are joining the cluster correctly...Pl see the content from one of the instance Server log below:

*************************************************************************************************************************

08:17:41,351 INFO [DefaultPartition] New cluster view for partition DefaultPartition (id: 1, delta: 1) : [10.9.10.164:1199, 10.9.10.164:1299]
08:17:41,352 INFO [DefaultPartition] I am (10.9.10.164:1199) received membershipChanged event:
08:17:41,352 INFO [DefaultPartition] Dead members: 0 ([])
08:17:41,352 INFO [DefaultPartition] New Members : 1 ([10.9.10.164:1299])
08:17:41,352 INFO [DefaultPartition] All Members : 2 ([10.9.10.164:1199, 10.9.10.164:1299])
08:17:41,431 INFO [RPCManagerImpl] Received new cluster view: [10.9.10.164:39244|1] [10.9.10.164:39244, 10.9.10.164:37412]
08:17:45,907 INFO [GroupMember] org.jboss.messaging.core.impl.postoffice.GroupMember$ControlMembershipListener@16acffa got new view [10.9.10.164:39244|1] [10.9.10.164:39244, 10.9.10.164:37412], old view is [10.9.10.164:39244|0] [10.9.10.164:39244]
08:17:45,908 INFO [GroupMember] I am (10.9.10.164:39244)
08:17:45,908 INFO [GroupMember] New Members : 1 ([10.9.10.164:37412])
08:17:45,908 INFO [GroupMember] All Members : 2 ([10.9.10.164:39244, 10.9.10.164:37412])

*************************************************************************************************************************

And that is correct too that am running a Java client outside the application server instances that makes a call to a Stateless session EJB deployed in RoundRobin fashion on the Jboss cluster.

I see that the lookup for the EJB is not load balanced,the call always goes to the first available instance in the comma separated list of server instances that I provided in the Provider_url while creating the InitialContext....

     env.setProperty("java.naming.factory.initial","org.jnp.interfaces.NamingContextFactory");
     env.setProperty("java.naming.provider.url","10.9.10.164:1199,10.9.10.164:1299");
     env.setProperty("java.naming.factory.url.pkgs","org.jboss.naming:org.jnp.interfaces");
     ic = new InitialContext(env);
Please let me know if you see any issues with this approach...

thnx!
Actions
3. Re: Problem in Load Balancing in a Jboss 5.1 Cluster

brian.stansberry Mar 16, 2010 10:20 AM (in response to rajsoni)

env.setProperty("java.naming.provider.url","10.9.10.164:1199,10.9.10.164:1299");

Those don't look like HA-JNDI ports; if you want load balanced JNDI operations you need to use HA-JNDI (e.g. env.setProperty("java.naming.provider.url","10.9.10.164:1200,10.9.10.164:1300");

But, not using HA-JNDI shouldn't cause your actual EJB invocations to not be load balanced, so perhaps there is something more going on here.
Actions
4. Re: Problem in Load Balancing in a Jboss 5.1 Cluster

rajsoni Mar 16, 2010 1:21 PM (in response to brian.stansberry)

Brian,

I also have an MDB deployed in the Jboss cluster.This MDB invokes the same stateless session bean and I see that the calls from the MDB are load balanced fine on both the instances.

The issue is when calling from my Java client where I see all the calls going to one instance only.

Just providing some more information underneath.Please let me know if you see something wrong :

Jboss.xml for Session bean in cluster:

<?xml version="1.0"?>
<jboss xmlns:xs="http://www.jboss.org/j2ee/schema"
       xs:schemaLocation="http://www.jboss.org/j2ee/schema jboss_5_0.xsd"
       version="5.0">
    <enterprise-beans>
          <session>
               <ejb-name>MktSearch</ejb-name>
               <jndi-name>MktSearchEJB</jndi-name>
               <local-jndi-name>LocalMktSearchEJB</local-jndi-name>
               
              
                  
                  
                      
                  
                  
                      
                  
              
            <configuration-name>Clustered Stateless SessionBean</configuration-name>
            </session>
      </enterprise-beans>
    <container-configurations>
     <container-configuration>
      <container-name>Clustered Stateless SessionBean</container-name>
      <call-logging>false</call-logging>
      <invoker-proxy-binding-name>clustered-stateless-unified-invoker</invoker-proxy-binding-name>
      <container-interceptors>
        <interceptor>org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor</interceptor>
        <interceptor>org.jboss.ejb.plugins.CleanShutdownInterceptor</interceptor>
        <interceptor>org.jboss.ejb.plugins.LogInterceptor</interceptor>
        <interceptor>org.jboss.ejb.plugins.security.PreSecurityInterceptor</interceptor>
        <interceptor>org.jboss.ejb.plugins.SecurityInterceptor</interceptor>
        
        <interceptor transaction="Container">org.jboss.ejb.plugins.TxInterceptorCMT</interceptor>
        <interceptor transaction="Container">org.jboss.ejb.plugins.CallValidationInterceptor</interceptor>
        <interceptor transaction="Container">org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor</interceptor>
        
        <interceptor transaction="Bean">org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor</interceptor>
        <interceptor transaction="Bean">org.jboss.ejb.plugins.TxInterceptorBMT</interceptor>
        <interceptor transaction="Bean">org.jboss.ejb.plugins.CallValidationInterceptor</interceptor>
        <interceptor>org.jboss.resource.connectionmanager.CachedConnectionInterceptor</interceptor>
      </container-interceptors>
      <instance-pool>org.jboss.ejb.plugins.StatelessSessionInstancePool</instance-pool>
      <instance-cache></instance-cache>
      <persistence-manager></persistence-manager>
      <container-pool-conf>
        <MaximumSize>100</MaximumSize>
      </container-pool-conf>
      <cluster-config>
         <partition-name>${jboss.partition.name:DefaultPartition}</partition-name>
         <home-load-balance-policy>org.jboss.ha.framework.interfaces.RoundRobin</home-load-balance-policy>
         <bean-load-balance-policy>org.jboss.ha.framework.interfaces.RoundRobin</bean-load-balance-policy>
      </cluster-config>
     </container-configuration>
    </container-configurations>

    
        
          
          
          
          
            
            
            
            
            
            
            
            
            
            
            
            
            
          
          
          
          
          
            
          
        
         
     
</jboss>

In the client call I have tried different options.One that I think should work is keeping the home interface for the looked up Stateless bean cached and calling home.create() to get the remote interface each time.
I don't see it working though....

Am running the code underneath in loop after getting the home interface:

   mktSearchHome = CoreLookup.getMktSearchHome();//gets the home interface if it is not null,a check above this

for(int i =0;i<15000;i++){
   mktSearch = mktSearchHome.create();//creates a new remote reference for each call
   data = mktSearch.search(filter);//Call methd on bean,this always seems to go to the first instance
   logger.info(" Number of search results : " + data.size());
   }

Please let me know if you see anything am doing incorrect...
thanks for your help!
Actions
5. Re: Problem in Load Balancing in a Jboss 5.1 Cluster

rajsoni Mar 16, 2010 2:36 PM (in response to rajsoni)

I also tried using the HAJNDI ports in Client with same result.....

env.setProperty("java.naming.provider.url","10.9.10.164:1200,10.9.10.164:1300");

In the JMX console I see the HAJNDI ports for my instances as 1200 and 1300 but in this case also all the calls for the Stateless Session EJB are going to the first node only.
Actions
6. Re: Problem in Load Balancing in a Jboss 5.1 Cluster

rajsoni Mar 16, 2010 5:18 PM (in response to rajsoni)

I've made some progress on this....I have been able to set up load balancing between separate client calls.....though if I run the same client and execute the method on clustered session bean (via its remote interface) in a for loop,all calls for that go to same instance....
Actions
7. Re: Problem in Load Balancing in a Jboss 5.1 Cluster

rajsoni Mar 17, 2010 9:08 AM (in response to rajsoni)

resolved this completely....Here's the finding if anyone else is stuck similarly....
Somewhere in the code in ServiceLocator that my client was calling,the Ejb home was getting cached....If the Home is cached,then load balancing was working for my different client calls(since look up was happing afresh then) with all calls from first client going to one instance and for the other to the other instance.....failover wasn't working fine in this case for the client calls(since home was cached and when the instance went down,the information in home for where to connect to wasnt updated to the remaining live node)....

When I removed the caching for Home in the ServiceLocator,then each client call does a home look up and creates a remote(usually this is an expensive operation) but once I made this change I see load balancing and failover working fine for repeated calls from the same client....

thanks!
Actions
8. Re: Problem in Load Balancing in a Jboss 5.1 Cluster

nstronics Jul 12, 2011 5:17 AM (in response to rajsoni)

Hi Raj

We are facing the same problem u face.still we ve not got the solution.. we are using Annotations to create clustered session beans.
Please instruct me how to configure servicelocator cache..
Thanks,
Santhosh
Actions

Go to original post