FirstAvailable LBP is sticky when not serialized

[fuzzy333]

JBoss 4.2.3.GA

I'm having some trouble getting this scenario working:

Web service client
Apache load balancer
Two JBoss nodes in a cluster, one primary, one fail over
One web service and one "worker" stateful session bean hosted in both JBoss nodes

The client invokes the web service, the call goes through the load balancer, the primary is selected unless it has failed. So far so good. The web service extracts a session id from the web service parameters, this is used to key the worker session bean reference. I need instances of this worker bean to get distributed between the two nodes to spread out the load. I configured the bean to use the FirstAvailable load balance policy, which is exactly what I need, but I find that the calls are not distributed between the nodes! The node is initially randomly selected, but once one is chosen it is used for all instances.

So I did a little digging, and in the FirstAvailable load balance policy I find that after initially selecting a target node, it is cached in a transient variable:

protected transient Object electedTarget = null;

However since one instance of this load balance policy is created (at startup from what I can tell, I'm not sure if it's shared between different bean type either, but I suspect not), this means that local (same vm) clients like my web service end up choosing the target once and sticking with it for all instances: it expects to get serialized to a remote client.

I could serialize the reference when acquiring it to reproduce the behavior of sending it to a remote client, but that seems patchy. Is there some other way of getting this working that I'm missing? Should I open a JIRA for this?

Thanks in advance!
Jonathan Tougas

[brian.stansberry]

I take it this is an EJB2 bean? EJB2 directly stores the client proxy in JNDI, so yeah I can see how all in-VM clients would end up with a shared reference to a single LoadBalancePolicy instance. EJB3 should work somewhat differently, as what's stored in JNDI is a javax.naming.Reference which results in an ObjectFactory instantiating a new LoadBalancePolicy instance each time the client does the JNDI lookup.

TBH, w/ EJB2 I think you are better off w/ your serialization/deserialization hack, as it's extremely unlikely the EJB2 impl will be changed. It would take a pretty significant change to fix this (basically changing to the EJB3 approach) and EJB2 is in maintenance mode.

Another problem you'll face is the EJB proxy includes logic to detect when the bean container is present locally, in VM. If it is, the call is routed to the local bean; it won't be load balanced. See "Why are calls between clustered session beans not load balanced even though load balancing policy is Round Robin?" on http://www.jboss.org/community/docs/DOC-9306.

[fuzzy333]

The test bean I'm using to setup this scenario is an EJB3 bean. Here it is for reference:

package com.foo.session;

import javax.ejb.Remote;
import javax.ejb.Stateful;

import org.apache.log4j.Logger;
import org.jboss.annotation.ejb.Clustered;
import org.jboss.ha.framework.interfaces.FirstAvailable;

@Stateful
@Clustered(loadBalancePolicy=FirstAvailable.class)
@Remote( value= Test2.class )
public class Test2Bean implements Test2 {

 private final static Logger log = Logger.getLogger( Test2Bean.class );

 private int count;

 public void setCount( int count ) {
 log.info( "setCount( " + count + " )" );
 this.count = count;
 }

 public void printCount() {
 log.info( "count=" + count );
 }
}

The load balance policy is only ever instantiated once. I copied the code from the FirstAvailable LBP to add some traces a few days ago to get a better idea of what's going on, and the constructor only ever got called once, during the app's initialization.

11:17:58,312 INFO [JmxKernelAbstraction] installing MBean: jboss.j2ee:ear=testEAR.ear,jar=testEJB.jar,name=Test2Bean,service=EJB3 with dependencies:
11:17:58,375 INFO [EJBContainer] STARTED EJB: com.foo.session.Test2Bean ejbName: Test2Bean
11:17:58,406 INFO [MyLBPolicy] LOAD BALANCE POLICY CONSTRUCTOR CALLED

I had already removed the IsLocal interceptor after having pulled most of my hair out ;) Thanks for mentioning it though.

[brian.stansberry]

Ah, sorry, you said you're using 4.2.3. The binding of EJB3 proxies in 4.2.3 works the same as EJB2. The change to using a javax.naming.Reference and an ObjectFactory is in 5.x. Can you try 5.1?

[fuzzy333]

I ran the same app on 5.1.0.GA after commenting out all lines in ejb3-interceptors-aop.xml that had IsLocal in them, and I could not get calls to the bean to get load balanced. I guess there's something else going on, but I don't have time to investigate further on this version.

I'm going to stick with 4.2.3.GA (I don't have a choice) cloning the value returned from the jndi lookup. I will surely revisit this in the future though.