WildFly 8.1 EJB Stateless load-balancing does not work
I hope you guys can help me to resolve this. I've been having this absolutely surreal experience trying to get EJB LB to work.
I have a domain with 3 cluster nodes: node1, node2 and node3. All located on different ports on the same host. I have several SLSB deployed on all of the nodes. The REST services inject and call the Remote interfaces of the SLSB hoping to have the calls load-balanced. This appears to be completely impossible.
I can see cluster initializing and everything is peachy on that front. However, I cannot wrap my mind around the following problem.
1) ClusterNodeSelector not being called - I'm experiencing the exact same problem - load-balancer is instantiated but is not being called (exception is only logged for stack trace purposes, not thrown; code is a copy of RandomClusterNodeSelector from EJB Client):
2014-06-29 20:35:04,505 INFO [com.csc.sm.jbosspoc.util.RandomClusterNodeSelector] (MSC service thread 1-12) constructed class com.csc.sm.jbosspoc.util.RandomClusterNodeSelector: java.lang.Exception
at com.csc.sm.jbosspoc.util.RandomClusterNodeSelector.<init>(RandomClusterNodeSelector.java:23)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [rt.jar:1.7.0_60]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) [rt.jar:1.7.0_60]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [rt.jar:1.7.0_60]
at java.lang.reflect.Constructor.newInstance(Constructor.java:526) [rt.jar:1.7.0_60]
at java.lang.Class.newInstance(Class.java:374) [rt.jar:1.7.0_60]
at org.jboss.as.ejb3.remote.EJBClientClusterConfig.<init>(EJBClientClusterConfig.java:77)
at org.jboss.as.ejb3.deployment.processors.EJBClientDescriptorMetaDataProcessor.createClientConfiguration(EJBClientDescriptorMetaDataProcessor.java:158)
at org.jboss.as.ejb3.deployment.processors.EJBClientDescriptorMetaDataProcessor.deploy(EJBClientDescriptorMetaDataProcessor.java:89)
at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:159) [wildfly-server-8.1.0.Final.jar:8.1.0.Final]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_60]
2) Cluster seems to be fully operational:
2014-06-29 20:35:07,998 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 30) ISPN000094: Received new cluster view: [master:node3/ejb|1] (3) [master:node3/ejb, master:node2/ejb, master:node1/ejb]
2014-06-29 20:35:07,999 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 30) ISPN000079: Cache local address is master:node1/ejb, physical addresses are [127.0.0.1:55300]
2014-06-29 20:35:08,001 INFO [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 30) ISPN000128: Infinispan version: Infinispan 'Infinium' 6.0.2.Final
2014-06-29 20:35:08,137 INFO [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 30) ISPN000031: MBeans were successfully registered to the platform MBean server.
2014-06-29 20:35:08,235 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 30) JBAS010281: Started dist cache from ejb container
2014-06-29 20:35:08,291 INFO [org.jboss.ejb.client.remoting] (default task-27) EJBCLIENT000017: Received server version 2 and marshalling strategies [river]
2014-06-29 20:35:08,294 INFO [org.jboss.ejb.client.remoting] (ejb-client-cluster-node-connection-creation-4-thread-1) EJBCLIENT000013: Successful version handshake completed for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@4b935832, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection@29745eac,channel=jboss.ejb,nodename=master:node3]} on channel Channel ID f42d3552 (outbound) of Remoting connection 1887d2f4 to /127.0.0.1:8380
3) I understand that this topic does not apply @Clustered applied to @Stateless: needed or not? since both the annotation and the jboss-ejb3.xml were deprecated in WildFly. I applied them anyway. They've been recognized and explicitly ignored, yet clustering doesn't work.
2014-06-29 20:35:03,993 WARN [org.jboss.as.ejb3] (MSC service thread 1-2) JBAS014266: The @Clustered annotation is deprecated and will be ignored.
2014-06-29 20:35:04,016 WARN [org.jboss.as.ejb3] (MSC service thread 1-2) JBAS014267: The <clustering xmlns="urn:clustering:1.0"/> element will be ignored.
4) In the WAR I have the following which renders weird results:
4.1) Lookup in @EJB on line 4 always fails with nothing logged. I checked for syntactical errors in lookup.
4.2) Line 30 is irrelevant, lookup on line 33 results in a valid proxy regardless, having exactly the same lookup string as used in @EJB.
4.3) Line 15 always invokes the remote of the host on which the web application is executed. Not a single time has the container load-balanced the call by invoking RandomClusterNodeSelector.
Path("/cluster")
public class ClusterService
{
@EJB(lookup = "ejb:jee-ear/jee-ejb//OperationsManagerBean!com.csc.sm.jbosspoc.service.RemoteOperationsManager")
private RemoteOperationsManager om;
@GET
@Path("/topology")
@Produces(MediaType.APPLICATION_JSON)
public Collection<String> getTopology()
{
LinkedHashSet<String> nodes = new LinkedHashSet<>();
for (;;) {
String s = om.getNode();
if (nodes.contains(s)) {
return nodes;
}
nodes.add(s);
}
}
@PostConstruct
public void invokeOnBean()
{
try {
final Hashtable props = new Hashtable();
// setup the ejb: namespace URL factory
props.put(Context.URL_PKG_PREFIXES, "org.jboss.ejb.client.naming");
// create the InitialContext
final Context context = new InitialContext(props);
om = (RemoteOperationsManager) context.lookup("ejb:" + "jee-ear/jee-ejb//"
+ OperationsManagerBean.class.getSimpleName() + "!" + RemoteOperationsManager.class.getName());
}
catch (Exception e) {
throw new RuntimeException(e);
}
}
}
UPDATE:
More info. I have an EAR containing EJB and WAR modules. WAR injects beans from EJB and invokes them. OperationsManagerBean has RemoteOperationsManager interface (annotated @Remote).
I just found out that if the OperationsManagerBean inject its own interface as follows, I will get a proxy inside om "Proxy for remote EJB StatelessEJBLocator{appName='jee-ear', moduleName='jee-ejb', distinctName='', beanName='OperationsManagerBean', view='interface com.csc.sm.jbosspoc.service.RemoteOperationsManager'}":
@EJB(lookup = "ejb:jee-ear/jee-ejb//OperationsManagerBean!com.csc.sm.jbosspoc.service.RemoteOperationsManager")
private RemoteOperationsManager om;
Same proxy is returned if looked up via invokeOnBean above. The above @EJB injection, does NOT work from inside a RESTful service though (invokeOnBean does return a proxy though). I assume it's a bug but please confirm.
No matter where, the proxy is for an org.jboss.ejb.client.EJBInvocationHandler, which contains locator "StatelessEJBLocator{appName='jee-ear', moduleName='jee-ejb', distinctName='', beanName='OperationsManagerBean', view='interface com.csc.sm.jbosspoc.service.RemoteOperationsManager'}" and weakAffinity "org.jboss.ejb.client.Affinity$NoAffinity".
UPDATE2:
I have debugged the container further and confirmed that StatelessSessionComponent.weakAffinity (line 84) is set correctly to ClusterAffinity for all my EJBs on deployment.
UPDATE3:
This is my first time ever looking at JBoss codebase but I believe that Stateless clustering is simply turned off and cannot work in WF8.1!
StatelessEJBLocator is ALWAYS constructed with Affinity.NONE.
/**
* Construct a new instance.
*
* @param viewType the view type
* @param appName the application name
* @param moduleName the module name
* @param beanName the bean name
* @param distinctName the distinct name
*/
public StatelessEJBLocator(final Class<T> viewType, final String appName, final String moduleName, final String beanName, final String distinctName) {
super(viewType, appName, moduleName, beanName, distinctName, Affinity.NONE);
}
Since affinity is always NONE, the org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(EJBClientInvocationContext) line 62 (line 1 below) will always be NONE. Therefore we branch to line org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(EJBClientInvocationContext) line 90 (line 29 below). EJBInvocationHandler.weakAffinity is NEVER SET to an instance of ClusterAffinity - there is simply no execution path that would facilitate it. Am I missing something?
final Affinity affinity = locator.getAffinity();
if (affinity instanceof NodeAffinity) {
final String nodeName = ((NodeAffinity) affinity).getNodeName();
if (excludedNodes.contains(nodeName)) {
throw Logs.MAIN.requiredNodeExcludedFromInvocation(locator, nodeName, invocationContext);
}
receiverContext = clientContext.requireNodeEJBReceiverContext(nodeName);
} else if (affinity instanceof ClusterAffinity) {
final Affinity weakAffinity = invocationContext.getInvocationHandler().getWeakAffinity();
if (weakAffinity instanceof NodeAffinity) {
final String nodeName = ((NodeAffinity) weakAffinity).getNodeName();
final EJBReceiver nodeReceiver;
// ignore the weak affinity if the node has been marked as excluded for this invocation context
if (excludedNodes.contains(nodeName)) {
logger.debug("Ignoring weak affinity on node " + nodeName + " since that node has been marked as excluded for invocation context " + invocationContext);
nodeReceiver = null;
} else {
nodeReceiver = clientContext.getNodeEJBReceiver(nodeName);
}
if (nodeReceiver != null && clientContext.clusterContains(((ClusterAffinity) affinity).getClusterName(), nodeReceiver.getNodeName())) {
receiverContext = clientContext.requireEJBReceiverContext(nodeReceiver);
} else {
receiverContext = clientContext.requireClusterEJBReceiverContext(invocationContext, ((ClusterAffinity) affinity).getClusterName());
}
} else {
receiverContext = clientContext.requireClusterEJBReceiverContext(invocationContext, ((ClusterAffinity) affinity).getClusterName());
}
} else if (affinity == Affinity.NONE) {
final Affinity weakAffinity = invocationContext.getInvocationHandler().getWeakAffinity();
if (weakAffinity instanceof NodeAffinity) {
final String nodeName = ((NodeAffinity) weakAffinity).getNodeName();
final EJBReceiver nodeReceiver;
// ignore the weak affinity if the node has been marked as excluded for this invocation context
if (excludedNodes.contains(nodeName)) {
logger.debug("Ignoring weak affinity on node " + nodeName + " since that node has been marked as excluded for invocation context " + invocationContext);
nodeReceiver = null;
} else {
nodeReceiver = clientContext.getNodeEJBReceiver(nodeName);
}
if (nodeReceiver != null) {
receiverContext = clientContext.requireEJBReceiverContext(nodeReceiver);
} else {
final EJBReceiver receiver = clientContext.requireEJBReceiver(invocationContext, locator.getAppName(), locator.getModuleName(), locator.getDistinctName());
receiverContext = clientContext.requireEJBReceiverContext(receiver);
}
} else if (weakAffinity instanceof ClusterAffinity) {
final EJBReceiverContext clusterReceiverContext = clientContext.getClusterEJBReceiverContext(invocationContext, ((ClusterAffinity) weakAffinity).getClusterName());
if (clusterReceiverContext != null) {
receiverContext = clusterReceiverContext;
} else {
final EJBReceiver receiver = clientContext.requireEJBReceiver(invocationContext, locator.getAppName(), locator.getModuleName(), locator.getDistinctName());
receiverContext = clientContext.requireEJBReceiverContext(receiver);
}
} else {
final EJBReceiver receiver = clientContext.requireEJBReceiver(invocationContext, locator.getAppName(), locator.getModuleName(), locator.getDistinctName());
receiverContext = clientContext.requireEJBReceiverContext(receiver);
}
UPDATE4:
The only possiblity for EJBInvocationHandler.weakAffinity to be set is via EJBInvocationHandler.setWeakAffinity, which in turn is only invoked outside of EJBInvocationHandler itself via EJBClientInvocationContext.getResult() line 270 (line 4 below). In my call path this will never happen:
resultDone = true;
final Affinity weakAffinity = getAttachment(AttachmentKeys.WEAK_AFFINITY);
if (weakAffinity != null) {
invocationHandler.setWeakAffinity(weakAffinity);
}
This is MethodInvocationMessageHandler:
// attach any weak affinity if available
Affinity weakAffinity = null;
if (locator instanceof StatefulEJBLocator && componentView.getComponent() instanceof StatefulSessionComponent) {
final StatefulSessionComponent statefulSessionComponent = (StatefulSessionComponent) componentView.getComponent();
weakAffinity = MethodInvocationMessageHandler.this.getWeakAffinity(statefulSessionComponent, (StatefulEJBLocator<?>) locator);
} else if (componentView.getComponent() instanceof StatelessSessionComponent) {
final StatelessSessionComponent statelessSessionComponent = (StatelessSessionComponent) componentView.getComponent();
weakAffinity = statelessSessionComponent.getWeakAffinity();
}
if (weakAffinity != null) {
attachments.put(Affinity.WEAK_AFFINITY_CONTEXT_KEY, weakAffinity);
}
This is MethodInvocationResultProducer:
// see if there's a weak affinity passed as an attachment. If yes, then attach it to the client invocation
// context
if (this.clientInvocationContext != null && attachments != null && attachments.containsKey(Affinity.WEAK_AFFINITY_CONTEXT_KEY)) {
final Affinity weakAffinity = (Affinity) attachments.get(Affinity.WEAK_AFFINITY_CONTEXT_KEY);
this.clientInvocationContext.putAttachment(AttachmentKeys.WEAK_AFFINITY, weakAffinity);
}
The AttachmentKeys.WEAK_AFFINITY are only manipulated by MethodInvocationMessageHandler and my execution path goes through LocalEjbReceiver and never touches remote invocation as a result! There are no more execution paths that may set weakAffinity in EJBInvocationHandler and allow a SLSB which is deployed across all servers in the cluster to actually be load-balanced via internal invocation.