0 Replies Latest reply on Jun 30, 2014 1:18 AM by Arcadiy Ivanov

    WildFly 8.1 EJB Stateless load-balancing does not work

    Arcadiy Ivanov Newbie

      I hope you guys can help me to resolve this. I've been having this absolutely surreal experience trying to get EJB LB to work.


      I have a domain with 3 cluster nodes: node1, node2 and node3. All located on different ports on the same host. I have several SLSB deployed on all of the nodes. The REST services inject and call the Remote interfaces of the SLSB hoping to have the calls load-balanced. This appears to be completely impossible.

       

      I can see cluster initializing and everything is peachy on that front. However, I cannot wrap my mind around the following problem.

       

      1) ClusterNodeSelector not being called - I'm experiencing the exact same problem - load-balancer is instantiated but is not being called (exception is only logged for stack trace purposes, not thrown; code is a copy of RandomClusterNodeSelector from EJB Client):

      2014-06-29 20:35:04,505 INFO  [com.csc.sm.jbosspoc.util.RandomClusterNodeSelector] (MSC service thread 1-12) constructed class com.csc.sm.jbosspoc.util.RandomClusterNodeSelector: java.lang.Exception

              at com.csc.sm.jbosspoc.util.RandomClusterNodeSelector.<init>(RandomClusterNodeSelector.java:23)

              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [rt.jar:1.7.0_60]

              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) [rt.jar:1.7.0_60]

              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [rt.jar:1.7.0_60]

              at java.lang.reflect.Constructor.newInstance(Constructor.java:526) [rt.jar:1.7.0_60]

              at java.lang.Class.newInstance(Class.java:374) [rt.jar:1.7.0_60]

              at org.jboss.as.ejb3.remote.EJBClientClusterConfig.<init>(EJBClientClusterConfig.java:77)

              at org.jboss.as.ejb3.deployment.processors.EJBClientDescriptorMetaDataProcessor.createClientConfiguration(EJBClientDescriptorMetaDataProcessor.java:158)

              at org.jboss.as.ejb3.deployment.processors.EJBClientDescriptorMetaDataProcessor.deploy(EJBClientDescriptorMetaDataProcessor.java:89)

              at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:159) [wildfly-server-8.1.0.Final.jar:8.1.0.Final]

              at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]

              at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_60]

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_60]

              at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_60]

       

      2) Cluster seems to be fully operational:

      2014-06-29 20:35:07,998 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 30) ISPN000094: Received new cluster view: [master:node3/ejb|1] (3) [master:node3/ejb, master:node2/ejb, master:node1/ejb]

      2014-06-29 20:35:07,999 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 30) ISPN000079: Cache local address is master:node1/ejb, physical addresses are [127.0.0.1:55300]

      2014-06-29 20:35:08,001 INFO  [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 30) ISPN000128: Infinispan version: Infinispan 'Infinium' 6.0.2.Final

      2014-06-29 20:35:08,137 INFO  [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 30) ISPN000031: MBeans were successfully registered to the platform MBean server.

      2014-06-29 20:35:08,235 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 30) JBAS010281: Started dist cache from ejb container

      2014-06-29 20:35:08,291 INFO  [org.jboss.ejb.client.remoting] (default task-27) EJBCLIENT000017: Received server version 2 and marshalling strategies [river]

      2014-06-29 20:35:08,294 INFO  [org.jboss.ejb.client.remoting] (ejb-client-cluster-node-connection-creation-4-thread-1) EJBCLIENT000013: Successful version handshake completed for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@4b935832, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection@29745eac,channel=jboss.ejb,nodename=master:node3]} on channel Channel ID f42d3552 (outbound) of Remoting connection 1887d2f4 to /127.0.0.1:8380


      3) I understand that this topic does not apply @Clustered applied to @Stateless: needed or not? since both the annotation and the jboss-ejb3.xml were deprecated in WildFly. I applied them anyway. They've been recognized and explicitly ignored, yet clustering doesn't work.

      2014-06-29 20:35:03,993 WARN  [org.jboss.as.ejb3] (MSC service thread 1-2) JBAS014266: The @Clustered annotation is deprecated and will be ignored.

      2014-06-29 20:35:04,016 WARN  [org.jboss.as.ejb3] (MSC service thread 1-2) JBAS014267: The <clustering xmlns="urn:clustering:1.0"/> element will be ignored.


      4) In the WAR I have the following which renders weird results:

      4.1) Lookup in @EJB on line 4 always fails with nothing logged. I checked for syntactical errors in lookup.

      4.2) Line 30 is irrelevant, lookup on line 33 results in a valid proxy regardless, having exactly the same lookup string as used in @EJB.

      4.3) Line 15 always invokes the remote of the host on which the web application is executed. Not a single time has the container load-balanced the call by invoking RandomClusterNodeSelector.

       

      Path("/cluster")
      public class ClusterService
      {
          @EJB(lookup = "ejb:jee-ear/jee-ejb//OperationsManagerBean!com.csc.sm.jbosspoc.service.RemoteOperationsManager")
          private RemoteOperationsManager om;
      
      
          @GET
          @Path("/topology")
          @Produces(MediaType.APPLICATION_JSON)
          public Collection<String> getTopology()
          {
              LinkedHashSet<String> nodes = new LinkedHashSet<>();
              for (;;) {
                  String s = om.getNode();
                  if (nodes.contains(s)) {
                      return nodes;
                  }
                  nodes.add(s);
              }
          }
      
      
         @PostConstruct
          public void invokeOnBean()
          {
              try {
                  final Hashtable props = new Hashtable();
                  // setup the ejb: namespace URL factory
                  props.put(Context.URL_PKG_PREFIXES, "org.jboss.ejb.client.naming");
                  // create the InitialContext
                  final Context context = new InitialContext(props);
                  om = (RemoteOperationsManager) context.lookup("ejb:" + "jee-ear/jee-ejb//"
                          + OperationsManagerBean.class.getSimpleName() + "!" + RemoteOperationsManager.class.getName());
              }
              catch (Exception e) {
                  throw new RuntimeException(e);
              }
          }
      }
      
      
      
      
      
      
      
      
      
      

       

      UPDATE:

      More info. I have an EAR containing EJB and WAR modules. WAR injects beans from EJB and invokes them. OperationsManagerBean has RemoteOperationsManager interface (annotated @Remote).

       

      I just found out that if the OperationsManagerBean inject its own interface as follows, I will get a proxy inside om "Proxy for remote EJB StatelessEJBLocator{appName='jee-ear', moduleName='jee-ejb', distinctName='', beanName='OperationsManagerBean', view='interface com.csc.sm.jbosspoc.service.RemoteOperationsManager'}":


          @EJB(lookup = "ejb:jee-ear/jee-ejb//OperationsManagerBean!com.csc.sm.jbosspoc.service.RemoteOperationsManager")

          private RemoteOperationsManager om;

       

      Same proxy is returned if looked up via invokeOnBean above. The above @EJB injection, does NOT work from inside a RESTful service though (invokeOnBean does return a proxy though). I assume it's a bug but please confirm.

       

      No matter where, the proxy is for an org.jboss.ejb.client.EJBInvocationHandler, which contains locator "StatelessEJBLocator{appName='jee-ear', moduleName='jee-ejb', distinctName='', beanName='OperationsManagerBean', view='interface com.csc.sm.jbosspoc.service.RemoteOperationsManager'}" and weakAffinity "org.jboss.ejb.client.Affinity$NoAffinity".

       

      UPDATE2:

      I have debugged the container further and confirmed that StatelessSessionComponent.weakAffinity (line 84) is set correctly to ClusterAffinity for all my EJBs on deployment.

       

      UPDATE3:

      This is my first time ever looking at JBoss codebase but I believe that Stateless clustering is simply turned off and cannot work in WF8.1!

       

      StatelessEJBLocator is ALWAYS constructed with Affinity.NONE.

       

          /**
           * Construct a new instance.
           *
           * @param viewType     the view type
           * @param appName      the application name
           * @param moduleName   the module name
           * @param beanName     the bean name
           * @param distinctName the distinct name
           */
          public StatelessEJBLocator(final Class<T> viewType, final String appName, final String moduleName, final String beanName, final String distinctName) {
              super(viewType, appName, moduleName, beanName, distinctName, Affinity.NONE);
          }
      
      
      

      Since affinity is always NONE, the org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(EJBClientInvocationContext) line 62 (line 1 below) will always be NONE. Therefore we branch to line org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(EJBClientInvocationContext) line 90 (line 29 below). EJBInvocationHandler.weakAffinity is NEVER SET to an instance of ClusterAffinity - there is simply no execution path that would facilitate it. Am I missing something?

       

                  final Affinity affinity = locator.getAffinity();
                  if (affinity instanceof NodeAffinity) {
                      final String nodeName = ((NodeAffinity) affinity).getNodeName();
                      if (excludedNodes.contains(nodeName)) {
                          throw Logs.MAIN.requiredNodeExcludedFromInvocation(locator, nodeName, invocationContext);
                      }
                      receiverContext = clientContext.requireNodeEJBReceiverContext(nodeName);
                  } else if (affinity instanceof ClusterAffinity) {
                      final Affinity weakAffinity = invocationContext.getInvocationHandler().getWeakAffinity();
                      if (weakAffinity instanceof NodeAffinity) {
                          final String nodeName = ((NodeAffinity) weakAffinity).getNodeName();
                          final EJBReceiver nodeReceiver;
                          // ignore the weak affinity if the node has been marked as excluded for this invocation context
                          if (excludedNodes.contains(nodeName)) {
                              logger.debug("Ignoring weak affinity on node " + nodeName + " since that node has been marked as excluded for invocation context " + invocationContext);
                              nodeReceiver = null;
                          } else {
                              nodeReceiver = clientContext.getNodeEJBReceiver(nodeName);
                          }
                          if (nodeReceiver != null && clientContext.clusterContains(((ClusterAffinity) affinity).getClusterName(), nodeReceiver.getNodeName())) {
                              receiverContext = clientContext.requireEJBReceiverContext(nodeReceiver);
                          } else {
                              receiverContext = clientContext.requireClusterEJBReceiverContext(invocationContext, ((ClusterAffinity) affinity).getClusterName());
                          }
                      } else {
                          receiverContext = clientContext.requireClusterEJBReceiverContext(invocationContext, ((ClusterAffinity) affinity).getClusterName());
                      }
                  } else if (affinity == Affinity.NONE) {
                      final Affinity weakAffinity = invocationContext.getInvocationHandler().getWeakAffinity();
                      if (weakAffinity instanceof NodeAffinity) {
                          final String nodeName = ((NodeAffinity) weakAffinity).getNodeName();
                          final EJBReceiver nodeReceiver;
                          // ignore the weak affinity if the node has been marked as excluded for this invocation context
                          if (excludedNodes.contains(nodeName)) {
                              logger.debug("Ignoring weak affinity on node " + nodeName + " since that node has been marked as excluded for invocation context " + invocationContext);
                              nodeReceiver = null;
                          } else {
                              nodeReceiver = clientContext.getNodeEJBReceiver(nodeName);
                          }
                          if (nodeReceiver != null) {
                              receiverContext = clientContext.requireEJBReceiverContext(nodeReceiver);
                          } else {
                              final EJBReceiver receiver = clientContext.requireEJBReceiver(invocationContext, locator.getAppName(), locator.getModuleName(), locator.getDistinctName());
                              receiverContext = clientContext.requireEJBReceiverContext(receiver);
                          }
                      } else if (weakAffinity instanceof ClusterAffinity) {
                          final EJBReceiverContext clusterReceiverContext = clientContext.getClusterEJBReceiverContext(invocationContext, ((ClusterAffinity) weakAffinity).getClusterName());
                          if (clusterReceiverContext != null) {
                              receiverContext = clusterReceiverContext;
                          } else {
                              final EJBReceiver receiver = clientContext.requireEJBReceiver(invocationContext, locator.getAppName(), locator.getModuleName(), locator.getDistinctName());
                              receiverContext = clientContext.requireEJBReceiverContext(receiver);
                          }
                      } else {
                          final EJBReceiver receiver = clientContext.requireEJBReceiver(invocationContext, locator.getAppName(), locator.getModuleName(), locator.getDistinctName());
                          receiverContext = clientContext.requireEJBReceiverContext(receiver);
                      }
      
      
      


      UPDATE4:

      The only possiblity for EJBInvocationHandler.weakAffinity to be set is via EJBInvocationHandler.setWeakAffinity, which in turn is only invoked outside of EJBInvocationHandler itself via EJBClientInvocationContext.getResult() line 270 (line 4 below). In my call path this will never happen:

       

                  resultDone = true;
                  final Affinity weakAffinity = getAttachment(AttachmentKeys.WEAK_AFFINITY);
                  if (weakAffinity != null) {
                      invocationHandler.setWeakAffinity(weakAffinity);
                  }
      
      


      This is MethodInvocationMessageHandler:

       

                             // attach any weak affinity if available
                              Affinity weakAffinity = null;
                              if (locator instanceof StatefulEJBLocator && componentView.getComponent() instanceof StatefulSessionComponent) {
                                  final StatefulSessionComponent statefulSessionComponent = (StatefulSessionComponent) componentView.getComponent();
                                  weakAffinity = MethodInvocationMessageHandler.this.getWeakAffinity(statefulSessionComponent, (StatefulEJBLocator<?>) locator);
                              } else if (componentView.getComponent() instanceof StatelessSessionComponent) {
                                  final StatelessSessionComponent statelessSessionComponent = (StatelessSessionComponent) componentView.getComponent();
                                  weakAffinity = statelessSessionComponent.getWeakAffinity();
                              }
                              if (weakAffinity != null) {
                                  attachments.put(Affinity.WEAK_AFFINITY_CONTEXT_KEY, weakAffinity);
                              }
      

      This is MethodInvocationResultProducer:

       

                      // see if there's a weak affinity passed as an attachment. If yes, then attach it to the client invocation
                      // context
                      if (this.clientInvocationContext != null && attachments != null && attachments.containsKey(Affinity.WEAK_AFFINITY_CONTEXT_KEY)) {
                          final Affinity weakAffinity = (Affinity) attachments.get(Affinity.WEAK_AFFINITY_CONTEXT_KEY);
                          this.clientInvocationContext.putAttachment(AttachmentKeys.WEAK_AFFINITY, weakAffinity);
                      }
      

       

      The AttachmentKeys.WEAK_AFFINITY are only manipulated by MethodInvocationMessageHandler and my execution path goes through LocalEjbReceiver and never touches remote invocation as a result! There are no more execution paths that may set weakAffinity in EJBInvocationHandler and allow a SLSB which is deployed across all servers in the cluster to actually be load-balanced via internal invocation.