EAP 6.1 EJB Remote Client using Clustering problems

[smccormick]

Alas, where to start?

I am trying to integrate with a new EAP 6.0.1 EJB Client jar that was provided to me. The client EJB jar/code was ported from WebLogic to JBoss EAP 6.0.1 (on the server) and needs to continue to support Tomcat and Weblogic, which it seems to be doing. We are currently using JBoss AS  5.1 as the remote client, but after running into major issues getting AS 5.1 to talk to EAP 6, we decided to port to EAP 6.1.x. Unfortunately even 6.1.x has issues. The first was the problem of modifying the context after it was created using the existing code in the client. I was finally able to overcome this using scoped context per https://access.redhat.com/site/solutions/231333. But there seems to be some problem with clustering and scoped contexts (or just clustering?) that I can't figure out. Specifically I get the following exception:

[RIPT_6EA8FE8D218940769ED05264B6500752][11:28:10.891] WARN(ConfigBasedEJBClientCont: 139) Could not register a EJB receiver for connection to [IP here]:4447: java.lang.RuntimeException: Operation failed with status WAITING

at org.jboss.ejb.client.remoting.IoFutureHelper.get(IoFutureHelper.java:93)

at org.jboss.ejb.client.remoting.ConnectionPool.getConnection(ConnectionPool.java:75)

at org.jboss.ejb.client.remoting.RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51)

at org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.setupEJBReceivers(ConfigBasedEJBClientContextSelector.java:130)

at org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.<init>(ConfigBasedEJBClientContextSelector.java:100)

at org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.<init>(ConfigBasedEJBClientContextSelector.java:68)

at org.jboss.ejb.client.naming.ejb.EjbNamingContext.createIdentifiableEjbClientContext(EjbNamingContext.java:256)

at org.jboss.ejb.client.naming.ejb.EjbNamingContext.setupScopedEjbClientContextIfNeeded(EjbNamingContext.java:123)

at org.jboss.ejb.client.naming.ejb.EjbNamingContext.<init>(EjbNamingContext.java:98)

at org.jboss.ejb.client.naming.ejb.ejbURLContextFactory.getObjectInstance(ejbURLContextFactory.java:38)

at org.jboss.as.naming.InitialContext.lookup(InitialContext.java:101)

at org.jboss.as.naming.NamingContext.lookup(NamingContext.java:183)

at org.jboss.as.naming.NamingContext.lookup(NamingContext.java:179)

at javax.naming.InitialContext.lookup(InitialContext.java:411)

at com.fares.common.util.jndi.ServiceLocatorCache.locateEJBRemoteProxy(ServiceLocatorCache.java:165)

at com.fares.common.util.jndi.ServiceLocatorCache.getEJBRemote(ServiceLocatorCache.java:104)

at com.fares.common.base.delegate.DelegateBase.getServiceHandleNew(DelegateBase.java:167)

at com.fares.common.base.delegate.DelegateBase.getServiceObjectHolder(DelegateBase.java:50)

at com.fares.smartsearch.ifc.delegate.SmartSearchBD.getPermissableMlsBoards(SmartSearchBD.java:324)

at com.corelogic.ript.coremls.vendor.services.CLiQService.getPermissibleUseMLSBoards(CLiQService.java:166)

at com.corelogic.ript.coremls.vendor.dto.CoreMLSPermissibleUseGeoCoverage.getPermissibleMLSBoards(CoreMLSPermissibleUseGeoCoverage.java:125)

at com.corelogic.ript.lmar.report.LMARReport.runReport(LMARReport.java:189)

at com.corelogic.ript.web.controllers.RiptServiceController.handleRequestInternal(RiptServiceController.java:189)

at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)

at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)

at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:900)

at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:827)

at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)

at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)

at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336)

at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)

at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653)

at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920)

at java.lang.Thread.run(Thread.java:722)

Having walked thru the client and Jboss code locally, this exception is preceded by a 5 second pause every time it happens, and for the application it happens about 6-8 times, which means the report takes an additional +30 seconds to execute, and sometime fails outright. Does anyone have any idea of why this is happening or what I might be doing wrong?

The code I am calling to get the context looks like:

private static Properties getEJBClientContextPropsEAP(String multipleHostIps,String remotingUserName,String remotingPassword)

{

  Properties env = new Properties();

  env.put("org.jboss.ejb.client.scoped.context", "true");

  env.put("javax.naming.Context.URL_PKG_PREFIXES", "org.jboss.ejb.client.naming");

  env.put("remote.clusters","ejb");

  env.put("reconnect.tasks.timeout","2000");

  env.put("remote.cluster.ejb.connect.timeout","2500");

  env.put("remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT", "false");

  env.put("jboss.naming.client.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT", "false"); 

  env.put("remote.cluster.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS","false");

  env.put("remote.cluster.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT","false");

  env.put("remote.cluster.ejb.connect.options.org.xnio.Options.SSL_ENABLED","false");

  env.put("remote.cluster.ejb.username",remotingUserName);

  env.put("remote.cluster.ejb.password",remotingPassword);

  env.put("deployment.node.selector","com.fares.common.util.jndi.RoundRobinDeploymentNodeSelector");

  appendClusterHostPropeties(multipleHostIps, env, remotingUserName, remotingPassword);

  return env;

}

...

     String[] urlValues = argProviderUrl.split(":");

     if(!SELECTOR_IS_SET.get())

     {

      Properties env = getEJBClientContextPropsEAP(argProviderUrl,argUser,argPass);

      ctx = new InitialContext(env);

     }

The strange thing is that some of the connections are working, and data is getting back but some seem to fail. The application that I can running is also multithreaded making several EJB calls at one time, though I have seen this error even when running them sequentially.

Thanks for you help in advance!

Stephen

[jaikiran]

It looks like the server listed isn't up or some firewall issue. What does:

telnet <IP here> 4447

give you from the machine where this code is failing?

[smccormick]

Verified the firewalls are open. Ran the code locally and on remote server with same results. I left the IP in the above to protect the innocent...sys eng hate when we give out IPs even if they are just locally subnet.

Single threaded applications seem to run fine. Running a multithreaded app sees these errors. I still get some successful results, only I get hung up on the future IO calls. I dug into the JBoss code (ConfigBasedEJBClientContextSelector.java) and it is here where the error occurs:

   private void setupEJBReceivers() throws IOException {
        if (!this.ejbClientConfiguration.getConnectionConfigurations().hasNext()) {
            // no connections configured so no EJB receivers to create
            return;
        }
        // create the endpoint
        final Endpoint endpoint = this.remotingEndpointManager.getEndpoint(this.ejbClientConfiguration.getEndpointName(), this.ejbClientConfiguration.getEndpointCreationOptions(), this.ejbClientConfiguration.getRemoteConnectionProviderCreationOptions());

        final Iterator<EJBClientConfiguration.RemotingConnectionConfiguration> connectionConfigurations = this.ejbClientConfiguration.getConnectionConfigurations();
        int successfulEJBReceiverRegistrations = 0;
        while (connectionConfigurations.hasNext()) {
            final EJBClientConfiguration.RemotingConnectionConfiguration connectionConfiguration = connectionConfigurations.next();
            final String host = connectionConfiguration.getHost();
            final int port = connectionConfiguration.getPort();
            final int MAX_RECONNECT_ATTEMPTS = 65535; // TODO: Let's keep this high for now and later allow configuration and a smaller default value
            // create a re-connect handler (which will be used on connection breaking down)
            final ReconnectHandler reconnectHandler = new EJBClientContextConnectionReconnectHandler(ejbClientContext, endpoint, host, port, connectionConfiguration, MAX_RECONNECT_ATTEMPTS);
            try {
                // wait for the connection to be established
                final Connection connection = this.remotingConnectionManager.getConnection(endpoint, host, port, connectionConfiguration);
                // create a remoting EJB receiver for this connection
                final EJBReceiver remotingEJBReceiver = new RemotingConnectionEJBReceiver(connection, reconnectHandler, connectionConfiguration.getChannelCreationOptions());
                // associate it with the client context
                this.ejbClientContext.registerEJBReceiver(remotingEJBReceiver);
                // keep track of successful registrations for logging purposes
                successfulEJBReceiverRegistrations++;
            } catch (Exception e) {
                // just log the warn but don't throw an exception. Move onto the next connection configuration (if any)
                logger.warn("Could not register a EJB receiver for connection to " + host + ":" + port, e);
                // add a reconnect handler for this connection
                if (reconnectHandler != null) {
                    this.ejbClientContext.registerReconnectHandler(reconnectHandler);
                    logger.debug("Registered a reconnect handler in EJB client context " + this.ejbClientContext + " for remote://" + host + ":" + port);
                }

            }
        }
        logger.debug("Registered " + successfulEJBReceiverRegistrations + " remoting EJB receivers for EJB client context " + this.ejbClientContext);
    }

May just be a symptom, but with out the ability to debug it (I can step thru it but not look at the variable contents since I didn't build it locally) it is difficult to tell

I am using a provided EJB client jar that I did not develop, which was ported by another team. I have verified that it works for Tomcat 7.0.0 without errors (but that part does not use a scoped context), but it fails for any version of EAP 6.1 using the scoped context. I cannot get their client code to work without using a scoped context. There is a lot of code and I hate to have to re-write it all. Do scoped contexts and clustering not like each other?

To try and use the same client across multiple containers we are looking at the MBean to determine the container and have different code for EAP. Here is the initialization portion, though there is a lot more code than this:

private static Properties getEJBClientContextPropsEAP(String multipleHostIps,String remotingUserName,String remotingPassword)

{

  Properties env = new Properties();

  env.put("org.jboss.ejb.client.scoped.context", "true");

  env.put("javax.naming.Context.URL_PKG_PREFIXES", "org.jboss.ejb.client.naming");

  env.put("remote.clusters","ejb");

  env.put("reconnect.tasks.timeout","2000");

  env.put("remote.cluster.ejb.connect.timeout","2500");

  env.put("remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT", "false");

  env.put("jboss.naming.client.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT", "false");

  env.put("remote.cluster.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS","false");

  env.put("remote.cluster.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT","false");

  env.put("remote.cluster.ejb.connect.options.org.xnio.Options.SSL_ENABLED","false");

  env.put("remote.cluster.ejb.username",remotingUserName);

  env.put("remote.cluster.ejb.password",remotingPassword);

  env.put("deployment.node.selector","com.fares.common.util.jndi.RoundRobinDeploymentNodeSelector");

  appendClusterHostPropeties(multipleHostIps, env, remotingUserName, remotingPassword);

  return env;

}

private static Properties getEJBClientContextProps(String multipleHostIps,String remotingUserName,String remotingPassword)

{

  Properties env = new Properties();

        env.put("jboss.naming.client.ejb.context", "true");

  env.put("remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED","false");

  env.put("remote.clusters","ejb");

  env.put("reconnect.tasks.timeout","2000");

  env.put("remote.cluster.ejb.connect.timeout","2500");

  env.put("remote.cluster.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS","false");

  env.put("remote.cluster.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT","false");

  env.put("remote.cluster.ejb.connect.options.org.xnio.Options.SSL_ENABLED","false");

  env.put("remote.cluster.ejb.username",remotingUserName);

  env.put("remote.cluster.ejb.password",remotingPassword);

  env.put("deployment.node.selector","com.fares.common.util.jndi.RoundRobinDeploymentNodeSelector");

  appendClusterHostPropeties(multipleHostIps, env, remotingUserName, remotingPassword);

  return env;

}

...

    /**
    *
    * @param argProviderUrl
    * @param argUser
    * @param argPass
    * @return
    * @throws NamingException
    */
    public static Context getInitialContext(String argProviderUrl, String argUser, String argPass) throws NamingException
    {
        Logger.logDebug(null, "Provider URL: " + argProviderUrl + ", User: " + argUser +
                              ", Passwd: " + argPass);

        Context ctx = checkAndCreateServerLocalEnviormentInitialContext();
        if(ctx != null)
        {
          Logger.logDebug(null,"System is ignoring all passed values (i.e. providerUrl,userName,pass) as caller application is running in same server instance.System will use locally created InitialConext.");
          return ctx;
        }
        Properties props = new Properties();
        try
  {
        if(null != argProviderUrl && null != argUser && null != argPass)
  {
    String productName = "";
 
    try
    {
    MBeanServer server = ManagementFactory.getPlatformMBeanServer();
    ObjectName name = new ObjectName("jboss.as:management-root=server");
    productName = (String) (server.getAttribute(name, "productName"));
    }
    catch (Exception ex)
    {
    // Just swallow the error, nothing we can do
    }

    // If we are JBoss EAP, then proceed
    if (EAP.equalsIgnoreCase(productName))
    {
    String[] urlValues = argProviderUrl.split(":");
    if(!SELECTOR_IS_SET.get())
    {
      Properties env = getEJBClientContextPropsEAP(argProviderUrl,argUser,argPass);
      ctx = new InitialContext(env);
    }

    }
    // Old/Normal EJB processing
    else
    {
    String[] urlValues = argProviderUrl.split(":");
    props.put(javax.naming.Context.URL_PKG_PREFIXES, "org.jboss.ejb.client.naming");
    ctx = new InitialContext(props);       
    if(!SELECTOR_IS_SET.get())
    {
      Properties env= getEJBClientContextProps(argProviderUrl,argUser,argPass);
        /*
      Properties env = new Properties();
      env.put("remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED", "false");
      env.put("remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS", "false");
      env.put("remote.connections", "default");
      env.put("remote.connection.default.host", urlValues[1].substring(2));
      env.put("remote.connection.default.port", urlValues[2]);
      env.put("remote.connection.default.connect.options.org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL", HEARTBEAT_INTERVAL);//system will send heart beat in in every 3 mins.
      env.put("remote.connection.default.connect.options.org.xnio.Options.KEEP_ALIVE","true");
      env.put("remote.connection.default.username", argUser);
      env.put("remote.connection.default.password", argPass);
      */
      Logger.logInfo(null, "EJBClientContext.setSelector(..) is not called, system will now set the selector and this will be used across the system");
      try
      {
      SELECTOR_LOCK.lock();
      SELECTOR_IS_SET.set(true);
      EJBClientConfiguration clientConfig = new PropertiesBasedEJBClientConfiguration(env);
      ContextSelector<EJBClientContext> contexSelector = new ConfigBasedEJBClientContextSelector(clientConfig);
      EJBClientContext.setSelector(contexSelector);
      }
      finally
      {
          SELECTOR_LOCK.unlock();
      }
    }
    }
        }
        Logger.logInfo(null, "non local InitialContext is returned from method.");
        return ctx;
        }
  catch(Exception ne)
  {
  Logger.logError(null, "\n ###### Unable to create context in parameterized getInitialContext : ##### ",ne);
 
  props = new Properties();
  props.put(Context.URL_PKG_PREFIXES, URL_PKG_PREFIXES);
  props.put("jboss.naming.client.connect.options.org.xnio.Options.KEEP_ALIVE","true");
  props.put("jboss.naming.client.connect.options.org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL", String.valueOf(1000));
  if (props.size() > 0)
  {
    return new InitialContext(props);
  }
  else
  {
    return new InitialContext();
  }
  }
    }

The temptation is to simply move to Tomcat, but I dislike changing containers just because one element of the system (which we will eventually replace with an XML service) isn't working. EAP seems to very snappy at loading and is exactly what we are looking for, other than the EJB issues.

Thanks

Stephen