Jboss 4.0.3Sp1 + Jdk 1.5 + Spring Framework 2.0.8 - Cluster restart problem

[cdsanchez]

Hello,

- I know I am talking about old versions but I don't have more options atm; I have a known problem about remote calling a EJB after restarting a Cluster.

- My architecture is:

- 1 Jboss partition with 2 nodes with a war application.

- 1 Jboss partition in Cluster with 2 nodes with a stateless EJB.

- No problem if i call to the remote EJB from the war application, it is working; when I restart the Jboss partition in Cluster, after that, the EJB client is unable to call the Remote EJB, it shows 1 error:

- The original exception seeing in Debug:

org.jboss.invocation.ServiceUnavailableException: Service unavailable

- The Object cachedHome which includes a org.jboss.proxy.ClientContainer has invalid reference (this proxy is not valid anymore)

The final stacktrace in log:

org.springframework.remoting.RemoteAccessException: Could not access remote service [ejb/EJBService]; nested exception is org.jboss.invocation.ServiceUnavailableException: Service unavailable.
Caused by:
org.jboss.invocation.ServiceUnavailableException: Service unavailable.
    at org.jboss.invocation.jrmp.interfaces.JRMPInvokerProxyHA.invoke(JRMPInvokerProxyHA.java:292)
    at org.jboss.invocation.InvokerInterceptor.invokeInvoker(InvokerInterceptor.java:227)
    at org.jboss.invocation.MarshallingInvokerInterceptor.invoke(MarshallingInvokerInterceptor.java:37)
    at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
    at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:55)
    at org.jboss.proxy.ejb.HomeInterceptor.invoke(HomeInterceptor.java:169)
    at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:86)
    at $Proxy63.create(Unknown Source)
    at sun.reflect.GeneratedMethodAccessor285.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.springframework.ejb.access.AbstractSlsbInvokerInterceptor.create(AbstractSlsbInvokerInterceptor.java:177)
    at org.springframework.ejb.access.AbstractRemoteSlsbInvokerInterceptor.newSessionBeanInstance(AbstractRemoteSlsbInvokerInterceptor.java:205)
    at org.springframework.ejb.access.SimpleRemoteSlsbInvokerInterceptor.getSessionBeanInstance(SimpleRemoteSlsbInvokerInterceptor.java:108)
    at org.springframework.ejb.access.SimpleRemoteSlsbInvokerInterceptor.doInvoke(SimpleRemoteSlsbInvokerInterceptor.java:74)
    at org.springframework.ejb.access.AbstractRemoteSlsbInvokerInterceptor.invoke(AbstractRemoteSlsbInvokerInterceptor.java:119)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
    at $Proxy52.doAvailabilitySearch(Unknown Source)

Of course I have the "jndi lookup during restart" problem too.

The question is if you know how to fix it; I don't know how to use the RetryInterceptor using Spring, I don't know how to use it.

Any ideas?

Thank you very much in advance

[cdsanchez]

More details:

The class org.springframework.ejb.access.SimpleRemoteSlsbInvokerInterceptor in the doInvoke method, once it tries to do the remote call to the EJB, throws an org.jboss.invocation.ServiceUnavailableException: Service unavailable

This is a RemoteException but in the process of checking which type of exception was to throw the necessary to refresh the cache (throw RmiClientInterceptorUtils.convertRmiAccessException(invocation.getMethod(), rex, isConnectFailure(rex), getJndiName()), the isConnectFailure check return false and then it is not possible to do the refresh, because:

RmiClientInterceptor.convertRmiAccessException, checks if isConnectFailure or not, it seems org.jboss.invocation.ServiceUnavailableException is not a type of possible connection failure, so, then it doesn't throw a RemoteConnectFailure, throws a RemoteAccessException.

Then, AbstractRemoteSlsbInvokerInterceptor (SimpleRemoteSlsbInvokerInterceptor parent) then doesn't try the handleRemoteConnectFailure method which ask for the refreshHomeOnConnectFailure which call to the refreshAndRetry method.

Maybe I should ask to Spring forums? Not sure where is the source, I was thinking to include into Spring the Jboss exception!! but well, I think is a bad idea.

Thank you

[brian.stansberry]

The RetryInterceptor wiki page at http://community.jboss.org/docs/DOC-11840 gives an example of how to configure your EJB to include the interceptor. Sounds like that's what you need. The fact that you're using Spring on the client side shouldn't affect that.

[cdsanchez]

Hello Brian,

Thank you for your reply.

I tried the RetryInterceptor but it doesn't work: I can see in my logs the trace:

2010-03-11 12:38:40,211 DEBUG [org.springframework.ejb.access.SimpleRemoteStatelessSessionProxyFactoryBean] Trying to create reference to remote EJB

It only shows this trace but doesn't continue, even I tried to debug it but it seems the JdkDynamicAopProxy class is null, maybe is this the problem?

Thank you

[cdsanchez]

Hello again,

It loops on waitOnInvokerProxy method from RetryInterceptor, never returns.

  private void waitOnInvokerProxy(InvocationContext ctx, InvocationType type)
   {
      if( trace )
         log.trace("Begin waitOnInvokerProxy");
      boolean isRemote = type == InvocationType.REMOTE;
      String jndiName = (String) ctx.getValue(InvocationKey.JNDI_NAME);
      if( isRemote == true )
         jndiName += "-RemoteInvoker";
      else
         jndiName += "-HomeInvoker";
     while( retry == true )
      {
         try
         {
            Thread.sleep(1000);
            InitialContext namingCtx = new InitialContext(retryEnv);
            if( trace )
               log.trace("Looking for invoker: "+jndiName);
            Invoker invoker = (Invoker) namingCtx.lookup(jndiName);
            if( trace )
               log.trace("Found invoker: "+invoker);
            ctx.setInvoker(invoker);
           break;
         }
         catch(Throwable t)
         {
            if( trace )
               log.trace("Failed to lookup proxy", t);
         }
      }
      if( trace )
         log.trace("End waitOnInvokerProxy");
   }

I am checking the environment properties, maybe i am loosing the java.naming.provider.url.

Thank you

[brian.stansberry]

Note the discussion of the JNDI Lookup Process on http://community.jboss.org/;jsessionid=B2AFA12BDB09F05E06A7D2B60D4FDEF0docs/DOC-11840 and of the org.jboss.naming.NamingContextFactory at http://community.jboss.org/;jsessionid=DAA90021F6569A9ADFA00BA8EF9B6057docs/DOC-11462

[cdsanchez]

Hello,

Thank you Brian for your reply.

Finally because I can't use newest version of Jboss, at the moment, I decided to override the RetryInterceptor class and change the original loop by a new loop with 3 retries; after third, it exits the loop; now it is working, another problem was the lookup parameters; the application doesn't use the jndi.properties standard file and first time in the Application starts it load the properties but, next time (for example in the EJB cluster restart) it was unable to find these properties so I've include the standard jndi.properties in the Application classpath.

Anyway I'll take a view to your links to get more understanding about how is the behavior.

Thank you very much again.

Just last question; how in the new RetryInterceptor/SingleRetry do you pass the retries and time wait parameters into the invoker definition into the jboss.xml?

Thank you very much in advance.