Some Issues with clustering on JBoss 4.0.4 and EJB stub cach

[ablevine1]

I am currently usign the JBoss4.0.4 and the EJB3 release that is included with it. I have been using this http://docs.jboss.org/jbossas/jboss4guide/r4/html/cluster.chapt.html as my basis for understanding how jboss clustering works, but I think my understanding of it may be wrong. I am currently running with a cluster of two app servers on the same partition called app1 and app2.

Here is my test:
1. My external client program connects to the cluster via HAJNDI(port 1100) and gets a client stub to a session bean by doing a lookup on an initial context.
2. It then makes some calls to the ejb using this stub and caches the stub in variable.
3. I see which of the app servers handled the request and shut it down but leave the other one running.
4. Now I use the stored stub on the client to make the same call to the cluster as before but it fails.

From the clustering doc mentioned above, I was under the impression that step 4 should have worked and that the stubs handled failover. Is this assumption incorrect, or am I missing something??

any help with this would be greatly appreciated.

[marcreis]

Can you post some more details ? (Code snippet, Config info (Server/Beans...)). Are you invoking on a SLSB or an SFSB ?

[ablevine1]

I am using a stateless session bean. Now it seems like it does reconnect to the second server if the first one goes down, but if all servers in the cluster go down and the client tries to connect when they are down, the client gets in a state where it fails every time, even after all servers are up again.

Also, what is the default Cluster configuration for an EJB3 slsb using the @Clustered annotation??

[marcreis]

If you want the client to not fail if all nodes of the cluster go down have a look at the client retry Interceptor http://wiki.jboss.org/wiki/Wiki.jsp?page=RetryInterceptor.
The Default for SLSB LB is RoundRobin afaik, are you looking for some specific detail?.

Sincerely

Marc

[ablevine1]

Thanks Marc. I've notice that the error I mentioned happens sometime only when one server is brought down, putting the client in a bad state where the cached stub continuously fails to reconnect properly. The only way to get the client program to work again when in that state, is to reconnect with a new initial context and lookup. I've put code in to do that, but was hoping that there was a better way.

[ablevine1]

After looking more closely at the retry interceptor wiki page, I see that it is geared towards an older ejb, specifically the jboss.xml. Anyone know what this file would look like for an EJB3.0 Session bean??

[marcreis]

I would also like to know about that :-)

[brian.stansberry]

Are your beans marked

<clustered>true</clustered>

in jboss.xml?

[brian.stansberry]

Sorry, I saw the 1st post in this thread and replied; didn't see the rest of the thread. I'll assume your beans are marked clustered, either via annotation or in the xml.

RetryInterceptor is not yet available in EJB3. Don't think it would solve your problem though. I'd need to know more details about your exact error (how does it fail on reconnect -- full stack trace please) to say much more.

[ablevine1]

Since I have the same ejb's in two different clusters, I am using the jboss.xml approach with

<clustered>true</clustered>

for each of my sesion beans.
see: http://www.jboss.com/index.html?module=bb&op=viewtopic&t=91189

The jboss failover code does not seem to be working though, and I get the following exception stack trace when I shutdown one of the servers in the cluster that the client had been making calls to:

java.lang.reflect.UndeclaredThrowableException
 at $Proxy1.loadObject(Unknown Source)
 at com.squaretrade.managed.AbstractManager.loadObject(AbstractManager.java:37)
 at com.squaretrade.person.PersonManager.loadPerson(PersonManager.java:52)
 at com.squaretrade.person.RecurringPersonLoader.main(RecurringPersonLoader.java:44)
Caused by: org.jboss.aop.NotFoundInDispatcherException: Object with oid: jboss.j2ee:ear=batch.ear,jar=beans-batch.jar,name=PersonManagerSessionImpl,service=EJB3 was not found in the Dispatcher
 at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:85)
 at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:828)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:681)
 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:358)
 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:412)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:239)
 at org.jboss.remoting.RemoteClientInvoker.invoke(RemoteClientInvoker.java:190)
 at org.jboss.remoting.Client.invoke(Client.java:525)
 at org.jboss.remoting.Client.invoke(Client.java:488)
 at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:55)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:61)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:55)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:65)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.stateless.StatelessRemoteProxy.invoke(StatelessRemoteProxy.java:102)
 ... 4 more

note that loadObject is the session bean call. and that is where it is failing. As soon as I create a new InitialContext using port 1100 and then lookup the session bean again, the call goes through. Although, I thought that the stub was supposed to do this for me.


here is the entry in jboss.xml for one of my sesion beans:

<session>
 <ejb-name>PersonManagerSessionImpl</ejb-name>
 <clustered>true</clustered>
 <cluster-config>
 <partition-name>${jboss.partition.name}</partition-name>
 <load-balance-policy>org.jboss.ha.framework.interfaces.FirstAvailable</load-balance-policy>
 </cluster-config>
</session>

[brian.stansberry]

There's another recent post of a problem with failover of beans configured with an xml override: http://www.jboss.com/index.html?module=bb&op=viewtopic&t=91658.

We're investigating the cause.

[brian.stansberry]

The cause is simple; support for XML configuration of clustering was added in EJB3 RC9 (see http://jira.jboss.com/jira/browse/EJBTHREE-569), which is after the EJB3 release included with 4.0.4.GA. You'd need to upgrade your EJB3 release.

Sorry for not knowing this :(

[ablevine1]

Sorry, I saw the 1st post in this thread and replied; didn't see the rest of the thread. I'll assume your beans are marked clustered, either via annotation or in the xml.

RetryInterceptor is not yet available in EJB3. Don't think it would solve your problem though. I'd need to know more details about your exact error (how does it fail on reconnect -- full stack trace please) to say much more.

Has this retry interceptor or some simlar concept become available for use with the jboss ejb3 implementation yet?? Currently, failover works fine in my client if at least one node in the server cluster is still running, however, when all the nodes in the cluster go down and then come up, the client now has a bad session stub reference and will throw exceptions. I then have to surround every call on the stub with error handling code that will clear my server connections and start with fresh ones. It would be nice to have the retry interceptor so that this excess code can be eliminated.