RoundRobin, new cluster member & SFSB == NPE

[pmadzik]

Hello,

I know that using RoundRobin policy for SFSB is not best idea, but nevertheless i don't expect behavior described below.

When new member joins to cluster, client crashes with NPE exception:

 [java] Exception in thread "main" java.lang.NullPointerException
 [java] at org.jboss.ejb3.stateful.StatefulInstanceInterceptor.invoke(StatefulInstanceInterceptor.java:61)
 [java] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 [java] at org.jboss.aspects.remoting.ReplicantsManagerInterceptor.invoke(ReplicantsManagerInterceptor.java:51)
 [java] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 [java] at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
 [java] at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:102)
 [java] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 [java] at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:47)
 [java] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 [java] at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
 [java] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 [java] at org.jboss.ejb3.stateful.StatefulContainer.dynamicInvoke(StatefulContainer.java:319)
 [java] at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
 [java] at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
 [java] at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:828)
 [java] at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:681)
 [java] at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:358)
 [java] at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:398)
 [java] at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:239)
 [java] at org.jboss.remoting.RemoteClientInvoker.invoke(RemoteClientInvoker.java:190)
 [java] at org.jboss.remoting.Client.invoke(Client.java:525)
 [java] at org.jboss.remoting.Client.invoke(Client.java:488)
 [java] at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:55)
 [java] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 [java] at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:77)
 [java] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 [java] at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:61)
 [java] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 [java] at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:53)
 [java] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 [java] at org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:77)
 [java] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 [java] at org.jboss.ejb3.stateful.StatefulClusteredProxy.invoke(StatefulClusteredProxy.java:103)
 [java] at $Proxy2.sayHelloFrom(Unknown Source)
 [java] at client.Client.main(Client.java:22)

Is it bug in jboss?
I use 4.0.5.GA-ejb3 version. I didn't change anything in default configuration.

Regards,
PM

[ben.wang]

It seems that the clustered cache instance in the new node is not ready and therefore the NPE. Can you send me your code for me to try? It'd best to have source as well. Thanks.

[pmadzik]

Hello,

It's my code:

CounterRemote.java

package cluster.counter;

import javax.ejb.Remote;

@Remote
public interface CounterRemote {
 public String showCounter();
}

CounterBean.java

package cluster.counter;

import java.net.InetAddress;
import java.net.UnknownHostException;

import javax.ejb.Stateful;

@Stateful
public class CounterBean implements CounterRemote {

 private int counter = 0;

 public String showCounter() {
 return "[" + getIpAddr() + "]\tcounter = " + (++counter);
 }

 private static String getIpAddr() {
 String tempIpAddr = "?.?.?.?";

 try {
 tempIpAddr = InetAddress.getLocalHost().getHostAddress();
 } catch (UnknownHostException e) {
 e.printStackTrace();
 }

 return tempIpAddr;
 }
}

CounterClient.java

package cluster.client;

import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;

import cluster.counter.CounterRemote;

public class CounterClient {
 public static void main(String[] args)
 {
 try {
 Context jndiContext = getInitialContext();
 Object ref = jndiContext.lookup("CounterBean/remote");
 CounterRemote counter = (CounterRemote) ref;

 for (int i = 0; i < ((args.length > 0) ? Integer.valueOf(args[0]) : 10); i++) {
 System.out.println(counter.showCounter());
 }
 } catch (NamingException ne) {
 ne.printStackTrace();
 }
 }

 public static Context getInitialContext() throws NamingException
 {
 return new InitialContext();
 }
}

jboss.xml

<?xml version="1.0"?>
<jboss
 xmlns="http://java.sun.com/xml/ns/javaee"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
 http://www.jboss.org/j2ee/schema/jboss_5_0.xsd"
 version="3.0">
 <enterprise-beans>
 <session>
 <ejb-name>CounterBean</ejb-name>
 <clustered>true</clustered>
 <cluster-config>
 <load-balance-policy>org.jboss.ha.framework.interfaces.RoundRobin</load-balance-policy>
 </cluster-config>
 </session>
 </enterprise-beans>
</jboss>

Error message is same as above except for two last lines (i changed slightly code)

 at $Proxy2.showCounter(Unknown Source)
 at cluster.client.CounterClient.main(CounterClient.java:18)

I have another problem with above code and FirstAvailable policy -- failover ends with error.
When i down node which serving client calls another node is starting serving for that client but after moment i get error (jteam = 10.1.0.85):

[10.1.0.85] counter = 1
...
[10.1.0.85] counter = 129
[10.1.0.85] counter = 130[10.1.2.71] counter = 131...
[10.1.2.71] counter = 357
[10.1.2.71] counter = 358
Exception in thread "main" javax.ejb.EJBException: java.lang.RuntimeException: org.jboss.cache.ReplicationException: rsp
=sender=jteam:33015, retval=null, received=false, suspected=true
 at org.jboss.ejb3.tx.Ejb3TxPolicy.handleExceptionInOurTx(Ejb3TxPolicy.java:69)
 at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:83)
 at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.stateful.StatefulInstanceInterceptor.invoke(StatefulInstanceInterceptor.java:83)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.remoting.ReplicantsManagerInterceptor.invoke(ReplicantsManagerInterceptor.java:51)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
 at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:102)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:47)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.stateful.StatefulContainer.dynamicInvoke(StatefulContainer.java:319)
 at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
 at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:828)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:681)
 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:358)
 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:412)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:239)
 at org.jboss.remoting.RemoteClientInvoker.invoke(RemoteClientInvoker.java:190)
 at org.jboss.remoting.Client.invoke(Client.java:525)
 at org.jboss.remoting.Client.invoke(Client.java:488)
 at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:55)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:77)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:61)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:53)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:77)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.stateful.StatefulClusteredProxy.invoke(StatefulClusteredProxy.java:103)
 at $Proxy2.showCounter(Unknown Source)
 at cluster.client.CounterClient.main(CounterClient.java:18)
Caused by: java.lang.RuntimeException: org.jboss.cache.ReplicationException: rsp=sender=jteam:33015, retval=null, receiv
ed=false, suspected=true
 at org.jboss.ejb3.cache.tree.StatefulTreeCache.replicate(StatefulTreeCache.java:180)
 at org.jboss.ejb3.cache.StatefulReplicationInterceptor.invoke(StatefulReplicationInterceptor.java:63)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
 at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.entity.ExtendedPersistenceContextPropagationInterceptor.invoke(ExtendedPersistenceContextPropa
gationInterceptor.java:57)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerIntercep
tor.java:54)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:46)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
 at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.stateful.StatefulInstanceInterceptor.invoke(StatefulInstanceInterceptor.java:83)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.remoting.ReplicantsManagerInterceptor.invoke(ReplicantsManagerInterceptor.java:51)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
 at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:102)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:47)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.stateful.StatefulContainer.dynamicInvoke(StatefulContainer.java:319)
 at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
 at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:828)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:681)
 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:358)
 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:412)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:239)
Caused by: org.jboss.cache.ReplicationException: rsp=sender=jteam:33015, retval=null, received=false, suspected=true
 at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4191)
 at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4114)
 at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4215)
 at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:110)
 at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:88)
 at org.jboss.cache.interceptors.ReplicationInterceptor.handleReplicatedMethod(ReplicationInterceptor.java:119)
 at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:83)
 at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
 at org.jboss.cache.interceptors.PassivationInterceptor.invoke(PassivationInterceptor.java:69)
 at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
 at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:345)
 at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:156)
 at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
 at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:157)
 at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5520)
 at org.jboss.cache.TreeCache.put(TreeCache.java:3678)
 at org.jboss.cache.TreeCache.put(TreeCache.java:3616)
 at org.jboss.ejb3.cache.tree.StatefulTreeCache.replicate(StatefulTreeCache.java:176)
 ... 35 more
Caused by: org.jboss.cache.SuspectException: Response suspected: sender=jteam:33015, retval=null, received=false, suspec
ted=true
 at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4185)
 ... 52 more

Regards,
PM

[ben.wang]

OK. please give me a day or two to get to it. Thanks!

[ben.wang]

Sorry to get back to you late.

I have finally got a chance to try to reproduce your problem. I am using the latest jboss-4.0 src (of which should be very close to 4.0.5.GA). But I can't reproduce your problem. I have the current cluster counter test running and then have a new node startup to join. No exception occurred.

However, I don't see the round robin going to the new node either. I will look into this part more.

[pmadzik]

Hello Ben,

I checked out latest svn version (rev 58156) of 4.0 branch (http://anonsvn.jboss.org/repos/jbossas/branches/Branch_4_0).

I can reproduce this problem (SFSB + RoundRobin + new node) in a few different environements (operating systems, networks).

And what about my second problem (SFSB + FirstAvailable + failover)?

Regards,
PM

[ben.wang]

No, I can't reproduce from my system. I have tried both RoundRobin and FirstAvailable. They failover just fine. I use two node cluster to restart one.

Here is my bean code snippet:

@Stateful(name="GangDispute")
@Clustered(loadBalancePolicy= org.jboss.ha.framework.interfaces.FirstAvailable.class)
//@Clustered
@Remote(GangDispute.class)
public class GangDisputeBean implements java.io.Serializable, GangDispute
{
 private long arrestedGuys = 0;

 public long newGuysArrested(long newArrests)
 {
 arrestedGuys+=newArrests;
 System.out.println("Total arrested guys: " + arrestedGuys + "( +" + newArrests + " arrests)");
 return arrestedGuys;
 }

 public long getNbArrestedGuys ()
 {
 return arrestedGuys;
 }

and client

 Client () throws Exception
 {
 //Get JNDI context
 Context naming = new InitialContext ();
 ClassLoader loader = Thread.currentThread().getContextClassLoader();
 Enumeration rsrcs = loader.getResources("jndi.properties");
 while( rsrcs.hasMoreElements() )
 {
 System.out.println(rsrcs.nextElement());
 }
 System.out.println("JNDI.env: "+naming.getEnvironment());
 System.out.println("Looking up GangDispute/remote");
 //Look up the Remote
 gd = (GangDispute)naming.lookup ("GangDispute/remote");
 }

 public void test ()
 {

 try
 {
 while (true)
 {
 long total = gd.newGuysArrested (1);
 System.out.println("Total number of arrested guys: " + total);

 try
 {
 synchronized (this)
 {
 this.wait (1000);

...

and jndi.prperties:

java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
java.naming.provider.url=jnp://localhost:1100
java.naming.factory.url.pkgs=org.jboss.naming

I am now using exactly 4.0.5.GA as well. How do you run yours?

[pmadzik]

I can run your code without errors. But when i decrease time between remote calls to minimum by comment out this fragment:

 public void test ()
 {

 try
 {
 while (true)
 {
 long total = gd.newGuysArrested (1);
 System.out.println("Total number of arrested guys: " + total);
 //try
 //{
 // synchronized (this)
 // {
 // this.wait (1000);
...

i reproduce both described problems.

Regards,
PM

[ben.wang]

OK, getting rid of sleep interval, I encountered Socket reset exception. I think this is coming from JBoss Remoting. So I will try to see if I can upgrade it to release 2.0.

However, I still don't see your exception though.

[pmadzik]

Hello,

Have you any suggestions what may be wrong in my configuration?

Are you sure that one may exclude some kind of race condition in JBoss (Cache) code?

Regards,
PM