9 Replies Latest reply on Nov 10, 2006 2:19 AM by Przemyslaw Madzik

    RoundRobin, new cluster member & SFSB == NPE

    Przemyslaw Madzik Newbie

      Hello,

      I know that using RoundRobin policy for SFSB is not best idea, but nevertheless i don't expect behavior described below.

      When new member joins to cluster, client crashes with NPE exception:

       [java] Exception in thread "main" java.lang.NullPointerException
       [java] at org.jboss.ejb3.stateful.StatefulInstanceInterceptor.invoke(StatefulInstanceInterceptor.java:61)
       [java] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       [java] at org.jboss.aspects.remoting.ReplicantsManagerInterceptor.invoke(ReplicantsManagerInterceptor.java:51)
       [java] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       [java] at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
       [java] at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:102)
       [java] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       [java] at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:47)
       [java] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       [java] at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
       [java] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       [java] at org.jboss.ejb3.stateful.StatefulContainer.dynamicInvoke(StatefulContainer.java:319)
       [java] at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
       [java] at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
       [java] at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:828)
       [java] at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:681)
       [java] at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:358)
       [java] at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:398)
       [java] at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:239)
       [java] at org.jboss.remoting.RemoteClientInvoker.invoke(RemoteClientInvoker.java:190)
       [java] at org.jboss.remoting.Client.invoke(Client.java:525)
       [java] at org.jboss.remoting.Client.invoke(Client.java:488)
       [java] at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:55)
       [java] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       [java] at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:77)
       [java] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       [java] at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:61)
       [java] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       [java] at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:53)
       [java] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       [java] at org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:77)
       [java] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       [java] at org.jboss.ejb3.stateful.StatefulClusteredProxy.invoke(StatefulClusteredProxy.java:103)
       [java] at $Proxy2.sayHelloFrom(Unknown Source)
       [java] at client.Client.main(Client.java:22)
      


      Is it bug in jboss?
      I use 4.0.5.GA-ejb3 version. I didn't change anything in default configuration.

      Regards,
      PM

        • 1. Re: RoundRobin, new cluster member & SFSB == NPE
          Ben Wang Master

          It seems that the clustered cache instance in the new node is not ready and therefore the NPE. Can you send me your code for me to try? It'd best to have source as well. Thanks.

          • 2. Re: RoundRobin, new cluster member & SFSB == NPE
            Przemyslaw Madzik Newbie

            Hello,

            It's my code:

            CounterRemote.java

            package cluster.counter;
            
            import javax.ejb.Remote;
            
            @Remote
            public interface CounterRemote {
             public String showCounter();
            }
            


            CounterBean.java
            package cluster.counter;
            
            import java.net.InetAddress;
            import java.net.UnknownHostException;
            
            import javax.ejb.Stateful;
            
            @Stateful
            public class CounterBean implements CounterRemote {
            
             private int counter = 0;
            
             public String showCounter() {
             return "[" + getIpAddr() + "]\tcounter = " + (++counter);
             }
            
             private static String getIpAddr() {
             String tempIpAddr = "?.?.?.?";
            
             try {
             tempIpAddr = InetAddress.getLocalHost().getHostAddress();
             } catch (UnknownHostException e) {
             e.printStackTrace();
             }
            
             return tempIpAddr;
             }
            }
            


            CounterClient.java
            package cluster.client;
            
            import javax.naming.Context;
            import javax.naming.InitialContext;
            import javax.naming.NamingException;
            
            import cluster.counter.CounterRemote;
            
            public class CounterClient {
             public static void main(String[] args)
             {
             try {
             Context jndiContext = getInitialContext();
             Object ref = jndiContext.lookup("CounterBean/remote");
             CounterRemote counter = (CounterRemote) ref;
            
             for (int i = 0; i < ((args.length > 0) ? Integer.valueOf(args[0]) : 10); i++) {
             System.out.println(counter.showCounter());
             }
             } catch (NamingException ne) {
             ne.printStackTrace();
             }
             }
            
             public static Context getInitialContext() throws NamingException
             {
             return new InitialContext();
             }
            }
            


            jboss.xml
            <?xml version="1.0"?>
            <jboss
             xmlns="http://java.sun.com/xml/ns/javaee"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
             http://www.jboss.org/j2ee/schema/jboss_5_0.xsd"
             version="3.0">
             <enterprise-beans>
             <session>
             <ejb-name>CounterBean</ejb-name>
             <clustered>true</clustered>
             <cluster-config>
             <load-balance-policy>org.jboss.ha.framework.interfaces.RoundRobin</load-balance-policy>
             </cluster-config>
             </session>
             </enterprise-beans>
            </jboss>
            


            Error message is same as above except for two last lines (i changed slightly code)
             at $Proxy2.showCounter(Unknown Source)
             at cluster.client.CounterClient.main(CounterClient.java:18)
            


            I have another problem with above code and FirstAvailable policy -- failover ends with error.
            When i down node which serving client calls another node is starting serving for that client but after moment i get error (jteam = 10.1.0.85):

            [10.1.0.85] counter = 1
            ...
            [10.1.0.85] counter = 129
            [10.1.0.85] counter = 130[10.1.2.71] counter = 131...
            [10.1.2.71] counter = 357
            [10.1.2.71] counter = 358
            Exception in thread "main" javax.ejb.EJBException: java.lang.RuntimeException: org.jboss.cache.ReplicationException: rsp
            =sender=jteam:33015, retval=null, received=false, suspected=true
             at org.jboss.ejb3.tx.Ejb3TxPolicy.handleExceptionInOurTx(Ejb3TxPolicy.java:69)
             at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:83)
             at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.ejb3.stateful.StatefulInstanceInterceptor.invoke(StatefulInstanceInterceptor.java:83)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.aspects.remoting.ReplicantsManagerInterceptor.invoke(ReplicantsManagerInterceptor.java:51)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
             at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:102)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:47)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.ejb3.stateful.StatefulContainer.dynamicInvoke(StatefulContainer.java:319)
             at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
             at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
             at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:828)
             at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:681)
             at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:358)
             at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:412)
             at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:239)
             at org.jboss.remoting.RemoteClientInvoker.invoke(RemoteClientInvoker.java:190)
             at org.jboss.remoting.Client.invoke(Client.java:525)
             at org.jboss.remoting.Client.invoke(Client.java:488)
             at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:55)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:77)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:61)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:53)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:77)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.ejb3.stateful.StatefulClusteredProxy.invoke(StatefulClusteredProxy.java:103)
             at $Proxy2.showCounter(Unknown Source)
             at cluster.client.CounterClient.main(CounterClient.java:18)
            Caused by: java.lang.RuntimeException: org.jboss.cache.ReplicationException: rsp=sender=jteam:33015, retval=null, receiv
            ed=false, suspected=true
             at org.jboss.ejb3.cache.tree.StatefulTreeCache.replicate(StatefulTreeCache.java:180)
             at org.jboss.ejb3.cache.StatefulReplicationInterceptor.invoke(StatefulReplicationInterceptor.java:63)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
             at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.ejb3.entity.ExtendedPersistenceContextPropagationInterceptor.invoke(ExtendedPersistenceContextPropa
            gationInterceptor.java:57)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerIntercep
            tor.java:54)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:46)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
             at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.ejb3.stateful.StatefulInstanceInterceptor.invoke(StatefulInstanceInterceptor.java:83)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.aspects.remoting.ReplicantsManagerInterceptor.invoke(ReplicantsManagerInterceptor.java:51)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
             at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:102)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:47)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.ejb3.stateful.StatefulContainer.dynamicInvoke(StatefulContainer.java:319)
             at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
             at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
             at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:828)
             at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:681)
             at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:358)
             at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:412)
             at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:239)
            Caused by: org.jboss.cache.ReplicationException: rsp=sender=jteam:33015, retval=null, received=false, suspected=true
             at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4191)
             at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4114)
             at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4215)
             at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:110)
             at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:88)
             at org.jboss.cache.interceptors.ReplicationInterceptor.handleReplicatedMethod(ReplicationInterceptor.java:119)
             at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:83)
             at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
             at org.jboss.cache.interceptors.PassivationInterceptor.invoke(PassivationInterceptor.java:69)
             at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
             at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:345)
             at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:156)
             at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
             at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:157)
             at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5520)
             at org.jboss.cache.TreeCache.put(TreeCache.java:3678)
             at org.jboss.cache.TreeCache.put(TreeCache.java:3616)
             at org.jboss.ejb3.cache.tree.StatefulTreeCache.replicate(StatefulTreeCache.java:176)
             ... 35 more
            Caused by: org.jboss.cache.SuspectException: Response suspected: sender=jteam:33015, retval=null, received=false, suspec
            ted=true
             at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4185)
             ... 52 more
            


            Regards,
            PM

            • 3. Re: RoundRobin, new cluster member & SFSB == NPE
              Ben Wang Master

              OK. please give me a day or two to get to it. Thanks!

              • 4. Re: RoundRobin, new cluster member & SFSB == NPE
                Ben Wang Master

                Sorry to get back to you late.

                I have finally got a chance to try to reproduce your problem. I am using the latest jboss-4.0 src (of which should be very close to 4.0.5.GA). But I can't reproduce your problem. I have the current cluster counter test running and then have a new node startup to join. No exception occurred.

                However, I don't see the round robin going to the new node either. I will look into this part more.

                • 5. Re: RoundRobin, new cluster member & SFSB == NPE
                  Przemyslaw Madzik Newbie

                  Hello Ben,

                  I checked out latest svn version (rev 58156) of 4.0 branch (http://anonsvn.jboss.org/repos/jbossas/branches/Branch_4_0).

                  I can reproduce this problem (SFSB + RoundRobin + new node) in a few different environements (operating systems, networks).

                  And what about my second problem (SFSB + FirstAvailable + failover)?

                  Regards,
                  PM

                  • 6. Re: RoundRobin, new cluster member & SFSB == NPE
                    Ben Wang Master

                    No, I can't reproduce from my system. I have tried both RoundRobin and FirstAvailable. They failover just fine. I use two node cluster to restart one.

                    Here is my bean code snippet:

                    @Stateful(name="GangDispute")
                    @Clustered(loadBalancePolicy= org.jboss.ha.framework.interfaces.FirstAvailable.class)
                    //@Clustered
                    @Remote(GangDispute.class)
                    public class GangDisputeBean implements java.io.Serializable, GangDispute
                    {
                     private long arrestedGuys = 0;
                    
                     public long newGuysArrested(long newArrests)
                     {
                     arrestedGuys+=newArrests;
                     System.out.println("Total arrested guys: " + arrestedGuys + "( +" + newArrests + " arrests)");
                     return arrestedGuys;
                     }
                    
                     public long getNbArrestedGuys ()
                     {
                     return arrestedGuys;
                     }
                    

                    and client
                     Client () throws Exception
                     {
                     //Get JNDI context
                     Context naming = new InitialContext ();
                     ClassLoader loader = Thread.currentThread().getContextClassLoader();
                     Enumeration rsrcs = loader.getResources("jndi.properties");
                     while( rsrcs.hasMoreElements() )
                     {
                     System.out.println(rsrcs.nextElement());
                     }
                     System.out.println("JNDI.env: "+naming.getEnvironment());
                     System.out.println("Looking up GangDispute/remote");
                     //Look up the Remote
                     gd = (GangDispute)naming.lookup ("GangDispute/remote");
                     }
                    
                     public void test ()
                     {
                    
                     try
                     {
                     while (true)
                     {
                     long total = gd.newGuysArrested (1);
                     System.out.println("Total number of arrested guys: " + total);
                    
                     try
                     {
                     synchronized (this)
                     {
                     this.wait (1000);
                    
                    ...
                    

                    and jndi.prperties:
                    java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
                    java.naming.provider.url=jnp://localhost:1100
                    java.naming.factory.url.pkgs=org.jboss.naming
                    


                    I am now using exactly 4.0.5.GA as well. How do you run yours?

                    • 7. Re: RoundRobin, new cluster member & SFSB == NPE
                      Przemyslaw Madzik Newbie

                      I can run your code without errors. But when i decrease time between remote calls to minimum by comment out this fragment:

                       public void test ()
                       {
                      
                       try
                       {
                       while (true)
                       {
                       long total = gd.newGuysArrested (1);
                       System.out.println("Total number of arrested guys: " + total);
                       //try
                       //{
                       // synchronized (this)
                       // {
                       // this.wait (1000);
                      ...
                      


                      i reproduce both described problems.

                      Regards,
                      PM

                      • 8. Re: RoundRobin, new cluster member & SFSB == NPE
                        Ben Wang Master

                        OK, getting rid of sleep interval, I encountered Socket reset exception. I think this is coming from JBoss Remoting. So I will try to see if I can upgrade it to release 2.0.

                        However, I still don't see your exception though.

                        • 9. Re: RoundRobin, new cluster member & SFSB == NPE
                          Przemyslaw Madzik Newbie

                          Hello,

                          Have you any suggestions what may be wrong in my configuration?

                          Are you sure that one may exclude some kind of race condition in JBoss (Cache) code?

                          Regards,
                          PM