0 Replies Latest reply on Apr 5, 2004 5:38 AM by Marco

    SFSB prolem during state replication in HA cluster ?

    Marco Newbie

      Hi folks,

      i'm writing a personal proof of concept focused on Session EJB high availability, this proof is based on two simple Session EJB the first is a stateless and the second, obviously, stateful with a simple conversation logic; source code and deplyment descriptors follows.

      To test EJB HA i've defined a JBoss cluster configuration made of three logical nodes running into DefaultPartition; a front end node FE1 running only a web application as client of remote EJBs; the other two back end nodes, BE1 and BE2, holds the same EJB package with clusterable tags sets to true.

      The cluster, only for development purpose, is running onto a unique machine; configurations follows. The same cluster configuration has been tested in Linux (RH9) and Win2K with identical "unexpected" results.

      Testing a front end node FE1 with Web app using HA-JNDI and only one back end node, say BE1, with EJB app, the cluster works fine with both kind of EJBs; but when turn ON the second EJB back end node BE2 ... the new BE2 node is recognized by the cluster, the stateless EJBs are still well balanced using default RoundRobin policy, the stateful EJB ... lets the cluster go 'crazy' in a unrepeatable way; it rarely works, the most fails; it seems to create an infinite number of stateful EJB instances going into a stack overflow exception on the other node respect the one is about to serve the request.

      Scenario example:

      1- the client on FE1 create a remote instance, the home created by HA-JNDI serves a new remote instance from BE1,
      2- the client start and consume a conversation with the remote instance, the EJB generates a random number,
      3- the client executes 'n' consuming calls to terminate the conversation then remove the remote instance,

      this is an excerpts from trace ...

      [All traces are from the Win2K run test, the Linux test results are equals ... :-( i was hoping in a Gates mistake !!!!]

      ... from BE2 node ...
      10:58:59,673 INFO [Server] JBoss (MX MicroKernel) [3.2.3 (build: CVSTag=JBoss_3_2_3 date=200311301445)] Started in 40s:87ms
      10:59:28,124 INFO [STDOUT] CTOR: TestHAEjbStatefulBean
      10:59:28,134 INFO [STDOUT] setSessionContext
      10:59:28,134 INFO [STDOUT] ejbCreate: bean [bean-one] with [1] items to consume.
      10:59:28,144 INFO [STDOUT] ejbPassivate
      10:59:28,164 INFO [STDOUT] getHandle: org.jboss.proxy.ejb.handle.StatefulHandleImpl@104a4b6
      10:59:28,204 INFO [STDOUT] ejbActivate
      10:59:28,274 INFO [STDOUT] whoAreYou: bean-one, requester: alfa
      10:59:28,274 INFO [STDOUT] isModified: false
      10:59:28,274 INFO [STDOUT] getMaxItems: bean-one, maxItems: 1
      10:59:28,274 INFO [STDOUT] isModified: false
      10:59:28,284 INFO [STDOUT] getNextItem: bean-one, nextItem: 1
      10:59:28,284 INFO [STDOUT] isModified: false
      10:59:28,284 INFO [STDOUT] ejbRemove: bean-one
      ...

      ... the other BE1 node still silent ...
      10:58:20,006 INFO [Server] JBoss (MX MicroKernel) [3.2.3 (build: CVSTag=JBoss_3_2_3 date=200311301445)] Started in40s:758ms
      10:58:48,687 INFO [DefaultPartition] New cluster view: 2 ([192.168.1.9:1099, 192.168.1.9:1199, 192.168.1.9:1299] delta: 1)
      10:58:48,697 INFO [DefaultPartition:ReplicantManager] Dead members: 0
      ...

      try a next click on Web app, this is a lucky click !!!

      ... the request hits yet the BE2 node ...
      11:05:13,010 INFO [STDOUT] CTOR: TestHAEjbStatefulBean
      11:05:13,010 INFO [STDOUT] setSessionContext
      11:05:13,010 INFO [STDOUT] ejbCreate: bean [bean-one] with [2] items to consume.
      11:05:13,010 INFO [STDOUT] ejbPassivate
      11:05:13,020 INFO [STDOUT] getHandle: org.jboss.proxy.ejb.handle.StatefulHandleImpl@e9493a
      11:05:13,070 INFO [STDOUT] ejbActivate
      11:05:13,160 INFO [STDOUT] whoAreYou: bean-one, requester: alfa
      11:05:13,160 INFO [STDOUT] isModified: false
      11:05:13,170 INFO [STDOUT] getMaxItems: bean-one, maxItems: 2
      11:05:13,170 INFO [STDOUT] isModified: false
      11:05:13,180 INFO [STDOUT] getNextItem: bean-one, nextItem: 2
      11:05:13,180 INFO [STDOUT] isModified: false
      11:05:13,180 INFO [STDOUT] getNextItem: bean-one, nextItem: 1
      11:05:13,180 INFO [STDOUT] isModified: false
      11:05:13,190 INFO [STDOUT] ejbRemove: bean-one
      ...

      ... the other BE1 node still silent ...
      ...

      try a further click on Web app, this is NOT a lucky click !!!

      ... this time the request is routed to BE1 node ...
      11:05:17,316 INFO [STDOUT] CTOR: TestHAEjbStatefulBean
      11:05:17,326 INFO [STDOUT] setSessionContext
      11:05:17,326 INFO [STDOUT] ejbCreate: bean [bean-one] with [2] items to consume.
      11:05:17,326 INFO [STDOUT] ejbPassivate
      11:05:17,336 INFO [STDOUT] getHandle: org.jboss.proxy.ejb.handle.StatefulHandleImpl@18e8fbd
      11:05:17,376 INFO [STDOUT] ejbActivate
      ...

      and BE1 stops works without say nothing while the BE2 node go crazy ...

      ... from BE2 trace ...
      11:05:17,416 INFO [STDOUT] CTOR: TestHAEjbStatefulBean
      11:05:17,416 INFO [STDOUT] setSessionContext
      11:05:17,466 INFO [STDOUT] CTOR: TestHAEjbStatefulBean
      11:05:17,466 INFO [STDOUT] setSessionContext
      11:05:17,466 INFO [STDOUT] CTOR: TestHAEjbStatefulBean
      11:05:17,466 INFO [STDOUT] setSessionContext

      ::::: ---->>> i've intentionally omitted the whole 1304 trace rows <<<---- ::::::

      11:05:19,059 INFO [STDOUT] CTOR: TestHAEjbStatefulBean
      11:05:19,059 INFO [STDOUT] setSessionContext
      11:05:19,059 INFO [STDOUT] CTOR: TestHAEjbStatefulBean
      11:05:19,059 INFO [STDOUT] setSessionContext
      ... and stops works ...

      consequently the Web app receive a RemoteException.

      ... from front end FE1 trace ...
      10:57:34,160 INFO [Server] JBoss (MX MicroKernel) [3.2.3 (build: CVSTag=JBoss_3_2_3 date=200311301445)] Started in 35s:210ms
      10:58:08,039 INFO [DefaultPartition] New cluster view (id: 1, delta: 1) : [192.168.1.9:1099, 192.168.1.9:1199]
      10:58:08,039 INFO [DefaultPartition:ReplicantManager] Dead members: 0
      10:58:48,697 INFO [DefaultPartition] New cluster view (id: 2, delta: 1) : [192.168.1.9:1099, 192.168.1.9:1199, 192.168.1.9:1299]
      10:58:48,697 INFO [DefaultPartition:ReplicantManager] Dead members: 0
      10:59:27,914 INFO [STDOUT] Using: java.naming.provider.url = localhost:1100
      10:59:27,914 INFO [STDOUT] Using: java.naming.factory.initial = org.jnp.interfaces.NamingContextFactory
      10:59:27,924 INFO [STDOUT] Using: result jsp = result.jsp
      10:59:28,094 INFO [STDOUT] name: alfa beanName: bean-one strLoops: 1
      11:05:12,980 INFO [STDOUT] name: alfa beanName: bean-one strLoops: 1
      11:05:17,296 INFO [STDOUT] name: alfa beanName: bean-one strLoops: 1
      11:05:19,389 ERROR [STDERR] java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
      java.rmi.NoSuchObjectException: Could not activate; CausedByException is
      :
      Could not get EJBObject; nested exception is:
      java.rmi.NoSuchObjectException: Could not activate; CausedByException is
      :
      Could not get EJBObject; nested exception is:
      java.rmi.NoSuchObjectException: Could not activate; CausedByException is
      :

      ::::: ---->>> i've intentionally omitted the whole identical trace rows <<<---- ::::::

      :
      Could not get EJBObject; nested exception is:
      java.rmi.NoSuchObjectException: Could not activate; CausedByException is

      Could not get EJBObject; nested exception is:
      java.rmi.NoSuchObjectException: Could not activate; CausedByException is
      :
      Could not get EJBObject; nested exception is:
      RuntimeErrorException: Error in MBean operation 'invoke(org.jboss.invocation.Invocation)' Cause: java.lang.StackOverflowError
      11:05:19,409 ERROR [STDERR] at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:292)
      11:05:19,409 ERROR [STDERR] at sun.rmi.transport.Transport$1.run(Transport.java:148)
      11:05:19,409 ERROR [STDERR] at java.security.AccessController.doPrivileged(Native Method)
      11:05:19,409 ERROR [STDERR] at sun.rmi.transport.Transport.serviceCall(Transport.java:144)
      11:05:19,409 ERROR [STDERR] at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460)
      11:05:19,409 ERROR [STDERR] at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701)
      11:05:19,409 ERROR [STDERR] at java.lang.Thread.run(Thread.java:534)
      11:05:19,409 ERROR [STDERR] at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:247)
      11:05:19,409 ERROR [STDERR] at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:223)
      11:05:19,409 ERROR [STDERR] at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:133)
      11:05:19,409 ERROR [STDERR] at org.jboss.invocation.jrmp.server.JRMPInvoker_Stub.invoke(Unknown Source)
      11:05:19,409 ERROR [STDERR] at org.jboss.invocation.jrmp.interfaces.JRMPInvokerProxyHA.invoke(JRMPInvokerProxyHA.java:189)
      11:05:19,409 ERROR [STDERR] at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:96)
      11:05:19,409 ERROR [STDERR] at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
      11:05:19,409 ERROR [STDERR] at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:45)
      11:05:19,409 ERROR [STDERR] at org.jboss.proxy.ejb.StatefulSessionInterceptor.invoke(StatefulSessionInterceptor.java:104)
      11:05:19,409 ERROR [STDERR] at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:85)
      11:05:19,409 ERROR [STDERR] at $Proxy45.remove(Unknown Source)
      11:05:19,409 ERROR [STDERR] at it.ibm.com.it42021.test.web.ServletTestHAJBossEjbStateful.execute(ServletTestHAJBossEjbStateful.java:164)
      11:05:19,409 ERROR [STDERR] at it.ibm.com.it42021.test.web.ServletTestHAJBossEjbStateful.doGet(ServletTestHAJBossEjbStateful.java:53)
      11:05:19,409 ERROR [STDERR] at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
      11:05:19,409 ERROR [STDERR] at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
      11:05:19,409 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
      11:05:19,409 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
      11:05:19,409 ERROR [STDERR] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
      11:05:19,409 ERROR [STDERR] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
      11:05:19,409 ERROR [STDERR] at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
      11:05:19,409 ERROR [STDERR] at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
      11:05:19,419 ERROR [STDERR] at org.jboss.web.tomcat.security.JBossSecurityMgrRealm.invoke(JBossSecurityMgrRealm.java:220)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.valves.CertificatesValve.invoke(CertificatesValve.java:246)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
      11:05:19,419 ERROR [STDERR] at org.jboss.web.tomcat.tc4.statistics.ContainerStatsValve.invoke(ContainerStatsValve.java:76)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
      11:05:19,419 ERROR [STDERR] at org.jboss.web.tomcat.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:78)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2417)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
      11:05:19,419 ERROR [STDERR] at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:65)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:577)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
      11:05:19,419 ERROR [STDERR] at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
      11:05:19,419 ERROR [STDERR] at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:197)
      11:05:19,419 ERROR [STDERR] at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:781)
      11:05:19,419 ERROR [STDERR] at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:549)
      11:05:19,419 ERROR [STDERR] at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:605)
      11:05:19,419 ERROR [STDERR] at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:677)
      11:05:19,419 ERROR [STDERR] at java.lang.Thread.run(Thread.java:534)
      11:05:19,429 ERROR [STDERR] Caused by: java.rmi.NoSuchObjectException: Could not activate; CausedByException is:
      Could not get EJBObject; nested exception is:
      java.rmi.NoSuchObjectException: Could not activate; CausedByException is
      :
      Could not get EJBObject; nested exception is:
      java.rmi.NoSuchObjectException: Could not activate; CausedByException is
      :

      ::::: ---->>> i've intentionally omitted the whole identical trace rows <<<---- ::::::

      :
      Could not get EJBObject; nested exception is:
      java.rmi.NoSuchObjectException: Could not activate; CausedByException is
      :
      Could not get EJBObject; nested exception is:
      RuntimeErrorException: Error in MBean operation 'invoke(org.jboss.invocation.Invocation)' Cause: java.lang.StackOverflowError
      11:05:19,429 ERROR [STDERR] at org.jboss.ejb.plugins.StatefulHASessionInstanceCache.get(StatefulHASessionInstanceCache.java:109)
      11:05:19,429 ERROR [STDERR] at org.jboss.ejb.plugins.StatefulSessionInstanceInterceptor.invoke(StatefulSessionInstanceInterceptor.java:210)
      11:05:19,429 ERROR [STDERR] at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:84)
      11:05:19,429 ERROR [STDERR] at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:267)
      11:05:19,429 ERROR [STDERR] at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:128)
      11:05:19,439 ERROR [STDERR] at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:191)
      11:05:19,439 ERROR [STDERR] at org.jboss.ejb.plugins.CleanShutdownInterceptor.invoke(CleanShutdownInterceptor.java:264)
      11:05:19,439 ERROR [STDERR] at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
      11:05:19,439 ERROR [STDERR] at org.jboss.ejb.StatefulSessionContainer.internalInvoke(StatefulSessionContainer.java:416)
      11:05:19,439 ERROR [STDERR] at org.jboss.ejb.Container.invoke(Container.java:700)
      11:05:19,439 ERROR [STDERR] at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
      11:05:19,439 ERROR [STDERR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      11:05:19,439 ERROR [STDERR] at java.lang.reflect.Method.invoke(Method.java:324)
      11:05:19,439 ERROR [STDERR] at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
      11:05:19,439 ERROR [STDERR] at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:546)
      11:05:19,439 ERROR [STDERR] at org.jboss.invocation.jrmp.server.JRMPInvokerHA.invoke(JRMPInvokerHA.java:163)
      11:05:19,439 ERROR [STDERR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      11:05:19,439 ERROR [STDERR] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      11:05:19,439 ERROR [STDERR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      11:05:19,439 ERROR [STDERR] at java.lang.reflect.Method.invoke(Method.java:324)
      11:05:19,439 ERROR [STDERR] at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261)
      11:05:19,439 ERROR [STDERR] at sun.rmi.transport.Transport$1.run(Transport.java:148)
      11:05:19,439 ERROR [STDERR] at java.security.AccessController.doPrivileged(Native Method)
      11:05:19,439 ERROR [STDERR] at sun.rmi.transport.Transport.serviceCall(Transport.java:144)
      11:05:19,439 ERROR [STDERR] at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460)
      11:05:19,439 ERROR [STDERR] at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701)
      11:05:19,439 ERROR [STDERR] ... 1 more
      ...

      the source code and configuration only for stateful EJB

      ========================= TestHAEjbStatefulBean.java ================================================
      package it.ibm.com.it42021.test.ejb;

      import java.net.InetAddress;
      import java.net.UnknownHostException;
      import java.rmi.RemoteException;
      import java.util.Random;

      import javax.ejb.EJBException;
      import javax.ejb.EJBHome;
      import javax.ejb.EJBObject;
      import javax.ejb.Handle;
      import javax.ejb.RemoveException;
      import javax.ejb.SessionBean;
      import javax.ejb.SessionContext;

      /**
      * @author ....
      */
      public class TestHAEjbStatefulBean implements SessionBean, TestHAEjbStatefulRemote {

      public SessionContext ctx = null;
      public String beanName = null;
      public int maxItems = 0;
      public int nextItem = 0;

      public TestHAEjbStatefulBean() {
      super();
      System.out.println("CTOR: TestHAEjbStatefulBean");
      }

      public void ejbActivate() throws EJBException, RemoteException {
      System.out.println("ejbActivate");
      }

      public void ejbPassivate() throws EJBException, RemoteException {
      System.out.println("ejbPassivate");
      }

      public void ejbRemove() throws EJBException, RemoteException {
      System.out.println("ejbRemove: "+beanName);
      }

      public void setSessionContext(SessionContext ctx) throws EJBException, RemoteException {
      this.ctx = ctx;
      System.out.println("setSessionContext");
      }

      public EJBHome getEJBHome() throws RemoteException {
      EJBHome home = null;

      if ( ctx != null ) {
      home = ctx.getEJBHome();
      }
      System.out.println("getEJBHome: "+home);

      return home;
      }

      public Handle getHandle() throws RemoteException {
      Handle handle = null;

      if ( ctx != null ) {
      EJBObject object = ctx.getEJBObject();
      if ( object != null ) {
      handle = object.getHandle();
      }
      }
      System.out.println("getHandle: "+handle);
      return handle;
      }

      public Object getPrimaryKey() throws RemoteException {
      Object key = null;

      if ( ctx != null ) {
      EJBObject object = ctx.getEJBObject();
      if ( object != null ) {
      key = object.getPrimaryKey();
      }
      }
      System.out.println("getPrimaryKey: "+key);
      return key;
      }

      public boolean isIdentical(EJBObject obj) throws RemoteException {
      System.out.println("isIdentical: "+obj);
      return false;
      }

      public void remove() throws RemoteException, RemoveException {
      System.out.println("remove: "+beanName);
      }

      public void ejbCreate(String beanName) {
      this.beanName = beanName;
      maxItems = new Random().nextInt( 3 ) + 1;
      nextItem = maxItems;
      System.out.println("ejbCreate: bean ["+ beanName + "] with [" +maxItems+"] items to consume.");
      }

      public String whoAreYou(String requester) throws RemoteException {
      String serverName = null;
      String answer = null;

      try {
      serverName = InetAddress.getLocalHost().getHostName();
      } catch (UnknownHostException e) {
      try {
      serverName = InetAddress.getLocalHost().getHostAddress();
      } catch (UnknownHostException e1) {
      serverName = "<unknown host>";
      }
      }
      answer = "[from requester="+requester+"] bean ["+beanName+"] from server ["+serverName+"]";

      System.out.println("whoAreYou: "+beanName+", requester: "+requester);

      return answer;
      }

      public int getMaxItems() throws RemoteException {
      System.out.println("getMaxItems: "+beanName+", maxItems: "+maxItems);
      return maxItems;
      }

      public int getNextItem() throws RemoteException {
      if ( nextItem < 0 ) {
      throw new RemoteException("No more items.");
      }
      System.out.println("getNextItem: "+beanName+", nextItem: "+nextItem);
      return nextItem--;
      }

      // JBoss 3.2.x
      public boolean isModified() {
      System.out.println("isModified: false");
      return false;
      }
      }

      ======================== TestHAEjbStatefulHomeRemote =====================================================
      package it.ibm.com.it42021.test.ejb;

      import java.rmi.RemoteException;

      import javax.ejb.CreateException;
      import javax.ejb.EJBHome;

      /**
      * @author ...
      */
      public interface TestHAEjbStatefulHomeRemote extends EJBHome {

      public TestHAEjbStatefulRemote create(String beanName) throws RemoteException, CreateException;

      }
      ======================== TestHAEjbStatefulRemote =====================================================
      package it.ibm.com.it42021.test.ejb;

      import java.rmi.RemoteException;

      import javax.ejb.EJBObject;

      /**
      * @author ...
      */
      public interface TestHAEjbStatefulRemote extends EJBObject {

      public String whoAreYou(String requester) throws RemoteException;
      public int getMaxItems() throws RemoteException;
      public int getNextItem() throws RemoteException;
      }

      ========================== ejb-jar.xml =====================================================

      <?xml version="1.0"?>

      <!DOCTYPE ejb-jar PUBLIC
      "-//Sun Microsystems, Inc.//DTD Enterprise JavaBeans 2.0//EN"
      "http://java.sun.com/dtd/ejb-jar_2_0.dtd">

      <ejb-jar>

      <enterprise-beans>


      <ejb-name>TestHAEjbStateless</ejb-name>

      it.ibm.com.it42021.test.ejb.TestHAEjbStatelessHomeRemote
      it.ibm.com.it42021.test.ejb.TestHAEjbStatelessRemote
      <ejb-class>it.ibm.com.it42021.test.ejb.TestHAEjbStatelessBean</ejb-class>
      <session-type>Stateless</session-type>
      <transaction-type>Container</transaction-type>
      <security-identity><use-caller-identity/></security-identity>




      <ejb-name>TestHAEjbStateful</ejb-name>

      it.ibm.com.it42021.test.ejb.TestHAEjbStatefulHomeRemote
      it.ibm.com.it42021.test.ejb.TestHAEjbStatefulRemote
      <ejb-class>it.ibm.com.it42021.test.ejb.TestHAEjbStatefulBean</ejb-class>
      <session-type>Stateful</session-type>
      <transaction-type>Container</transaction-type>
      <security-identity><use-caller-identity/></security-identity>



      </enterprise-beans>

      </ejb-jar>
      ========================== jboss.xml =====================================================
      <?xml version="1.0"?>

      <!-- Esempio di descrittore per EJB in HA -->


      <enterprise-beans>


      <ejb-name>TestHAEjbStateless</ejb-name>
      <jndi-name>ejb/TestHAEjbStateless</jndi-name>

      true




      <ejb-name>TestHAEjbStateful</ejb-name>
      <jndi-name>ejb/TestHAEjbStateful</jndi-name>

      true

      <!-- i've tried with and without spefic infos about cluster-config -->
      <cluster-config>
      <partition-name>DefaultPartition</partition-name>
      <home-load-balance-policy>org.jboss.ha.framework.interfaces.RoundRobin</home-load-balance-policy>

      <!-- i've tried all possible combinations -->
      <bean-load-balance-policy>org.jboss.ha.framework.interfaces.RoundRobin</bean-load-balance-policy>
      <!-- bean-load-balance-policy>org.jboss.ha.framework.interfaces.FirstAvailable</bean-load-balance-policy -->
      <!-- bean-load-balance-policy>org.jboss.ha.framework.interfaces.FirstAvailableIdenticalAllProxies</bean-load-balance-policy -->

      <session-state-manager-jndi-name>/HASessionState/Default</session-state-manager-jndi-name>
      </cluster-config>



      </enterprise-beans>


      ============================= ServletTestHAJBossEjbStateful ==========================================
      package it.ibm.com.it42021.test.web;


      import it.ibm.com.it42021.test.ejb.TestHAEjbStatefulHomeRemote;
      import it.ibm.com.it42021.test.ejb.TestHAEjbStatefulRemote;
      import it.ibm.com.it42021.test.ejb.TestHAEjbStatelessHomeRemote;

      import java.io.IOException;
      import java.rmi.RemoteException;
      import java.util.Properties;

      import javax.ejb.CreateException;
      import javax.ejb.RemoveException;
      import javax.naming.Context;
      import javax.naming.InitialContext;
      import javax.naming.NamingException;
      import javax.servlet.ServletConfig;
      import javax.servlet.ServletException;
      import javax.servlet.http.HttpServlet;
      import javax.servlet.http.HttpServletRequest;
      import javax.servlet.http.HttpServletResponse;

      /**
      * @author ...
      */
      public class ServletTestHAJBossEjbStateful extends HttpServlet {

      private static final String _paramNameProviderUrl = "providerUrl";
      private static final String _paramNameResultJsp = "resultJsp";
      private static final String _defaultJBossHAJNDIServer = "localhost:1100";
      private static final String _defaultJBossNCtxFactory = "org.jnp.interfaces.NamingContextFactory";
      private static final String _ejbJndiName = "ejb/TestHAEjbStateful";
      private static final String _requesterName = "requesterName";
      private static final String _beanName = "beanName";
      private static final String _loops = "loops";
      private static final String _answerBean = "answerBean";
      private static String _resultJsp = "result.jsp";

      private InitialContext ctx = null;
      private TestHAEjbStatefulHomeRemote home = null;
      private TestHAEjbStatefulRemote bean = null;

      public ServletTestHAJBossEjbStateful() {
      super();
      }

      protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
      execute(request, response);
      }

      protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
      execute(request, response);
      }

      public void destroy() {
      super.destroy();
      if (ctx != null) {
      try {
      ctx.close();
      } catch (NamingException e) {
      }
      }
      }

      public void init(ServletConfig config) throws ServletException {
      super.init(config);

      String providerUrl = null;
      String param = null;

      param = config.getInitParameter(_paramNameProviderUrl);
      if (param != null) {
      providerUrl = param;
      } else {
      // default HA-JNDI for JBoss
      providerUrl = _defaultJBossHAJNDIServer;
      }

      param = config.getInitParameter(_paramNameResultJsp);
      if (param != null) {
      _resultJsp = param;
      }

      Properties jndiProps = new Properties();
      jndiProps.put(Context.PROVIDER_URL, providerUrl);
      jndiProps.put(Context.INITIAL_CONTEXT_FACTORY, _defaultJBossNCtxFactory);

      System.out.println("Using: " + Context.PROVIDER_URL+" = "+providerUrl);
      System.out.println("Using: " + Context.INITIAL_CONTEXT_FACTORY+" = "+_defaultJBossNCtxFactory);
      System.out.println("Using: result jsp = "+_resultJsp);

      try {
      ctx = new InitialContext(jndiProps);

      // get home here to use balancing
      home = (TestHAEjbStatefulHomeRemote) ctx.lookup(_ejbJndiName);

      } catch (NamingException e) {
      throw new ServletException(e);
      }
      }

      protected void execute(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
      String result = null;
      String name = request.getParameter(_requesterName);
      String beanName = request.getParameter(_beanName);
      String strLoops = request.getParameter(_loops);

      System.out.println("name: "+name+" beanName: "+beanName+" strLoops: "+strLoops);

      int loops = 1;
      try {
      loops = Integer.parseInt(strLoops);
      } catch (NumberFormatException e) {
      }

      if (name == null) {
      name = "anonimo";
      }

      if (beanName == null) {
      beanName = "bean anonimo";
      }

      StringBuffer sb = new StringBuffer();
      sb.append("<table border=\"1\">");

      bean = null;
      try {

      for (int i = 0; i < loops; i++) {
      bean = home.create(beanName);
      result = testEjb(name);
      sb.append(""+(i+1)+"");
      sb.append(result);
      sb.append("");
      try {
      bean.remove();
      } catch (RemoveException e2) {
      e2.printStackTrace();
      }
      bean = null;
      }
      } catch (RemoteException e1) {
      sb.append(""+e1.getMessage()+"");
      } catch (CreateException e1) {
      sb.append(""+e1.getMessage()+"");
      } finally {
      if ( bean != null ) {
      try {
      bean.remove();
      } catch (RemoteException e2) {
      e2.printStackTrace();
      } catch (RemoveException e2) {
      e2.printStackTrace();
      }
      }
      }

      sb.append("");

      request.setAttribute(_answerBean, sb.toString());

      request.getRequestDispatcher(_resultJsp).forward(request, response);
      }

      private String testEjb(String name) {
      String result = null;

      try {
      result = bean.whoAreYou(name);
      int max = bean.getMaxItems();
      result += " max items ["+max+"]";
      for (int i = 0; i < max; i++) {
      result += " <"+bean.getNextItem()+">";
      }
      } catch (RemoteException e) {
      result = e.getMessage();
      }

      return result;
      }
      }

      ================= cluster-service.xml (from BE1 configs, BE2 is identical) =======================================

      <?xml version="1.0" encoding="UTF-8"?>

      <!-- ===================================================================== -->
      <!-- -->
      <!-- Sample Clustering Service Configuration -->
      <!-- -->
      <!-- ===================================================================== -->





      <!-- ==================================================================== -->
      <!-- Cluster Partition: defines cluster -->
      <!-- ==================================================================== -->



      <!-- Name of the partition being built -->
      DefaultPartition
      <!-- Determine if deadlock detection is enabled -->
      False
      <!-- The JGroups protocol configuration -->


      <!-- UDP: if you have a multihomed machine,
      set the bind_addr attribute to the appropriate NIC IP address -->
      bind_addr="192.168.1.9"
      <!-- UDP: On Windows machines, because of the media sense feature
      being broken with multicast (even after disabling media sense)
      set the loopback attribute to true -->
      <UDP mcast_addr="228.1.2.3" mcast_port="45566"
      ip_ttl="64" ip_mcast="true"
      mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
      ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
      loopback="true" />
      <PING timeout="2000" num_initial_members="3"
      up_thread="true" down_thread="true" />
      <MERGE2 min_interval="5000" max_interval="10000" />
      <FD shun="true" up_thread="true" down_thread="true"
      timeout="2500" max_tries="5" />
      <VERIFY_SUSPECT timeout="3000" num_msgs="3"
      up_thread="true" down_thread="true" />
      <pbcast.NAKACK gc_lag="50" retransmit_timeout="300,600,1200,2400,4800"
      up_thread="true" down_thread="true" />
      <pbcast.STABLE desired_avg_gossip="20000"
      up_thread="true" down_thread="true" />
      <UNICAST timeout="5000" window_size="100" min_threshold="10"
      down_thread="true" />
      <FRAG frag_size="8192"
      down_thread="true" up_thread="true" />
      <pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
      shun="true" print_local_addr="true" />
      <pbcast.STATE_TRANSFER up_thread="true" down_thread="true" />




      <!-- ==================================================================== -->
      <!-- HA Session State Service for SFSB -->
      <!-- ==================================================================== -->


      jboss:service=DefaultPartition
      <!-- Name of the partition to which the service is linked -->
      DefaultPartition
      <!-- JNDI name under which the service is bound -->
      /HASessionState/Default
      <!-- Max delay before cleaning unreclaimed state.
      Defaults to 30*60*1000 => 30 minutes -->
      0


      <!-- ==================================================================== -->
      <!-- HA JNDI -->
      <!-- ==================================================================== -->


      jboss:service=DefaultPartition
      <!-- Name of the partition to which the service is linked -->
      DefaultPartition
      <!-- bind address of HA JNDI RMI endpoint -->
      ${jboss.bind.address}
      <!-- RmiPort to be used by the HA-JNDI service
      once bound. 0 => auto. -->
      0
      <!-- Port on which the HA-JNDI stub is made available -->
      11100
      <!-- Backlog to be used for client-server RMI
      invocations during JNDI queries -->
      50

      <!-- Multicast Address and Group used for auto-discovery -->
      230.0.0.4
      1102

      <!-- IP Address to which should be bound: the Port, the RmiPort and
      the AutoDiscovery multicast socket. -->
      <!-- Client socket factory to be used for client-server
      RMI invocations during JNDI queries -->
      <!--attribute name="ClientSocketFactory">custom</attribute-->
      <!-- Server socket factory to be used for client-server
      RMI invocations during JNDI queries -->
      <!--attribute name="ServerSocketFactory">custom</attribute-->




      ${jboss.bind.address}
      <!--
      0
      custom
      custom
      -->


      <!-- ==================================================================== -->
      <!-- Distributed cache invalidation -->
      <!-- ==================================================================== -->


      jboss:service=DefaultPartition
      jboss.cache:service=InvalidationManager
      jboss.cache:service=InvalidationManager
      DefaultPartition
      DefaultJGBridge




      =======================================================================

      ... other useful infos ...
      ===============================================================================================
      == JBoss version
      ===============================================================================================
      ...
      09:58:41,250 INFO [Server] Starting JBoss (MX MicroKernel)...
      09:58:41,260 INFO [Server] Release ID: JBoss [WonderLand] 3.2.3 (build: CVSTag=JBoss_3_2_3 date=200311301445)
      09:58:41,260 DEBUG [Server] Using config: org.jboss.system.server.ServerConfigImpl@ecd7e
      09:58:41,260 DEBUG [Server] Server type: class org.jboss.system.server.ServerImpl
      09:58:41,260 INFO [Server] Home Dir: C:\jboss-cluster
      09:58:41,260 INFO [Server] Home URL: file:/C:/jboss-cluster/
      ...

      ===============================================================================================
      == JGroups version
      ===============================================================================================
      C:\jboss-cluster\server\backend-nodo1\lib>java -cp jgroups.jar org.jgroups.Version

      Version: 2.2.0
      CVS: $Id: Version.java,v 1.1.1.1 2003/09/09 01:24:08 belaban Exp $
      History: (see doc/history.txt for details)

      Test and demo programs works fine.

      A - java org.jgroups.demos.Draw
      B - java org.jgroups.tests.McastReceiverTest -mcast_addr 224.10.10.10 -port 5555
      java org.jgroups.tests.McastSenderTest -mcast_addr 224.10.10.10 -port 5555

      ===============================================================================================

      To execute the cluster on the same machine i've opportunely changed these files for each one of FE1, BE1 and BE2 nodes

      /<jboss-installation-directory>/server/\conf\jboss-minimal.xml
      /<jboss-installation-directory>/server/\conf\jboss-service.xml
      /<jboss-installation-directory>/server/\conf\jacorb.properties
      /<jboss-installation-directory>/server/\deploy\http-invoker.sar\META-INF\jboss-service.xml
      /<jboss-installation-directory>/server/\deploy\jbossweb-tomcat41.sar\META-INF\jboss-service.xml
      /<jboss-installation-directory>/server/\deploy\jms\oil-service.xml
      /<jboss-installation-directory>/server/\deploy\jms\oil2-service.xml
      /<jboss-installation-directory>/server/\deploy\jms\uil2-service.xml
      /<jboss-installation-directory>/server/\deploy\snmp-adaptor.sar\managers.xml
      /<jboss-installation-directory>/server/\deploy\snmp-adaptor.sar\META-INF\jboss-service.xml
      /<jboss-installation-directory>/server/\deploy\cluster-service.xml

      the changes was about ports numbers to avoid conflicts on the same machine IP.
      ===============================================================================================

      The mixed s.o. test (Linux + Win2K) has not be done ... i don't want to win another nightmare !!!

      Until here this may seems a tutorial on how to test a cluster ;-) but ...

      As from correctness of stateless EJB availability and balancing the cluster node communication seem to works,
      the problem 'seems' to be about stateful EJB state propagation, should i configure some specific for sharing EJB states ?

      Is there something wrong in 'cluster-service.xml' configuration ?

      Can anyone advice me on the next serious action to put it at work ?

      Bela, hey Bela ... i'm talking to You ;-)

      P.S. Sorry for a so huge message, but is difficult to describe a so strange behaviour.


      Marco.