3 Replies Latest reply on Apr 7, 2008 11:00 PM by brian.stansberry

    How do servlets communicate with session beans across cluste

    yugantshah

      Hello,

      I am working on a project wherein I have to access beans that are being deployed in different clusters.

      Consider Cluster A containing web components.
      Cluster B containing EJB components.

      How do I lookup the session bean from the servlet.
      Is there any specific configuration required.
      Is there any document/ebook/website that can give information on this.
      Any help on this would be appreciated.

      Thanks in advance.
      Yugant Shah

        • 1. Re: How do servlets communicate with session beans across cl
          brian.stansberry

          Inside your servlet, you need to do a JNDI lookup to the other cluster. Probably best thing is to use HA-JNDI with autodiscovery enabled so you aren't forced to statically list the JNDI hostnames/ports of the ejb cluster members:

          Properties p = new Properties();
          p.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory");
          p.put(Context.URL_PKG_PREFIXES, "jboss.naming:org.jnp.interfaces");
          // Ensure we only find servers on the "EJBPartition"
          p.put("jnp.partitionName", "EJBPartition");
          Context ctx = new InitialContext(p);
          MyEJB ejb = (MyEJB) ctx.lookup("myEjbName");
          


          See also http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossHAJNDIUseCluster . Note that the examples on that page describe using "localhost:1100" which is based on the idea you want to find the HA-JNDI for the same cluster the client is in. If you were to use the jboss-web.xml resource-ref mapping described on that page, for the jndi-name element you would have to replace localhost:1100 with the hostname:port of *one* of the "EJBPartition" servers. Problem with doing that is if that server isn't running for some reason, you won't be able to find any of the other servers and can't do the lookup, even though other servers in the EJBPartition are running. To avoid that problem, don't try and use resource-ref mapping; just do the lookup programatically as shown above.

          • 2. Re: How do servlets communicate with session beans across cl
            yugantshah

            Hello,
            Thanks for reply.

            I have done the look up programmatically as said below.

            The code works.

            I have one more issue, while testing one scenario:
            The current configuration is:

            I am having Two clusters clusterA and clusterB

            clusterA is having the war file and this is deployed one separate machine(say machine X) in a JBOSS instance.

            clusterB is having the ear. This is deployed as farm deployment and the there are two JBOSS instances on separete machines.(say machine Y and Z)
            So, there are three machines each having JBOSS instance.
            Further, Apache HTTP server is deployed on machine X
            Cluster A has one node and B has two nodes.

            The test scenario is :
            we have a JSP page which makes a call to the session bean.
            The session bean has a for loop that counts and prints counter till 40.
            We hit one request
            1. One node in clusterB starts processing(Node Y)
            2. Now, we down this node.
            3. The request is shifted to other node(Node Z)
            4. We start the previous node(Node Y) and then close the node(Node Z) which is processing
            Expected. The node A should again start processing the request
            Actual: error occurs on the node X of clusterA i.e where the war file is deployed.

            NOTE:
            Also, pLease confirm if Jboss handles such test scenario.
            Is there any limit while toggling the request between the nodes.
            Error:

            008-04-01 13:31:55,340 WARN [org.jgroups.protocols.UDP] discarded message from different group "clusterB-EntityCache" (our group is "clusterA-EntityCache"). Sender was 192.168.0.15:2163
            2008-04-01 13:31:56,068 DEBUG [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[a554da, socket://192.168.0.13:3873] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1192 ms
            2008-04-01 13:31:56,068 DEBUG [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[a554da, socket://192.168.0.13:3873] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1192 ms
            2008-04-01 13:31:56,068 DEBUG [org.jboss.remoting.InvokerRegistry] decremented SocketClientInvoker[a554da, socket://192.168.0.13:3873]'s count, current count 1
            2008-04-01 13:31:56,068 WARN [com.ecebs.hops.messaging.MessagingRequestHandlerServlet] Exception occured while executing the Download message request
            java.lang.RuntimeException: cluster invocation failed, last exception was:
            at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:166)
            at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:67)
            at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:53)
            at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            at org.jboss.ejb3.remoting.ClusteredIsLocalInterceptor.invoke(ClusteredIsLocalInterceptor.java:55)
            at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            at org.jboss.ejb3.stateless.StatelessClusteredProxy.invoke(StatelessClusteredProxy.java:112)
            at $Proxy87.handleDownloadMessageRequest(Unknown Source)
            at com.ecebs.hops.messaging.MessagingRequestHandlerServlet.doGet(MessagingRequestHandlerServlet.java:161)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
            at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
            at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
            at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
            at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
            at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:437)
            at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:366)
            at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
            at java.lang.Thread.run(Thread.java:595)
            Caused by: org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [socket://192.168.0.13:3873/]
            at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:532)
            at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
            at org.jboss.remoting.Client.invoke(Client.java:1634)
            at org.jboss.remoting.Client.invoke(Client.java:548)
            at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:62)
            at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:77)
            ... 30 more
            Caused by: java.net.ConnectException: Connection refused: connect
            at java.net.PlainSocketImpl.socketConnect(Native Method)
            at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
            at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
            at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
            at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
            at java.net.Socket.connect(Socket.java:520)
            at org.jboss.remoting.transport.socket.SocketClientInvoker.createSocket(SocketClientInvoker.java:187)
            at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:815)
            at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:525)
            at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
            at org.jboss.remoting.Client.invoke(Client.java:1634)
            at org.jboss.remoting.Client.invoke(Client.java:548)
            at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:62)
            at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:77)
            at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:67)
            at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:53)
            at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            at org.jboss.ejb3.remoting.ClusteredIsLocalInterceptor.invoke(ClusteredIsLocalInterceptor.java:55)
            at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            at org.jboss.ejb3.stateless.StatelessClusteredProxy.invoke(StatelessClusteredProxy.java:112)
            at $Proxy87.handleDownloadMessageRequest(Unknown Source)
            at com.ecebs.hops.messaging.MessagingRequestHandlerServlet.doGet(MessagingRequestHandlerServlet.java:161)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
            at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
            at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
            at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
            at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
            at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:437)
            at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:366)
            at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
            at java.lang.Thread.run(Thread.java:595)
            at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:74)
            ... 32 more
            2008-04-01 13:31:56,068 DEBUG [org.jboss.remoting.InvokerRegistry] removed SocketClientInvoker[a554da, socket://192.168.0.13:3873] from registry
            2008-04-01 13:31:56,068 DEBUG [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[a554da, socket://192.168.0.13:3873] disconnecting ...
            2008-04-01 13:31:56,068 WARN [com.ecebs.hops.messaging.MessagingRequestHandlerServlet] Exception occured while executing the Download message request
            java.lang.RuntimeException: cluster invocation failed, last exception was:
            at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:166)
            at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:67)
            at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:53)
            at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            at org.jboss.ejb3.remoting.ClusteredIsLocalInterceptor.invoke(ClusteredIsLocalInterceptor.java:55)
            at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            at org.jboss.ejb3.stateless.StatelessClusteredProxy.invoke(StatelessClusteredProxy.java:112)
            at $Proxy87.handleDownloadMessageRequest(Unknown Source)
            at com.ecebs.hops.messaging.MessagingRequestHandlerServlet.doGet(MessagingRequestHandlerServlet.java:161)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
            at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
            at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
            at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
            at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
            at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:437)
            at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:366)
            at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
            at java.lang.Thread.run(Thread.java:595)
            Caused by: org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [socket://192.168.0.13:3873/]
            at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:532)
            at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
            at org.jboss.remoting.Client.invoke(Client.java:1634)
            at org.jboss.remoting.Client.invoke(Client.java:548)
            at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:62)
            at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:77)
            ... 30 more
            Caused by: java.net.ConnectException: Connection refused: connect
            at java.net.PlainSocketImpl.socketConnect(Native Method)
            at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
            at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
            at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
            at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
            at java.net.Socket.connect(Socket.java:520)
            at org.jboss.remoting.transport.socket.SocketClientInvoker.createSocket(SocketClientInvoker.java:187)
            at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:815)
            at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:525)
            at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
            at org.jboss.remoting.Client.invoke(Client.java:1634)
            at org.jboss.remoting.Client.invoke(Client.java:548)
            at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:62)
            at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:77)
            at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:67)
            at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:53)
            at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            at org.jboss.ejb3.remoting.ClusteredIsLocalInterceptor.invoke(ClusteredIsLocalInterceptor.java:55)
            at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            at org.jboss.ejb3.stateless.StatelessClusteredProxy.invoke(StatelessClusteredProxy.java:112)
            at $Proxy87.handleDownloadMessageRequest(Unknown Source)
            at com.ecebs.hops.messaging.MessagingRequestHandlerServlet.doGet(MessagingRequestHandlerServlet.java:161)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
            at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
            at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
            at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
            at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
            at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:437)
            at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:366)
            at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
            at java.lang.Thread.run(Thread.java:595)
            at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:74)
            ... 32 more
            2008-04-01 13:31:56,332 WARN [org.jgroups.protocols.UDP] discarded message from different group "clusterB-EntityCache" (our group is "clusterA-EntityCache"). Sender was 192.168.0.15:2163
            2008-04-01 13:31:57,199 WARN [org.jgroups.protocols.UDP] discarded message from different group "Tomcat-clusterB" (our group is "Tomcat-clusterA"). Sender was 192.168.0.15:2146
            2008-04-01 13:31:58,190 WARN [org.jgroups.protocols.UDP] discarded message from different group "Tomcat-clusterB" (our group is "Tomcat-clusterA"). Sender was 192.168.0.15:2146
            2008-04-01 13:32:03,782 DEBUG [org.jboss.web.tomcat.service.session.JBossCacheManager] Looking for sessions that have expired ...
            2008-04-01 13:32:04,804 WARN [org.jgroups.protocols.UDP] discarded message from different group "clusterB" (our group is "clusterA"). Sender was 192.168.0.15:2152

            Thanks in advance,
            Yugant Shah

            • 3. Re: How do servlets communicate with session beans across cl
              brian.stansberry

              What's I believe is happening here is as follows:

              When you look up your SLSB in JNDI, the servlet actually downloads a cluster-aware proxy to the EJB. The proxy includes information about the topology for that EJB, i.e. that it is deployed on {Y, Z}.

              That topology gets updated 1) whenever you make a request that fails (node that failed is remove from list on client side) 2) if you make a request that succeeds and the server detects that your client side proxy is out of sync with what the server knows 3) if you download a new proxy by doing a new JNDI lookup.

              So, let's go through your scenario, showing what the client sees as the cluster topology, and what the server side situation actually is.

              Legend for the following:

              Action : { client-side view of topology } : {actual server-side topology}

              So:

              1) download proxy via JNDI lookup : { Y, Z } : { Y, Z}
              2) make request: { Y, Z } : { Y, Z}
              3) kill Y, request fails over to Z: { Z } : { Z}
              4) start Y, request to Z ongoing: { Z} : { Y, Z }
              5) kill Z, ongoing request to Z aborted: {} : { Y, Z }

              In step 4/5, your client never made a new request to the cluster so it was never informed that Y was available. When Z fails, the client sees no available servers and fails.

              If you had made a new request to Z after step 4 but before 5, you would have gotten a new topology { Y, Z }, and then failover would have worked in step 5. If you did a new JNDI lookup you'd also get an updated topology.

              Our EJB2 clustering code has a feature called the RetryInterceptor that can deal with this situation. See http://wiki.jboss.org/wiki/RetryInterceptor .

              There is no analogue yet for EJB3, as the way the proxies work is different. JIRA for adding one is http://jira.jboss.com/jira/browse/EJBTHREE-1002 .