4 Replies Latest reply on Oct 21, 2009 12:01 PM by brian.stansberry

Cluster invocation error

mohitanchlia Oct 19, 2009 12:17 PM

We have stateless bean which is clustered. We have more than 8 nodes in a cluster and that bean is supposed to be clustered accross 8 nodes. Problem is that when one of the nodes is in process of going down for maintenance then the user request fails while connecting that node. What I don't understand is why would user requests fail when the bean is supposed to be clustered? Shouldn't the proxy object go to the next node in the cluster and process the request instead of returning an error?

Caused by: java.lang.RuntimeException: cluster invocation failed, last exception was:
 at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:166)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:61)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:53)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.remoting.ClusteredIsLocalInterceptor.invoke(ClusteredIsLocalInterceptor.java:55)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.stateless.StatelessClusteredProxy.invoke(StatelessClusteredProxy.java:112)
 at $Proxy284.findFilingsByCmTokenId(Unknown Source)
 ... 61 more
Caused by: org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [socket://10.4.2.37:3873/?numberOfRetries=1]
 at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:532)
 at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
 at org.jboss.remoting.Client.invoke(Client.java:1634)
 at org.jboss.remoting.Client.invoke(Client.java:548)
 at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:62)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:77)
 ... 70 more
Caused by: java.net.ConnectException: Connection refused
 at java.net.PlainSocketImpl.socketConnect(Native Method)
 at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
 at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
 at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
 at java.net.Socket.connect(Socket.java:516)
 at org.jboss.remoting.transport.socket.SocketClientInvoker.createSocket(SocketClientInvoker.java:187)
 at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:815)
 at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:525)
 at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
 at org.jboss.remoting.Client.invoke(Client.java:1634)
 at org.jboss.remoting.Client.invoke(Client.java:548)
 at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:62)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:77)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:61)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:53)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.remoting.ClusteredIsLocalInterceptor.invoke(ClusteredIsLocalInterceptor.java:55)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.stateless.StatelessClusteredProxy.invoke(StatelessClusteredProxy.java:112)
 at $Proxy284.findFilingsByCmTokenId(Unknown Source)
 at sun.reflect.GeneratedMethodAccessor174.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at com.intuit.engine.efp.efe.filer.FilingProcessorDelegate$FilingProcessorFipProxy.invoke(FilingProcessorDelegate.java:246)
 at $Proxy285.findFilingsByCmTokenId(Unknown Source)
 at com.intuit.engine.efp.efe.filer.FilingProcessorDelegate.findFilingsByCmTokenId(FilingProcessorDelegate.java:120)
 at com.intuit.engine.efp.efe.filer.FipFilerStrategy.sync(FipFilerStrategy.java:529)
 at com.intuit.engine.efp.efe.filer.server.common.FilerService.sync(FilerService.java:222)
 at sun.reflect.GeneratedMethodAccessor173.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at com.sun.xml.ws.api.server.InstanceResolver$1.invoke(InstanceResolver.java:246)
 at com.sun.xml.ws.server.InvokerTube$2.invoke(InvokerTube.java:146)
 at com.sun.xml.ws.server.sei.EndpointMethodHandler.invoke(EndpointMethodHandler.java:257)
 at com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(SEIInvokerTube.java:93)
 at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:598)
 at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:557)
 at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:542)
 at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:439)
 at com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:243)
 at com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:470)
 at com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:244)
 at com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(ServletAdapter.java:135)
 at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doGet(WSServletDelegate.java:129)
 at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(WSServletDelegate.java:160)
 at com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(WSServlet.java:75)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at com.intuit.engine.efp.efe.filer.filters.servlet.EmptyRespElementFilter.doFilter(EmptyRespElementFilter.java:73)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at com.intuit.engine.efp.efe.filer.filters.servlet.FixPortFilter.doFilter(FixPortFilter.java:68)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at com.intuit.engine.efp.efe.filer.filters.servlet.ClientAppVerifyFilter.doFilter(ClientAppVerifyFilter.java:74)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at com.intuit.engine.efp.efe.filer.filters.servlet.ValidatorFilter.doFilter(ValidatorFilter.java:269)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at com.intuit.engine.efp.efe.filer.filters.servlet.ExtractPayloadFilter.doFilter(ExtractPayloadFilter.java:90)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
 at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
 at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
 at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
 at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:437)
 at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:381)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:451)
 at java.lang.Thread.run(Thread.java:595)
 at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:74)
 ... 72 more

1. Re: Cluster invocation error

brian.stansberry Oct 20, 2009 5:41 PM (in response to mohitanchlia)

What AS release is this?
Actions
2. Re: Cluster invocation error

brian.stansberry Oct 20, 2009 5:50 PM (in response to mohitanchlia)

That exception is thrown when the ClusterChooserInterceptor cannot find any more targets to try; the "Caused by: org.jboss.remoting.CannotConnectException" is telling you what the exception was from the last target it attempted. So the question is why there are no other targets to try if you've got an eight node cluster and only one node is being shut down.
Actions
3. Re: Cluster invocation error

mohitanchlia Oct 21, 2009 10:43 AM (in response to mohitanchlia)

We are on AS 4.2. But all other targets are up at that point and they are in the cluster.
Actions
4. Re: Cluster invocation error

brian.stansberry Oct 21, 2009 12:01 PM (in response to mohitanchlia)

What's the call pattern? Were the other nodes up and available when the client dowloaded the ejb proxy from JNDI or last invoked on it? (Last == the time before the call that failed).
Actions

Go to original post