Service Unavailable Error from Tomcat While accessing JBoss

[bbalaji]

Hi,

We have clustered environment setup but for testing we use one instance of Apache and one instance of Tomcat and one Instance of JBoss, all in different machines. Right now we are load testing this environment with simple application, which has servlet in Tomcat looking for SLSB bean to insert records into DB through CMP's. All EJB's are clustered.

We are using Tomcat 5 and JBoss 3.2.3 and JDK 1.4
Max pool size for EJB's are set to 1000.


When load tested for 100 concurrent users through OpenSTA tool we are getting the following error

java.rmi.RemoteException: Service unavailable.
at org.jboss.invocation.jrmp.interfaces.JRMPInvokerProxyHA.invoke(JRMPInvokerProxyHA.java:301)
at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:96)
at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:45)
at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:100)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:85)
at $Proxy2.saveEmployees(Unknown Source)
at com.clusttest.web.servlets.EmployeeServlet.doPost(EmployeeServlet.java:81)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:763)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:284)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:204)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:257)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:151)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:564)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:245)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:199)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:151)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:564)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:195)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:151)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:164)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:149)
at org.apache.catalina.cluster.tcp.ReplicationValve.invoke(ReplicationValve.java:182)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:149)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:564)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:156)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:151)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:564)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:972)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:206)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:324)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:395)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:673)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:615)
at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:786)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
at java.lang.Thread.run(Thread.java:534)

Your help is much appreciated.

Thanks,
Balaji

[jarnbjo]

"bbalaji" wrote:

java.rmi.RemoteException: Service unavailable.
at org.jboss.invocation.jrmp.interfaces.JRMPInvokerProxyHA.invoke(JRMPInvokerProxyHA.java:301)

It's not easy to tell exactly what's breaking, since JBoss is so "kind" to discard the original exception before throwing the RemoteException with a useless message text. This is done after the proxy has failed to invoke the method on any of the cluster nodes (it tries all nodes if allowed), but there can be lots of reasons for this. Network problems, marshalling exceptions, server overload and so on.

We've developed a modified version of the JRMPInvokerProxyHA class to wrap the last causing exception in the RemoteException and you may have a look at it. Note that only the last causing exception is wrapped (there may be different exceptions from different cluster nodes) and treat it as a dirty hack and restrict the use to development or testing systems. You can download the classes from:

http://jarnbjo.de/o2-lacosa-jbossextension.jar

and place the jar file in $JBOSS_HOME/server/all/lib and somewhere accessible for your client/web-application. Then change the cluster configuration (cluster-service.xml) to use com.o2.lacosa.jboss.proxy.JRMPInvokerHA instead of org.jboss.invocation.jrmp.server.JRMPInvokerHA.

Tor