JBoss & 500 connections
Hi
I was trying to stress-test our application using The Grinder (http://grinder.sourceforge.net/). Each thread makes several direct EJB calls which result in quite complex business logic triggered, mostly causing DB load.
Everything went smoothly for 10,50,100,250 concurrent requests, but attempt to run 500 threads effectively kills JBoss. Grinder's error log says:
5/21/06 10:58:03 AM (thread 449): Aborting thread due to net.grinder.engine.process.JythonScriptExecutionException: Jython error encountered creating per-threa
d test runner object
net.grinder.engine.process.JythonScriptExecutionException: Jython error encountered creating per-thread test runner object
at net.grinder.engine.process.JythonScript$JythonRunnable.<init>(JythonScript.java:130)
at net.grinder.engine.process.GrinderThread.run(GrinderThread.java:104)
at java.lang.Thread.run(Thread.java:595)
Caused by: com.supportwizard.utils.ejb.SWEJBException: (server23_1148234283868) com.supportwizard.utils.ejb.SWEJBException error during JRMP connection establi
shment; nested exception is:
java.net.SocketException: Connection reset
at net.grinder.engine.process.JythonScriptExecutionException.stripPyException(JythonScriptExecutionException.java:58)
at net.grinder.engine.process.JythonScriptExecutionException.<init>(JythonScriptExecutionException.java:48)
... 3 more
Caused by: com.supportwizard.utils.ejb.SWEJBException: (server23_1148234283868) com.supportwizard.utils.ejb.SWEJBException error during JRMP connection establi
shment; nested exception is:
java.net.SocketException: Connection reset
at com.supportwizard.seance.Seance.constructUserSeance(Seance.java:1808)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java)
at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java)
at org.python.core.PyObject.__call__(PyObject.java)
at org.python.core.PyObject.invoke(PyObject.java)
at org.python.pycode._pyx0.__init__$2(../devlib/grinder/scripts/randomcaseeditapi.py:32)
at org.python.pycode._pyx0.call_function(../devlib/grinder/scripts/randomcaseeditapi.py)
at org.python.core.PyTableCode.call(PyTableCode.java)
at org.python.core.PyTableCode.call(PyTableCode.java)
at org.python.core.PyTableCode.call(PyTableCode.java)
at org.python.core.PyFunction.__call__(PyFunction.java)
at org.python.core.PyInstance.__init__(PyInstance.java)
at org.python.core.PyClass.__call__(PyClass.java)
at org.python.core.PyObject.__call__(PyObject.java)
at net.grinder.engine.process.JythonScript$JythonRunnable.<init>(JythonScript.java:127)
... 2 more
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
at java.io.DataInputStream.readByte(DataInputStream.java:241)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:215)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:171)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:94)
at org.jboss.invocation.jrmp.server.JRMPInvoker_Stub.invoke(Unknown Source)
at org.jboss.invocation.jrmp.interfaces.JRMPInvokerProxy.invoke(JRMPInvokerProxy.java:119)
at org.jboss.invocation.InvokerInterceptor.invokeInvoker(InvokerInterceptor.java:227)
at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:167)
at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:55)
at org.jboss.proxy.ejb.HomeInterceptor.invoke(HomeInterceptor.java:169)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:86)
at $Proxy4.findByName(Unknown Source)
at com.supportwizard.seance.Seance.constructUserSeance(Seance.java:1801)
Here Seance:1801 is a finder call on a home interface.
In server.log I can see autodeployer stopping due to JBAS-3107 and then it fails to start because of JBAS-2573 (both fixed in 4.0.4, but I'm still on 4.0.3sp1). No explicit errors show up there prior to
10:58:04,144 WARN [otocol.file.FileURLLister] Could not list directory '/usr/local/EnterpriseWizard/jboss/server/sw/deploy', reason unknown
message.
The question is, what can cause such behaviour? I think that it can be native threads limit, but I'm not sure.
System info:
SUSE LINUX 10.0 (X86-64)
Release ID: JBoss [Zion] 4.0.3SP1 (build: CVSTag=JBoss_4_0_3_SP1 date=200510231054)
Java VM: Java HotSpot(TM) 64-Bit Server VM 1.5.0_06-b05,Sun Microsystems Inc.
2 x Dual Core AMD Opteron(tm) Processor 270
4GB RAM
Thanks