EJB remote invocation server bottleneck
Hello,
I'm performing ejb remote invocation benchmarks on JBoss AS 7.20.Final, calling ejb deployed in one JBoss instance from another JBoss instance, all instances are in domain on a single 20-core host.
EJB has following annotations:
@Stateless
@Clustered
@ConcurrencyManagement(ConcurrencyManagementType.BEAN)
@TransactionManagement(TransactionManagementType.BEAN)
Client just invokes EJB in multiple threads, server basically doing nothing.
At first I observed a bottleneck in the client, which was limited by 4 threads, but I resolved this by adding
<worker-thread-pool task-core-threads="20" task-max-threads="40" write-threads="20" read-threads="20"/>
to <subsystem xmlns="urn:jboss:domain:remoting:1.1">
Also I've added
<remote connector-ref="remoting-connector" thread-pool-name="default">
<channel-creation-options>
<option name="MAX_OUTBOUND_MESSAGES" value="400" type="remoting"/>
<option name="MAX_INBOUND_MESSAGES" value="400" type="remoting"/>
<option name="MAX_OUTBOUND_CHANNELS" value="400" type="remoting"/>
<option name="MAX_INBOUND_CHANNELS" value="400" type="remoting"/>
</channel-creation-options>
</remote>
<thread-pools>
<thread-pool name="default">
<max-threads count="400"/>
<keepalive-time time="1000" unit="milliseconds"/>
</thread-pool>
</thread-pools>
When I run my benchmark in 1 client - 1 server configuration (everything is still at the same host), I get 9k requests per second at most, and according to measurements, server instance uses just 3 CPU cores. Adding more server instances to the same host makes performance to scale linearly, that is 8 server instances gives 32k rps, still using 3 cores per server instance.
My question is how to achieve this maximum performance in a single process? Why each server instance is limited by just 3 threads?
Test sources including JBoss configs are available at https://github.com/DeltaFlight/ejbTest