Performance saturation regarding jboss-ejb-client

[mmois]

Hello,

at the moment we are developing an ejb client that has to upload hundreds of documents as fast as possible to our ejb application running inside JBoss AS 7.1.1.Final. I have written a client that uses a configurable number of threads to push the documents via ejb remote call to a statefull session bean. The ejb client calls are implemented as described here using jboss-ejb-client: https://docs.jboss.org/author/display/AS71/EJB+invocations+from+a+remote+client+using+JNDI

While measuring the performance, I am wondering that there seems to be a specific boundary on upload speed, while the CPU on the client machine as well as the CPU on the JBoss machine don't go to their limits. The network speed is about 10MB/s, and therefore far away from the practical limit of a gigabit connection. Also the write speed of the database is far from its limits, due to the small network throughput. Increasing the number of threads from one step by step only increases the throughput for the first additional threads. After that, adding more threads doesn't help. In the meantime the CPU of the JBoss server nevers goes beyond 20% load. The profiler shows that all worker threads for the stateful ejbs idle most of the time. The same is true for my client application.

On my search for parameters that help me tuning the upload speed, I found this document about the configuration parameters of jboss-ejb-client: https://docs.jboss.org/author/display/EJBCLIENT/Overview+of+Client+properties. I stubmled upon the sentence "

There is only one connection from the client (JVM) to the server to handle all invocations.". But as the document mentioned above about EJB invocations already states, there is no further documentation about the jboss-ejb-client project at the moment. Looking into the sources of the jboss-ejb-client project, I saw that they use only one jboss-remoting3 channel for each configuration connection. I already tried to add the same connection twice, but this didn't increase the speed. I also increased the parameter MAX_OUTBOUND_MESSAGES, with no effect too.

In order to be sure that we have no other effect that lets the performance run into saturation, I setup a simple test project to examine the problem under laboratory conditions, i.e. it uploads a random byte array to a statefull session bean. The byte array isn't stored, so the speed of the harddrive doesn't have an impcat here. You can find the sources on github: https://github.com/mmois/jboss-ejb-client-performance. But the project shows the same effect.

So my question is, are there any specific configuration options that help to speed up the document upload (e.g. options to configure the number of TCP/IP connections per channel)?

On the other hand, what is the recommended way, when you want to do remote ejb calls that have to upload a lot of data? Maybe the jboss-ejb-client approach isn't the right one?

Thanks in advance,

Martin

[jaikiran]

Martin, welcome to the forums.

We had a significant performance bug in EJB client (and the server side) in 7.1.1.Final. It's been fixed in a later release. So please test this against the newest release listed here http://www.jboss.org/jbossas/downloads

[jaikiran]

While testing with the newer version, remember to update your client classpath to use the newer libraries from the downloaded version.

[mmois]

Hello Jaikiran,

thank you for your fast response.

I downloaded  JBoss EAP 6.1.0.Alpha1 (AS 7.2.0.Alpha1-redhat-4) and updated my client maven dependency jboss-as-ejb-client-bom to version 7.2.0.Alpha1-SNAPSHOT. This uses the version 1.0.10.Final of jboss-ejb-client instead of 1.0.5.Final as the 7.1.1.Final did.

But now my client doesn't work at all, though I just changed the maven dependencies. It seems to hang on a closed message window:

2013-04-07 09:09:26,915 TRACE pool-1-thread-1 org.jboss.remoting.remote - Message window is closed, waiting

2013-04-07 09:09:26,915 TRACE Remoting "config-based-ejb-client-endpoint" read-1 org.xnio.channels.framed - Did not read a length

2013-04-07 09:09:26,916 TRACE Remoting "config-based-ejb-client-endpoint" read-1 org.jboss.remoting.remote - No message ready; returning

2013-04-07 09:09:26,916 TRACE Remoting "config-based-ejb-client-endpoint" read-1 org.xnio.nio.selector - Beginning select on sun.nio.ch.WindowsSelectorImpl@58caafcc

You can find the whole TRACE log on my newly github branch for my test project: https://github.com/mmois/jboss-ejb-client-performance/blob/jboss-7.2.0.Alpha1-redhat-4/client/performance_test.log.

I found this JIRA entry https://issues.jboss.org/browse/EJBCLIENT-45, stating that the handshake timeout is not configureable. Thus, I already tried out version 1.0.12.Final of jboss-ejb-client and set the configuration property invocation.timeout to 10sec, but this didn't help.

Any ideas on that? Thanks in advance.

Regards

Martin

[jaikiran]

mmois wrote:

Hello Jaikiran,

thank you for your fast response.

I download  JBoss EAP 6.1.0.Alpha1 (AS 7.2.0.Alpha1-redhat-4) and updated my client maven dependency jboss-as-ejb-client-bom to version 7.2.0.Alpha1-SNAPSHOT.

Don't use the snapshot. Use this maven repository http://maven.repository.redhat.com/earlyaccess/all/ which has the correct version http://maven.repository.redhat.com/earlyaccess/all/org/jboss/as/jboss-as-ejb-client-bom/7.2.0.Alpha1-redhat-4/

[mmois]

Thank you for pointing me to the earlyaccess maven repository. I updated the dependency and it works again.

The performance has also improved by factor 2: Instead of 10MB/s my test tool now runs with 21.2MB/s.

This is a nice improvement, but again far away from the practical limit of 115MB/s on our gigabit ethernet connection. Are there any further optimizations I can make to improve throughput even more?

Regards

Martin

[jaikiran]

I haven't looked at your project yet. I'll take a look sometime during the week and see if I can find anything.

[mmois]

Hi Jaikiran,

that would be great. Thanks in advance for that.

By the way, I also tried to start multiple instances of my test tool and it shows that the instances share the throughput. Starting e.g. four parallel instances, I reach the same throughput in sum over all four as with one instance. Thus, using additional JVMs doesn't solve the problem, too.

Regards

Martin

[mmois]

Hi Jaikiran,

have you found some time to investigate the source code of my example project? The problem is urgent for us, thus we appreciate any help.

Thanks in advance.

Regards

Martin

[jaikiran]

No, I haven't yet found the time. I do have it on my TODO list, but I can't assure of any immediate help, sorry.

[jaikiran]

One thing you could do is profile the application and see if it shows any performance bottlenecks.

[mmois]

Hello Jaikiran,

we have performed some profiling and got the following view:

All of our threads have a huge difference between total time and CPU time (e.g. 16482ms to 368ms), indicating that they are waiting for some sychronized section. Looking deeper into the strack trace, the problem seems to come from the BufferWriter implementation inside the class OutboundMessage of the jboss-remoting3 library. Inside the method accecpt() of the BufferWriter the profiler shows huge amounts of Object.wait() time.

A look into the source code reveales, that the access to the ConnectedMessageChannel is synchronized. This documentation here https://docs.jboss.org/author/display/EJBCLIENT/Overview+of+Client+properties states, that there is only one channel for the jboss-ejb-client library. Thus, my conclusion is, that the write access to this one channel is the bottleneck. Would you agree with that?

So even an increased amount of remoting writer threads doesn't increase the throughput, as they are all working on one channel whose access is synchronized. We have tried to increase the number of writer threads (hard coded within a patched version of xnio, as there is no option to configure that, right?), but there wasn't any positive effect.

Do you think it is possible to increase the number of channels that jboss-ejb-client uses or even make this configurable? As far as I know, there is no such option. Should I create a JIRA task for that?

Regards

Martin

[jaikiran]

Have you tried increasing the window sizes org.jboss.remoting3.TRANSMIT_WINDOW_SIZE on client side and org.jboss.remoting3.RECEIVE_WINDOW_SIZE on server side?

[jaikiran]

Set them as channel creation options, by the way.

[mmois]

I have added the RECEIVE_WINDOW_SIZE as a property to the standalone.xml as follows:

<subsystem xmlns="urn:jboss:domain:remoting:1.1">

  <connector name="remoting-connector" socket-binding="remoting">

    <properties>

      <property name="org.jboss.remoting3.RemotingOptions.RECEIVE_WINDOW_SIZE" value="104857600" />

    </properties>

                              </connector>

</subsystem>

In my jboss-ejb-client.properties I added the following line:

remote.connectionprovider.create.options.org.jboss.remoting3.RemotingOptions.TRANSMIT_WINDOW_SIZE=104857600

But this didn't change anything.