-
15. Re: Performance saturation regarding jboss-ejb-client
jaikiran Apr 11, 2013 9:22 AM (in response to mmois)Those aren't set at the right levels. On the client side it needs to be:
remote.connection.<connection-name>.channel.options.org.jboss.remoting3.RemotingOptions.TRANSMIT_WINDOW_SIZE=<your value>
replace <connection-name> with the appropriate name that you have used in those properties.
On the server side, it has to be in the EJB3 subsystem:
<subsystem xmlns="urn:jboss:domain:ejb3:2.0"> ... <remote connector-ref="remoting-connector" thread-pool-name="default"> <channel-creation-options> <option name="RECEIVE_WINDOW_SIZE" value="your value" type="remoting"/> </channel-creation-options> ...
-
16. Re: Performance saturation regarding jboss-ejb-client
mmois Apr 12, 2013 2:57 AM (in response to mmois)Hi Jaikiran,
thank you for fast response again. I adjusted the option, but the effect was minimal. In one or two runs I had a performance gain of ~1MB/s when I set the option on the client and on the server side to 10MB. But again, this doesn't reach the bandwidth of our connection. The profiler shows the same picture as before, a lot of wait time within org.jboss.remoting3.remote.OutboundMessage$1.accept(org.xnio.Pooled, boolean).
Regards
Martin
-
17. Re: Performance saturation regarding jboss-ejb-client
mmois Apr 12, 2013 11:52 AM (in response to mmois)Hi Jaikiran,
in hope to get a solution, I have created a JIRA issue for this problem: https://issues.jboss.org/browse/EJBCLIENT-75.
Regards
Martin
-
18. Re: Performance saturation regarding jboss-ejb-client
jaikiran Apr 16, 2013 7:59 AM (in response to mmois)Martin, creating a JIRA at this point won't make much of a difference since we haven't yet figured out if it is a bug and if it is then which project needs to be fixed. So continuing the discussion here is the best path.
Could you please attach the latest profiler snapshots which show the problem?
-
19. Re: Performance saturation regarding jboss-ejb-client
jaikiran Apr 16, 2013 8:02 AM (in response to jaikiran)Can you also commit your standalone*.xml that you are using, to your github repo?
-
20. Re: Performance saturation regarding jboss-ejb-client
mmois Apr 16, 2013 4:45 PM (in response to mmois)Hi Jaikiran,
thank you for returning to this thread.
I have commited my standalone.xml to the github repo: https://github.com/mmois/jboss-ejb-client-performance/blob/cde8252466d48bca358dba06c1996dc9ab94be5d/server/jboss.config/standalone.xml.
The following snapshot shows the stack trace of one of my threads, with huge differences between CPU time and execution time:
The next snapshot displays the "hot spots":
-
21. Re: Performance saturation regarding jboss-ejb-client
jaikiran Apr 22, 2013 2:43 AM (in response to mmois)I've pushed a few commits to my branch (which is the fork of your test application) here https://github.com/jaikiran/jboss-ejb-client-performance/commits/jboss-7.2.0.Alpha1-redhat-4 which fixes the configurations. My quick testing has shown the performance numbers, being reported by your application, to be twice or more of what were reported before those changes.
-
22. Re: Performance saturation regarding jboss-ejb-client
mmois Apr 22, 2013 5:37 PM (in response to mmois)Hello Jaikiran,
have you made some measurements by varying the number of threads or the message size? Here are some examples from my computer with your updated configuration:
- 100 iterations, 1 Thread , 1MB message size: 18,573551 MB/s.
- 100 iterations, 5 Threads, 1MB message size: 28,130978 MB/s
- 100 iterations, 10 Threads, 1MB message size: 28,797696 MB/s
- 100 iterations, 15 Threads, 1MB message size: 29,388715 MB/s
- 100 iterations, 20 Threads, 1MB message size: 29,129890 MB/s
- 100 iterations, 25 Threads, 1MB message size: 28,325723 MB/s
- 100 iterations, 1 Thread, 3MB message size: 23,557126 MB/s
- 100 iterations, 5 Threads, 3MB message size: 29,619098 MB/s
- 100 iterations, 10 Threads, 3MB message size: 28,561908 MB/s
- 100 iterations, 15 Threads, 3MB message size: 29,250219 MB/s
- 100 iterations, 20 Threads, 3MB message size: 28,974170 MB/s
As the figures show, there seems to be a specific limit, which you cannot overcome by adding further threads or by increasing the message size. But when I use iperf (http://sourceforge.net/projects/iperf/) to measure the bandwidth of my loopback interface, I see that 72,3 MB/s are possible. Thus, the measured bandwidth is about the factor 2 away from the practical limit measured with iperf. This is why i stated that there seems to be some kind of saturation.
Can you measure the bandwidth of your network connection and perform some tests by varying the number of threads and the message size and post your results here?
Regards
Martin
-
23. Re: Performance saturation regarding jboss-ejb-client
mmois Apr 23, 2013 10:51 AM (in response to mmois) -
24. Re: Performance saturation regarding jboss-ejb-client
jaikiran Apr 24, 2013 6:35 AM (in response to mmois)I'll take a look at this tonight. I think I see where the problem is.
-
25. Re: Performance saturation regarding jboss-ejb-client
jukka_pekkanen May 21, 2013 8:22 AM (in response to jaikiran)Hi
I have been struggling with EJB performance lately, and noticed that this discussion covers exactly the things I have been experiencing as well - after trying to tune everything possible (same settings as proposed in the discussion) and doing profiling, it still looks like the single tcp/ip connection is the ultimate bottleneck. Remoting read-thread getting blocked by EJB-default-threads, which in turn are blocking each other at BufferPipeOutputStream.send / OutboundMessage.accept.
Have you managed to reach any conclusion?
-
26. Re: Performance saturation regarding jboss-ejb-client
mmois May 22, 2013 5:23 AM (in response to mmois)Hi Jukka,
we haven't reached any conclusion yet. My JIRA issue has been closed and the response to Jaikiran's last posting is still outstanding.
In the meantime we have also investigated a clustered setup. In this case the jboss-ejb-client library will dispatch the remote calls to both application servers, but the limitation seen with one application server still remains. Thus, the problem must definitely lie on the client side.
Maybe you can reopen the JIRA issue and describe in a comment your problem?
Regards
Martin
-
27. Re: Performance saturation regarding jboss-ejb-client
jaikiran May 22, 2013 6:26 AM (in response to mmois)mmois wrote:
Maybe you can reopen the JIRA issue and describe in a comment your problem?
Regards
Martin
That JIRA was closed because based on what I could see it wasn't a bug in the EJB client project. As for what I concluded of my testing, I got busy with some other important tasks and couldn't finish it. I'll have to get back to this.
-
28. Re: Performance saturation regarding jboss-ejb-client
deltaflight Oct 28, 2013 9:24 AM (in response to jaikiran)Is there any progress yet? I have nearly the same issue.
-
29. Re: Performance saturation regarding jboss-ejb-client
wdfink Oct 28, 2013 10:11 AM (in response to deltaflight)Hi Alexander,
welcome to the forums.
Could you create a new thread (link to this one) and give the details of what you are doing and which version do you use?
From the point here I would recommend to use EAP6.2.0.Alpha or later or WIldFly, depend on what you need for production.
In that versions the fixes which are mentioned by jaikiran.jai_forums2005.yahoo.co.in are included.