1 2 3 Previous Next 30 Replies Latest reply on Oct 15, 2014 9:14 AM by cdbrown Go to original post
      • 15. Re: Performance saturation regarding jboss-ejb-client
        jaikiran

        Those aren't set at the right levels. On the client side it needs to be:

         

        remote.connection.<connection-name>.channel.options.org.jboss.remoting3.RemotingOptions.TRANSMIT_WINDOW_SIZE=<your value>
        

         

        replace <connection-name> with the appropriate name that you have used in those properties.

         

         

        On the server side, it has to be in the EJB3 subsystem:

        <subsystem xmlns="urn:jboss:domain:ejb3:2.0">
            ...
            <remote connector-ref="remoting-connector" thread-pool-name="default">
                <channel-creation-options>
                    <option name="RECEIVE_WINDOW_SIZE" value="your value" type="remoting"/>
                </channel-creation-options>
            ...
        
        • 16. Re: Performance saturation regarding jboss-ejb-client
          mmois

          Hi Jaikiran,

           

          thank you for fast response again. I adjusted the option, but the effect was minimal. In one or two runs I had a performance gain of ~1MB/s when I set the option on the client and on the server side to 10MB. But again, this doesn't reach the bandwidth of our connection. The profiler shows the same picture as before, a lot of wait time within org.jboss.remoting3.remote.OutboundMessage$1.accept(org.xnio.Pooled, boolean).

           

          Regards

          Martin

          • 17. Re: Performance saturation regarding jboss-ejb-client
            mmois

            Hi Jaikiran,

             

            in hope to get a solution, I have created a JIRA issue for this problem: https://issues.jboss.org/browse/EJBCLIENT-75.

             

            Regards

            Martin

            • 18. Re: Performance saturation regarding jboss-ejb-client
              jaikiran

              Martin, creating a JIRA at this point won't make much of a difference since we haven't yet figured out if it is a bug and if it is then which project needs to be fixed. So continuing the discussion here is the best path.

               

              Could you please attach the latest profiler snapshots which show the problem?

              • 19. Re: Performance saturation regarding jboss-ejb-client
                jaikiran

                Can you also commit your standalone*.xml that you are using, to your github repo?

                • 20. Re: Performance saturation regarding jboss-ejb-client
                  mmois

                  Hi Jaikiran,

                   

                  thank you for returning to this thread.

                   

                  I have commited my standalone.xml to the github repo: https://github.com/mmois/jboss-ejb-client-performance/blob/cde8252466d48bca358dba06c1996dc9ab94be5d/server/jboss.config/standalone.xml.

                   

                  The following snapshot shows the stack trace of one of my threads, with huge differences between CPU time and execution time:

                  netbeans_profiler_stack_thread.PNG

                   

                  The next snapshot displays the "hot spots":

                  netbeans_profiler_hotspots.PNG

                  • 21. Re: Performance saturation regarding jboss-ejb-client
                    jaikiran

                    I've pushed a few commits to my branch (which is the fork of your test application) here https://github.com/jaikiran/jboss-ejb-client-performance/commits/jboss-7.2.0.Alpha1-redhat-4 which fixes the configurations. My quick testing has shown the performance numbers, being reported by your application, to be twice or more of what were reported before those changes.

                    • 22. Re: Performance saturation regarding jboss-ejb-client
                      mmois

                      Hello Jaikiran,

                       

                      have you made some measurements by varying the number of threads or the message size? Here are some examples from my computer with your updated configuration:

                       

                      - 100 iterations,  1 Thread , 1MB message size: 18,573551 MB/s.

                      - 100 iterations,  5 Threads, 1MB message size: 28,130978 MB/s

                      - 100 iterations, 10 Threads, 1MB message size: 28,797696 MB/s

                      - 100 iterations, 15 Threads, 1MB message size: 29,388715 MB/s

                      - 100 iterations, 20 Threads, 1MB message size: 29,129890 MB/s

                      - 100 iterations, 25 Threads, 1MB message size: 28,325723 MB/s

                       

                      - 100 iterations,  1 Thread, 3MB message size: 23,557126 MB/s

                      - 100 iterations,  5 Threads, 3MB message size: 29,619098 MB/s

                      - 100 iterations, 10 Threads, 3MB message size: 28,561908 MB/s

                      - 100 iterations, 15 Threads, 3MB message size: 29,250219 MB/s

                      - 100 iterations, 20 Threads, 3MB message size: 28,974170 MB/s

                       

                      As the figures show, there seems to be a specific limit, which you cannot overcome by adding further threads or by increasing the message size. But when I use iperf (http://sourceforge.net/projects/iperf/) to measure the bandwidth of my loopback interface, I see that 72,3 MB/s are possible. Thus, the measured bandwidth is about the factor 2 away from the practical limit measured with iperf. This is why i stated that there seems to be some kind of saturation.

                       

                      Can you measure the bandwidth of your network connection and perform some tests by varying the number of threads and the message size and post your results here?

                       

                      Regards

                      Martin

                      • 23. Re: Performance saturation regarding jboss-ejb-client
                        mmois

                        Just to underline my last measurements, here a screenshot of Window's Task Manager showing the network throughput on our gigabit connection while running the tool with even higher values than you used:

                         

                        taskmanager.PNG

                         

                        Regards

                        Martin

                        • 24. Re: Performance saturation regarding jboss-ejb-client
                          jaikiran

                          I'll take a look at this tonight. I think I see where the problem is.

                          • 25. Re: Performance saturation regarding jboss-ejb-client
                            jukka_pekkanen

                            Hi

                             

                            I have been struggling with EJB performance lately, and noticed that this discussion covers exactly the things I have been experiencing as well - after trying to tune everything possible (same settings as proposed in the discussion) and doing profiling, it still looks like the single tcp/ip connection is the ultimate bottleneck. Remoting read-thread getting blocked by EJB-default-threads, which in turn are blocking each other at BufferPipeOutputStream.send / OutboundMessage.accept.

                             

                            Have you managed to reach any conclusion?

                            • 26. Re: Performance saturation regarding jboss-ejb-client
                              mmois

                              Hi Jukka,

                               

                              we haven't reached any conclusion yet. My JIRA issue has been closed and the response to Jaikiran's last posting is still outstanding.

                               

                              In the meantime we have also investigated a clustered setup. In this case the jboss-ejb-client library will dispatch the remote calls to both application servers, but the limitation seen with one application server still remains. Thus, the problem must definitely lie on the client side.

                               

                              Maybe you can reopen the JIRA issue and describe in a comment your problem?

                               

                              Regards

                              Martin

                              • 27. Re: Performance saturation regarding jboss-ejb-client
                                jaikiran

                                mmois wrote:

                                 

                                 

                                Maybe you can reopen the JIRA issue and describe in a comment your problem?

                                 

                                Regards

                                Martin

                                That JIRA was closed because based on what I could see it wasn't a bug in the EJB client project. As for what I concluded of my testing, I got busy with some other important tasks and couldn't finish it. I'll have to get back to this.

                                • 28. Re: Performance saturation regarding jboss-ejb-client
                                  deltaflight

                                  Is there any progress yet? I have nearly the same issue.

                                  • 29. Re: Performance saturation regarding jboss-ejb-client
                                    wdfink

                                    Hi Alexander,

                                    welcome to the forums.

                                     

                                    Could you create a new thread (link to this one) and give the details of what you are doing and which version do you use?

                                    From the point here I would recommend to use EAP6.2.0.Alpha or later or WIldFly, depend on what you need for production.

                                    In that versions the fixes which are mentioned by jaikiran.jai_forums2005.yahoo.co.in are included.