1 2 Previous Next 20 Replies Latest reply on Jan 17, 2012 8:31 AM by prasad.deshpande

    Performance drop in nightly builds?

    prasad.deshpande

      Hi Guys,

       

      I'm just wondering if anyone else has noticed this, there is some performance drop in the nightly builds as compared woth AS7.final release.

       

      I've deployed my application on AS7 final & performed some tests that access EJB's & some database access entire test took around 21-26 seconds to complete various runs of the same test.

       

      Then I performed same test on nightly builds & noticed that it takes somewhere around 34-37 seconds.

       

      Just want to check if anyone too has found this observation, and if yes, can we do something about it since main feature of AS7 is performance?

       

      Thanks,

      Prasad

        • 1. Re: Performance drop in nightly builds?
          jaikiran

          Do you have more details on what is being tested and how? Any sample application which shows this? We would surely like to see what's going on.

          • 2. Re: Performance drop in nightly builds?
            prasad.deshpande

            I can not attach my running application here, but I've attached a test case here.

             

            What I've done is created a sample multi-threaded program TestMultiThread.java & invoking this test from a servlet. Architecture of the test is like follows. You have a loop running for 10 iterations, in each iteration, it will create 10 threads & then each thread will access my applications API & will do some task at the backend.

             

            In the backend, it uses session/entity beans & database connection from datasource to perform insert/update/select.

             

            Timing that I've recodred for the same test on AS 7.0.0.Final is

             

            14:57:36,687 INFO  [stdout] (http-localhost-127.0.0.1-8080-1) Running test...

            14:58:24,281 INFO  [stdout] (http-localhost-127.0.0.1-8080-1) Completed execution in: 46860 ms

            14:58:59,421 INFO  [stdout] (http-localhost-127.0.0.1-8080-1) Completed execution in: 35140 ms

            14:59:29,828 INFO  [stdout] (http-localhost-127.0.0.1-8080-1) Completed execution in: 30407 ms

            14:59:56,937 INFO  [stdout] (http-localhost-127.0.0.1-8080-1) Completed execution in: 27109 ms

            15:00:21,765 INFO  [stdout] (http-localhost-127.0.0.1-8080-1) Completed execution in: 24828 ms

            15:00:46,312 INFO  [stdout] (http-localhost-127.0.0.1-8080-1) Completed execution in: 24547 ms

            15:01:10,375 INFO  [stdout] (http-localhost-127.0.0.1-8080-1) Completed execution in: 24063 ms

            15:01:33,296 INFO  [stdout] (http-localhost-127.0.0.1-8080-1) Completed execution in: 22921 ms

            15:01:55,234 INFO  [stdout] (http-localhost-127.0.0.1-8080-1) Completed execution in: 21938 ms

            15:02:15,984 INFO  [stdout] (http-localhost-127.0.0.1-8080-1) Completed execution in: 20734 ms

            15:02:16,000 INFO  [stdout] (http-localhost-127.0.0.1-8080-1) Test complete.

            and on today's nightly build (supposed to be 7.0.1)

             

            15:22:20,109 INFO  [stdout] (http--0.0.0.0-8080-1) Running test...

            15:23:04,984 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 44391 ms

            15:23:43,375 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 38391 ms

            15:24:17,968 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 34593 ms

            15:24:52,375 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 34407 ms

            15:25:26,625 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 34235 ms

            15:26:03,250 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 36625 ms

            15:26:37,609 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 34359 ms

            15:27:11,515 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 33906 ms

            15:27:48,656 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 37125 ms

            15:28:22,734 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 34063 ms

            15:28:22,750 INFO  [stdout] (http--0.0.0.0-8080-1) Test complete.

             

            You can see the difference clearly. If you need any specific info, please let me know. apart from handing over my ear to you, I'll try to do everything to give you every bit you need.

            • 3. Re: Performance drop in nightly builds?
              prasad.deshpande

              Also, some more information that might be helpful to you,

               

              This application also uses MDB's, but since @MessageDriven annotation was ignored in AS 7.0.0.Final, so to have same effect in yesterday's nightly build, I removed all my MDB's so atleast there performace can be equated with 7.0.0.Final, otherwise with MDB's on timing was even worse, it was

              11:47:17,171 INFO  [stdout] (http--0.0.0.0-8080-1) Running test...

              11:49:02,500 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 104844 ms

              11:50:11,203 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 68688 ms

              11:51:09,453 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 58250 ms

              11:52:05,484 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 56031 ms

              11:52:56,125 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 50641 ms

              11:53:47,906 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 51781 ms

              11:54:38,953 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 51047 ms

              11:55:27,062 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 48094 ms

              11:56:17,968 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 50906 ms

              11:57:05,687 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 47719 ms

              11:57:05,718 INFO  [stdout] (http--0.0.0.0-8080-1) Test complete.

               

              I've as well attached my standalone.xml to help you with the subsystem that my server is using.

              • 4. Re: Performance drop in nightly builds?
                jaikiran

                I will have a look at this, but when running the tests, just make sure that you have configured the right pool size for your SLSB and MDB (check the standalone-preview.xml EJB3 subsystem for details). By default, the pool size is capped at a upper limit of 20.

                • 5. Re: Performance drop in nightly builds?
                  prasad.deshpande

                  To be honest, I'm not in favour of changing configuration parameters for the test, simply because on 7.0.0.Final & on nightly build too I ran test with keeping default values for parameters. So unless pool size is changed in nightly builds since 7.0.0.Final, I'd like to keep parameters same so as to have real comparison between two versions.

                   

                  So for real comparison, I've kept hardware/software details same (apart from two versions of AS7), and that's where I found this observation.

                  • 6. Re: Performance drop in nightly builds?
                    jaikiran

                    The standalone-efp.xml that you attached shows that you have disabled pooling for your beans (i.e. there's no pooling configuration in your EJB3 subsystem):

                     

                    <subsystem xmlns="urn:jboss:domain:ejb3:1.1"/>
                    

                     

                     

                    So effectively this will now result in a new bean instance being created and destroyed on every invocation to the bean. Depending on how heavy the beans are (like the code in @PostConstruct etc...) this can have an adverse effect. In 7.0.0 we always had a strict max pool (hard coded) with max size of 20. So if you want to compare the performance then don't disable pooling in your configuration. Just use the default pooling config that comes in standalone.xml of 7.0.1 and see if the runs show improvement.

                    • 7. Re: Performance drop in nightly builds?
                      prasad.deshpande

                      Thanks Jaikiran,

                       

                      Yes, performance seems to be better after adding pool size. I'll give you some timings here..

                       

                      With pool size 20 & MDB's enabled following was the result.

                      10:45:22,312 INFO  [stdout] (http--0.0.0.0-8080-1) Running test...

                      10:46:31,031 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 68000 ms

                      10:47:15,703 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 44672 ms

                      10:47:52,625 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 36922 ms

                      10:48:28,875 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 36250 ms

                      10:49:05,312 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 36422 ms

                      10:49:37,968 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 32640 ms

                      10:50:10,296 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 32312 ms

                      10:50:43,296 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 33000 ms

                      10:51:17,671 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 34359 ms

                      10:51:49,718 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 32047 ms

                      10:51:49,734 INFO  [stdout] (http--0.0.0.0-8080-1) Test complete.

                       

                      With pool size 20 & MDB's disabled following was result.

                       

                      11:06:37,015 INFO  [stdout] (http--0.0.0.0-8080-1) Running test...

                      11:07:08,484 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 30828 ms

                      11:07:33,390 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 24906 ms

                      11:07:53,531 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 20141 ms

                      11:08:13,265 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 19734 ms

                      11:08:32,265 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 19000 ms

                      11:08:53,015 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 20750 ms

                      11:09:12,250 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 19235 ms

                      11:09:30,984 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 18734 ms

                      11:09:52,218 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 21234 ms

                      11:10:11,781 INFO  [stdout] (http--0.0.0.0-8080-1) Completed execution in: 19563 ms

                      11:10:11,796 INFO  [stdout] (http--0.0.0.0-8080-1) Test complete.

                       

                      Now, if you see just with MDB's on/off result difference is noticable. I'll do some profiling on my MDB's to check bottleneck.

                      • 8. Re: Performance drop in nightly builds?
                        jaikiran

                        When you say MDBs are enabled, do you actually send messages and process message through the MDBs? If yes, then I would expect a difference in the numbers when you compare it with MDB being disabled, since that processing will no longer happen.

                        • 9. Re: Performance drop in nightly builds?
                          prasad.deshpande
                          When you say MDBs are enabled, do you actually send messages and process message through the MDBs?

                          When I say disabling MDB's that mean I'm removing them from ejb.jar, so there won't be any processing of message, however message is still sent to the topic.

                           

                          If yes, then I would expect a difference in the numbers when you compare it with MDB being disabled, since that processing will no longer happen.

                           

                          Difference in numbers I'm expecting too, but this seems to be quite a lot, without MDB's it's in a range 19-21sec whereas with MDB's 32-36sec that's like almost 70% increase, which sounds little odd to me. I'm still in the process of profiling it, let's see what comes up...

                          • 10. Re: Performance drop in nightly builds?
                            jaikiran

                            Prasad Deshpande wrote:

                             

                            I'm still in the process of profiling it, let's see what comes up...

                            Sure. Let us know if you find anything interesting. Without the application, it's a bit difficult for me to speculate what might be wrong.

                            • 11. Re: Performance drop in nightly builds?
                              smarlow

                              If you could run both AS versions with load for a few minutes, while profiling (http://community.jboss.org/wiki/ProfilingAS7UsingOProfileOnFedora).  That would give us a nice description of where we are spending more time while running the application.

                               

                              The nice thing about oprofile, is that it produces a brief (native level) description of where time is spent in the Java application.  In other words, running oprofile, doesn't add noticeable overhead (there is some but not as much as other profilers that I have used).  If your not on Fedora 15, there are probably other profilers that could be used.

                              • 12. Re: Performance drop in nightly builds?
                                prasad.deshpande

                                Scott, Jaikiran, I've attached result of CPU views for now, If this intersts you. If you need specific details, please let me know, I'll try to generate that. Meanwhile, I'm trying to get Fedora on other system for oprofile.

                                • 13. Re: Performance drop in nightly builds?
                                  prasad.deshpande

                                  Thanks Jaikiran,

                                   

                                  Here is the lastest standalone-efp.xml for Cr1b, if that helps..!

                                  • 14. Re: Performance drop in nightly builds?
                                    jaikiran

                                    Would it be possible for you to attach the server side application (with the beans) too?

                                    1 2 Previous Next