-
1. Re: Performance drop in nightly builds?
jaikiran Aug 17, 2011 10:00 AM (in response to prasad.deshpande)Do you have more details on what is being tested and how? Any sample application which shows this? We would surely like to see what's going on.
-
2. Re: Performance drop in nightly builds?
prasad.deshpande Aug 17, 2011 10:40 AM (in response to jaikiran)I can not attach my running application here, but I've attached a test case here.
What I've done is created a sample multi-threaded program TestMultiThread.java & invoking this test from a servlet. Architecture of the test is like follows. You have a loop running for 10 iterations, in each iteration, it will create 10 threads & then each thread will access my applications API & will do some task at the backend.
In the backend, it uses session/entity beans & database connection from datasource to perform insert/update/select.
Timing that I've recodred for the same test on AS 7.0.0.Final is
14:57:36,687 INFO [stdout] (http-localhost-127.0.0.1-8080-1) Running test...
14:58:24,281 INFO [stdout] (http-localhost-127.0.0.1-8080-1) Completed execution in: 46860 ms
14:58:59,421 INFO [stdout] (http-localhost-127.0.0.1-8080-1) Completed execution in: 35140 ms
14:59:29,828 INFO [stdout] (http-localhost-127.0.0.1-8080-1) Completed execution in: 30407 ms
14:59:56,937 INFO [stdout] (http-localhost-127.0.0.1-8080-1) Completed execution in: 27109 ms
15:00:21,765 INFO [stdout] (http-localhost-127.0.0.1-8080-1) Completed execution in: 24828 ms
15:00:46,312 INFO [stdout] (http-localhost-127.0.0.1-8080-1) Completed execution in: 24547 ms
15:01:10,375 INFO [stdout] (http-localhost-127.0.0.1-8080-1) Completed execution in: 24063 ms
15:01:33,296 INFO [stdout] (http-localhost-127.0.0.1-8080-1) Completed execution in: 22921 ms
15:01:55,234 INFO [stdout] (http-localhost-127.0.0.1-8080-1) Completed execution in: 21938 ms
15:02:15,984 INFO [stdout] (http-localhost-127.0.0.1-8080-1) Completed execution in: 20734 ms
15:02:16,000 INFO [stdout] (http-localhost-127.0.0.1-8080-1) Test complete.
and on today's nightly build (supposed to be 7.0.1)
15:22:20,109 INFO [stdout] (http--0.0.0.0-8080-1) Running test...
15:23:04,984 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 44391 ms
15:23:43,375 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 38391 ms
15:24:17,968 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 34593 ms
15:24:52,375 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 34407 ms
15:25:26,625 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 34235 ms
15:26:03,250 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 36625 ms
15:26:37,609 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 34359 ms
15:27:11,515 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 33906 ms
15:27:48,656 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 37125 ms
15:28:22,734 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 34063 ms
15:28:22,750 INFO [stdout] (http--0.0.0.0-8080-1) Test complete.
You can see the difference clearly. If you need any specific info, please let me know. apart from handing over my ear to you, I'll try to do everything to give you every bit you need.
-
TestMultiThread.java.zip 1.2 KB
-
-
3. Re: Performance drop in nightly builds?
prasad.deshpande Aug 17, 2011 10:52 AM (in response to jaikiran)Also, some more information that might be helpful to you,
This application also uses MDB's, but since @MessageDriven annotation was ignored in AS 7.0.0.Final, so to have same effect in yesterday's nightly build, I removed all my MDB's so atleast there performace can be equated with 7.0.0.Final, otherwise with MDB's on timing was even worse, it was
11:47:17,171 INFO [stdout] (http--0.0.0.0-8080-1) Running test...
11:49:02,500 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 104844 ms
11:50:11,203 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 68688 ms
11:51:09,453 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 58250 ms
11:52:05,484 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 56031 ms
11:52:56,125 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 50641 ms
11:53:47,906 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 51781 ms
11:54:38,953 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 51047 ms
11:55:27,062 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 48094 ms
11:56:17,968 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 50906 ms
11:57:05,687 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 47719 ms
11:57:05,718 INFO [stdout] (http--0.0.0.0-8080-1) Test complete.
I've as well attached my standalone.xml to help you with the subsystem that my server is using.
-
standalone-efp.xml 15.2 KB
-
-
4. Re: Performance drop in nightly builds?
jaikiran Aug 18, 2011 1:09 AM (in response to prasad.deshpande)I will have a look at this, but when running the tests, just make sure that you have configured the right pool size for your SLSB and MDB (check the standalone-preview.xml EJB3 subsystem for details). By default, the pool size is capped at a upper limit of 20.
-
5. Re: Performance drop in nightly builds?
prasad.deshpande Aug 18, 2011 4:19 AM (in response to jaikiran)To be honest, I'm not in favour of changing configuration parameters for the test, simply because on 7.0.0.Final & on nightly build too I ran test with keeping default values for parameters. So unless pool size is changed in nightly builds since 7.0.0.Final, I'd like to keep parameters same so as to have real comparison between two versions.
So for real comparison, I've kept hardware/software details same (apart from two versions of AS7), and that's where I found this observation.
-
6. Re: Performance drop in nightly builds?
jaikiran Aug 18, 2011 5:06 AM (in response to prasad.deshpande)The standalone-efp.xml that you attached shows that you have disabled pooling for your beans (i.e. there's no pooling configuration in your EJB3 subsystem):
<subsystem xmlns="urn:jboss:domain:ejb3:1.1"/>
So effectively this will now result in a new bean instance being created and destroyed on every invocation to the bean. Depending on how heavy the beans are (like the code in @PostConstruct etc...) this can have an adverse effect. In 7.0.0 we always had a strict max pool (hard coded) with max size of 20. So if you want to compare the performance then don't disable pooling in your configuration. Just use the default pooling config that comes in standalone.xml of 7.0.1 and see if the runs show improvement.
-
7. Re: Performance drop in nightly builds?
prasad.deshpande Aug 18, 2011 7:04 AM (in response to jaikiran)Thanks Jaikiran,
Yes, performance seems to be better after adding pool size. I'll give you some timings here..
With pool size 20 & MDB's enabled following was the result.
10:45:22,312 INFO [stdout] (http--0.0.0.0-8080-1) Running test...
10:46:31,031 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 68000 ms
10:47:15,703 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 44672 ms
10:47:52,625 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 36922 ms
10:48:28,875 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 36250 ms
10:49:05,312 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 36422 ms
10:49:37,968 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 32640 ms
10:50:10,296 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 32312 ms
10:50:43,296 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 33000 ms
10:51:17,671 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 34359 ms
10:51:49,718 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 32047 ms
10:51:49,734 INFO [stdout] (http--0.0.0.0-8080-1) Test complete.
With pool size 20 & MDB's disabled following was result.
11:06:37,015 INFO [stdout] (http--0.0.0.0-8080-1) Running test...
11:07:08,484 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 30828 ms
11:07:33,390 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 24906 ms
11:07:53,531 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 20141 ms
11:08:13,265 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 19734 ms
11:08:32,265 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 19000 ms
11:08:53,015 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 20750 ms
11:09:12,250 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 19235 ms
11:09:30,984 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 18734 ms
11:09:52,218 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 21234 ms
11:10:11,781 INFO [stdout] (http--0.0.0.0-8080-1) Completed execution in: 19563 ms
11:10:11,796 INFO [stdout] (http--0.0.0.0-8080-1) Test complete.
Now, if you see just with MDB's on/off result difference is noticable. I'll do some profiling on my MDB's to check bottleneck.
-
8. Re: Performance drop in nightly builds?
jaikiran Aug 18, 2011 7:33 AM (in response to prasad.deshpande)When you say MDBs are enabled, do you actually send messages and process message through the MDBs? If yes, then I would expect a difference in the numbers when you compare it with MDB being disabled, since that processing will no longer happen.
-
9. Re: Performance drop in nightly builds?
prasad.deshpande Aug 18, 2011 7:55 AM (in response to jaikiran)When you say MDBs are enabled, do you actually send messages and process message through the MDBs?
When I say disabling MDB's that mean I'm removing them from ejb.jar, so there won't be any processing of message, however message is still sent to the topic.
If yes, then I would expect a difference in the numbers when you compare it with MDB being disabled, since that processing will no longer happen.
Difference in numbers I'm expecting too, but this seems to be quite a lot, without MDB's it's in a range 19-21sec whereas with MDB's 32-36sec that's like almost 70% increase, which sounds little odd to me. I'm still in the process of profiling it, let's see what comes up...
-
10. Re: Performance drop in nightly builds?
jaikiran Aug 18, 2011 8:02 AM (in response to prasad.deshpande)Prasad Deshpande wrote:
I'm still in the process of profiling it, let's see what comes up...
Sure. Let us know if you find anything interesting. Without the application, it's a bit difficult for me to speculate what might be wrong.
-
11. Re: Performance drop in nightly builds?
smarlow Aug 18, 2011 9:26 AM (in response to prasad.deshpande)If you could run both AS versions with load for a few minutes, while profiling (http://community.jboss.org/wiki/ProfilingAS7UsingOProfileOnFedora). That would give us a nice description of where we are spending more time while running the application.
The nice thing about oprofile, is that it produces a brief (native level) description of where time is spent in the Java application. In other words, running oprofile, doesn't add noticeable overhead (there is some but not as much as other profilers that I have used). If your not on Fedora 15, there are probably other profilers that could be used.
-
12. Re: Performance drop in nightly builds?
prasad.deshpande Aug 18, 2011 10:23 AM (in response to smarlow)Scott, Jaikiran, I've attached result of CPU views for now, If this intersts you. If you need specific details, please let me know, I'll try to generate that. Meanwhile, I'm trying to get Fedora on other system for oprofile.
-
CallTree.zip 271.4 KB
-
-
13. Re: Performance drop in nightly builds?
prasad.deshpande Jan 16, 2012 12:32 PM (in response to prasad.deshpande)Thanks Jaikiran,
Here is the lastest standalone-efp.xml for Cr1b, if that helps..!
-
standalone-efp.xml 16.5 KB
-
-
14. Re: Performance drop in nightly builds?
jaikiran Jan 17, 2012 4:35 AM (in response to prasad.deshpande)Would it be possible for you to attach the server side application (with the beans) too?