JBoss Portal Stress Results

[walbar]

Hi,

I would like to share with you some results I gathered from stress testing JBoss Portal 2.4 and 2.6DR on JBoss AS 4.0.5.GA.

My CPU (running the Server) is
AMD Sempron 2800+ 1.6Ghz. with 1Gb of RAM
Windows Server 2003 SP1

Client CPU (running the Stress Tool) is a similar one. They are LAN connected. Results were obtained using Microsoft Web Application Stress Tool.

The difference between both version is quite high. Is there any other performance results you can point me to?. Do you think they are accurate or I misconfigured something? Are these results close to what we should expect from JBoss Portal with this configuration?

Thanks in advance for your opinions,

Waldemar

Overview ================================================================================
Report name: JBossPortal_2.4
Run on: 11/10/2006 6:46:42 PM
Run length: 00:02:50

Web Application Stress Tool Version:1.1.293.1

Number of test clients: 1

Number of hits: 756
Requests per Second: 6.30

Socket Statistics
--------------------------------------------------------------------------------
Socket Connects: 756
Total Bytes Sent (in KB): 181.62
Bytes Sent Rate (in KB/s): 1.51
Total Bytes Recv (in KB): 8552.99
Bytes Recv Rate (in KB/s): 71.29

Socket Errors
--------------------------------------------------------------------------------
Connect: 0
Send: 0
Recv: 0
Timeouts: 0

RDS Results
--------------------------------------------------------------------------------
Successful Queries: 0


Script Settings ================================================================================
Server: 10.152.0.63
Number of threads: 15

Test length: 00:02:00
Warmup: 00:00:25
Cooldown: 00:00:25

Use Random Delay: No

Follow Redirects: Yes
Max Redirect Depth: 15


Clients used in test ================================================================================
localhost

Clients not used in test ================================================================================


Result Codes
Code Description Count
================================================================================
200 OK 756


Page Summary
Page Hits TTFB Avg TTLB Avg Auth Query
================================================================================
GET /portal/ 756 2324.73 2365.84 No No

Overview ================================================================================
Report name: 11/10/2006 5:55:05 PM
Run on: 11/10/2006 5:55:05 PM
Run length: 00:02:50

Web Application Stress Tool Version:1.1.293.1

Number of test clients: 1

Number of hits: 116
Requests per Second: 0.96

Socket Statistics
--------------------------------------------------------------------------------
Socket Connects: 116
Total Bytes Sent (in KB): 27.87
Bytes Sent Rate (in KB/s): 0.23
Total Bytes Recv (in KB): 1687.10
Bytes Recv Rate (in KB/s): 14.02

Socket Errors
--------------------------------------------------------------------------------
Connect: 0
Send: 0
Recv: 0
Timeouts: 0

RDS Results
--------------------------------------------------------------------------------
Successful Queries: 0


Script Settings ================================================================================
Server: 10.152.0.63
Number of threads: 15

Test length: 00:02:00
Warmup: 00:00:25
Cooldown: 00:00:25

Use Random Delay: No

Follow Redirects: Yes
Max Redirect Depth: 15


Clients used in test ================================================================================
localhost

Clients not used in test ================================================================================


Result Codes
Code Description Count
================================================================================
200 OK 116


Page Summary
Page Hits TTFB Avg TTLB Avg Auth Query
================================================================================
GET /portal/ 116 14794.99 15167.84 No No

[roy.russo]

Its actually not the portal-core. Running the same test app on my devlaptop (WinXPSP2, 2GB, 2.13GHz), I ended up with similar results for 2.6DR. I then went in to the admin screen and removed the CMS Portlet, and here is what I have:

Overview
================================================================================
Report name: 11/10/2006 5:38:43 PM
Run on: 11/10/2006 5:38:43 PM
Run length: 00:02:50

Web Application Stress Tool Version:1.1.293.1

Notes
--------------------------------------------------------------------------------
JBoss Portal 2.6DR

Number of test clients: 1

Number of hits: 1649
Requests per Second: 13.73

Socket Statistics
--------------------------------------------------------------------------------
Socket Connects: 1649
Total Bytes Sent (in KB): 391.32
Bytes Sent Rate (in KB/s): 3.26
Total Bytes Recv (in KB): 16923.18
Bytes Recv Rate (in KB/s): 140.91

Socket Errors
--------------------------------------------------------------------------------
Connect: 0
Send: 0
Recv: 0
Timeouts: 0

RDS Results
--------------------------------------------------------------------------------
Successful Queries: 0


Script Settings
================================================================================
Server: localhost
Number of threads: 15

Test length: 00:02:00
Warmup: 00:00:25
Cooldown: 00:00:25

Use Random Delay: No

Follow Redirects: Yes
Max Redirect Depth: 15


Clients used in test
================================================================================
localhost

Clients not used in test
================================================================================


Result Codes
Code Description Count
================================================================================
200 OK 1649


Page Summary
Page Hits TTFB Avg TTLB Avg Auth Query
================================================================================
GET /portal 1649 1076.51 1092.96 No No

We're going to look in to this, as its not behaving as planned.. You didn't say which 2.4 version you were running against, but everything from 2.4.1 to 2.6DR now has Cache available in the CMS.

[walbar]

Hi Roy,

Thanks a lot for your quick answer.

We have performed the tests deploying JBoss 2.4.0, and 2.6.0-DR1 on JBoss AS 4.0.5.GA.

What we are wondering is why 2.6 is slower than 2.4. Maybe because of its development state?

I'll perform a new test hitting a page without the CMS Portlet as soon as I get to the office on Monday. I'll post my results then.

Best Regards,

Waldemar

[roy.russo]

Well, keep in mind that 2.6 was pushed out by the portal team developers earlier than planned, because we wanted feedback on the direction we were taking wrt usability.

As for the CMS, Sohil will be profiling it this week and see where the problem is. I'll tell him to post here if he finds anything.

[walbar]

Hi,

I've performed the test without the CMS Portlet and I've got similar results (13 pages/sec.).

Besides, I have activated a higher level of logging and found that for rendering '/portal' page it makes extensive use of hibernate, many times going beyond the cache and making actual queries to the database (even without the CMS portlet). I guess this could be a performance bottleneck. I believe that after the first page render there aren't reasons to have new accesses to the DB, layout and such can be cached. I wonder how much we can tune JBoss Portal to get better performance in a production environment. In fact, how much can we push it beyond the 13p/s?.

Regards,

Waldemar

[roy.russo]

"walbar" wrote:
Besides, I have activated a higher level of logging and found that for rendering '/portal' page it makes extensive use of hibernate, many times going beyond the cache and making actual queries to the database (even without the CMS portlet).

Depends where in the DB its accessing. Remember, that Hib does not allow cacheing Blobs. Im not sure about others parts of the DB schema, but the CMS (I realize you removed it) does makes extensive use of Blobs (stupid jackrabbit architecture!), so we were forced to add another level cache.

Sohil should be able to jprofile things a bit better than my random shots in the dark posts here, and Ill have him post here. ;-)

[walbar]

Thanks a lot for your answers, we will be looking forward to see Sohil news :-)

Anyway, let's assume we don't care about the CMS Portlet. What about Portal Core performance? Shall we assume that it is difficult to get more than 13 p/s with this configuration? Shall we assume that the database is going to be hit even when the page doesn't have the CMS Portlet? Is there some cache technology that is still to be included?

Regards,

Waldemar

[roy.russo]

After talking with Julien, he is telling me that 2.6DR has not been fine-tuned. Some tables will likely change... WSRP and CMS security come to mind, so he chose to left the Hibernate level in a development state. I would expect our Beta (not this coming Alpha) to be fine-tuned.

[walbar]

Hi,

Thanks a lot for your answer. Following your line of thinking I tested 2.4 home page without the CMS Portlet and got 20 p/s. Do you think that 2.6 will give us this number of pages once it is fine-tuned? May we expect a better number after the changes to the tables you mention?

Regards,

Waldemar

[roy.russo]

Sohil found two things in the 2.6 codebase and they've been fixed (and backported to 2.4):

1. CMS regexp I added for removal of everything except *. The regexp was being run every time, over the cached object, regardless if the tag existed.

2. Portal ResourceBundle loader seemed to be buggy/slow.

He ran the windows test tool on 2.6 and now has this (WITH the CMS):

Overview

================================================================================

Report name: 11/13/2006 8:35:32 PM

Run on: 11/13/2006 8:35:32 PM

Run length: 00:02:51



Web Application Stress Tool Version:1.1.293.1



Number of test clients: 1



Number of hits: 2713

Requests per Second: 22.57



Socket Statistics

--------------------------------------------------------------------------------

Socket Connects: 2713

Total Bytes Sent (in KB): 643.81

Bytes Sent Rate (in KB/s): 5.35

Total Bytes Recv (in KB): 42361.48

Bytes Recv Rate (in KB/s): 352.35



Socket Errors

--------------------------------------------------------------------------------

Connect: 0

Send: 0

Recv: 0

Timeouts: 0



RDS Results

--------------------------------------------------------------------------------

Successful Queries: 0





Script Settings

================================================================================

Server: localhost

Number of threads: 1



Test length: 00:02:00

Warmup: 00:00:25

Cooldown: 00:00:25



Use Random Delay: No



Follow Redirects: Yes

Max Redirect Depth: 15





Clients used in test

================================================================================

localhost



Clients not used in test

================================================================================





Result Codes

Code Description Count

================================================================================

200 OK 2713





Page Summary

Page Hits TTFB Avg TTLB Avg Auth Query

================================================================================

GET /portal 2713 39.32 42.92 No No

As we get closer to 2.6Beta, expect throughput to increase.

[julien1]

I have added a couple of improvements in 2.6 trunk, would you mind to test it and see what you get ?

"walbar" wrote:
Hi,

Thanks a lot for your answer. Following your line of thinking I tested 2.4 home page without the CMS Portlet and got 20 p/s. Do you think that 2.6 will give us this number of pages once it is fine-tuned? May we expect a better number after the changes to the tables you mention?

Regards,

Waldemar

[walbar]

Hi,

I've performed the test on the new version from the trunk. I get:

With the CMS Portlet: 15 p/s
Without the CMS Portlet: 31p/s

Yesterday I was trying to find out where in your code there were the SQL queries logued by hibernate. I was suspecting that they were the bottleneck but the source code is quite big and I had not much success. Anyway I see you have worked on it and now only the cache is hit without actually reaching the DB.

I guess that another possible bottleneck are the JTA transactions that are continuously being opened and closed. I would like to try to remove them in order to perform more tests, would you point me to the config or code file where this could be done (if possible)?. I wouldn't mind to lack some functionality during the tests.

Regards,

Waldemar

[roy.russo]

"walbar" wrote:

I've performed the test on the new version from the trunk. I get:

With the CMS Portlet: 15 p/s
Without the CMS Portlet: 31p/s

Which version?

"walbar" wrote:

I guess that another possible bottleneck are the JTA transactions that are continuously being opened and closed. I would like to try to remove them in order to perform more tests, would you point me to the config or code file where this could be done (if possible)?. I wouldn't mind to lack some functionality during the tests.

umm... that would kill most of the bundled portlets. Do you intend to only test the core?

[walbar]

Those tests were performed on the 2.6 trunk version Julien mentioned above.

In order to use JBoss Portal in my project I need better performance. I'm trying to guess how much it could give by tuning, caching, etc. From 0.96 p/s (in my first post) to 15p/s on version 2.6 there is a great improve but I would like to know if we would be able to push it further.

You are right, if removing JTA will break most of the protlets, then I guess that test is useless since it would be too far from the real environment.

Regards,

Waldemar

[walbar]

Anyway we don't need JTA in production since most of our portlets' logic would be read-only. We would use JTA during Staging, and in that phase we are not interested in performance yet.

So, I'm still interested in knowing where I could remove JTA transactions. Would you help me?

Thanks a lot,

Waldemar