Performance tests status

[sergeypk]

I am able to run performance tests against all four providers (JBoss MQ, JBM, ActiveMQ, QPid). I have used JBoss 4.0.5, ActiveMQ 4.1.1, QPid M1, and Messaging version "whatever I had built at the moment", which is something after 1.2.0. I didn't tune ActiveMQ nor QPid in any significant way. For Messaging and JBoss MQ I used the destinations that are deployed by the messaging-perf build script.

Problems:

JBossMQ can't handle high throughput and dies, leaving itself in a non-operable state so no further tests can be run until it's cleaned up. I clean it up by stopping JBoss, removing data, log, tmp, and work directories from the configuration, and then restarting JBoss.

QPid also fails under high loads (both throughput and just high number of messages sent), eating a lot of memory and consuming 100% CPU for a long time without making any progress. I had it even throw an OOME once.

ActiveMQ locks up if a lot of messages are sent to a queue without anybody consuming them at the same time. The "Measured Send Rate vs. Intended Send Rate Remote" test locks up on ActiveMQ because of this.

Messaging seems to run fine so far but I haven't yet tried all the tests.

[sergeypk]

I have a question. Do we want to spend time automating those tests so that we can for example leave them running for the night, or is it fine to just make them work to be able to get some results from time to time?

The reason I'm asking is that if we want to automate, I think it would be useful if the tests could start the relevant provider, run a test against it, then stop it (and maybe clean up) - this would prevent a failing provider from affecting the others (if a provider consumes a significant amount of RAM, all others are probably going to be slowed down so the comparison would not be fair).

Or is not needed at this point and we just want to be able to get some nice graphs and are willing to do some manual work to get them?

Also, do we want to spend time finding out why QPid and ActiveMQ don't work with our tests?

[timfox]

"sergeypk" wrote:
I have a question. Do we want to spend time automating those tests so that we can for example leave them running for the night, or is it fine to just make them work to be able to get some results from time to time?

Ideally we should have these automated, but I don't want to spend a lot of time on this, so for now, just getting results ad hoc is sufficient.

Also, do we want to spend time finding out why QPid and ActiveMQ don't work with our tests?

Perhaps ActiveMQ is throttling? Many messaging providers have "quotas" or "flow control" where a particular user is only allowed to send a max. of x messages per sec.

I wouldn't spend too long on this either.

[sergeypk]

I updated the Wiki page at http://www.jboss.com/wiki/Edit.jsp?page=JBossJMSNewPerformanceBenchmark to include my changes to the performance framework and the information about non-JBoss providers.

Tim, assign a new task to me :-)