About the performance framework
The Messaging Performance Framework runs a set of performance tests against different messaging providers and creates graphs from the results. Tests can be run against any JMS-compliant provider, at the moment we have tried JBossMQ, JBossMessaging, Apache ActiveMQ, and Apache Qpid.
Checking out and building the performance framework
The Messaging Performance Framework source code is available in the JBoss SVN repository at https://svn.jboss.org/repos/messaging/projects/perf/trunk. In order to use the framework, you first need to check out the source code and build the project. The location of the framework's source code will be referred to as $PERF_HOME throughout the rest of the document.
svn co https://svn.jboss.org/repos/messaging/projects/perf/trunk messaging-perf cd $PERF_HOME ant
Note that you will need to have ant-contrib installed since the build file makes use of it.
Using the performance framework
This document contains step-by-step instructions on how to use Messaging Performance Framework to run a simple performance test. The test consists in sending bursts of 1000 0 Kilobytes non-persistent messages to all provider instances while gradually increasing the send rate (200 messages/sec, 400 messages/sec, etc) and measuring the receive rate. At the end, the framework generates the graph representing the receive rate as function of the send rate for each tested provider.
For JBoss providers, this document assumes that there are two JBoss 4.0.3 (or higher) instances up and running. One JBoss instance, which will be referred to as the messaging instance runs a scoped JBoss Messaging deployment and the other, which will be referred to as jbossmq runs a standard JBossMQ deployment.
This document assumes that JBoss instances run on the same physical machine. The messaging instance uses the default port range, and the jbossmq instance uses the ServiceBindingManager's "ports-01" range (JNDI server listens on 1199 instead of 1099 and in general services use their default port values + 100). For simplicity, we could assume that directories containing the alternate JBoss configurations are also named messaging and jbossmq and live in the same $JBOSS_HOME/server directory. However, the JBoss instances do not necessarily have to run on the same physical box.
This is the sequence of steps required to generate a result set:
1. Build and deploy a JBoss Messaging scoped SAR.
In order to use JBoss Messaging with a JBoss 4.x instance, you need to build a scoped Messaging deployment archive and also prepare a customized JBoss configuration. Instructions on how to do that are available here. Use the messaging directory for deployment. Alternatively, you can create the messaging configuration by using the release bundle installation script. Instruction on how to do that are available at http://labs.jboss.com/portal/jbossmessaging/docs/index.html.
2. Create a JBossMQ test configuration
Create a new $JBOSS_HOME/server/jbossmq directory, based on the default configuration. The only necessary customization is to enable ServiceBindingManager to avoid port conflicts. Use the "ports-01" port set (see conf/jboss-service.xml).
3. Start both JBoss instances
From two different terminals:
cd $JBOSS_HOME/bin ./run -c messaging
cd $JBOSS_HOME/bin ./run -c jbossmq
4. Set up and run other providers
Consult the documentation of the providers to configure and launch their message brokers. For ActiveMQ disable the use of JMX so that it doesn't try to use port 1099 already taken by the JBoss instances. Most providers can be configured through JNDI properties set in perf.xml.
5. Set up the executors
The performance framework relies on distributed executors to send messages into the providers being tested. The executors can run standalone in their own VM and act as "remote" JMS connections, or colocated, in which case they are deployed as JBoss services and simulate "colocated" JMS connections. Non-JBoss providers only have remote executors.
Set the providers property in build-perf.xml to the space-separated list of providers you are going to run tests against. Under the lib/providers directory create a new directory for each provider (using the name you specified in the build file) and place the required client JARs for each provider under this directory. Subdirectories are allowed, so it's often enough to just copy the lib directory of the third-party provider into lib/providers/provider-name. The remote executors for the given provider will have all JARs found under this directory on their classpath.
For JBossMQ, place jnp-client.jar, jbossmq-client.jar, and concurrent.jar from $JBOSS_HOME/client to lib/providers/jbossmq. For Messaging, use jnp-client.jar and messaging-client.jar.
5. Deploy the executors
In order to correctly deploy the colocated executors, the framework relies on the JBOSS_HOME environment variable. It assumes directories $JBOSS_HOME/server/messaging and $JBOSS_HOME/server/jbossmq exist.
cd $PERF_HOME ant sar ant start-executors
5. Deploy destinations
Some comparison performance tests only make sense when different providers' destinations are tuned to be "equivalent". If you need such destinations, add them to messaging-destinations-service.xml, jbossmq-destinations-service.xml, etc and then deploy these destinations with
ant deploy-destinations
6. Configure the test run
TODO: this only shows Messaging and JBoss MQ.
Make sure your perf.xml <providers> section looks similar to:
<provider name="JBossMessaging"> <factory>org.jnp.interfaces.NamingContextFactory</factory> <url>jnp://localhost:1099</url> <pkg>org.jboss.naming:org.jnp.interfaces</pkg> <executor name="REMOTE" url="rmi://localhost:7777/remote-messaging"></executor> <executor name="REMOTE2" url="rmi://localhost:7777/remote-messaging2"></executor> <executor name="COLOCATED" url="rmi://localhost:7777/local-messaging"></executor> <executor name="COLOCATED2" url="rmi://localhost:7777/local-messaging2"></executor> </provider> <provider name="JBossMQ"> <factory>org.jnp.interfaces.NamingContextFactory</factory> <url>jnp://localhost:1199</url> <pkg>org.jboss.naming:org.jnp.interfaces</pkg> <executor name="REMOTE" url="rmi://localhost:7777/remote-jbossmq"></executor> <executor name="REMOTE2" url="rmi://localhost:7777/remote-jbossmq2"></executor> <executor name="COLOCATED" url="rmi://localhost:7777/local-jbossmq"></executor> <executor name="COLOCATED2" url="rmi://localhost:7777/local-jbossmq2"></executor> </provider>
Also, the performance test configuration should be similar to:
<performance-test name="Throughput O KB Message Non-Persistent Non-Transactional, 1 sender, 1 receiver"> <graph> <x job="send" result="yes" metric="rate" label="measured send rate (msg/s)"></x> <y job="receive" result="yes" metric="rate" label="measured receive rate (msg/s)" ></y> </graph> <message-size>0</message-size> <messages>10000</messages> <drain></drain> <parallel> <send rate="200" executor="COLOCATED"></send> <receive executor="COLOCATED2"></receive> </parallel> <parallel> <send rate="400" executor="COLOCATED"></send> <receive executor="COLOCATED2"></receive> </parallel> <parallel> <send rate="800" executor="COLOCATED"></send> <receive executor="COLOCATED2"></receive> </parallel> <parallel> <send rate="1000" executor="COLOCATED"></send> <receive executor="COLOCATED2"></receive> </parallel> <parallel> <send rate="1500" executor="COLOCATED"></send> <receive executor="COLOCATED2"></receive> </parallel> <parallel> <send rate="2000" executor="COLOCATED"></send> <receive executor="COLOCATED2"></receive> </parallel> <parallel> <send rate="2500" executor="COLOCATED"></send> <receive executor="COLOCATED2"></receive> </parallel> <parallel> <send rate="3000" executor="COLOCATED"></send> <receive executor="COLOCATED2"></receive> </parallel> <execution provider="JBossMessaging"></execution> <execution provider="JBossMQ"></execution> </performance-test>
7. Run the test
ant run
A successful test run should generate output similar to
ovidiu@Gatecrasher /cygdrive/c/work/src/cvs/jboss-head/jms/perf $ ant Buildfile: build.xml init: compile: rmi-sar: [delete] Deleting: C:\work\src\cvs\jboss-head\jms\perf\output\classes\META-INF\jboss-service.xml [copy] Copying 1 file to C:\work\src\cvs\jboss-head\jms\perf\output\classes\META-INF [jar] Building jar: C:\work\src\cvs\jboss-head\jms\perf\output\lib\rmi-executor.sar [delete] Deleting: C:\work\src\cvs\jboss-head\jms\perf\output\classes\META-INF\jboss-service.xml [copy] Copying 1 file to C:\work\src\cvs\jboss-head\jms\perf\output\classes\META-INF [jar] Building jar: C:\work\src\cvs\jboss-head\jms\perf\output\lib\rmi-executor-scoped.sar sar: run: runner: [java] 01:53:28,389 INFO [PerformanceTest] [java] 01:53:28,389 INFO [PerformanceTest] Performance Test "Throughput O KB Message Non-Persistent Non-Transactional, 1 sender, 1 receiver" [java] 01:53:28,479 INFO [PerformanceTest] [java] 01:53:28,479 INFO [PerformanceTest] executor rmi://localhost:7777/standalone on-line [java] 01:53:28,499 INFO [PerformanceTest] executor rmi://localhost:7777/colocated-messaging on-line [java] 01:53:28,499 INFO [PerformanceTest] executor rmi://localhost:7777/colocated-messaging2 on-line [java] 01:53:28,499 INFO [PerformanceTest] [java] 01:53:28,499 INFO [PerformanceTest] Execution 1 (provider JBossMessaging) [java] 01:53:30,082 INFO [PerformanceTest] 0. DRAIN JOB drained 0 messages [java] 01:53:30,082 INFO [PerformanceTest] 1. PARALLEL [java] 01:53:39,085 INFO [PerformanceTest] SEND JOB sent 1000 messages in 4677 ms at a rate of 213.81 messages/sec [java] 01:53:39,085 INFO [PerformanceTest] RECEIVE JOB received 1000 messages in 4697 ms at a rate of 212.9 messages/sec [java] 01:53:39,085 INFO [PerformanceTest] 2. PARALLEL [java] 01:53:41,708 INFO [PerformanceTest] SEND JOB sent 1000 messages in 2473 ms at a rate of 404.37 messages/sec [java] 01:53:41,708 INFO [PerformanceTest] RECEIVE JOB received 1000 messages in 2483 ms at a rate of 402.74 messages/sec [java] 01:53:41,708 INFO [PerformanceTest] 3. PARALLEL [java] 01:53:43,070 INFO [PerformanceTest] SEND JOB sent 1000 messages in 1231 ms at a rate of 812.35 messages/sec [java] 01:53:43,070 INFO [PerformanceTest] RECEIVE JOB received 1000 messages in 1091 ms at a rate of 916.59 messages/sec [java] 01:53:43,070 INFO [PerformanceTest] 4. PARALLEL [java] 01:53:45,234 INFO [PerformanceTest] SEND JOB sent 1000 messages in 1252 ms at a rate of 798.72 messages/sec [java] 01:53:45,234 INFO [PerformanceTest] RECEIVE JOB received 1000 messages in 2042 ms at a rate of 489.72 messages/sec [java] 01:53:45,244 INFO [PerformanceTest] [java] 01:53:45,244 INFO [PerformanceTest] executor rmi://localhost:7777/standalone on-line [java] 01:53:45,254 INFO [PerformanceTest] executor rmi://localhost:7777/colocated-jbossmq on-line [java] 01:53:45,264 INFO [PerformanceTest] executor rmi://localhost:7777/colocated-jbossmq2 on-line [java] 01:53:45,264 INFO [PerformanceTest] [java] 01:53:45,264 INFO [PerformanceTest] Execution 2 (provider JBossMQ) [java] 01:53:45,524 INFO [PerformanceTest] 0. DRAIN JOB drained 0 messages [java] 01:53:45,524 INFO [PerformanceTest] 1. PARALLEL [java] 01:53:50,802 INFO [PerformanceTest] SEND JOB sent 1000 messages in 4957 ms at a rate of 201.73 messages/sec [java] 01:53:50,802 INFO [PerformanceTest] RECEIVE JOB received 1000 messages in 5068 ms at a rate of 197.32 messages/sec [java] 01:53:50,802 INFO [PerformanceTest] 2. PARALLEL [java] 01:53:53,445 INFO [PerformanceTest] SEND JOB sent 1000 messages in 2553 ms at a rate of 391.7 messages/sec [java] 01:53:53,445 INFO [PerformanceTest] RECEIVE JOB received 1000 messages in 2563 ms at a rate of 390.17 messages/sec [java] 01:53:53,445 INFO [PerformanceTest] 3. PARALLEL [java] 01:53:55,078 INFO [PerformanceTest] SEND JOB sent 1000 messages in 1272 ms at a rate of 786.16 messages/sec [java] 01:53:55,078 INFO [PerformanceTest] RECEIVE JOB received 1000 messages in 1302 ms at a rate of 768.05 messages/sec [java] 01:53:55,078 INFO [PerformanceTest] 4. PARALLEL [java] 01:53:56,279 INFO [PerformanceTest] SEND JOB sent 1000 messages in 1151 ms at a rate of 868.81 messages/sec [java] 01:53:56,279 INFO [PerformanceTest] RECEIVE JOB received 1000 messages in 1171 ms at a rate of 853.97 messages/sec init: compile: charter: runner: [java] 01:53:58,893 INFO [Runner] charts created BUILD SUCCESSFUL Total time: 33 seconds
The graph generated by the test is linked to output/results/benchmark-results.html
8. Undeploy the executors and destinations
ant stop-executors ant undeploy-destinations
Known problems
Several messaging providers have problems with performance tests.
JBoss MQ sometimes fails under large throughput. The failure is not easily recoverable, JBoss MQ has to be brought back to life by stopping the JBoss instance, removing data directory of the jbossmq configuration, and restarting.
Qpid M1 fails on large volumes, it consumes large amounts of memory and uses 100% CPU for long periods without making any progress. Some tests cause Qpid to OOME.
ActiveMQ can only send 33826 zero-sized messages to a queue, attempts to send more messages block until the previous messages are received. I haven't found any way to configure this behavior. There is a test in the test suite which attempts to send 50000 messages in the queue and receive them later, ActiveMQ will hang on it.
-
Comments