concurrent consuming - amq - jvm based limitation?

[kwakfuse]

We are testing performance of our newly developed software. We are using camel/activemq extensively. camel 2.5, amq 5.4.2

We have found a strange performance limitation.

Here is description of our test.

For test purposes we have stripped down our application to diagnose this problem.

Each machine mentioned is 8 core/24 gig mem Dell Poweredge R410 running RHEL5.5

We have a java application reading raw binary messages from a backspool file, encoding the messages into xml and sending to a camel jms endpoint.

The java application can easily send messages at a rate over 3500 messages per second. Our target is to process 3000 messages per seconds over multiple machines. We are hoping to process at minimum 1000 messages per machine in a cluster(hopefully more).

The broker is running on a separate machine.

For our test, when the camel endpoint receives a message, it will delay a specified(property file) number of milliseconds.

We can set concurrent consumers on the camel endpoint via property file as well.

As expected, we saw the longer the processing time, the more consumers were needed.

For a 20ms processing time adding consumers increased performance until we got to around 27 consumers. Any more and performance got worse. 27 consumers gave us around 1000 messages per second.

In reality our most time critical application camel route will take more than 20ms so this is a problem.

At first we thought there was a broker problem. Even though we have no flow control configured we saw the java app?s producer slow down when we added consumers to the camel endpoint.

We then duplicated the consuming camel application on another machine and we got almost double performance. Then we moved this second jvm to the original machine. This left us with 2 consuming camel endpoints, each in their own jvm on one machine. This also gave us almost double the performance. This makes it seem like there is NOT a broker problem. It sounds like a jvm resource problem on the consuming side which causes the broker to push back on the producer. Again?we have no flow control configured so I?m not sure what would cause this.

btw. jms pool on java producer side is configured to 8 connections/100 sessions. jms pool in camel consuming side is configured 8 connections/500 sessions

Edited by: kwakfuse on Mar 27, 2011 5:21 PM

Edited by: kwakfuse on Mar 28, 2011 7:23 PM

[garytully]

camel-jms layers over spring-jms and concurrent consumers in a spring container share an underlying JMS connection if the CACHE_LEVEL != CACHE_NONE.

My guess is that the difference between the second jvm test and increased concurrent consumers is that a second connection is in the mix when you introduce another jvm.

You could add a second route, or change the cache level to CACHE_NONE and each concurrent consumer will get its own connection, which will improve the parallelism.