We are running ESB 4.8 on the 5.1 app server.
Our app takes in a messages in a CBR and then routes to different services (in different ESB deployments) based on what type of message it is. These services then transform the message and do something with it (i.e. output to file, insert a record into a DB, etc.).
We are currently trying to nail down performance issues we have been seeing while developing this service setup.
First off -- we have tried using InVM, and the performance is great, but we need the ability to have fail over in case of processing error and persistence for the messages. Also, while using InVM for all of our services, we saw message changes from one service show up in a different service when using the StaticRouter or StaticWiretap (pass by value may be necessary, but that point is moot, we need persistence).
Now, we have changed all of our listeners to use the jms listeners with varying levels of maxThreads (anywhere from 10 to 50 on places where we see bottlenecks, and these bottlenecks occur ONLY when going across service boundaries, which makes sense when you have more messages queued than readers available.)
Our process is that we dump X number of messages onto the bus and based on X, we know how many messages we are supposed to get on the output.
So for X = 100, we watch the messages go across the bus fairly quickly, but there are always 4-5 messages that take 30 or more seconds to make it through processing when most messages are making it through in 200-300 milliseconds. 90% of the messages have made it through in 5 or so seconds, but then we have to wait another 25 for the rest of the messages to be pulled off of one of the service queues (message aware listener) and then processed. Its almost like the reader pool for that listener (tried anywhere from 10-50 maxThreads, same issue every time), stops reading messages off of the queue for X amount of time.
Is there an easy way for us to be able to track where bottlenecks may be occurring in the ESB source? Where is the code that handles how messages are read off of the message-aware queues? We have tried using StaticRouter, StaticWiretap, our own custom router that just uses ServiceInvoker, the JMSRouter all with similar issues.