1 2 Previous Next 23 Replies Latest reply on May 19, 2010 7:52 AM by davesiracusa

    Performance Issues With Routing (ESB 4.8)

      We are running ESB 4.8 on the 5.1 app server.

       

      Our app takes in a messages in a CBR and then routes to different services (in different ESB deployments) based on what type of message it is. These services then transform the message and do something with it (i.e. output to file, insert a record into a DB, etc.).

       

      We are currently trying to nail down performance issues we have been seeing while developing this service setup.

       

      First off -- we have tried using InVM, and the performance is great, but we need the ability to have fail over in case of processing error and persistence for the messages. Also, while using InVM for all of our services, we saw message changes from one service show up in a different service when using the StaticRouter or StaticWiretap (pass by value may be necessary, but that point is moot, we need persistence).

       

      Now, we have changed all of our listeners to use the jms listeners with varying levels of maxThreads (anywhere from 10 to 50 on places where we see bottlenecks, and these bottlenecks occur ONLY when going across service boundaries, which makes sense when you have more messages queued than readers available.)

       

      Our process is that we dump X number of messages onto the bus and based on X, we know how many messages we are supposed to get on the output.

       

      So for X = 100, we watch the messages go across the bus fairly quickly, but there are always 4-5 messages that take 30 or more seconds to make it through processing when most messages are making it through in 200-300 milliseconds. 90% of the messages have made it through in 5 or so seconds, but then we have to wait another 25 for the rest of the messages to be pulled off of one of the service queues (message aware listener) and then processed. Its almost like the reader pool for that listener (tried anywhere from 10-50 maxThreads, same issue every time), stops reading messages off of the queue for X amount of time.

       

      Is there an easy way for us to be able to track where bottlenecks may be occurring in the ESB source? Where is the code that handles how messages are read off of the message-aware queues? We have tried using StaticRouter, StaticWiretap, our own custom router that just uses ServiceInvoker, the JMSRouter all with similar issues.

        • 1. Re: Performance Issues With Routing (ESB 4.8)
          tfennelly

          You can try using the monitoring tools (see admin guide) and can also have a look at using something like JProfiler (what I use to track down  bottlenecks).

          • 2. Re: Performance Issues With Routing (ESB 4.8)

            The odd thing that I have seen is that the messages that don't get put on take almost exactly 30 seconds (30015 milliseconds, 30000 milliseconds) even if it is on my local dev machine or our hardcore server (16 core, 72 gb memory). Is there anywhere that you can think of that might cause a message to NOT get read off for 30 seconds (some setting somewhere?)

             

            Message was edited by: Brent Merle

            • 3. Re: Performance Issues With Routing (ESB 4.8)
              kconner

              Two comments come to mind

               

              First, InVM should not deliver messages to the wrong service.  If this is happening then there must be a bug somewhere.  Could you provide a log file for this?  Do you have a specific example which shows this?  Are these messages requests or replies?

               

              Second, the 30 second delay will be the delay specified to the deliverSync, after which it will retry.  This suggests that the original service did not respond and that the service invoker is delivering the message a subsequent time.

               

              If you enable DEBUG then it should show you what is happening within the pipelines, is there anything in the log files?

               

              Kev

              • 4. Re: Performance Issues With Routing (ESB 4.8)

                the InVM messages were not being delivered to the wrong service, it is like the references to the message are being passed to two places in the action chain.

                 

                i.e.

                 

                message A

                 

                A goes to a StaticRouter, the services being routed to have the InVM transport on. A gets to service 1 before service 2 and is transformed in 1. When service 2 sees the message it is the transformed version, which service 2 is not expecting, throwing weird XML parsing errors (content not allowed in prolog, etc.)

                 

                This does not always happen, but it happened enough to where it is not feasible for us to use. When I use only message aware listeners riding on JMS instead of InVM, I don't see any of these XML-based errors, so I have to assume something is not perfect with InVM.

                 

                I can't show my code, but I can tell you our service makeup for this situation:

                 

                Gateway service:

                Basic service with both listeners (message aware and gateway) have maxThread=1

                a ContentBasedRouter with cbrAlias=XPATH has two possible outputs

                (Every service beyond the gateway service has InVM enabled, with maxThreads >= 10)

                A message going through the CBR CAN be sent to two services (i.e. it can match two XPath expressions)

                When it matches the first, it goes to another service which uses StaticRouter to go to another service which handles that type of message

                When it matches the second, it goes to another service which uses StaticRouter which handles the seconds type of message.

                For the first type of message, (here is where the InVM screwiness happens), we use StaticRouter again to send to TWO services. The first uses Smooks to apply an XSL to the message, the second uses a custom action to transform the XML to an output file format. Sometimes, the second action sees the transformed message that resulted from the Smooks transformation for the first of the two services. I can tell this because when I output the message that is throwing the error in the second service, it is the transformed version of the message that resulted from the Smooks action in the first service.

                 

                Again, all services but the gateway service, use InVM as the transport method, with varying levels of maxThreads. We usually saw this happen when we tried to pump 100 or more messages through the pipeline, not when we just tried one.

                 

                To your second question, as far as I can tell, we are only using deliverAsync, we have no reason to get a response from any of our calls; our implementation is purely message transformation and delivery to different types of consumer applications.

                • 5. Re: Performance Issues With Routing (ESB 4.8)
                  mvecera

                  Dear Brent,

                   

                  I'm really interested in your problem. Could you please send me your jboss-esb.xml file with necessary action classes and a sample message(s)? I'll give it a try and inform you about my findings. Please use my email (mvecera at redhat dot com).

                   

                  Thanks,

                  Martin Vecera

                  JBoss Quality Assurance Engineer

                  • 6. Re: Performance Issues With Routing (ESB 4.8)

                    Martin,

                     

                    I sent those files to you on Tuesday, were you able to reproduce the issue I have been describing with InVM?

                     

                    Brent

                    • 7. Re: Performance Issues With Routing (ESB 4.8)
                      kconner

                      If the services are modifying the body of the message then you should have them ask for it to be 'passByValue' rather than 'passByReference', unless you know that they are the only recipient.

                       

                      If you are passing the message by reference then only a subset is copied, the body is not one of them.  Both services will see the same message body and, therefore, run interference with each other.

                       

                      Have you specified passByValue semantics for those services?

                       

                      Kev

                      1 of 1 people found this helpful
                      • 8. Re: Performance Issues With Routing (ESB 4.8)

                        I've tried using passByValue, but that turned up some other error and I didn't try any further to diagnose what that error was. We have decided to not use InVM transport at all since losing messages is not an option for our implementation.

                         

                        If it is the case that we should be using passByValue instead of the default in the case that the body is changing with multiple recipients, then I think the Programmers Guide should be a little bit more explicit about what the certain "situations" are.

                         

                        As far as I saw in the Programmers Guide, there are two paragraphs  that talk about this issue of passByValue with InVM transport and there isn't much explanation in the way of "if I use InVM or JMS as my transport, the same functionality should exist... right?" i.e. using a StaticRouter sending to InVM or JMS listeners shouldn't matter.

                         

                        I appreciate your candor on this issue and it looks like you addressed the InVM problems we saw, but I am still curious about that 30 second delay on some messages.

                        • 9. Re: Performance Issues With Routing (ESB 4.8)
                          mvecera

                          Hello,

                           

                          unfortunately, this was quite a busy week. I'll try to have a look on it the next week.

                           

                          Thanks,

                          Martin

                          • 10. Re: Performance Issues With Routing (ESB 4.8)
                            mvecera

                            Hello,

                             

                            I just quickly went through the zip file and I realized that there is no CBR used - just a static router. Do you see the same issue with this router as well? What is the message you use (it can really matter)?

                             

                            Cheers,

                            Martin

                            • 11. Re: Performance Issues With Routing (ESB 4.8)

                              Yes -- it happens with both.

                               

                              The message is JBOSS_XML with the body as a string (our XML).

                              • 12. Re: Performance Issues With Routing (ESB 4.8)
                                mvecera

                                I see. I'm still on it but I had more important tasks to do, so postponed to the next week once more...

                                Cheers,

                                Martin

                                • 13. Re: Performance Issues With Routing (ESB 4.8)
                                  mvecera

                                  Hello,

                                   

                                  I went through it and I realized that we have routing scenarios covered by our tests already. I was not able to reproduce your issue. I suggest you to check the following items:

                                  • Are the JMS messages persistent? If yes, monitor the DB usage.
                                  • Use VisualVM to monitor GC activity
                                  • Monitor blocked threads to see if there is a concurrency issue
                                  • Do you use JMS JCA provider?

                                   

                                  It is hard to determine the root cause if you do not provide the complete working example...

                                   

                                  Cheers,

                                  Martin

                                  • 14. Re: Performance Issues With Routing (ESB 4.8)
                                    davesiracusa

                                    I recently performed my own testing of JMS performance.  Here is a recap:

                                     

                                    I created chained services consist of 3 ESB services, all JMS based (transacted), the first service simply routes to the second, and the second to the third/final service which simulated an arbitrary amount of work.  Basically I wanted see how asynchronous service tiers affect latency.

                                     

                                    I used apache jmeter to perform testing.  In the case of JMS testing we used the out-of-box “JMS Point to Point” sampler.  We also developed our own aggregated version of the “JMS Point to Point” sampler so we could send unique messages to ESB.  This allowed us to correlate messages as it left jmeter and traversed services within SOA-P.

                                     

                                    In all cases we performed simulated work within the SOA-P service and recorded statistics using the jmeter “Summary Report” listener.  All these tests were performed on a single laptop, where the laptop hosted jmeter, SOA-P and Oracle.

                                     

                                    I increased the number of threads for the any gateway listeners and jms-based esb listener.  I found that specifying the number of threads for the gateway listener was not as important as specifying it for the ESB-aware listener.  It appears the real work is done in the ESB-aware worker thread.  Message en-queuing was very fast, de-queuing may take up to 200 milliseconds due to its hard coded poll period.

                                     

                                    I found similar hangups, for me performance was restored when I increased the number of pooled jms connections.

                                    This can be found in file jbossesb-properties.xml.  For my testing I increased it to 200, from 20.

                                             <property name="org.jboss.soa.esb.jms.connectionPool" value="20" />

                                     

                                    In the end I verified that unique threads were being used by printing the threadid.

                                    I watched the thread and jms pool behavior via JMXConsole and all seemed fine.

                                    1 2 Previous Next