9 Replies Latest reply on Aug 3, 2010 8:03 AM by Jeff DeLong

    ESB CEP support for multiple event streams

    Jeff DeLong Master

      CEP applications often are designed to correlate messages from multiple event streams. A common example (used often the Drools CEP presentations and documentation) is a Online Trading System, which has a "Home Broker Stream" and a "Stock Trader Stream".

       

      An Enterprise Service Bus such as JBoss ESB can provide a good architecture to support this type of CEP application. The developer could create three ESB Serivces. The first would create the Drool stateful knowledge session (and somehow publish this object to make it available for the other services), insert static facts, and call fireUntilHalt. The second service would represent the Home Broker Stream. It would receive trade requests from the Home Broker (say via a SOAP message), transform the XML into Java BuyOrderEvent objects, and insert these events into the stateful knowledge session via the Home Broker Stream entry point. The third service would be similar to the second service, except that it would handle messages from the Stock Trader, which for the sake of this example, come in as XML via a JMS transport, get transformed into Java  BuyAckEvent objects and inserted into the session via the Stock Trader Stream entry point.

       

      The rationale for having multiple services is to be able to handle different protocols and perform different transformations.

        • 1. Re: ESB CEP support for multiple event streams
          David Ward Master

          Thanks Jeff.  It's good to break this out into its own conversation, as I believe everything else in this thread has been addressed (for 4.9).  Let me spend some brain-cycles on this and respond in a bit.

          • 2. Re: ESB CEP support for multiple event streams
            Jeff DeLong Master

            In reviewing the original thread I think there is another point that was raised that deserves addressing, and that it scalability. I will start another thread and email Edson so that he can join in.

            • 3. Re: ESB CEP support for multiple event streams
              David Ward Master

              Not sure that another thread is necessary for the performance/scalability discussion.  It's kinda ingrained as an aspect of this use case, right?

               

              On a somewhat related note, I'm concerned about thread safety.  Gut first reaction to support what you want would be to have another action (or another configuration property of this action), which specifies the a combination of the original ruleProperties (which is the "key" to how we cache stateful sessions today) PLUS an entry-point name perhaps?  The problem with that, in my mind, is synchronization, as stateful sessions are not thread-safe.  In BusinessRulesProcessor, we currently lock() on access to the cached session.  To have multiple actions being executed by services in different threads lock on the same stateful session is not very condusive to CEP applications, where Very High throughput is expected.

              • 4. Re: ESB CEP support for multiple event streams
                David Ward Master

                Just to keep readers of this thread updated, Jeff and I received this information from Edson:

                 

                "[Drools Fusion] is designed to support one thread per entry-point.  So, if a session has the main entry point, plus entry points A, B and C, there could be 4 threads feeding events/facts in the working memory.  Just each entry-point must not have more than one thread concurrently executing actions on it."

                 

                This is good news.  Thanks, Edson.  I will come up with a proposed plan of attack tomorrow morning and respond back here for comments.

                • 5. Re: ESB CEP support for multiple event streams
                  Kevin Conner Master

                  Please keep in mind that all new development has missed the cut-off for EBS 4.9 and will have to wait until later.

                   

                  Also, the majority of bugs have now missed cut-off with only those deemed 'critical' being allowed in to the release schedule for 4.9.

                   

                  Kev

                  • 6. Re: ESB CEP support for multiple event streams
                    Jeff DeLong Master

                    I think what we are discussing can be implemented post ESB 4.9; it is an important requirement but also one whose design needs a lot of thought. The JIRAs that have been resolved go a long way to solidifying ESB support for CEP. We should continue this discussion as part of planning for next release.

                    • 7. Re: ESB CEP support for multiple event streams
                      David Ward Master

                      Kev,

                       

                      I created this jira to track this feature request: https://jira.jboss.org/browse/JBESB-3396

                       

                      I originally set it as 4.9 CP1, but if you think it should be set to a different Fix Version, please feel free to change it.

                       

                      Thanks,

                      David

                      • 8. Re: ESB CEP support for multiple event streams
                        David Ward Master

                        I think we should start discussing how this would best be implemented.  The big questions is, should we do it in BusinessRulesProcessor, or create a new action?  Let me explain....

                         

                        We could add a configuration option to BusinessRulesProcessor, saying this action instance is only for adding facts into the specified entry-point.  This could prove very confusing, however, considering the ability to specify entry-point names within the object-path(s) configuration section is already there.  What happens if those two names don't align?  ConfigurationException thrown, probably, but why give users the ability to mess up like that?

                         

                        We could create a new action, perhaps called "EntryPointInserter" (or whatever).  Obviously it would need a config option which somehow identifies the stateful session created by BusinessRulesProcessor.  For clarity of purpose, it's ONLY job would be to insert facts into ONE entry-point.  There would still have to be thread-synchronization done, however.  The most  annoying one would be because you can already insert into entry-points  from BusinessRulesProcessor, what happens if that gets done at the same  time this separate action tries to?  So, we'd have to be cautious of this.

                         

                        Thoughts?

                        • 9. Re: ESB CEP support for multiple event streams
                          Jeff DeLong Master

                          I like the idea of a separate action for adding events into the entry point.

                           

                          The use would need to make sure that they configured each EntryPointINserter and the BusinessRulesProcessor with different entry points.