ESB CEP support for multiple event streams

[jeffdelong]

CEP applications often are designed to correlate messages from multiple event streams. A common example (used often the Drools CEP presentations and documentation) is a Online Trading System, which has a "Home Broker Stream" and a "Stock Trader Stream".

An Enterprise Service Bus such as JBoss ESB can provide a good architecture to support this type of CEP application. The developer could create three ESB Serivces. The first would create the Drool stateful knowledge session (and somehow publish this object to make it available for the other services), insert static facts, and call fireUntilHalt. The second service would represent the Home Broker Stream. It would receive trade requests from the Home Broker (say via a SOAP message), transform the XML into Java BuyOrderEvent objects, and insert these events into the stateful knowledge session via the Home Broker Stream entry point. The third service would be similar to the second service, except that it would handle messages from the Stock Trader, which for the sake of this example, come in as XML via a JMS transport, get transformed into Java  BuyAckEvent objects and inserted into the session via the Stock Trader Stream entry point.

The rationale for having multiple services is to be able to handle different protocols and perform different transformations.

[dward]

Thanks Jeff.  It's good to break this out into its own conversation, as I believe everything else in this thread has been addressed (for 4.9).  Let me spend some brain-cycles on this and respond in a bit.

[jeffdelong]

In reviewing the original thread I think there is another point that was raised that deserves addressing, and that it scalability. I will start another thread and email Edson so that he can join in.

[dward]

Not sure that another thread is necessary for the performance/scalability discussion.  It's kinda ingrained as an aspect of this use case, right?

On a somewhat related note, I'm concerned about thread safety.  Gut first reaction to support what you want would be to have another action (or another configuration property of this action), which specifies the a combination of the original ruleProperties (which is the "key" to how we cache stateful sessions today) PLUS an entry-point name perhaps?  The problem with that, in my mind, is synchronization, as stateful sessions are not thread-safe.  In BusinessRulesProcessor, we currently lock() on access to the cached session.  To have multiple actions being executed by services in different threads lock on the same stateful session is not very condusive to CEP applications, where Very High throughput is expected.

[dward]

Just to keep readers of this thread updated, Jeff and I received this information from Edson:

"[Drools Fusion] is designed to support one thread per entry-point.  So, if a session has the main entry point, plus entry points A, B and C, there could be 4 threads feeding events/facts in the working memory.  Just each entry-point must not have more than one thread concurrently executing actions on it."

This is good news.  Thanks, Edson.  I will come up with a proposed plan of attack tomorrow morning and respond back here for comments.

[kconner]

Please keep in mind that all new development has missed the cut-off for EBS 4.9 and will have to wait until later.

Also, the majority of bugs have now missed cut-off with only those deemed 'critical' being allowed in to the release schedule for 4.9.

Kev

[jeffdelong]

I think what we are discussing can be implemented post ESB 4.9; it is an important requirement but also one whose design needs a lot of thought. The JIRAs that have been resolved go a long way to solidifying ESB support for CEP. We should continue this discussion as part of planning for next release.

[dward]

Kev,

I created this jira to track this feature request: https://jira.jboss.org/browse/JBESB-3396

I originally set it as 4.9 CP1, but if you think it should be set to a different Fix Version, please feel free to change it.

Thanks,

David

[dward]

I think we should start discussing how this would best be implemented.  The big questions is, should we do it in BusinessRulesProcessor, or create a new action?  Let me explain....

We could add a configuration option to BusinessRulesProcessor, saying this action instance is only for adding facts into the specified entry-point.  This could prove very confusing, however, considering the ability to specify entry-point names within the object-path(s) configuration section is already there.  What happens if those two names don't align?  ConfigurationException thrown, probably, but why give users the ability to mess up like that?

We could create a new action, perhaps called "EntryPointInserter" (or whatever).  Obviously it would need a config option which somehow identifies the stateful session created by BusinessRulesProcessor.  For clarity of purpose, it's ONLY job would be to insert facts into ONE entry-point.  There would still have to be thread-synchronization done, however.  The most  annoying one would be because you can already insert into entry-points  from BusinessRulesProcessor, what happens if that gets done at the same  time this separate action tries to?  So, we'd have to be cautious of this.

Thoughts?

[jeffdelong]

I like the idea of a separate action for adding events into the entry point.

The use would need to make sure that they configured each EntryPointINserter and the BusinessRulesProcessor with different entry points.