Use of JBPM and Drools to build a near real time applications to process million of events a day

[joao.pereira]

I'm not sure that this is the correct place to expose this, as I'm seeking opinions about a stack of technologies that are being recommended to me to build an application. Let me just give some background. My system will receive around 1 million (let's assume a peak of 15 events per second) of "external" events per day, coming from a mobile telecommunications network. The system will need to do some tasks depending on a series of simple rules applied to incoming event. Some of these tasks can be executed in parallel to increase throughput. These tasks generally involve updating some database and communicating with external services, where some type of orchestration is required. The business rules will not change frequently, nor the processes, i.e., which external services will be notified, maybe the body of requests to these external service need some change time to time. Now, I'm being asked to build such application using Jboss JBPM and Drools, mainly because there’s some lack of knowledge about these processes and rules, and people are trying go with very flexible application where they can design the processes and the rules to have the system doing what they want. JBPM and Drools are being recommended as the technologies to build such kind of application. I must also say that this system requires to be up 24/7 as it is critical to the business and it will require to update constantly the network elements to have the network running. Each event must be processed near real time, or as fast as possible and there is not manual tasks involved, everything should be automated. I read from JBPM 5 documentation that even if I design a business process with parallel activities, they will be technically executed in the same thread. Maybe JBPM was not designed to handle such type of requirements and that's why I'm seeking some advice from someone that built a system with similar requirements using JBPM and Drools and what challenges will I face doing this system with the recommended technologies. Thanks.

[salaboy21]

Hi Joao,

It sounds as a really good ecosystem for drools and jBPM.

Business Processes are usually more high level and you don't really need to care about how they are executed. If you are interested in very low level workflows, you can definitely spawn two different processes into two different threads, that will do the work for you.

I think that one of the important things that you need to do, is to first understand drools and jBPM in depth to be able to define the architecture that fits for your use case. Memory wise, you will need to answer questions like for example: is the JVM able to create 15 objects per second and keep all of them in memory for a week? Answering these kind of questions will help you to decide how to break apart your problem into different subproblems and how drools and jbpm fits into that.

Drools & jBPM has being designed to be flexible enough to work in almost every scenario, so I'm sure that they can help you a great deal into your current project, but you will need to spend some time learning about the projects in order to leverage their benefits.

HTH

[arif.mohammed]

Earlier I had worked in Telecom domain particularly in OSS where we have built custom Events Processing Engine using java. We had different types of rules(Supression rule, event correlation etc) for processing events and producing alarms. Since I have gone through the development of that engine I see the CEP feature of jBPM+drools is perfect fit which provides you the out of box solution.

[arif.mohammed]

But you need to check how does jBPM+drools will scale horizontally and see how you can integrate some in memory caching mechanisms like terracotta cache (not sure whether jBPM+drools can support this) instead of DB persistence