10 Replies Latest reply on Sep 13, 2012 5:21 PM by David Ward

    collecting and processing of JBPM5 BAM data

    Jeff Bride Novice

      Hi,

        we've recently had a customer request for, among other things, capturing events from a jbpm5 process engine executing in SY to support business activity monitoring.

        the purpose of this post is to discuss what this might look like in SY

       

        JPAWorkingMemoryDbLogger

           jbpm5 currently ships with the following class to capture and persist audit trail events from the process engine :  https://github.com/droolsjbpm/jbpm/blob/master/jbpm-bam/src/main/java/org/jbpm/process/audit/JPAWorkingMemoryDbLogger.java

           JPAWorkingMemoryDbLogger persists using the same transaction and JPA persistence-unit that the jbpm5 process engine is operating in.

           Although JPAWorkingMemoryDbLogger has been demonstrated to slow down the process engine in performance tests .... the implementation is straight-forward and there are use-cases where it is may be important.

           recommend that JPAWorkingMemoryDbLogger be supported in SY and optionally enabled in the switchyard.xml (ie:    <bam type="JPA" />    )

       

       

        AsyncBamProducer / Consumer

           in load tests, we have found the jbpm5 process engine to be fast.  in environments where the engine is experiencing more load .... it's important to avoid anything that could potentially slow it down.

           we have observed that JPAWorkingMemoryDbLogger reduces the speed of the engine by a factor of 2-3. 

          Subsequently, another possibility for capturing and processing of audit trail events is an async approach where events are sent to a JMS broker optimized for speed (ie:  hornetq or MRG-M)

           on a different thead (most likely operating on a different jvm on a different OS), a JMS consumer receives the events and persists in batch to a data warehouse

           this architecture has the dual benifit of relieving the single greatest bottleneck on the jbpm5 process engine as well as persisting those events in a data warehouse specifically tuned for that purpose.

           an implementation of the AsyncBamProducer can be found here:  https://github.com/jboss-sso/processFlowProvision/blob/master/common/src/main/java/org/jboss/processFlow/bam/AsyncBAMProducer.java

           recommend that AsyncBAMProducer (or something similar) be supported in SY and optionally enabled in switchyard.xml (ie:  <bam type="ASYNC" /> )

       

       

        KnowledgeRuntimeLoggerFactory

            somewhat related to business activity monitoring is the KnowledgeRuntimeLogger functionality that drools provides to capture low-level rules engine events.

            this is already supported in SY and exposed as a configurable option in switchyard.xml (ie:  <audit type="CONSOLE" />  )

            this drools knowledgesession logging is useful.

            recommend that this functionality remain in SY and that BAM processing options (as mentioned in the above two) be in addition to the existing <audit .... /> config .

       

      thoughts ?

       

      thanks!   jeff

        • 1. Re: collecting and processing of JBPM5 BAM data
          Keith Babo Master

          Hey Jeff,

           

          Chatted with David about this briefly this afternoon and I believe he's going to have a look at it tomorrow.  In terms of the BAM event producer via JMS, I would prefer to not bind it directly to JMS in the component.  There are two other mechanisms that might be suitable:

           

          1) Use our internal event producer API to publish the event and then register a separate consumer with it's own thread pool to consume the events and push out to a queue.

          2) Make it possible to call a SY service to publish audit events.

           

          Both of the above options probably require a bit more conversation, so we could always split it out as a separate JIRA and proceed with the other changes mentioned here.

           

          Thanks very much for the contribution!

           

          cheers,

          keith

          • 2. Re: collecting and processing of JBPM5 BAM data
            David Ward Master

            I agree with Keith that the BAM event producer stuff requires more discussion and should probably break that off into it's own.  Keith, you seem to have some ideas on how you migth want that to work. Did you want to create a JIRA for it?

             

            Regarding the work to integrate org.jbpm.process.audit.JPAWorkingMemoryDbLogger, as you did here: http://goo.gl/yguf5 , that can be done in a relatively straightforward manner.  Shall we just start with that?  If so, I can create a JIRA and pick out the stuff I want from Jeff's contribution. (Thanks Jeff!)  Shouldn't be too much work.

            • 3. Re: collecting and processing of JBPM5 BAM data
              Jeff Bride Novice

              keith / David,

                thanks for the replies.

                i agree with starting with work to integrate jbpm5's JPAWorkingMemoryDbLogger.  However, my initial attempt at this is not what we want  ... so please disregard those commits.

                depending on your priorities David .... if you have time to implement this quickly that would be excellent.

                otherwise, if you don't think you'll be able to get around to this for awhile .... I could also take a second stab at this and submit a pull request for your review when i've completed.

               

              thank you!

               

              jeff

              • 4. Re: collecting and processing of JBPM5 BAM data
                David Ward Master

                Can you please elaborate on why your initial attempt "is not what we want"?

                • 5. Re: collecting and processing of JBPM5 BAM data
                  Jeff Bride Novice

                  sure.  Originally, my thought was to support JPAWorkingMemoryDbLogger as an additional option to the existing audit types (ie:   <audit type="JPA" /> )

                  when configuring a SY service, users would have to select one from :    CONSOLE, FILE, THREADED_FILE or JPA

                   

                  jeff d. pointed out however that what we really want is the ability to potentially handle org.drools.event.process.ProcessEventListener events independent of the other three drools audit related interfaces :

                      org.drools.event.WorkingMemoryEventListener

                      org.drools.event.AgendaEventListener

                      org.drools.event.RuleBaseEventListener

                   

                  meaning, we want to allow a SY user the option to enable capture of all  events (as per all 4 drools interfaces mentioned above) and process as per the existing selection:   CONSOLE, FILE or THREADED_FILE

                  the use case for this data would be typically for debugging of the rules / process engine.

                   

                  in addition to, we want to allow a SY user the option to enable capture of BAM events (org.drools.event.process.ProcessEventListener only ) and process immediately via JPA or async as per what Keith has in mind.

                   

                  does that clarify ?

                  • 6. Re: collecting and processing of JBPM5 BAM data
                    David Ward Master

                    It does, thanks. Hmmm...  A few thoughts:

                    1. I think this should now be one jira instead of two, given that I'm now leaning toward a more generic EventListener solution.
                    2. Per #1, it might be good to allow the user to specify any number of <eventListener>s in the <implementation.rule> and <implementation.bpm> sections.  The behind-the-scenes code would figure out what type of EventListener the class is, and register it correctly based on that.
                    3. Wouldn't it be org.drools.event.rule.WorkingMemoryEventListener instead of org.drools.event.WorkingMemoryEventListener, and org.drools.event.rule.AgendaEventListener instead of org.drools.event.AgendaEventListener, so that we're using knowledge-api instead of drools-core?  Or maybe per #2 above, we support implementing either...
                    • 7. Re: collecting and processing of JBPM5 BAM data
                      Jeff Bride Novice

                      that sounds good David.

                      yes .. you are correct about working with knowledge-api instead of drools-core.  sorry about that.

                      • 8. Re: collecting and processing of JBPM5 BAM data
                        Jeff Bride Novice

                        David / Keith ... i've created a JIRA for work discussed in this forum:   https://issues.jboss.org/browse/SWITCHYARD-974

                        • 9. Re: collecting and processing of JBPM5 BAM data
                          David Ward Master

                          Thanks Jeff.  I've taken ownership of it.

                          • 10. Re: collecting and processing of JBPM5 BAM data
                            David Ward Master

                            Jeff, please read the comment on https://issues.jboss.org/browse/SWITCHYARD-974.  I've added a generic eventListener registration configuration to both the Rules and BPM components, as well as adding the jbpm-bam jar file to the AS7 module, so this should provide what you require. Once the pull requests get processed, the jars get deployed via maven to the nexus repo, and the next Hudson build completes, I would appreciate if you could give it a shot.  Thanks!