1 2 Previous Next 28 Replies Latest reply on Nov 26, 2008 5:50 AM by kukeltje

    Analytical and operational dashboards

    heiko.braun

      This refers to https://jira.jboss.org/jira/browse/JBPM-2.
      (Don't laugh, some day I can tell my kids "... and I did JBPM-2, really'")

      Ok ladies and gentlemen, no more talking about BI and BAM.
      From now on we talk about the analytical and the operational dashboard.
      The former offers analytical insights, mostly based on historical data, i.e. the number of process executions compared to last month, while the later will help monitoring operations, i.e. alerting when a job cannot be executed after a number of retries.

      To begin with I'll wrap up, what people posted to the JIRA:


      1. jBPM should store the duration and start date of a state/process in the logs
      2. Write some detailed info on wiki
      3. Show processinstance and data
      4. day/week/month overviews
      5. near realtime data per process
      6. produce graphs of day/week/month overviews

      Some more ideas i got from a requirements document...
      - Number of cases processed in through a particular process definition
      - Number of cases flowing through a particular activity
      - Longest work item delay by activity
      - Workitems awaiting a particular task
      - Workitems in a specific worklist claimed and/or completed by a particular user
      - Longest end-to-end delay
      - Mean time to approve by activity by participant
      - Mean end-to-end completion time.



      Before going full blown on statistics a simple extensible audit trail would be nice, pretty much every commercial workflow toolset has this available in
      one way or another in both their standard clients as well as in their api's. So to me a full blown graphical console doesnt add much to end-users. It is more important to make the jbpm data available through an easy to use api set then to build a console wich wont satisfy all users. The current api command fails in my view on this one as it only offers an all or nothing aproach to getting processlog instances on a process instance....


        • 1. Re: Analytical and operational dashboards
          heiko.braun

           


          So to me a full blown graphical console doesnt add much to end-users. It is more important to make the jbpm data available through an easy to use api


          I don't agree with this. IMO the API is not necessary because it has a DB schema already, which is ready for consumption. Actually I think quiet the opposite of what you say. Adding an API atop of the schema would just narrow the possibilities to query according to users demands.

          However, it is right that customization plays a huge part here. But to begin with I think we should be able to define a set of common analytical and operational questions user typical ask and build that into a console.




          • 2. Re: Analytical and operational dashboards
            heiko.braun

             


            ...produce graphs of day/week/month overviews


            That's a too unspecific. We can produce graphs of information that's put into relation.
            I.e. comparing the number of process instances this month with the number of process instances last month.

            What are common questions people ask themselves when looking at the console,
            or the process logs? In what context to problems appear?



            • 3. Re: Analytical and operational dashboards
              kukeltje

              JBPM-2, you solved it, I reported it ;-) (as well as JBPM-1 btw)

              That's a too unspecific. We can produce graphs of information that's put into relation.
              I.e. comparing the number of process instances this month with the number of process instances last month.
              And this for comparing weeks, days, is that to unspecific? It's just like downdrilling. Basic view is e.g. comparing months (like in the dow-jones...ahhh bad example nowadays ;-) ). Clicking on a month then you see the weeks of this month, or select start and end week (like a sliding window) and downdrilling shows a day with e.g. the hour as the unit)

              In what context to problems appear?

              Sometimes in the context of a month, sometimes a week, sometimes a day. And they are not always 'problems' sometimes it is detecting a trend so more people can be hired the comming day's. Current values (may different ones like number of instances, duration etc) visible in relation to the average of a certain period (which in turn is debatable)

              • 4. Re: Analytical and operational dashboards
                kukeltje

                 

                "heiko.braun@jboss.com" wrote:

                So to me a full blown graphical console doesnt add much to end-users. It is more important to make the jbpm data available through an easy to use api


                I don't agree with this. IMO the API is not necessary because it has a DB schema already, which is ready for consumption. Actually I think quiet the opposite of what you say. Adding an API atop of the schema would just narrow the possibilities to query according to users demands.

                However, it is right that customization plays a huge part here. But to begin with I think we should be able to define a set of common analytical and operational questions user typical ask and build that into a console.


                What is important is that (from my experience) users want to drill down to individual case data when using the dashboard for both operational data and analytical data (if it is still available). Since we advise people not to put all data in jBPM (as opposed to other BPM suppliers to a certain extend), but use their own domain model and storage, there should be e.g. a basic api that returns all businesskeys related to that period or an individual case (a business key being the FK between the domain and the processinstance). That way users can analyse bottlenecks more easily. Since I cannot imagine the jBPM console showing the domain data, an api with basic methods does sound interesting. Users can embed the basic operational/analytical data in their own apps then. Basic being the keyword!!

                For more complex analytical dashboards, I personally advice my customer (hope there will be more) to use a real BI/BAM (sorry) product like Pentaho or Jaspersoft, since they often need this for other analysis as well, but that is just me. And since I saw you are using jaspersoft, maybe things will turn around.

                • 5. Re: Analytical and operational dashboards
                  heiko.braun

                   

                  And this for comparing weeks, days, is that to unspecific?


                  Yes, indeed. What infomration fdo you want to comapre? Process instances, number of failed deployments, exceptions, or the average number of console users?

                  • 6. Re: Analytical and operational dashboards
                    kukeltje

                    You already mentioned those:


                    Number of cases processed in through a particular process definition
                    - Number of cases flowing through a particular activity
                    - Longest work item delay by activity
                    - Workitems awaiting a particular task
                    - Workitems in a specific worklist claimed and/or completed by a particular user
                    - Longest end-to-end delay
                    - Mean time to approve by activity by participant
                    - Mean end-to-end completion time.


                    And mean/avg should also contain stdv since that tells me more

                    All this in 'windows' e.g. week 1-6 or feb-july. What is shown can be debated upon and may be even parameter as well. e.g. show avg duration on a day when compare weeks or just weekly averages

                    'longest' is not that interesting. avg and stdv are


                    • 7. Re: Analytical and operational dashboards
                      kukeltje

                      sorry for wrong quoting

                      • 8. Re: Analytical and operational dashboards
                        tom.baeyens

                        for me it's also very hard to see what concrete information charts are necessary to detect problems. and what kind of navigation is necessary in those scenarios.

                        since we have our expert users in the jbpm meeting in two weeks, i think that spending some time on the analythics and operational dashboards there would be good to bring their expertise into the mix.

                        • 9. Re: Analytical and operational dashboards
                          salaboy21

                          I don't know if my words count in here but for my experience I think a good beginning will be some good tutorial of extracting data from the jBPM database, I mean, something to show to the new people what kind of information will they have when they start using jBPM.

                          So I was thinking to do a nice introduction about that when I read this topic.

                          But for the future I think a nice and customizable UI + Clean APIs would be nice.
                          For nice a customizable UI + Clean APIs I mean something like a Cube, where you can fill it and then play with the offline data, and choose the correct dimensions to query.
                          The offline data part I think that is very important, because we don' t wanna have users playing with the production database.

                          I hope this word helps with something...




                          • 10. Re: Analytical and operational dashboards
                            krisverlaenen

                            At this point, I believe the database itself could almost be considered an API, it defines the data that could be reasoned on. Currently there is need to put a Java API on top of that, as it will become very difficult to create one that is both generic enough to support all use cases and specific enough to offer simple methods to end users. I do believe though that it could be useful to define a "minimal" schema (the minimal data that the db should contain to make it useful), and think about making that extensible so people can add more information later (and be able to use that extra information somehow in the analysis).

                            Are there any plans to support business indicators (KPIs)? Because people usually don't really reason about "the number of process instances", more about "outstanding orders", "the time to complete an order", etc. This might be somewhat related to CEP: creating higher-level business events from lower-level process events. Just a thought for the future though, to support business users. Simple operational dashboards for admins are probably the first step towards that.

                            Heiko: I'd love to hear about your ideas / technology / progress on this next week, you don't be surprised if I ask you that question ;)

                            Kris

                            • 11. Re: Analytical and operational dashboards
                              kukeltje

                              A low level API is indeed not interesting enough certainly not in relation to the amount of work it will take. What (imo) is interesting though is getting access to outcome of the queries and to the images. The company I work(ed) for want's to show this data in the webapp of the specific application for several reasons:
                              - marketing wise (look and feel)
                              - authorization wise (often already implemented in that app, implementing it also in the/a/one of the consoles is double (or more) work
                              - functionality wise (they can decide what to show, what not etc.) Give labels a name etc (see also response below)


                              Are there any plans to support business indicators (KPIs)? Because people usually don't really reason about "the number of process instances", more about "outstanding orders",


                              Since this is very process specific, (labels), information from processdefinitions (process name, task name etc.) can (should?) be used to overcome this.

                              From my (not very broad, but still) experience, the time to complete an order (one specific I assume you mean here) is only partly interesting. It is combinations of data that tells a manager how to act.

                              Some small examples:
                              - number of open orders goes up, average of the duration of handling an (from the moment is is really being acted upon) stays the same, then the conclusion is that there are more orders coming in then can be handled. 'Downdrilling' is not needed
                              - If the number of open goes up, but the average also goes up and the average of a processdef. (including stdv) also goes up, then there most likely is a bottleneck in an external actor (system or human) being used. Downdrilling to the individual times of nodes is interesting then to see which task/node/... is causing the problem and downdrilling to get detailed info of e.g. this tasknode can be needed, e.g. is on human always slower than others, is a generic system always slow, or is it related to data!. The latter may sound strange, but we've had an issue where a fraud system required a licenseplate number to be passed to it where the customers told us they did not want a check on validity (e.g. is the combination of characters a probably existing one). So many users used '000000' for that in specific circumstances that the system became slow when searching for possible fraudulent actions with this license plate number).

                              To me, these are 'fairly simple' to implement when using e.g. Pentaho and these are operational. Thresholds are needed to be able to have a clean/empty dashboard when everything is within configured thresholds. The tresholds themselves can come from analytical data



                              • 12. Re: Analytical and operational dashboards
                                heiko.braun

                                 


                                Are there any plans to support business indicators (KPIs)?


                                If they get too domain specific, no. At least not in the beginning. A good dashboard is always customized to the users needs. I am talking about information the can be derived from default BPM terms, because we are not offering a lot of customization in the beginning. Don't underestimate it though. People have the capabilities to read "between the lines". What you described with order example, can be derived from non domain specific figures as well. IMO "outstanding orders" can be as well expressed as a particular derivation or exception in the process execution without being to much tight to that particular domain.

                                The goal should be that people with specific knowledge are able to "recognize" patterns that put them in alert state or at least raise their interest.


                                • 13. Re: Analytical and operational dashboards
                                  heiko.braun

                                  Roland actually posted some examples that are close to what I mean:


                                  If the number of open goes up, but the average also goes up and the average of a processdef. (including stdv) also goes up, then there most likely is a bottleneck in an external actor (system or human) being used.


                                  • 14. Re: Analytical and operational dashboards
                                    kukeltje

                                    Yes, but if on a list of processdefinitions and the number of running instances is preceded by the process name (in the definition) it is almost what Kris wants I think. It just does not say

                                    Open Orders: 20

                                    But just

                                    <processdefinition name>: 20

                                    For me that is close enough. If they want more detailed descriptions, customisable they can build their own screens/reports or use data gathered in a kind of generic way from an basic api

                                    1 2 Previous Next