12 Replies Latest reply on Feb 5, 2009 2:36 PM by kukeltje

    Dynamic Fork + External Async WS invocation + problem

    clandestino_bgd

      Hi people, in short:

      My Process:
      Just after Start state, I have dynamic fork (Action Handler defined on Node, very similar to the one on wiki). In each child I have a Task node, on which event (task-create) I have an ActionHandler which invokes WS asynchronously, by passing taskInstanceId as parameter.
      When WS finishes the operation, it invokes callback method which ends the task instance.

      And the problem:
      When I start the process instance, if I have a significant number of children, it happens that the 1st WS does the callback before all child tokens are generated, which results in NPE when I try to get task instance in callback method.

      Ok, seems, that new tokens and created task instances are not persisted in DB, but only in memory and this is according to docs, but can I somehow change the default behavior and support the described scenario?
      Or maybe somebody has an idea how to support "concurrent" WS invocations without explicitely call to each of them?

      Retrying callback or WS timeout can fix the problem, but this is configurable on WS side, not JBPM.

      Any idea, much appreciated.
      best,
      milan

        • 1. Re: Dynamic Fork + External Async WS invocation + problem
          kukeltje

           

          "clandestino_bgd" wrote:

          which results in NPE when I try to get task instance in callback method.


          Stacktrace?

          "clandestino_bgd" wrote:

          Ok, seems, that new tokens and created task instances are not persisted in DB , but only in memory and this is according to docs,
          because of the delay? or not at all?

          "clandestino_bgd" wrote:

          but can I somehow change the default behavior and support the described scenario?

          What default behaviour? You can try to generate all tasknodes async...

          "clandestino_bgd" wrote:

          Or maybe somebody has an idea how to support "concurrent" WS invocations without explicitely call to each of them?


          It is not a WS problem, but a generic one...


          • 2. Re: Dynamic Fork + External Async WS invocation + problem
            clandestino_bgd

            Hi, 1st, thanks for looking into this.
            process looks like this: S-F-T-J-E
            S: Start state,
            D: Dynamic Fork (based on WF variable creates N new tokens),
            T: Task Node (on create has Action Handler that calls WS), Task ends when WS make a callback, child tokens wait in
            J: Join.
            E: End state

            - NPE in callback method:

            JbpmTaskInstance taskInstance = (JbpmTaskInstance) context.getTaskInstance(taskInstanceId);
            log.debug("documentId: " + taskInstance.getDocumentId());
            


            context.getTaskInstance() returns null, nothing particulary intersesting in stacktrace, simply task instance with that ID cannot be found neither in context nor in DB in that moment.

            My understanding of the subject, and please correct me if I am wrong is that in Dynamic Fork which looks like this:
            for (int i = 0; i < numberOfDocuments; i++) {
            String documentId = documentList.get(i);
            // build a new token name
            String tokenName = JPDLConstants.DOCUMENT_PREFIX+ documentId;
            final Token newToken = new Token(rootToken, tokenName);
            newToken.setTerminationImplicit(true);
            context.getJbpmContext().getSession().save(newToken);
            final ExecutionContext newExecutionContext = new ExecutionContext(newToken);
            newExecutionContext.getJbpmContext().getSession().save(newToken);
            node.leave(newExecutionContext);
            }
            


            after new ExecutionContext is constructed and node.leave(context) is called for Token #1, the task instance in that token is created, action handler is invoked and then the next token is processed in for loop, but task instance is still not persisted in DB physically.

            What I would like to have is when I call this action handler and WS in it, to be able to save that task instance in DB, so callback method can find it immediately.

            As far as I know, the default behavior is to propagate execution in ALL outgoing paths, until the wait state is reached in all of the child tokens and then to persist everything in DB.

            Making task node async does not remove the problem.
            Again as far as I understand async continuations, you just spawn another thread but it does not commit stuff in DB. In other words, in both cases (regular or async task node). task instances will not be created in DB until all of them are not created, and that will happen when all child token paths are calculated while leaving Fork node.

            Is this a nonsense? if yes, please help me understand.

            Btw, It is completely clear that this problem is not related to WS, but to any async invocation.

            Thank you for any hint/idea/comment/question.
            milan

            • 3. Re: Dynamic Fork + External Async WS invocation + problem
              kukeltje

              all correct.... what you might try is to to create all the tasks first, and then call node.leave on each of them. Might give you some headstart and be able to have them in the DB before the ws calls return...

              • 4. Re: Dynamic Fork + External Async WS invocation + problem
                clandestino_bgd

                 

                what you might try is to to create all the tasks first, and then call node.leave on each of them. Might give you some headstart and be able to have them in the DB before the ws calls return...
                


                Thanks Ronald.
                I thought about something similar, like I create, but not start tasks in Dynamic Fork, and put Action Handler which invokes WS on task-start event.
                Then in Dynamic Fork, when I create all child tokens, I iterate again through all of child tokens, find unfinished task instances and then start them all.
                That will save me some time, but it does not mean I will manage to win in the race with our superfast WS, since Action Handler which invokes WS calls some business logic, etc.

                And to be honest, it does not look very nice. I would also pretty much destroy the applicability of my Dynamic Fork node to other types of processes, which do not have this scenario.

                If somebody has better and cleaner idea or have problem solved in the past, I would be happy to hear it.

                thanks,
                milan

                • 5. Re: Dynamic Fork + External Async WS invocation + problem
                  kukeltje

                  there is a nicer/cleaner solution, but it is a more complex one. Use jms in between with retries. That is what jms was meant for. Your services do not have to retry then.

                  • 6. Re: Dynamic Fork + External Async WS invocation + problem
                    clandestino_bgd

                    Hi,
                    thanks Ronald.


                    there is a nicer/cleaner solution, but it is a more complex one. Use jms in between with retries. That is what jms was meant for. Your services do not have to retry then.


                    What you say is quite simple and actually web service itself should not talk to process anyway. Sending message to JMS Queue, which has ability to retry ending the task instance is much better.

                    Anyway, is there any other solution except retry in one way or the another? Or, in other words, can this scenario be accomplished in the world of nodes and action handlers?

                    thanks,
                    milan



                    • 7. Re: Dynamic Fork + External Async WS invocation + problem
                      kukeltje

                      sure... make your webservice slower

                      • 8. Re: Dynamic Fork + External Async WS invocation + problem
                        clandestino_bgd

                         


                        sure... make your webservice slower


                        Sorry, I can't. My EAI Bus is superfast :)

                        And now seriously, you don't think that this scenario can be supported by WF design and Action handlers?


                        • 9. Re: Dynamic Fork + External Async WS invocation + problem
                          kukeltje

                          I've been breaking my head over this but unfortunately nothing came to mind....

                          well... nothing is not realy true....

                          I get the impression you want something like: http://eaipatterns.com/BroadcastAggregate.html

                          or am I wrong? Maybe you should do it in your super fast EAI bus then and have a singe esb node in your process ;-)

                          • 10. Re: Dynamic Fork + External Async WS invocation + problem
                            kukeltje

                            I've been breaking my head over this but unfortunately nothing came to mind....

                            well... nothing is not realy true....

                            I get the impression you want something like: http://eaipatterns.com/BroadcastAggregate.html

                            or am I wrong? Maybe you should do it in your super fast EAI bus then and have a singe esb node in your process ;-)

                            • 11. Re: Dynamic Fork + External Async WS invocation + problem
                              clandestino_bgd

                               


                              I get the impression you want something like: http://eaipatterns.com/BroadcastAggregate.html


                              Looks I want something like that.


                              or am I wrong? Maybe you should do it in your super fast EAI bus then and have a singe esb node in your process ;-)


                              Quite true, I used the buzzword EAI bus, influenced by one of your answers on this list, which made me laugh :)
                              Implementing this scenario in e.g Jboss ESB seems like the best solution, but few days before project delivery, I simply cannot do that.
                              I used JBPM JPDL for orchestrating in house WSs both sync and async. Then I realised that in the case of many child tokens and async invocation, this problem can appear, especially if client wants to add some other WS, where I do not have control for retry and timeout.
                              And because I belive that everything can be done with good design and appropriate use of Action handlers, I post the question on this list.

                              Thank you for spending time on this, anyway.
                              best,
                              milan


                              • 12. Re: Dynamic Fork + External Async WS invocation + problem
                                kukeltje

                                No problem... it was enlightening for me to...