4 Replies Latest reply on Feb 18, 2009 12:54 PM by Jeff Yu

    Service validators and redelivery

    Gary Brown Master

      The current Service Validator mechanism uses a global filter to observe messages being passed between ESB service descriptors. If the endpoint associated with the message has a Service Validator configured, then the Service Validator is used to determine if that message is appropriate in the context of the business transaction (i.e. is it a known message type, is it being sent/received in the correct order, etc).

      The problem is that once the service validator has indicated that the message is valid, and the filter passes the message on for delivery to its destination service descriptor, the message is considered to have been processed.

      However if while processing the message, the destination service descriptor has a problem that results in an exception being thrown, the ESB will automatically enter a retry cycle (unless this is turned off). Each time the message is redelivered, the Service Validator would now consider it to be invalid, as it has already received such a message and flagged it as ok - therefore the service validator's state machine for that session has moved on.

      If the service validators are configured in 'active' mode, the detection of the error would result in a CourierException being thrown from the global filter, thus preventing the subsequent processing of the redelivered message - which may happen multiple times until the retry count expires.

      One solution to this would be to detect that it is a redelivered message (I assume the Message has a redelivery flag???) - which can be used to ignore any service validator error.

      However this would not resolve the problem if the initial processing of the message (when first delivered) resulted in other messages being sent.

      Even if the redelivery Service Validator error is ignored, each of the sent messages (which will not be considered as re-deliveries, but fresh messages), will also result in Service Validator errors, as the service validators for those destinations would have previously processed the sent messages from the first invocation of the service. So again - if the service validators are configured in 'active' mode, then their delivery would be blocked.

      This may be acceptable behaviour? If those messages were already sent to the target service descriptors, and possibly have already been processed, then maybe we don't want duplicates to be sent? If considered ok, then the only downside is that the Service Validator would report an out of sequence message - however this could potentially be traced back to the redelivery of the previous message.

      One of the reasons for this issue is that the validation of the message, and subsequent processing of the message, are not performed in an atomic transaction - which could be rolled back if a failure was to occur. Not sure that this would be possible, given the current approach used for the Service Validators - i.e. global filters.

      Therefore I believe there are only three solutions are the current time:

      a) Disable the redelivery mechanism - but this could prevent the business apps from recovering in the event of periodic failure.

      b) Only set the Service Validators in passive reporting mode.

      c) Detect redelivery messages and ignore service validation failures for these messages

      If (b) or (c) is taken, then it may mean some spurious out of sequence errors are reported, but it might be possible to precede the errors with a warning that a 'redelivered message is being re-processed, which may result in some out of sequence errors being generated'.

        • 1. Re: Service validators and redelivery
          Jeff Yu Master

          sorry, I am not clear on the 'active' and the 'passive' mode in service validator?

          Yes, the redelivery mechanism is an issue for our validator. I am thinking about taking the b) or c) approach, as you said above, it doesn't make sense for us to turn this off as users might used this feature before.


          • 2. Re: Service validators and redelivery
            Gary Brown Master

            Active = if a message is considered to be invalid, the Service Validator will prevent it from being delivered to the target service descriptor (i.e. so only valid messages are delivered to the service descriptors).

            Passive = the Service Validator will only report erronous messages, it won't prevent them from being delivered to target service descriptors, and therefore they will continue to be processed by the business app.

            What do you think about the side effect of using (c).

            For example,

            Service Descriptor A (SDA) is implemented to send messages to Service Descriptor B (SDB), before returning a reply to the invoker of SDA.

            However when SDA is invoked with a message, it sends the messages to SDB but fails when returning a reply to the invoker (e.g. destination invalid). This means that the original request, and the message sent to SDB have both been validated by the Service Validators.

            When the message is re-delivered to SDA, the Service Validator would realise that it was a re-delivery, and therefore ignore the service validator error. But, when the message is sent to SDB, the Service Validator would flag this as an error (as the message to SDB was previously delivered), and therefore block this message.

            SDA then proceeds to successfully send the reply to its invoker.

            The outcome of this is that the business transaction has processed the request, sent a message to SDB, and returned a reply. Without the send message to SDB being blocked, the re-delivery would have resulted in two messages being sent to SDB for the same business transaction.

            Is blocking the second delivery of the message to SDB ok?

            What happens currently in this situation, is it assumed that the second processing of the message to SDB is appropriate?

            One issue - if async delivery, then blocking the second message would be possible, as it won't impact the sender (SDA). However, sync delivery would result in the invoker receiving the exception, which may then prevent it from sending the reply - so this blocking approach would cause problems if using sync delivery.

            Not sure how to get around this one? Possibly only passive monitoring will be possible - otherwise blocking some messages may impact the re-delivery mechanism, preventing a recovery.

            • 3. Re: Service validators and redelivery
              Gary Brown Master

              The only other way, if (c) is used with active Service Validators, then possibly we don't block messages related to sync invocation?

              Is it possible for a global filter to understand whether the message being processed is either the request or reply from a sync delivery?

              • 4. Re: Service validators and redelivery
                Jeff Yu Master

                I was thinking about that maybe we need to implement both b) and c) option, and then let users choose which one they want to use. With regard to the question of either the request or reply from a sync delivery.. I need to check it later.