Failover MessageRedelivery

[kurtstam]

I checked in the redelivery code. So if your delivery fails it will store the message in the message store and schedule it for redelivery.

Check out the new jboss-esb.xml in the jbosses.esb archive, look at the new redelivery service

<service category="JBossESB-Internal" name="RedeliverService" description="Scheduled Service to Redeliver Messages">
 <listeners>
 <scheduled-listener name="redeliver-scheduled-listener" scheduleidref="5-min-trigger" composer="org.jboss.soa.esb.schedule.RedeliverEventMessageComposer" />
 </listeners>
 <actions>
 <action name="RedeliverMessagesAction" class="org.jboss.soa.esb.actions.MessageRedeliverer" />
 </actions>
 </service>

This redelivery mechanism only kicks in for async delivery. For synchronous message delivery I simply let the delivery fail, so the client can deal with the delivery failure.

I hope this makes sense to everyone. I will be adding docs tomorrow, but please let me know if you have any feedback/suggestions.

thx!

--Kurt

[burrsutter]

How might this situtation arise? "if your delivery fails"

Static-Router, CBR, MessageFilter, the wiretap actions?

[kurtstam]

I redelivery accounts for the situation where:

1- the jms queue you are trying to use has gone AWAL. So the ServiceInvoker will still find the EPR in the Registry, but at that point it is a 'dead' EPR. When the JMS provider comes back online your message should get redelivered.

2. The destination service is on a different box and the service EPR (or even the service) has not yet registered with the registry. Delivery will be retried until either successful or the max number of retries are reached, after which it is stored in the MessageStore with classification DLQ.

--K

[beve]

Hi Kurt,

this looks nice! :)

We are going to use this but I'm a little bit confused about how the messages that we want redelivered should be added to the MessageStore.
We have actions pipelines and when an error occurs and it makes sense to attempt to redeliver, do we simply add the message to the store using a classification name for the pipeline in question.

And then configure a redelivery service like this:

<service category="JBossESB-Internal" name="RedeliverService" description="Scheduled Service to Redeliver Messages">
 <listeners>
 <scheduled-listener name="redeliver-scheduled-listener" scheduleidref="5-min-trigger" composer="org.jboss.soa.esb.schedule.RedeliverEventMessageComposer" />
 </listeners>
 <actions>
 <action name="RedeliverMessagesAction" class="org.jboss.soa.esb.actions.MessageRedeliverer">
 <property name="classification" value="process1"/>
 </action>
 </actions>
 </service>

Is this the way we should be doing this?

Thanks,

Daniel

[kurtstam]

Well this should not happen between actions in the pipeline, but only when routing between services. The code base is now refactored (well almost) to always use the ServiceInvoker for sending messages between services. Now look at the code in asyncDelivery:

 message.getProperties().setProperty(MessageStore.CLASSIFICATION, MessageStore.CLASSIFICATION_RDLVR);
 message.getProperties().setProperty(DELIVER_TO, service);
 deliverToDeadLetterService(message);

If delivery fails it sets the TO service in a property as well as telling it to store it in the MessageStore under classification RDLVR.

Now the Redeliver Service periodically wakes up and tries to redeliver all the messages in the RDLVR classification. It uses the setting for the TO service to figure out where to send it. Note that this is bound to the service address and not the EPR. For each retry it will pull the latest EPR information for that service.

Going to work on making the maxRedeliveries configurable now :).

Does that help?

--Kurt

[marklittle]

This redelivery mechanism only kicks in for async delivery. For synchronous message delivery I simply let the delivery fail, so the client can deal with the delivery failure.

I hope this makes sense to everyone. I will be adding docs tomorrow, but please let me know if you have any feedback/suggestions.

Nice work Kurt.

[beve]

Hi Kurt,

thx for the quick response!

This is how we would like to use this (if it's possible).
We have an action pipeline that has 3 actions:
1. Validate xml
2. Post to WebService
3. Update database

Now, if the validation fails (step 1) we can set the FaultTo to let the calling system know somethings wrong.
If the posting to the WebService fails for some reason (step 2) it make sense to try to redeliver at a later time.

We would like to be able to have different scheduling intervals for different action pipelines and thought that step 2's exception handler would add the message to the MessageStore with a classification specific to that action pipeline. Then we would specify the redelivery service in the same jboss-esb.xml and use the classification property to specify which messages this redelivery service should care about.

Is this a valid way to use the redelivery service?

Thanks,

Daniel

[kurtstam]

Hi Daniel,

Now, if the validation fails (step 1) we can set the FaultTo to let the calling system know somethings wrong.
If the posting to the WebService fails for some reason (step 2) it make sense to try to redeliver at a later time.

So far so good :)

We would like to be able to have different scheduling intervals for different action pipelines and thought that step 2's exception handler would add the message to the MessageStore with a classification specific to that action pipeline.

The redeliverService is configured in the jbossesb.esb, and there is only one. It looks for msgs in the RDLVR classification, and tries to redeliver them on it's configured schedule. So all services share the same redeliverService. I never thought you'd like to make the redeliver schedule different for different services.

That said, like you said, we can define other redeliverDeliver services, and make the classification configurable. Then we could set that classification in the message and the ServiceInvoker could look to see if that property is set and if so use it.

Can you explain a little more why you would need different redeliver schedules?

--Kurt

[marklittle]

That said, like you said, we can define other redeliverDeliver services, and make the classification configurable. Then we could set that classification in the message and the ServiceInvoker could look to see if that property is set and if so use it.

I don't think we need any further extensions to this capability for this release. What you've put in place so far is good and should easily cover the 80/20 case for the initial release. We should definitely consider other use cases that we can't solve, but push any other support to after 4.2.x.

[marklittle]

Daniel, if you and Kurt come up with use cases/additional requirements, JIRA them so we don't lose them.

[beve]

Hi,

sorry for not replying earlier...

Can you explain a little more why you would need different redeliver schedules?

In our case we think it would be nice to have the declaration of the redelivery specified in the same jboss-esb.xml file, so that there is only one configuration file to examine.

Also, we have some services that need high availability and therefore would like to configure the redelivery on a tight interval in case a services that the ESB uses becomes unreachable, it should be retried very frequently, say every ten seconds. While other services might only need to be retried every hour or so.
I was thinking that having a single schedule for all redelivery might affect the performance of the ESB.

Thanks,

Daniel

[kurtstam]

http://jbossesb.blogspot.com/2007/08/service-clustering.html for some back ground info on this.

[beve]

Ok, now I get it. The current redelivery service deals with redelivery between ESB calls.

Our concern is scheduling redelivery/re-executing of a service upon an error in the action pipleline. An example of this would be a call to a web service that is temporarily unavailable.

Our current solution is to save the incoming Message object to the MessageStore, as the first action in the action pipeline. Upon sucessful completion of the pipeline the message will be removed.

If an exception occurs in the pipeline an exception handler will retrieve the message from the message store, re-add the retrieved message, with a different classification, to the MessageStore. The classification will match the classification that the redelivery service is configured to look for.

To get this to work I need to make a few minor changes.
I added a method to RedeliverStore that takes a classification as a parameter:

public interface RedeliverStore extends MessageStore {
...
public boolean redeliver(URI uid,String classification) throws MessageStoreException;

I'd also like to make the classification on RedeliverEventMessageComposer a configuratble property which would save us from having to create our own composer.

Does this look alright? Does anyone have a better way on doing this?

We do plan to build an error handling process that will take care of automatic redelivery and where we will also be able to manually trigger redelivery from tasks in jbpm, but our time schedule will not allow this at the moment. This is really just an effort to get automatic redelivery to work with the least amout of effort and buy us some time.

Thanks,

Daniel

[marklittle]

Why don't you use transacted JMS sessions to pull the message, work on it and then commit the transaction once it has been sent on to the Web Service? If you can't do that, roll back the transaction and the message goes back on the queue for later.

[beve]

Why don't you use transacted JMS sessions

Well, we also have the use case when it does not make sense to redeliver the message, like when a transformation or validation fails. The redelivery service can be configured with a retry count which we could then take advantage of. This is not an optimal solution as we will still be trying to redeliver/re-execute a message that we know will fail.

While we are on the subject of transacted JMS Sessions, we are using the JMSGatewayListener and as far as I can see these JMSSessions are created without transactions and AUTO_ACKNOWLEDGE. But there is a JIRA (http://jira.jboss.com/jira/browse/JBESB-439), which I'm sure you know:)
Lets say we make this configurable so that the JMSGatewayListener's sessions are transacted, and also make the ack mode be configurable.
How do we commit the transaction upon sucessfully completing the action pipeline?

Thanks,

Daniel