JBMESSAGING-126 -- JMS expiration

[weston.price]

Currently the DLQ handler in the JMS/JCA adapter *eats* expired messages. As a result, the messages dissapear. I was wondering if we want to add an expiration destination to the JMS activation spec where at the very least we could send them to the DLQ or another specified destination.

Of course we could get pretty crazy by having mapped expiry destinations something on the lines of 'if the message has X properties and is expired send to this destination etc, if not, this destination etc' but for starters it might be best to send them somewhere.

[timfox]

Sounds like a nice idea. How hard would it be to add?

I'm also curious how the DLQ works.

JBM as of 1.0.2 has it's own DLQ logic (nothing to do with JCA), which will stick the message in a DLQ specified in the jboss messaging config after a certain number of retries.

As I understand it, the MDB container also has it's own DLQ logic. (Which is what you're talking about)

Under what circumstances would stuff go into that DLQ? Do we need to unite the two concepts somewhat?

[weston.price]

A rudimentary implementation would be fairly straightforward. Something along the lines of sending them off to the DLQ or a specified expiration destination.

Is the JBM DLQ always active? I am a bit confused about the 'two' DLQ implementations and which one takes precendence. Does JBM do DLQ handling even with the existence of a connection consumer for a connection?

Maybe we should think about 'uniting' these. Either that or I am just missing something (more likely).

The MDB container in the case of JCA inflow pretty much does nothing in terms of DLQ handling, transactions etc. This is all handled on my end. Basically the JMS/JCA adapter unifies the old ASF and JMSContainerInvoker functionality.

The JCA DLQ handler detects redelivery attempts and either allows onMessage() to execute or shuffles the message off to the DLQ which is specified as a property of the ActivationSpec. This is where the expiry stuff happens as well.

[timfox]

JBM itself needs it's own DLQ logic since we won't always have JCA. E.g. in the standalone server case.

Before a message is sent to a message listener, or before a call to receive() returns we check the delivery counts property (JMSXDeliveryCount) of the message.

If it has exceeded the max delivery attempts as specified in the server peer config then it is sent to the DLQ.

In the case of an MDB - this AFAIK is still implemented as a message listener (please correct me if I am wrong) and delivery is handled by JBM so the same will apply.

Delivery count can be incremented in the following situations:

a) A RuntimeException is thrown from onMessage and ack mode = auto ack or dups ok, retry is immediately attempted (in JBM code).

b) Session gets closed before acking messages so messages are nacked back to the destination.

BTW I'm unclear as to how you keep track of how many times delivery has been attempted in the JCA layer? JMSXDeliveryCount is an optional property not supported by all messaging systems, so if you rely on that your code won't necessary work with other messaging systems.

[weston.price]

JBM itself needs it's own DLQ logic since we won't always have JCA. E.g. in the standalone server case.

Yep. Makes sense. So I wasn't too far off base.

BTW I'm unclear as to how you keep track of how many times delivery has been attempted in the JCA layer? JMSXDeliveryCount is an optional property not supported by all messaging systems, so if you rely on that your code won't necessary work with other messaging systems.

The DLQ handler is layered with a JBoss specific handler being called first to handle redelivery. First we check for the existence of the JMS_JBOSS_REDELIVERY_LIMIT and JMS_JBOSS_REDELIVERY_COUNT. If we can find the count we compare it againts the ActivationSpec property DLQMaxResent. If it's over the limit, the JBoss specific DLQ handles the message.

If we don't execute the above, the GenericHandler kicks in. We try and check the JMSXDeliveryCount first. If it exists, all is well and we use that as a marker for re-delivery attempts.

Now, heres the trick :-).

If that property does not exist, we increment a counter based on the cached ID for that message. So, basically in order:

JBoss specific
Generic JMSXDeliveryCount
Cached id/count mapping

Interestingly enough the two conditions you specify:

a) A RuntimeException is thrown from onMessage and ack mode = auto ack or dups ok, retry is immediately attempted (in JBM code).

b) Session gets closed before acking messages so messages are nacked back to the destination.

should actually never occur in JCA land for the following reasons:

a) I never propogate a RuntimeException from onMessage. Effectively I trap it and deal with it locally via Transaction demarcation.

b)Sessions are never closed during delivery, only in the case where they have reached an idle timeout, or the adapter itself is being stopped.

So, effectively it sounds like I am 'jumping in the way' of JBM DLQ. Of course you do redeliver in the case of a commit/rollback on either an XA or local JMS transaction but in this case the JCA DLQ handler would grab it because it is not one of the conditions above. At least that's how I have it in my head.

[weston.price]

Ahhh wait... I see. Sorry, I'm a tool.

a) A RuntimeException is thrown from onMessage and ack mode = auto ack or dups ok, retry is immediately attempted (in JBM code).

This is a bit different. I believe in the case of a transacted session the MessageCallbackHandler does nothing correct? It sounds like you are talking about JBM handling non transacted scenarios for DLQ.

[timfox]

The DLQ handler is layered with a JBoss specific handler being called first to handle redelivery. First we check for the existence of the JMS_JBOSS_REDELIVERY_LIMIT and JMS_JBOSS_REDELIVERY_COUNT.

You mean JBoss MQ specific? JBoss Messaging doesn't have these properties.

If we can find the count we compare it againts the ActivationSpec property DLQMaxResent. If it's over the limit, the JBoss specific DLQ handles the message.

If we don't execute the above, the GenericHandler kicks in. We try and check the JMSXDeliveryCount first. If it exists, all is well and we use that as a marker for re-delivery attempts.

Now, heres the trick :-).

If that property does not exist, we increment a counter based on the cached ID for that message. So, basically in order:

JBoss specific
Generic JMSXDeliveryCount
Cached id/count mapping

So you maintain a map of id to count? How long do you keep items in the map for? It could grow pretty big. Also wouldn't survive restart I assume.

[timfox]

Ahhh wait... I see. Sorry, I'm a tool.

a) A RuntimeException is thrown from onMessage and ack mode = auto ack or dups ok, retry is immediately attempted (in JBM code).

This is a bit different. I believe in the case of a transacted session the MessageCallbackHandler does nothing correct? It sounds like you are talking about JBM handling non transacted scenarios for DLQ.

If a RuntimeException is thrown from onMessage and the session is transacted it is ignored as per JMS spec and delivery of the next message continues.

Perhaps we should have an option to turn off JCA dlq functionality if we know the provider supports its own? Should this be the default?

[weston.price]

You mean JBoss MQ specific? JBoss Messaging doesn't have these properties.

I wondered about this. Based on this, the JMSXDeliveryCount will be consulted first and then the cached id in attempting to determine if the message should be sent to the DLQ. Effectively with JBM, no JBoss specific properties are consulted.

If a RuntimeException is thrown from onMessage and the session is transacted it is ignored as per JMS spec and delivery of the next message continues.

From the JMS spec:

Transacted Session - the next message for the listener is delivered. The client
can either commit or roll back the session (in other words, a
RuntimeException does not automatically rollback the session).

This is what we do, with the default behavior being that a rollback occurs. This can be altered via an ActivationSpec property to simply commit the message in the case of a RuntimeException when the transaction attribute of the MDB is BMT or CMT with NOT_SUPPORTED. CMT/REQUIRED is always rolledback and redelivered as per the EJB spec.

Perhaps we should have an option to turn off JCA dlq functionality if we know the provider supports its own? Should this be the default?

You can already do this it's a simple property (useDLQ) of the JMSActivationSpec.

[timfox]

This is what we do, with the default behavior being that a rollback occurs. This can be altered via an ActivationSpec property to simply commit the message in the case of a RuntimeException when the transaction attribute of the MDB is BMT or CMT with NOT_SUPPORTED. CMT/REQUIRED is always rolledback and redelivered as per the EJB spec.

The exception is caught in JBM code so I don't think you'll have a chance to do anything with it.

We have to do this to be spec compliant.

[weston.price]

Put another way:

If an EJB developer wants DLQ functionality for an MDB, he/she has to use the JMS/JCA DLQ implementation. Since we have made the decision to always execute message delivery in the context of a transaction (local or XA), JBM would not handle the case where a RuntimeException occurs. It would simply ignore the message and move on.

While the JMS spec makes note that RuntimeExceptions should not occur in MessageListeners, EJB(X) does not make this assumption. The spec even goes so far as to *require* that message be redelivered depending upon the transaction attributes.

For CMT/REQUIRED MDB's the choice is easy, the specification requires that the transaction be rolled back and the message be redelivered. For BMT/CMT-NOT_SUPPORTED, the spec is all but useless. Effectively the MDB executes in an 'unspecified transaction context'. There are no restrictions on how the container/JCA/integration layer can handle this situation but there are also no real guidlines unfortunately.

Of course for a standalone JBM deployment, the situation is much different; at that point it's just plain ole JMS ;-)

[timfox]

I don't get it.

How can you handle the exception if we catch it in our layer so it never bubbles up to you?

Here is the code:

catch (RuntimeException e)
 {
 long id = m.getMessage().getMessageID();

 log.error("RuntimeException was thrown from onMessage, " + id + " will be redelivered", e);

 // See JMS 1.1 spec 4.5.2

 if (ackMode == Session.AUTO_ACKNOWLEDGE || ackMode == Session.DUPS_OK_ACKNOWLEDGE)
 {
 // We redeliver a certain number of times
 if (tries < maxDeliveries)
 {
 tries++;
 }
 else
 {
 log.error("Max redeliveries has occurred for message: " + m.getJMSMessageID());

 //postdeliver will do a cancel rather than an ack which will cause the ref to end
 //up in the dlq

 cancel = true;

 break;
 }
 }
 else
 {
 // Session is either transacted or CLIENT_ACKNOWLEDGE
 // We just deliver next message
 if (trace) { log.trace("ignoring exception on " + id); }

 break;
 }

[weston.price]

The exception is caught in JBM code so I don't think you'll have a chance to do anything with it.

It's not actually. The MessageCallbackHandler invokes onMessage() on the listener. For JCA that is the JmsServerSession who calls the MessageEndpoint.onMessage() method. At this point you are in the EJB3 interceptor chain. If the MDB throws a RuntimeException it bubbles back up to the JmsServerSession where I handle it with a rollback or a commit.

As far as the MessageCallbackHandler is concerned, an exception was never thrown.

[weston.price]

Callchain:

AsfAspect.handleRun()->MessageCallbackHandler.callOnMessage()->JmsServerSession().onMessage()->MDB.onMessage()

Exception is thrown from MDB which is caught in the following code in the JMSServerSesssion

 public void onMessage(Message message)
 {
 try
 {
 endpoint.beforeDelivery(JmsActivation.ONMESSAGE);

 try
 {
 if (dlqHandler == null || dlqHandler.handleRedeliveredMessage(message) == false)
 {
 MessageListener listener = (MessageListener)endpoint;


 listener.onMessage(message);
 }
 }
 finally
 {
 endpoint.afterDelivery();

 if (dlqHandler != null)
 dlqHandler.messageDelivered(message);
 }
 }

 catch (Throwable t)
 {
 //MDB RUNTIME EXCEPTION HANDLED HERE!
 log.error("Unexpected error delivering message " + message, t);

 if(txnStrategy != null)
 txnStrategy.error();

 }


 }

So, the exception never makes it back to the MessageCallbackHandler. So the code

 if (trace) { log.trace("calling listener's onMessage(" + m + ")"); }

 listener.onMessage(m);

 if (trace) { log.trace("listener's onMessage() finished"); }

 break;

executes with no issues.

[timfox]

Duh. I am an idiot.

I have been doing powerpoint slides for JBW for 12 hours solid and it is eating into my brain :)