Bridge under load throws IllegalStateException

[mclu]

Hi again!

I have my bridge up and running.
2 servers (A and B) and a middleware server (M).
on both servers there are 3 queues. The middlewareserver bridges them to its local queues. I am using QOS 2 (Transacted)
Messageflow is like
-A send to lokal.
-Bridged from A to M
-M processing and send processed to lokalQueue(M)
-Message bridged from M to B
-B processing message and send result to localQueue(B)
-Bridged to M
-result stored.

When I send a lot of messages I receive a stack traces is like:

14:41:44,250 ERROR [ClientConsumer] Failed to deliver message
java.lang.IllegalStateException: There is already a delivery set for auto ack
 at org.jboss.jms.client.state.SessionState.setAutoAckInfo(SessionState.java:371)
 at org.jboss.jms.client.container.SessionAspect.handlePreDeliver(SessionAspect.java:258)
 at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect18.invoke(SessionAspect18.java)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$preDeliver_5548846795541436298.invokeNext(ClientSessionDelegate$preDeliver_5548846795541436298.java)
 at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$preDeliver_5548846795541436298.invokeNext(ClientSessionDelegate$preDeliver_5548846795541436298.java)
 at org.jboss.jms.client.delegate.ClientSessionDelegate.preDeliver(ClientSessionDelegate.java)
 at org.jboss.jms.client.container.ClientConsumer.callOnMessage(ClientConsumer.java:150)
 at org.jboss.jms.client.container.ClientConsumer$ListenerRunner.run(ClientConsumer.java:941)
 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89)
 at java.lang.Thread.run(Thread.java:619)

Its caused by the Bridge. My breakpoint stops on 3 threads related
my 3 queues. One example is listed below:
Variables of ClientConsumer.callOnMessage(SessionDelegate, MessageListener, String, String, boolean, MessageProxy, int, int, SessionDelegate, boolean):

sess ClientSessionDelegate (id=315)
listener Bridge$SourceListener (id=508)
consumerID "a1-tcoi696f-1-l3lf696f-nkdb51-q4z24a"
queueName "local_incomingSyncRequests"
isConnectionConsumer false
m TextMessageProxy (id=507)
ackMode 0
maxDeliveries 10
connectionConsumerSession null
shouldAck true
deliveryInfo DeliveryInfo (id=367)

And SessionState Vars:

this SessionState (id=310)
 acknowledgeMode 0
 asfMessages LinkedList<E> (id=395)
 autoAckInfo DeliveryInfo (id=399)
 callbackHandlers HashMap<K,V> (id=403)
 children HashSet<E> (id=408)
 clientAckList ArrayList<E> (id=396)
 currentTxId MessagingXid (id=419)
 delegate ClientSessionDelegate (id=315)
 dupsOKBatchSize 1000
 executor ClearableQueuedExecutor (id=423)
 parent ConnectionState (id=425)
 recoverCalled false
 sessionID "91-3hmi696f-1-l3lf696f-nkdb51-q4z24a"
 sessionListener null
 transacted true
 xa true
 xaResource MessagingXAResource (id=445)
info DeliveryInfo (id=367)
 connectionConsumerSession null
 consumerId "a1-tcoi696f-1-l3lf696f-nkdb51-q4z24a"
 msg TextMessageProxy (id=507)
 queueName "local_incomingSyncRequests"
 shouldAck true

I have seen some references in the Design forum and version 1.2.
Do you have an Idea?

Oh forgot: I am using 1.4 cr2 and jboss 4.2.1GA..

Greets
Markus Lutum

[mclu]

Hi again

I don't know if this will help, but in those error cases the
SessionAspect.handlePreDeliver() called
isXAAndConsideredNonTransacted(state) which returns true because the SessionState.sessionListener is null !?!

The Comment of isXAAnd.... sais:
http://jira.jboss.org/jira/browse/JBMESSAGING-946 - To accomodate TCK and the MQ behavior we should behave as non transacted, AUTO_ACK when there is no transaction enlisted
However when the Session is being used by ASF we should consider the case where we will convert LocalTX to GlobalTransactions. This function helper will ensure the condition that needs to be tested on this aspect


Don't know why and what this is all about...

Greets
Markus

[mclu]

My Problem is solved but maybe still some bug in the messaging.

My middle server (M) also has an internal Topic.
I registered some durable subscribers and because I can not use JmsXA Connectionfactory I used XAConnectionFactory. JmsXA is not possible because then I can not use setClientId() (J2ee Spec).

It seems that this is the root of this error. Means using XAConnectionFactory internally with heavy load and all the other configured and described stuff.

My workarround:
I changed my dynamic subscriber registration via a service to one single MDB which now handles all messages. Performance loss but now it uses JTA via JmsXA and everythink works fine...

Greets
Markus Lutum

[mclu]

Sorry... forget my last post....
the move to the MDB has nothing to do with the error and the error is still there.

Damm...

still trying to find out whats going on....

[mclu]

Ah.... I can reproduce it more often!!

Is has to do with the Bridge configuration MaxBatchSize and MaxBatchTime.

In my initial setup I set

<!-- The maximum number of messages to consume from the source before sending to the target -->
 <attribute name="MaxBatchSize">5</attribute>

 <!-- The maximum time to wait (in ms) before sending a batch to the target even if MaxBatchSize is not exceeded.
 -1 means wait forever -->
 <attribute name="MaxBatchTime">1000</attribute>

With this setting it fails once each 100 message (approx)

I haved changed the values to
MaxBatchSize 2
and
MaxBatchTime 200
and it seems to happen now each 2-10 message...

What do you need to test this on your side or how should I send you a TestCase. What are the testCase guidelines?

[clebert.suconic]

Can you pack an example with some description on how to reproduce it?

You could create something based on the examples.. and send your configs...

You could attach your example to this JIRA:

http://jira.jboss.org/jira/browse/JBMESSAGING-1068

[mclu]

Ok!

I will try to create a smaller testset.
But this will take at least one week because I have to interrupt my project to fix some bugs first...

thx!
Markus

[timfox]

http://jira.jboss.com/jira/browse/JBMESSAGING-1068

[timfox]

I believe this is related to http://www.jboss.com/index.html?module=bb&op=viewtopic&t=98577&postdays=0&postorder=asc&start=40

The problem occurs because of this code in SessionAspect:

 /** http://jira.jboss.org/jira/browse/JBMESSAGING-946 - To accomodate TCK and the MQ behavior
 * we should behave as non transacted, AUTO_ACK when there is no transaction enlisted
 * However when the Session is being used by ASF we should consider the case where
 * we will convert LocalTX to GlobalTransactions.
 * This function helper will ensure the condition that needs to be tested on this aspect
 * */
 private boolean isXAAndConsideredNonTransacted(SessionState state)
 {
 return state.isXA() && (state.getCurrentTxId() instanceof LocalTx) && (state.getDistinguishedListener() == null);
 }

This means the converstion of work done outside the tx will only happen when using MDBs.

In the case of using JTA without MDBs state.getDistinguishedListener() will be null.

[timfox]

mclu-

Can you try setting maxBatchTime to -1 and tell me if the problem still occurs?

[mclu]

Will test this tomorrow....

[timfox]

I have tried to replicate this locally but cannot.

If you could try and replicate in a test program ASAP that would be a great help, otherwise it's unlikely the fix (if required) will be ready in time for the next release.

[mclu]

Hi Tim!

I don't have much time currently to create an easy testcase because of project priorities. So I have to fight fist to get more time on this (with my wife and my Boss :-) )

But meanwhile I have tested the MaxBatchTime -1 setting and shaped the problem a little.

Again my setup but now only the relevant parts:
- 2 Servers A and B
- A should send messages to B. B processes the messages and sometimes send result to A.

Technical architecture
-A holds a queue Q1 which is bridged to B Bridged_Q1
-On B there is MDB processing the Messages in Bridged_Q1 and send results to Q2.
-Results are bridged to A Bridged_Q2
-on A there is a MDB which stores results.

So each side holds 2 queues. The bridge for both sides are deployed on A.

Testsituation:
about 900 Messages are waiting in Q1
When I now hotdeploy my bridge on A it starts the above processing:
MaxBatchSize is always set to 2

Result:
With maxBatchTime 200 or 300 I have a lot of errors. With 100 it is less!.
With -1 there is no Problem anymore...

It must have something to do if there are 2 directions with messages or two bridges are active at the same time for the same server. My third Server (which only send messages to A) have no bridging problem.

thx again for your effords!!
Markus Lutum

[timfox]

I think I know what the problem is by looking at the code, I just haven't managed to replicate here, I guess I just haven't managed to get the race.

Anyhow, I can add a "fix" even though I can't replicate, hopefully will have this done in before the end of the week. If you could then try that out, we'll put it in the next release.

[mclu]

Cool...
Yes! I can try it out (hopefully) if this then will help.
Just tell me how I can get the fix then...
Greets...