paging problem with orderingGroup enabled

[mclu]

Hi Howard!

I solved the orderinggroup handling by using one group and enabled it via connectionfactory.
But now after my Server starts up and reloads the paged messages it stops after the first set of messages are received.

Means:
To Test I have set fullsize to 20 and pagesize and downcache to 5.
My queue has 2000 messages.

After I received the first 20 messages the system stops. The database still contains messages and while debugging I also see that each 5 consumed messages the JDBCPersistenceManager.getPagedReferenceInfos() is called to load and add the next paged messages. I also saw that the internal messageRef List contains these reloaded messages.

But still, after the 20th message the consuming stops.

I first thought that its because of my self written transaction aware consumer that also receives message in order. But after I refactored this back to MessageDrivenBean the same error occurs.

My setup:
Queue:

 <mbean code="org.jboss.jms.server.destination.QueueService"
 name="jboss.messaging.destination:service=Queue,name=incomingSyncRequests"
 xmbean-dd="xmdesc/Queue-xmbean.xml">
 <depends optional-attribute-name="ServerPeer">
 jboss.messaging:service=ServerPeer
 </depends>
 <attribute name="FullSize">20</attribute>
 <attribute name="PageSize">5</attribute>
 <attribute name="DownCacheSize">5</attribute>
 </mbean>

MDB:

@MessageDriven(activationConfig = {
@ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),
@ActivationConfigProperty(propertyName = "destination", propertyValue = IMConstants.QUEUE_INCOMINGSYNCREQUESTS),
@ActivationConfigProperty(propertyName = "maxSession", propertyValue = "1"),
@ActivationConfigProperty(propertyName = "maxMessages", propertyValue = "1") })
public class GlobalMessageDispatcherBean implements MessageListener {
....

CF:

 <mbean code="org.jboss.jms.server.connectionfactory.ConnectionFactory" name="jboss.messaging.connectionfactory:service=ConnectionFactory" xmbean-dd="xmdesc/ConnectionFactory-xmbean.xml">
 <depends optional-attribute-name="ServerPeer">jboss.messaging:service=ServerPeer</depends>
 <depends optional-attribute-name="Connector">jboss.messaging:service=Connector,transport=sslbisocket</depends>
 <depends>jboss.messaging:service=PostOffice</depends>

 <attribute name="JNDIBindings">
 <bindings>
 <binding>/ConnectionFactory</binding>
 <binding>/XAConnectionFactory</binding>
 <binding>java:/ConnectionFactory</binding>
 <binding>java:/XAConnectionFactory</binding>
 </bindings>
 </attribute>
 <attribute name="EnableOrderingGroup">true</attribute>
 <attribute name="DefaultOrderingGroupName">IMSDefaultGroup</attribute>
 </mbean>

Maybe it has to do with this singleton MDB configuration but it should work with it too, or?

Using JBM 1.4.5 build 7766

[mclu]

Hi Howard!

I investigated my problem more deeply.

First I have to say, that the OrderingGroup of bridged messages are overwritten by the setting in the connection factory.
Not that nice I think.
So I send a message with ordering group G1 to a Queue. This message is bridged to another server where ordering group attributes are set at the connection factory to group GDefault. Then the messages have the group GDefault iso G1.
Bug or a feature :-) ?


Second:
I have added some tracing to the OrderingGroupMonitor to see what is registered and what completed.

While my server starts up it loaded the first 20 paged messages and adds them to the OrderingGroupMonitor. Then it starts handling the messages and calls completed on the Monitor, which removed the ref from the group.
After 5 messages are handled the JDBCPersistenceManager loads the next 5 paged messages and adds them.
That behaviour is fine!
But if a message is added to the queue (new message comes in) which have the same orderinggroup, this is registered too on the Monitor. But the message itself is added to the DownCache.
This seams to be a bug or?
If a message is added to a "full" queue which is paged then this should not be registered to the monitor, or?

Maybe this is the reason, why the delivery stops....

The behaviour you can see below:

20 times registering:

11:28:50,359 TRACE @main [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154740253458432' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:28:50,359 TRACE @main [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154740257849345' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:28:50,359 TRACE @main [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154740259258370' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:28:50,359 TRACE @main [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154740260470787' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:28:50,359 TRACE @main [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154740261191684' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:28:50,359 TRACE @main [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154740262076421' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:28:50,359 TRACE @main [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154740262600710' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:28:50,359 TRACE @main [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154740263747591' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:28:50,359 TRACE @main [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154740264697864' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:28:50,359 TRACE @main [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154740265484297' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:28:50,359 TRACE @main [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154740267089930' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:28:50,359 TRACE @main [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154740268236811' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:28:50,359 TRACE @main [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154740269252620' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:28:50,359 TRACE @main [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154740270006285' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:28:50,359 TRACE @main [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154740270530574' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:28:50,359 TRACE @main [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154740272136207' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:28:50,359 TRACE @main [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154740273217552' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:28:50,359 TRACE @main [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154740274888721' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:28:50,359 TRACE @main [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154740275707922' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:28:50,359 TRACE @main [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154740276363283' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'

//Bridge starts adding messages from another server (BUG?)
11:29:08,828 TRACE @WorkerThread#2[10.4.16.38:1529] [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154743382081536' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:29:09,031 TRACE @WorkerThread#2[10.4.16.38:1529] [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154743382966273' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:29:09,109 TRACE @WorkerThread#2[10.4.16.38:1529] [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154743383293954' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'

// first message is send
11:29:10,000 TRACE @Thread-27 [org.jboss.messaging.core.impl.OrderingGroupMonitor] marked sending MID '41154740253458432' for OrdGrp 'IMSDefaultGroup'

// other are added meanwhile
11:29:10,203 TRACE @WorkerThread#0[10.4.16.38:1523] [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154743387783179' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:29:10,421 TRACE @WorkerThread#0[10.4.16.38:1523] [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154743388667916' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'
11:29:10,718 TRACE @WorkerThread#0[10.4.16.38:1523] [org.jboss.messaging.core.impl.OrderingGroupMonitor] restistered MID '41154743389880333' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'

// first message completed
11:29:13,781 TRACE @WorkerThread#0[10.4.16.38:1523] [org.jboss.messaging.core.impl.OrderingGroupMonitor] completed MID '41154740253458432' for OrdGrp 'IMSDefaultGroup' and Destination 'incomingSyncRequests'

// second marked sending...
11:29:13,781 TRACE @WorkerThread#0[10.4.16.38:1523] [org.jboss.messaging.core.impl.OrderingGroupMonitor] marked sending MID '41154740257849345' for OrdGrp 'IMSDefaultGroup'
 and so on.

[gaohoward]

Hi Markus,

There may be some issue to consider here regarding to paging. In the case where the message of the foremost order has been paged and the subsequent messages are filling up the memory list of the queue, the message delivery will be stop as we only iterate through the memory list during each delivery. We stop at the end of each iteration and those are paged are not iterated.

I'll think about the case and try out some test.

Thanks
Howard

[gaohoward]

Re: So I send a message with ordering group G1 to a Queue. This message is bridged to another server where ordering group attributes are set at the connection factory to group GDefault. Then the messages have the group GDefault iso G1.
Bug or a feature :-) ?

>>> would be a bug definitely. The message's group id should not be overwriten.



Re: But if a message is added to the queue (new message comes in) which have the same orderinggroup, this is registered too on the Monitor. But the message itself is added to the DownCache.
This seams to be a bug or?

>>>No it's not a bug, it is so designed. Paged references are loaded from the beginning and added to the end.

RE: If a message is added to a "full" queue which is paged then this should not be registered to the monitor, or?

>>>I need to think about it.

[mclu]

Ok...

And please also make sure that those added messages while handling the paged messages added at the end of the queue. So preserve message order also in case of paging.

Thx...

[gaohoward]

Hi Markus,

https://jira.jboss.org/jira/browse/JBMESSAGING-1726

About the delivery stop, I think the problem is in ChannelSupport.InMemoryCallback.afterCommit():

promptDelivery = (promptDelivery || monitor.messageCompleted(ref));

This line has its flaw, you can see if promptDelivery is true, the second expression won't be evaluated (i.e. monitor.messageCompleted(ref)). But it must be executed in order to unregister a finished message. so

https://jira.jboss.org/jira/browse/JBMESSAGING-1727

Also I think you are right in that the paged references won't need registered (save memory). We can only registered at loading their loading time. But this doesn't affect functionality.

https://jira.jboss.org/jira/browse/JBMESSAGING-1728

[mclu]

I also saw this wron check. But I had no idea what promptDelivery is all about so I have not followed it.
Anyway. What do you think of the timeline? Do you have to fix other bugs first?
I dont wanna have a concrete date. Just an hint to plan my release cycles too.

THX

[mclu]

Hi Howard.

OrderingGroup Names are not changed anymore.

But if I do my testcases in my testenvironment the handling still stops!!

I have 2000 Messages on Server A
On Server B I have a bridge which pulls messages from A to B

My queues are configured to fullsize=20 page und downcache=5.
Then:

- I start up my 2 servers.
- The Bridge on B starts sending messages from A to B.
- I add extra messages via a testclient to the queues on A and/or B

- Then the Bridge stops pulling messages from A to B. (I think because of added message on A)
- And also my message handler stops handling the pulled messages from the local queue on B. (because of extra message on B?!)

I tried to restart the bridge (stop/start) but this has no effect!

[mclu]

forgot to say....
Using build 7797 and set OrderingGroup via the ConnectionFactory on both servers.

[mclu]

Sorry. next message because I can not edit previous once :-)
I just debugged to the code and while it handles and then stops the promptDelivery is never true! So this could not be the Problem.

[gaohoward]

Hi Markus,

Thanks for the info. Can you confirm the version you are using? I just did a svn up and the version is only 7794. but you said it's 7797?

Howard