Messages being delivered out of sequence

[dave.stubbs]

I've just had a problem with MQ in a recent soak test where batches of transactions were not received in the order they were written to the queue.

I've got a very long running test, pushing and pulling over 4 million transactions. MQ generally fills up its storage and throttles but this is normally OK.

In this instance I stopped the application filling the queue and allowed the client to continue.

Each record received is written to a file with an auto incermenting ID and a timestamp so I can tell precisely what order the replies were received in, each message had a sequence number as well so I could verify all three components to prove successful operation.

Like I said everything seemed to be going fine, I'd got up to 900K records when suddenly I noticed the ID and transaction ref got way out of sync.

By ordering the file I could see that random chunks of transactions had not been delivered in order at the correct time, but were delivered at some time later in the correct order (by this time preceeded by 30,000 or so other transactions).

The timestamps also indicated these were received processed some 50 minutes after they should have been processed as well.

I've attached the data for you to see for yourself. I've ordered it by the sequence ID used to publish the transactions so you can see how the actual receiving ID and timestamps are out of sync.

The queues are persisted and we're using them transactionally as well.

This is the Windows version of 5.2.0.2

The config we use is the vanilla version stright from the install only the memory is ramped up to 4GB for the policy entriy for queue.

This is the first time I've seen this bug in 6 months of testing, though I've seen many other strange issues in this time (posted on these boards) but this is the most serious issue I've had to date.

Any insight / bug fixes welcomed

[gseben]

Hi Dave,

Not long ago there was an edge case with the failover transport where messages would end up being delivered out of order (see link)

http://fusesource.com/issues/browse/MB-456

Your case however, seems to be different. I'm not aware of bugs reported that match this description. If you attach the test case that you used to reproduce this problem we can use that to find the cause of the problem.

-Gio

[dave.stubbs]

I wish these problems were that simple.

Every time I get an issue like this, I've never been able to reproduce it again. It is very frustrating.

Is it because I'm running on a Windows server that things are generally so unreliable.

I had a problem earlier today where three queues stopped being read from. I stopped and restarted the client with no change. But when I stopped and restarted the server they started flowing through again as if nothing had happened.

But can I repeat it, no. As I said, very frustrating.