How to know when all messages have been processed?

tim.shaw Mar 25, 2008 10:33 AM

Hi,

We need to implement a parallelising system using messaging - work is handled by message beans.

The upstream and message processing works fine - we've run cluster performance tests on 200000 messages and scalability is fine (and linear across 3 systems in a cluster).

The problem is to determine the best way of knowing when all the messages have been processed.

Initial attempts resulted in the thread http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4138673#4138673 which wasn't much use.

I am loathe to use a semaphore approach - we know the number of messages before we start, so each message could decrement this - as this would introduce contention on the semaphore (however implemented I think).

There doesn't appear to be a way of querying the queue.

We can't stick a 'final' message onto the queue as this could be processed before all the others have been processed (each message can take a while).

I have some other ideas, but they're all a bit twiddly : entry in DB for each completion ... count entries)

Just wondering if anyone has had the same problem - and what suggestions there are for solving it?

Many Thanks

tim

1. Re: How to know when all messages have been processed?

adrian.brock Mar 31, 2008 6:08 AM (in response to tim.shaw)

Any solution that uses non-volatile storage to hold the "count" of messages
processed is going to fail when the server with that storage crashes unless you
are clever about how you design the protocol.

e.g. you could collate the responses by sending jms responses to a client
that is designed for the purpose. e.g. uses CLIENT_ACK to wait for all
reponses in a group (defined in the message properties) before considering the work done
and then doing whatever is required.
Actions