Message redistribution: what if a node never comes back?

[kennardconsulting]

Hi guys,

I have been staring at the HornetQ documentation for a while and can't seem to grok this point.

Given that HornetQ uses local queues stored on the file system of each node, and that the producer distributes messages in some kind of even fashion to each node, what happens if a node goes down and never returns? Are all messages queued up on that node now lost? The manual mentions 'message redistribution' and 'starvation' but how can the other nodes access the 'dead' node to find out what messages were queued up there in order to redistribute them?

I read the bit about 'shared storage', but I assume this is just to backup a node? I tried starting a cluster with all nodes pointing to the same shared storage and that behaved very strangely! I assume the idea is that each active node is meant to have its own, isolated storage?

Thanks,

Richard.

[ataylor]

yes, if a node goes down then any messages on this node will not be available. For this you need to enable failover by configuring a backup node for each live server.

[kennardconsulting]

Andy,

Many thanks for your reply!

This would seem a loss of functionality compared to JBoss Messaging? Specifically, on JBoss 5.1.0.GA we had 3 nodes that were all sharing the processing of, say, 3 thousand messages (out of a DB-backed queue). If one of these nodes went down after say, 500 messages, the others would still 'take up the slack' and complete processing of all 3 thousand. They may dynamically decide to finish off 1.25 thousand each.

But now it would seem that processing will stop after, say, 2 thousand messages and wait until the stuck node comes back up, whereupon it can finish processing its remaining 500 messages? Given that one of the nodes may fail overnight, and needs to be manually brought back up, this is a bit awkward.

Is there any way to configure HornetQ to function as JBoss Messaging did? To complete processing of all messages, even if one node goes down?

Richard.

[ataylor]

This would seem a loss of functionality compared to JBoss Messaging? Specifically, on JBoss 5.1.0.GA we had 3 nodes that were all sharing the processing of, say, 3 thousand messages (out of a DB-backed queue). If one of these nodes went down after say, 500 messages, the others would still 'take up the slack' and complete processing of all 3 thousand. They may dynamically decide to finish off 1.25 thousand each.

It is not a loss of functionality as you can obtain the same functionality by providing backup servers. Its more conig and extra servers yes but with this comes higher reliablity and better scalability. With JbossMessaging you have a single point of failure, if your db or db connection goes down you are basically stuffed. ALso using a journal gives you much better performance

But now it would seem that processing will stop after, say, 2 thousand messages and wait until the stuck node comes back up, whereupon it can finish processing its remaining 500 messages? Given that one of the nodes may fail overnight, and needs to be manually brought back up, this is a bit awkward.

Backup servers will automatically start once a live server goes down

Is there any way to configure HornetQ to function as JBoss Messaging did? To complete processing of all messages, even if one node goes down?

As above

[kennardconsulting]

Andy,

Okay thanks for your informative answers.

This is a bit of a 'mindset shift' to what we're used to, so we'll have to give it some more thought!

Regards,

Richard.

[ataylor]

we have done a lot of work around this to make things easier to configure, this was mostly for eap platform and is in hornetq 2.2.2, it should be possible to locate live and backup servers on the same AS instance. what versions are you using

[kennardconsulting]

I am using the stock JBoss 6.0.0.Final.

Regards,

Richard.

[ataylor]

ok, that uses hornetq 2.1.2 so you can either have dedicated backup servers ot move to hornetq 2.2.2, or wait for as 7

[kennardconsulting]

We can certainly consider moving to 2.2.2. Can you point me to some docs that explain what you mean by 'it should be possible to locate live and backup servers on the same AS instance'?

Richard.

[ataylor]

currently this is  only in the eap manual altho if you checkout trunk you can build it, its under docs/