JBossDeveloper

Tim wrote:
We just need to extend the model slightly to cope with this.

I have committed some changes regarding server side failover.

In summary, these changes consist of:

I - Method failover(int) on DefaultClusteredPostOffice..
Still not integrated on JGroups:viewsChange, but that it's a trivial change (I hope). I will integrate it after I have done some more testing

II - Signature changes on PostOffice/ClusteredPostOffice to accept bindinging from other nodeIds than the current NodeId.

III - Changes on DefaultRouter to accept more than one local queue (created when DefaultClusteredPostOffice.failover is caleld).


Also... It looks like we will have to send messages received from client to server. This is because we will need to remove references on the database. I'm still investigating as I'm seeing messages still on the database after being received.

Also, you will realize a couple of TODOs on the code that are there just for tests. For example I'm ignoring some exceptions if a ACK is not found. This is just a temporary solution until I figure out what to do with pre-received messageIds. (send, ignore them, or find what's going on sometimes).

I will solve the messageId problems by tomorrow for sure, so I'm fine with this, it's just a matter of getting the job done.


Cheers,

Clebert Suconic

Just for reference, this method from DefaultClusteredPostOffice is going to be the heart of serverSide failover:

 public void failOver(int nodeId) throws Exception
 {
 log.info("Preparing failover against node " + nodeId);
 Map subMaps = (Map)nameMaps.get(new Integer(nodeId));
 ArrayList namesToRemove = new ArrayList();
 for (Iterator iterNames = subMaps.entrySet().iterator();iterNames.hasNext();)
 {
 Map.Entry entry = (Map.Entry)iterNames.next();
 Binding binding = (Binding )entry.getValue();
 if (binding.getQueue().isClustered())
 {
 ClusteredQueue queue = (ClusteredQueue) binding.getQueue();
 if (!queue.isLocal())
 {
 namesToRemove.add(entry);
 }
 }
 }

 for (Iterator iterNames = namesToRemove.iterator();iterNames.hasNext();)
 {
 Map.Entry entry = (Map.Entry)iterNames.next();
 DefaultBinding binding = (DefaultBinding)entry.getValue();
 RemoteQueueStub stub = (RemoteQueueStub)binding.getQueue();
 this.removeBinding(nodeId,(String)entry.getKey());

 Binding newBinding = this.createBinding(this.nodeId,nodeId,binding.getCondition(),stub.getName(),stub.getChannelID(),stub.getFilter(),stub.isRecoverable());
 LocalClusteredQueue clusteredQueue = (LocalClusteredQueue )newBinding.getQueue();
 clusteredQueue.deactivate();
 clusteredQueue.load();
 clusteredQueue.activate();
 addBinding(newBinding);
 }
 }

Another point we will have to think about is how to resume the server back.

Say.. you have serverA (the failedNode) and serverB (the server that will receive serverA's queues)

When serverA resumes after an eventual crash, all of his queues are loaded at serverB. They will need to be transfered back to serverA, and probably we will have to send a message to all clients to reconnect back into serverA after serverA resumed.

Maybe we could transfer queues as they reapear on serverA, but I guess there is a problem since javax.jms.Queue could be listened by more than one client.

Any ideas besides redirecting clients back on ServerA?

I'm answering your two latest posts here...

Tim wrote:
No need for the method to public - it will only be called from inside the post office

Tim wrote:
If the failover is driven from inside the postoffice then shouldn't need this on the interface?

Tim wrote:
Doesn't sound right to me. I'll have a look at the code.

Tim wrote:
There should be no need to check if the queue is clustered and not local, they will *always* be clustered and not local in the name map for a remote node, so this is redundant.

Tim wrote:
The correct type is Binding, not DefaultBinding?

Tim wrote:
The remove needs to be broadcast across the cluster so other nodes can remove the binding too

Tim wrote:
Isn't there going to be a problem creating two bindings with the same name?

"timfox" wrote:
This is a good point, and one that I covered in the design WIKI page: http://wiki.jboss.org/wiki/Wiki.jsp?page=NewHADesignOverview (see the end of the page)

Transferring queues back to a node if it recovers is a very difficult thing to do. Since consumers may be connected to the queues on the new node we can't just move them back to the old node.

Theoretically this is possible by effectively failing them over again to the original node, however in this case any non persistent messages unacked in their sessions will get redelivered so this is not acceptable, if the sessions have any unacked np messages.

This makes the whole thing complex, too complex to do initially. Eventually, if we decide to implement this it could also be used in the general case for the redistribution of connections across the cluster to best utilise available resources. But this is a nice to have and the pain to gain ratio is very high.

It's always good to have a very nice night of sleep...

I literally dreamed with an idea today :-) (Man... I'm getting that geek already :-) )

I was not sure what was that crazy dream... kind of hard to explain it but on my dream my bed was being invaded by HashMaps of Bindings, nodeIds and routers, and they were crashing with each other like the Brazilian flight 1907 (http://en.wikipedia.org/wiki/Gol_Transportes_A%C3%A9reos_Flight_1907)

Than I woke up and realized... I needed another map for failedNodes...

This way we could have duplicated names only on failedNodes... the client will need the channelId to be re-connected to a failedNode... and redistribution policies will take care of cases where the client also died.


I talked with Tim and I will be playing with this today and tomorrow, and it seems that this will solve the conceptual problems I'm having at this moment with these duplicates.


Kind of funny story, isn't? :-)


Clebert

ummmmm, Clebert .... this is a public forum, everybody can read it :)

Never mind, you go ahead, I am sure you'll figure out a way to overcome those invading Bindings, nodeIDs and routers :)

Tim wrote:
Any objections?