Packet Replication and failover

Each node in the cluster has a backup node url (Location) configured for it.

This is configured in jbm-configuration.xml <backup-node-url>tcp:hostname:port</backup-node>

(Potentially in the future we will allow the configuration of multiple backup nodes for a single active node, but first we will concentrate on the simplest case.)

As commands arrive from a client they are forwarded to the backup node, any replies from the backup node are forwarded back to the client. Effectively the live node maintains a connection to the backup node for every connection from the client. As session and consumers are created by the client, all these commands are replicated to the backup so the backup maintains an exact replica of the connection, session and consumers created on the live node. The exception being consumed messages not being sent from backup to live node since this is unnecessary.

The backup connection will have a flag "isbackup" set to true which determines if it actually sends messages to its client.

When a connection fails between client and server it is possible that inflight messages either from the client or server may be lost and need to be resent.

Consequently both the client and server number each packet they send with a unique and increasing contiguent id. They also maintain a buffer of packets they have sent. As packets are received in the other direction from the client or server (it is symmetrical) as well as their packet id, they contain the id of the last packet successfully received and processed from the other side. This can be used to clear down the client or servers resend buffer since it now knows those packets have been processed and won't need to be resent.

If a client or server has received many packets but hasn't sent a packet the other way for a while, then it can send an unsolicited "PacketAcknowledgeMessage" with the id. The number of packets to wait for before sending such a message can be configured on the connection factory, with the attribute packetAcknowledgeBufferSize.

When a client detects a failure in the network connection, it sends a ReAttachConnectionMessage to the backup node with the id of the connection. While this process is happening the client connection is frozen preventing any new packets being sent.

The client knows the backup node Location since this is configured as an attribute on the connection factory.

The server sets the backup flag on the connection to false.

The ReAttachConnectionMessage contains the id of the last packet received on the client side, the server responds with the id of the last message received on the server side.

Based on these two numbers, both the client and server can resend packets from their resend buffers, and clear out the remaining packets.

Once this has been done the connection can be unfrozen, and activity can continue as normal.