Shared and non-shared persistent stores

[timfox]

I think we need to be able distinguish between shared and non-shared persistent stores.

A shared persistent store (probably a RDBMS) can be seen by all nodes on a LAN, but a non-shared persistent store (e.g. a file based store) is only usable by a single node (but probably lives on a SAN for reliability).

When we send (or ack) a message to a destination in a cluster, and the active queue node is a different node to the sending node, then if the destination's persistent store is shared we can persist the message/ack (if reliable) *before* sending to the active node, otherwise we have to wait until it reaches the active node before it gets persisted.

[timfox]

Non shared persistent stores bring their own set of issues.

If different nodes on the cluster have their own stores, then when sending a reliable message to a topic which has multiple durable subscriptions, (or queue with replicas), then we need to use our custom 2PC multicast protocol described in an earlier thread, to ensure reliability.

If all nodes are using the same shared store, then life is much easier since we can update the database once and then do a 1PC multicast. This is going to be more performant too.

To make our life easier we could only support shared persistent stores, although I think this is perhaps too much of a limitation since this will prevent us from using file based stores for instance.