It was just jgroups configuration error. It works now.
I am using the following configuration:
<transaction syncCommitPhase="true" syncRollbackPhase="true"/>
<property name="location" value="../store/"/>
<hash numOwners="2" rehashWait="120000" rehashRpcTimeout="900000"/>
With the above configuration, when I start 2 servers and perform a remoteCache.put(k1, v1) I am expecting the put to be blocked until it is both replicated and persisted on both the instances, otherwise throw an error when either of those fails.
But, it does not throw an error even when I shut down 1 instance.
Prasanth, I believe you don't see any error because the client retried the operation on the other server and succeeded.
DIST mode doesn't require numOwners nodes in order to work, as long as there is one node in the cluster your put will succeed.