Check Infinispan state transfer on server startup

[manmeetjboss]

Our application is running in distributed mode. The same EAR needs to be deployed on different servers. In EAR, code is written that loads the infinispan cache from DB. If we deploy same EAR on multiple servers, each EAR will load data from DB.

So how can we check that if infinispan cache is loaded by one EAR on one server,  another EAR doesn't load the cache again from DB but instead joins the server1 and all data gets transferred from server1 to server2.

[william.burns]

I assume you are using the preload configuration option for the loader.

The preload step is performed before the cache joins into the cluster.  This is to ensure that data is properly rehashed to other nodes when not using a shared loader.  In the case of a shared loader there is no way currently to only perform it once without changing the config so only a single node has preload configured to true.  If you have a large data set maybe instead it would be more beneficial to just not enable preload and use passivation?  But that depends on how many values you will need in comparison to your data set at a given time.

[manmeetjboss]

In case of shared loader, if each node has different configuration, then how automatic failover cases will be handled in distributed environment. For example, there are three nodes in cluster, say Node A, Node B and Node C. Node A has preload=true, whereas other two has preload=false. Now during deployment, one has to take care that Node A always starts first, afterwords Node B and Node C joins cluster so that data from Node A gets transferred to Node B and Node C (via infinispan state transfer mechanism)

However, in case of outage if all server goes down and by somehow Node A can't come up automatically, whereas other two servers are able to come up, their local cache will be empty and hence application will not work properly.

We have solved this problem using HA Singleton. We have created HA singleton service whose duty is to load data from DB into infinispan cache. Since it is singleton per cluster so only master node (which starts first) will load the data from DB in cache whereas other nodes won't load the data from DB again and their cache gets populated via state transfer from master node.

If node A goes down, either node B or C becomes new master. At this point, singleton service will be created on that new master node which again tries to load data from DB in cache, but that can be prevented by putting a check if cache is not empty, then don't load the data from DB.

With this approach, it doesn't matter which node starts first.

Please suggest is this solution okay?

[william.burns]

However, in case of outage if all server goes down and by somehow Node A can't come up automatically, whereas other two servers are able to come up, their local cache will be empty and hence application will not work properly.

This isn't true (unless you are using special flags via AdvancedCache.withFlag).  In this case if a miss occurs in the memory cache it would then check the loader to see it if exists.  Preload is there just to improve performance by warming the cache up so that data would get more hits.

It sounds like what you do though should definitely help increase hit/miss ratio of the in memory cache.