How to shut down an embedded Infinispan node in a clustered environment whithout loosing data ?
mgeorg Aug 31, 2015 7:24 AMHello,
how can I shutdown an embedded Infinispan node in a cluster without loosing data ?
The problem is as follows:
We would like to use Infinispan 7.2.1 as a clustered and distributed key/value store
for about 10 to 20 million keys in one cache, the value size differs from 1k to 128k byte.
At the moment, I'm testing in a cluster with three nodes and I would like to shutdown
one node and restart it again after a while, but without loosing data. In the documentation
there is no explanation how to achieve this. When testing, I'm sending data to the cluster
with an external program, generating random keys with UUID's as value. The data is send to
all nodes, and after saving it in the Infinispan key/value store, it is send back for controlling.
When shutting down the whole cluster, the data is saved on disk, so after that we can
compare the saved data against the send back data to see if data loss has occured.
This will happen when shutting down only one node and stopping the other two nodes after
a while, then for example with 4.882.752 keys in the disk store there are missing 132 keys.
When shutting down, I got exceptions like "failed to process local variable for Key='xyz': could
not lock key 'xyz' because of: Timed out waiting for topology 5" for about 90 keys, nearly 15
seconds after stopping the node (I think this is the default value for 'remote-timeout' in
'distributed-cache' configuration).
The cache configuration is as follows:
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:7.1 http://www.infinispan.org/schemas/infinispan-config-7.1.xsd"
xmlns="urn:infinispan:config:7.1">
<!-- defining the network communication via JGroups -->
<jgroups>
<stack-file name="extern-file" path="../config/jgroups-udp.xml" />
</jgroups>
<cache-container default-cache="default" statistics="false">
<transport stack="extern-file" lock-timeout="240000" cluster="RSS Test" machine="c1" rack="2"/>
<distributed-cache name="KeyValueProvider" owners="2" mode="SYNC" segments="60">
<partition-handling enabled="true"/>
<state-transfer enabled="true" timeout="240000" chunk-size="10240" await-initial-transfer="true"/>
<locking isolation="REPEATABLE_READ" acquire-timeout="120000" concurrency-level="64"/>
<transaction mode="NON_XA" locking="PESSIMISTIC" stop-timeout="60000" complete-timeout="120000" transaction-manager-lookup="org.infinispan.transaction.lookup.GenericTransactionManagerLookup"/>
<store-as-binary keys="true" values="true"/>
</distributed-cache>
</cache-container>
</infinispan>
Code for cache declaration is:
private final EmbeddedCacheManager mCacheManager;
private final AdvancedCache<String,ExternalizableCacheObject> mCache;
Cache creation is like this, without loading data:
mCacheManager = new DefaultCacheManager(initializationFile); // initializationFile contains the name of the cache configuration file
mCache = cache.getAdvancedCache().withFlags(Flag.SKIP_CACHE_LOAD);
Cache stopping is:
mCache.stop();
mCacheManager.stop();
Is there something missing in the code or the configuration ?
Thanks for all advices and answers,
Michael