CacheException: Initial state transfer timed out for cache
dex80526 Dec 30, 2013 3:11 PMI have a 3 node cluster with replication configuration. I am seeing the "Initial state transfer timed out exception" during the initial start up of the cluster. This is definitely related to the number of entries in the cache store (we used rdmb/apache derby as persistent store). The time out exception happens when there are about 1 million cache entries.
I already have a very big timeout value in the configure (see below).
To me this is a critical scaling issue. I saw this timeout happens more frequently when the size of cluster (number of nodes) goes up to 5 or higher.
Anyone has any suggestion to work around this?
Here are the configuration for the cache:
....
<default>
<locking concurrencyLevel="5000" isolationLevel="READ_COMMITTED" lockAcquisitionTimeout="60000" useLockStriping="false" writeSkewCheck="false"/>
<transaction cacheStopTimeout="30000" eagerLockSingleNode="false" lockingMode="OPTIMISTIC" syncCommitPhase="false" syncRollbackPhase="false" transactionManagerLookupClass="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup" useEagerLocking="false"/>
<deadlockDetection enabled="true" spinDuration="1000"/>
<jmxStatistics enabled="true"/>
</default>
<namedCache name="KEYCHAIN">
<clustering mode="replication">
<stateTransfer chunkSize="10000" fetchInMemoryState="true" timeout="400000"/>
<async asyncMarshalling="true" replQueueInterval="1000" replQueueMaxElements="1000" useReplQueue="true"/>
</clustering>
<expiration lifespan="-1" maxIdle="-1" wakeUpInterval="-1"/>
<loaders passivation="false" preload="false" shared="false">
<store class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore" fetchPersistentState="false" purgeOnStartup="false">
<properties>
<property name="stringsTableNamePrefix" value="ISPN_STRING_TABLE"/>
<property name="idColumnName" value="ID_COLUMN"/>
<property name="dataColumnName" value="DATA_COLUMN"/>
<property name="timestampColumnName" value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType" value="BIGINT"/>
<property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory"/>
<property name="datasourceJndiLocation" value="java:comp/env/jdbc/UP_DB"/>
<property name="idColumnType" value="VARCHAR(255)"/>
<property name="dataColumnType" value="BLOB"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart" value="true"/>
....