RpcManagerImpl:113 unexpected error while replicating
I am using 4.2.1.CR4, Sun JVM 6 and Tomcat 6 and I built a simple web application as an Infinispan POC. The app allows me to do basic operation on a cache including put values, delete values, get values, see cache inventory, see cluster information, clear a cache, and ceate a new cache dynamically. I deploy the WAR and start one Tomcat instance and everything appears to work as expected. I start a second Tomcat instance and it can see the cache and I can retrieve cached values from the cache created on the first instance. So far so good. Then I try to add a cache dynamically "cacheD" and the following exception is thrown while replicating. I can ask for the cluster information on either node and "cacheD" exist and has inventory. I have seen this app allow me to create a few caches dynamically before this exception is thrown.
I am very new to Infinispan ( 1 week ) but this smells a bit like a race condition while definining a new cache and the background replication threads are trying to distribute the cache information to other nodes. OR I am not understanding proper implementation.
Any help is greatly appreciated.
I have included my code to create the cache dynamically and the resulting configuration values below.
exception from my log file:
2011-03-13 07:57:40,068 [] INFO DistributedMemoryCache:79 - INFINISPAN: cacheName [cacheD] exist...returning
2011-03-13 07:57:40,193 [] ERROR RpcManagerImpl:113 - unexpected error while replicating
org.infinispan.manager.NamedCacheNotFoundException: Cache: cacheD
at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:123)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:159)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:144)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:578)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:489)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:365)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771)
at org.jgroups.JChannel.up(JChannel.java:1465)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:954)
at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:478)
at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:265)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:190)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:419)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:419)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:888)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
at org.jgroups.protocols.UNICAST.up(UNICAST.java:310)
at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:806)
at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:671)
at org.jgroups.protocols.BARRIER.up(BARRIER.java:120)
at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:169)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:269)
at org.jgroups.protocols.MERGE2.up(MERGE2.java:210)
at org.jgroups.protocols.Discovery.up(Discovery.java:292)
at org.jgroups.protocols.PING.up(PING.java:67)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1093)
at org.jgroups.protocols.TP.access$100(TP.java:56)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1633)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1615)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
2011-03-13 07:58:44,832 [] INFO DistributedMemoryCache:79 - INFINISPAN: checking for cacheName [cacheD]
2011-03-13 07:58:44,832 [] INFO DistributedMemoryCache:79 - INFINISPAN: cacheName [cacheD] exist...returning
cache configuration file:
<?xml version="1.0" encoding="UTF-8"?>
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:4.2 http://www.infinispan.org/schemas/infinispan-config-4.2.xsd"
xmlns="urn:infinispan:config:4.2">
<global>
<transport
clusterName="edgeCluster"
transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport"/>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<jmxStatistics enabled="true"/>
<clustering mode="distribution">
<l1 enabled="true" lifespan="60000"/>
<hash numOwners="2" rehashRpcTimeout="120000"/>
<async/>
</clustering>
</default>
</infinispan>
how I create cache dynamically:
/*
* see if the cache already exist
*/
if( manager.getCacheNames( ) != null && manager.getCacheNames( ).contains( cacheName ) )
{
return manager.getCache( cacheName );
}
else
{
/*
* define the new cache
*/
Configuration configuration = manager.getCache( ).getConfiguration( );
manager.defineConfiguration( cacheName, configuration );
/*
* get the new cache
*/
Cache cache = manager.getCache( cacheName );
/*
* add a LoggingListener
*/
cache.addListener( new LoggingListener( ) );
/*
* dump cache configuration values so I can see the configuration
*/
configurationValues( cache.getName( ), cache.getConfiguration( ) );
/*
* return the new cache
*/
return cache;
Configuration: [cacheD]
-------------------------------
getCacheModeString = DIST_ASYNC
getCacheLoaderManagerConfig = CacheLoaderManagerConfig{shared=false, passivation=false, preload='false', cacheLoaderConfigs.size()=0}
getCacheMode = DIST_ASYNC
getCacheStopTimeout = 30000
getClass = class org.infinispan.config.Configuration
getConcurrencyLevel = 32
getConsistentHashClass = org.infinispan.distribution.ch.DefaultConsistentHash
getCustomInterceptors = []
getDeadlockDetectionSpinDuration = 100
getEvictionMaxEntries = -1
getEvictionStrategy = NONE
getEvictionThreadPolicy = DEFAULT
getEvictionWakeUpInterval = 5000
getExpirationLifespan = -1
getHashFunctionClass = org.infinispan.util.hash.MurmurHash2
getIsolationLevel = READ_COMMITTED
getL1Lifespan = 60000
getLockAcquisitionTimeout = 10000
getName = null
getNumOwners = 2
getRehashRpcTimeout = 120000
getRehashWaitTime = 60000
getReplQueueClass = org.infinispan.remoting.ReplicationQueueImpl
getReplQueueInterval = 5000
getReplQueueMaxElements = 1000
getStateRetrievalInitialRetryWaitTime = 500
getStateRetrievalLogFlushTimeout = 60000
getStateRetrievalMaxNonProgressingLogWrites = 100
getStateRetrievalNumRetries = 5
getStateRetrievalRetryWaitTimeIncreaseFactor = 2
getStateRetrievalTimeout = 10000
getSyncReplTimeout = 15000
getTransactionManagerLookup = null
