RpcDispatcher error in distributed mode cluster

[infini8]

I have deployed three cache nodes for testing purposes.

Following code is run on all nodes :

CacheManager cacheManager = new DefaultCacheManager("gui-demo-cache-config.xml");

Cache<String, Float> cache = cacheManager.getCache();

cache.put("key5", 0.1F);

config file is as :

<?xml version="1.0" encoding="UTF-8"?>

<infinispan

      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

      xsi:schemaLocation="urn:infinispan:config:4.0 http://www.infinispan.org/schemas/infinispan-config-4.0.xsd"

      xmlns="urn:infinispan:config:4.0">

   <global>

      <transport transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport" clusterName="demoCluster">

      </transport>

   </global>

   <default name="defaultCache">

      <clustering mode="distribution">

         <l1 enabled="false" lifespan="600000"/>

         <hash numOwners="2" rehashRpcTimeout="120000"/>

         <sync/>

      </clustering>

   </default>

</infinispan>

After deploying 2 nodes, the third node sometimes throws the following exception on cache.put();

SEVERE: Execution error:

org.infinispan.CacheException: java.io.NotSerializableException: RpcDispatcher returned a null.  This is most often caused by args for ClusteredGetCommand not being serializable.

at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:124)

at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:401)

at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:100)

at org.infinispan.distribution.DistributionManagerImpl.retrieveFromRemoteSource(DistributionManagerImpl.java:232)

at org.infinispan.interceptors.DistributionInterceptor.realRemoteGet(DistributionInterceptor.java:131)

at org.infinispan.interceptors.DistributionInterceptor.remoteGetAndStoreInL1(DistributionInterceptor.java:119)

at org.infinispan.interceptors.DistributionInterceptor.remoteGetBeforeWrite(DistributionInterceptor.java:241)

at org.infinispan.interceptors.DistributionInterceptor.handleWriteCommand(DistributionInterceptor.java:280)

at org.infinispan.interceptors.DistributionInterceptor.visitPutKeyValueCommand(DistributionInterceptor.java:162)

at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

at org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:198)

at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)

at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)

at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:183)

at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:132)

at org.infinispan.interceptors.DistTxInterceptor.visitPutKeyValueCommand(DistTxInterceptor.java:76)

at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)

at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)

at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)

at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:269)

at org.infinispan.CacheDelegate.put(CacheDelegate.java:434)

at org.infinispan.CacheDelegate.put(CacheDelegate.java:205)

at com.germinait.test.Test.main(Test.java:15)

This exception is not always thrown. I could neither figure out the cause, nor find any solution in available discussions.

Thanks, in advance, for any help.

[mircea.markus]

Can you reproduce this through an unit test? Glad to take a look

[infini8]

System Specs: Inter Single core 3.00GHz; 2GB RAM

I have created three jar files & attached the same to this post. Each have their source codes attached in the jar.

infinispanNode1.jar

infinispanNode2.jar

infinispanNode3.jar

command to be executed: java -cp$infinispan-dependencies & infinispanNode1.jar$ com.company.test.ClusterScaleTest

Only difference in their source code is:

node1 puts entries with key=1 to 100

node2 puts entries with key=101 to 200

node2 puts entries with key=201 to 300

Node1 & 2 go live with the following commands with no issues.

java -cp$infinispan-dependencies & infinispanNode1.jar$ com.company.test.ClusterScaleTest

java -cp$infinispan-dependencies & infinispanNode2.jar$ com.company.test.ClusterScaleTest

While trying to run the third node

java -cp$infinispan-dependencies & infinispanNode3.jar$ com.company.test.ClusterScaleTest

It throws:

log4j:WARN No appenders could be found for logger (org.infinispan.factories.ComponentRegistry).

log4j:WARN Please initialize the log4j system properly.

Exception in thread "main" org.infinispan.CacheException: java.io.NotSerializableException: RpcDispatcher returned a null.  This is most often caused by args for ClusteredGetCommand not being serializable.

        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:124)

        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:401)

        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:100)

        at org.infinispan.distribution.DistributionManagerImpl.retrieveFromRemoteSource(DistributionManagerImpl.java:232)

        at org.infinispan.interceptors.DistributionInterceptor.realRemoteGet(DistributionInterceptor.java:131)

        at org.infinispan.interceptors.DistributionInterceptor.remoteGetAndStoreInL1(DistributionInterceptor.java:119)

        at org.infinispan.interceptors.DistributionInterceptor.remoteGetBeforeWrite(DistributionInterceptor.java:241)

        at org.infinispan.interceptors.DistributionInterceptor.handleWriteCommand(DistributionInterceptor.java:280)

        at org.infinispan.interceptors.DistributionInterceptor.visitPutKeyValueCommand(DistributionInterceptor.java:162)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

        at org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:198)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)

        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

        at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:183)

        at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:132)

        at org.infinispan.interceptors.DistTxInterceptor.visitPutKeyValueCommand(DistTxInterceptor.java:76)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

        at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)

        at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)

        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

        at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:269)

        at org.infinispan.CacheDelegate.put(CacheDelegate.java:434)

        at org.infinispan.CacheDelegate.put(CacheDelegate.java:205)

        at com.company.test.ClusterScaleTest.main(ClusterScaleTest.java:34)

Caused by: java.io.NotSerializableException: RpcDispatcher returned a null.  This is most often caused by args for ClusteredGetCommand not being serializable.

        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:326)

        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:122)

        ... 30 more

[kapilnayar1]

Hi Mircea,

I am also getting this error when running 3 nodes and accessing the cache using concurrent threads.

Any updates/ fixes.

Kapil