3 Replies Latest reply on Sep 16, 2010 2:39 PM by Kapil Nayar

    RpcDispatcher error in distributed mode cluster

    infini8 infini8 Newbie

      I have deployed three cache nodes for testing purposes.

       

      Following code is run on all nodes :

      CacheManager cacheManager = new DefaultCacheManager("gui-demo-cache-config.xml");

      Cache<String, Float> cache = cacheManager.getCache();

      cache.put("key5", 0.1F);

       

      config file is as :

      <?xml version="1.0" encoding="UTF-8"?>

      <infinispan

            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

            xsi:schemaLocation="urn:infinispan:config:4.0 http://www.infinispan.org/schemas/infinispan-config-4.0.xsd"

            xmlns="urn:infinispan:config:4.0">

       

         <global>

            <transport transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport" clusterName="demoCluster">

            </transport>

         </global>

       

         <default name="defaultCache">

            <clustering mode="distribution">

               <l1 enabled="false" lifespan="600000"/>

               <hash numOwners="2" rehashRpcTimeout="120000"/>

               <sync/>

            </clustering>

         </default>

      </infinispan>

       

      After deploying 2 nodes, the third node sometimes throws the following exception on cache.put();

       

      SEVERE: Execution error:
      org.infinispan.CacheException: java.io.NotSerializableException: RpcDispatcher returned a null.  This is most often caused by args for ClusteredGetCommand not being serializable.
      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:124)
      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:401)
      at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:100)
      at org.infinispan.distribution.DistributionManagerImpl.retrieveFromRemoteSource(DistributionManagerImpl.java:232)
      at org.infinispan.interceptors.DistributionInterceptor.realRemoteGet(DistributionInterceptor.java:131)
      at org.infinispan.interceptors.DistributionInterceptor.remoteGetAndStoreInL1(DistributionInterceptor.java:119)
      at org.infinispan.interceptors.DistributionInterceptor.remoteGetBeforeWrite(DistributionInterceptor.java:241)
      at org.infinispan.interceptors.DistributionInterceptor.handleWriteCommand(DistributionInterceptor.java:280)
      at org.infinispan.interceptors.DistributionInterceptor.visitPutKeyValueCommand(DistributionInterceptor.java:162)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      at org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:198)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:183)
      at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:132)
      at org.infinispan.interceptors.DistTxInterceptor.visitPutKeyValueCommand(DistTxInterceptor.java:76)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)
      at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)
      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
      at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:269)
      at org.infinispan.CacheDelegate.put(CacheDelegate.java:434)
      at org.infinispan.CacheDelegate.put(CacheDelegate.java:205)
      at com.germinait.test.Test.main(Test.java:15)

       

      This exception is not always thrown. I could neither figure out the cause, nor find any solution in available discussions.

      Thanks, in advance, for any help.

        • 1. Re: RpcDispatcher error in distributed mode cluster
          Mircea Markus Master

          Can you reproduce this through an unit test? Glad to take a look

          • 2. Re: RpcDispatcher error in distributed mode cluster
            infini8 infini8 Newbie

            System Specs: Inter Single core 3.00GHz; 2GB RAM

             

            I have created three jar files & attached the same to this post. Each have their source codes attached in the jar.

            infinispanNode1.jar

            infinispanNode2.jar

            infinispanNode3.jar

             

            command to be executed: java -cp$infinispan-dependencies & infinispanNode1.jar$ com.company.test.ClusterScaleTest

             

            Only difference in their source code is:

            node1 puts entries with key=1 to 100

            node2 puts entries with key=101 to 200

            node2 puts entries with key=201 to 300

             

            Node1 & 2 go live with the following commands with no issues.

            java -cp$infinispan-dependencies & infinispanNode1.jar$ com.company.test.ClusterScaleTest

            java -cp$infinispan-dependencies & infinispanNode2.jar$ com.company.test.ClusterScaleTest

             

            While trying to run the third node

            java -cp$infinispan-dependencies & infinispanNode3.jar$ com.company.test.ClusterScaleTest

             

            It throws:

             

            log4j:WARN No appenders could be found for logger (org.infinispan.factories.ComponentRegistry).
            log4j:WARN Please initialize the log4j system properly.
            Exception in thread "main" org.infinispan.CacheException: java.io.NotSerializableException: RpcDispatcher returned a null.  This is most often caused by args for ClusteredGetCommand not being serializable.
                    at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:124)
                    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:401)
                    at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:100)
                    at org.infinispan.distribution.DistributionManagerImpl.retrieveFromRemoteSource(DistributionManagerImpl.java:232)
                    at org.infinispan.interceptors.DistributionInterceptor.realRemoteGet(DistributionInterceptor.java:131)
                    at org.infinispan.interceptors.DistributionInterceptor.remoteGetAndStoreInL1(DistributionInterceptor.java:119)
                    at org.infinispan.interceptors.DistributionInterceptor.remoteGetBeforeWrite(DistributionInterceptor.java:241)
                    at org.infinispan.interceptors.DistributionInterceptor.handleWriteCommand(DistributionInterceptor.java:280)
                    at org.infinispan.interceptors.DistributionInterceptor.visitPutKeyValueCommand(DistributionInterceptor.java:162)
                    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
                    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
                    at org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:198)
                    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
                    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
                    at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
                    at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
                    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
                    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
                    at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:183)
                    at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:132)
                    at org.infinispan.interceptors.DistTxInterceptor.visitPutKeyValueCommand(DistTxInterceptor.java:76)
                    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
                    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
                    at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)
                    at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)
                    at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
                    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
                    at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:269)
                    at org.infinispan.CacheDelegate.put(CacheDelegate.java:434)
                    at org.infinispan.CacheDelegate.put(CacheDelegate.java:205)
                    at com.company.test.ClusterScaleTest.main(ClusterScaleTest.java:34)
            Caused by: java.io.NotSerializableException: RpcDispatcher returned a null.  This is most often caused by args for ClusteredGetCommand not being serializable.
                    at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:326)
                    at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:122)
                    ... 30 more
            • 3. Re: RpcDispatcher error in distributed mode cluster
              Kapil Nayar Newbie

              Hi Mircea,

               

              I am also getting this error when running 3 nodes and accessing the cache using concurrent threads.

               

              Any updates/ fixes.

               

              Kapil