9 Replies Latest reply on Feb 3, 2011 8:42 AM by itayhindy

    ERROR org.infinispan.remoting.InboundInvocationHandlerImpl  - Defined caches

    itayhindy

      Hi,

       

      I am running stress tests on my servers that uses infinspan in cluster and distribution mode. During the tests I restarted one of the servers and kept the test running. I get the next error on the restarted server every few seconds and it does not recover (I am not able to work with the caches on that server):

      [2011-02-02 11:11:45,551] [OOB-7,infinispan-cluster,nqapp4-50937] ERROR org.infinispan.remoting.InboundInvocationHandlerImpl  - Defined caches: [users_cache, ban_users_cache, messages_cache, chat_diff_cache, base_room_cache, sequence_cache]

       

      Any Idea?

       

      thank in advance

        • 1. ERROR org.infinispan.remoting.InboundInvocationHandlerImpl  - Defined caches
          itayhindy

          More info:

          The replicated caches contains a lot of data and there is also the below exception:

           

          Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'chatService' defined in ServletContext resource [/WEB-INF/applicationContext.xml]: Invocation of init method failed; nested exception is org.infinispan.CacheException: Unable to invoke method public void org.infinispan.distribution.DistributionManagerImpl.waitForJoinToComplete() throws java.lang.Throwable on object

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1362)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:540)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:485)

                  at java.security.AccessController.doPrivileged(Native Method)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:455)

                  at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:251)

                  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:169)

                  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:248)

                  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:170)

                  at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:268)

                  ... 45 more

          Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.distribution.DistributionManagerImpl.waitForJoinToComplete() throws java.lang.Throwable on object

                  at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:174)

                  at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:889)

                  at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:687)

                  at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:589)

                  at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:150)

                  at org.infinispan.CacheDelegate.start(CacheDelegate.java:317)

                  at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:516)

                  at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:439)

                  at com.attractv.tvhive.common.cache.InfinispanCacheFactory.getCache(InfinispanCacheFactory.java:55)

                  at com.attractv.tvhive.service.impl.mngt.MngtChatServiceImpl.init(MngtChatServiceImpl.java:61)

                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                  at java.lang.reflect.Method.invoke(Method.java:597)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1427)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1396)

                  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1359)

                  ... 54 more

          Caused by: java.lang.reflect.InvocationTargetException

                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                  at java.lang.reflect.Method.invoke(Method.java:597)

                  at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:171)

                  ... 70 more

          Caused by: org.infinispan.CacheException: Unexpected exception

          at org.infinispan.distribution.JoinTask.performRehash(JoinTask.java:142)

                  at org.infinispan.distribution.RehashTask.call(RehashTask.java:53)

                  at org.infinispan.distribution.RehashTask.call(RehashTask.java:33)

                  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

                  at java.util.concurrent.FutureTask.run(FutureTask.java:138)

                  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                  at java.lang.Thread.run(Thread.java:619)

          Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout for nqapp3-51805

                  at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:49)

                  at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:414)

                  at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:101)

                  at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:125)

                  at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:230)

                  at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:217)

                  at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:200)

                  at org.infinispan.distribution.JoinTask.performRehash(JoinTask.java:132)

           

          thanks

          • 2. ERROR org.infinispan.remoting.InboundInvocationHandlerImpl  - Defined caches
            itayhindy

            If I stop my stress test then the replication works, it seems that trying to restart a server while the stress test is running and the caches are changing  all the time cause the replication for the restarted server to throw the errors and not load.

            Has anyone got those errors?

            • 3. ERROR org.infinispan.remoting.InboundInvocationHandlerImpl  - Defined caches
              mircea.markus

              Can you try increasing replTimeout configuration, perhaps to 20 then 30 seconds and try again?

              • 4. ERROR org.infinispan.remoting.InboundInvocationHandlerImpl  - Defined caches
                itayhindy

                I am working with async replication, can I set replTimeout fro async?

                • 5. Re: ERROR org.infinispan.remoting.InboundInvocationHandlerImpl  - Defined caches
                  itayhindy

                  Also

                  "ERROR org.infinispan.remoting.transport.jgroups.JGroupsTransport  Read past end of file"

                  and

                  Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws org.infinispan.statetransfer.StateTransferException on object

                          at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:174)

                          at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:889)

                          at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:687)

                          at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:589)

                          at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:150)

                          at org.infinispan.CacheDelegate.start(CacheDelegate.java:317)

                          at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:516)

                          at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:439)

                          at com.attractv.tvhive.common.cache.InfinispanCacheFactory.getCache(InfinispanCacheFactory.java:55)

                          at com.attractv.tvhive.service.impl.mngt.MngtChatServiceImpl.init(MngtChatServiceImpl.java:58)

                          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                          at java.lang.reflect.Method.invoke(Method.java:597)

                          at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1427)

                          at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1396)

                          at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1359)

                          ... 54 more

                  Caused by: java.lang.reflect.InvocationTargetException

                          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                          at java.lang.reflect.Method.invoke(Method.java:597)

                          at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:171)

                          ... 70 more

                  Caused by: org.infinispan.statetransfer.StateTransferException: Unable to fetch state on startup

                          at org.infinispan.remoting.rpc.RpcManagerImpl.retrieveState(RpcManagerImpl.java:186)

                          at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:129)

                          ... 75 more

                   

                   

                  exceptions in the log in case the stress test is running when restarting server.

                   

                  The only way to solve it is to stop the test when doing restart, however, because of the above problem we can not start using infinispan in our production servers that have heavy traffic all the time.

                  • 6. ERROR org.infinispan.remoting.InboundInvocationHandlerImpl  - Defined caches
                    mircea.markus

                    state transfer is happens sync even when your cache is configured to run async.

                    • 7. Re: ERROR org.infinispan.remoting.InboundInvocationHandlerImpl  - Defined caches
                      itayhindy

                      Thanks for the response.

                      Where can I set the replTimeout?

                       

                      This is my cache configuration:

                       

                          <namedCache name="room_cache">

                              <expiration lifespan="-1" maxIdle="-1"/> 

                              <invocationBatching enabled="true"/>

                              <clustering mode="replication">

                                  <async />

                                  <stateRetrieval fetchInMemoryState="true" timeout="60000"/>

                              </clustering>

                          </namedCache>

                      • 8. Re: ERROR org.infinispan.remoting.InboundInvocationHandlerImpl  - Defined caches
                        mircea.markus

                        <clustering>

                        <sync replTimeout="20000"/>

                        </clustering>

                        • 9. ERROR org.infinispan.remoting.InboundInvocationHandlerImpl  - Defined caches
                          itayhindy

                          Sorry but I did not understand.

                          Where you want me to write:

                          <clustering>

                          <sync replTimeout="20000"/>

                          </clustering>

                           

                          Inside my cache? It is already configure to be async? I can not set it to both sync and async toghter

                           

                              <namedCache name="room_cache">

                                  <expiration lifespan="-1" maxIdle="-1"/> 

                                  <invocationBatching enabled="true"/>

                                  <clustering mode="replication">

                                      <async />

                                      <stateRetrieval fetchInMemoryState="true" timeout="60000"/>

                                  </clustering>

                              </namedCache>