7 Replies Latest reply on May 27, 2011 11:14 AM by galder.zamarreno

    Why I got org.infinispan.tree.NodeKey serialization error in 5.0.0.cr when using distributed cache?

    yangju

      In jboss AS 6 Final (I upgraded its infinispan libraries from 4.2 to 5.0.0.cr). I have two nodes in the cluster. If I just enable one node, everything is fine. But as soon as more than one nodes in the cluster, when jboss is started, it gives the following errors:

      Why infinispan has anything to do with tree node?

       

      16:52:05,793 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] ISPN00095: Caught while responding to state transfer request: org.infinispan.statetransfer.StateTransferException: java.io.NotSerializableException: org.infinispan.tree.NodeKey

          at org.infinispan.statetransfer.StateTransferManagerImpl.generateInMemoryState(StateTransferManagerImpl.java:364) [:5.0.0.CR1]

          at org.infinispan.statetransfer.StateTransferManagerImpl.generateState(StateTransferManagerImpl.java:163) [:5.0.0.CR1]

          at org.infinispan.remoting.InboundInvocationHandlerImpl.generateState(InboundInvocationHandlerImpl.java:242) [:5.0.0.CR1]

          at org.infinispan.remoting.transport.jgroups.JGroupsTransport.getState(JGroupsTransport.java:564) [:5.0.0.CR1]

          at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:690) [:2.12.0.Final]

          at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771) [:2.12.0.Final]

          at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:135) [:2.12.0.Final]

          at org.jgroups.JChannel.up(JChannel.java:1484) [:2.12.0.Final]

          at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074) [:2.12.0.Final]

          at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:477) [:2.12.0.Final]

          at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER$StateProviderHandler.process(STREAMING_STATE_TRANSFER.java:651) [:2.12.0.Final]

          at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER$StateProviderThreadSpawner$1.run(STREAMING_STATE_TRANSFER.java:580) [:2.12.0.Final]

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) [:1.6.0_24]

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [:1.6.0_24]

          at java.lang.Thread.run(Unknown Source) [:1.6.0_24]

      Caused by: java.io.NotSerializableException: org.infinispan.tree.NodeKey

          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:972) [:]

          at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:423) [:]

          at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.writeObject(ImmortalCacheEntry.java:126) [:5.0.0.CR1]

          at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.writeObject(ImmortalCacheEntry.java:123) [:5.0.0.CR1]

          at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:361) [:5.0.0.CR1]

          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:147) [:]

          at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:423) [:]

          at org.infinispan.marshall.MarshallUtil.marshallCollection(MarshallUtil.java:49) [:5.0.0.CR1]

          at org.infinispan.marshall.exts.SetExternalizer.writeObject(SetExternalizer.java:62) [:5.0.0.CR1]

          at org.infinispan.marshall.exts.SetExternalizer.writeObject(SetExternalizer.java:47) [:5.0.0.CR1]

          at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:361) [:5.0.0.CR1]

          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:147) [:]

          at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:423) [:]

          at org.infinispan.marshall.jboss.GenericJBossMarshaller.objectToObjectStream(GenericJBossMarshaller.java:121) [:5.0.0.CR1]

          at org.infinispan.marshall.VersionAwareMarshaller.objectToObjectStream(VersionAwareMarshaller.java:156) [:5.0.0.CR1]

          at org.infinispan.statetransfer.StateTransferManagerImpl.generateInMemoryState(StateTransferManagerImpl.java:362) [:5.0.0.CR1]

          ... 14 more

      Caused by: an exception which occurred:

          in object org.infinispan.tree.NodeKey@ff816b1c

              -> toString = NodeKey{contents=STRUCTURE, fqn=/}

          in object org.infinispan.container.entries.ImmortalCacheEntry@2270abac

              -> toString = ImmortalCacheEntry{cacheValue=ImmortalCacheValue{value=AtomicHashMap{delegate=org.infinispan.util.FastCopyHashMap@f795b2}}} ImmortalCacheEntry{key=NodeKey{contents=STRUCTURE, fqn=/}}

          in object java.util.HashSet@68dca9fb

              -> toString = [ImmortalCacheEntry{cacheValue=ImmortalCacheValue{value=AtomicHashMap{delegate=org.infinispan.util.FastCopyHashMap@f795b2}}} ImmortalCacheEntry{key=NodeKey{contents=STRUCTURE, fqn=/}}, ImmortalCacheEntry{cacheValue=ImmortalCacheValue{value=AtomicHashMap{delegate=org.infinispan.util.FastCopyHashMap@7e84d99c}}} ImmortalCacheEntry{key=NodeKey{contents=DATA, fqn=/}}]

        • 1. Why I got org.infinispan.tree.NodeKey serialization error in 5.0.0.cr when using distributed cache?
          sannegrinovero

          Hi,

          it seems you're using the tree api, maybe indirectly.

          All keys and values in Infinispan need to be either Serialiable or have an Externalizer registered; the org.infinispan.tree.NodeKey is not serializable but org.infinispan.tree.LifecycleCallbacks should be invoked at cachemanager start to register the proper externalizer; it seems from your stacktrace that this didn't happen.

          Is it possible you're mixing some older library with never 5.0.0.CR1 ? please use Tattletale to check your classpath.

          • 2. Why I got org.infinispan.tree.NodeKey serialization error in 5.0.0.cr when using distributed cache?
            yangju

            Thanks, Sanne, I will double check my classpath for the jars. It could be that one of the jars in the jboss common lib directory needs to be updated too for infinispan.

             

            May I piggy back another separate issue? I am doing loading testing using too nodes (distribution mode) with jboss 6 and infinispan. If I don't know the key and I just want to ask for the next available entry in the cache, it always gives me the entry in the local jvm. How can I also get any entry from the other node if this node does not have any entry left? I am using the distributed cache to evenly distribute work (entries), so worker threads from each node work independently but can see work acoss the cluster.

             

            I have had the locking issue if I use programmatically created cache with jboss 6 infinispan's hibernate 2nd level cache. If I use them together, I would get locktimeout exception for the hibernate cache, even I make the hibernate cache local only, I still got this exception.

            Any direction is appreaciated.

             

            Thanks.

            • 3. Re: Why I got org.infinispan.tree.NodeKey serialization error in 5.0.0.cr when using distributed cache?
              sannegrinovero
              May I piggy back another separate issue? I am doing loading testing using too nodes (distribution mode) with jboss 6 and infinispan. If I don't know the key and I just want to ask for the next available entry in the cache, it always gives me the entry in the local jvm. How can I also get any entry from the other node if this node does not have any entry left? I am using the distributed cache to evenly distribute work (entries), so worker threads from each node work independently but can see work acoss the cluster.

              Sounds like you're running some task which needs to process all keys on all nodes; you should use the map/reduce API for that. If that's not the case, use the Query module to index special attributes so you can find entities by this attribute (Map/Reduce API can be used as well for that). It's not possible if you don't have the key to retrieve a list of all known keys.

               

               

              I have had the locking issue if I use programmatically created cache with jboss 6 infinispan's hibernate 2nd level cache. If I use them together, I would get locktimeout exception for the hibernate cache, even I make the hibernate cache local only, I still got this exception.

              Any direction is appreaciated.

              I don't know about which issue you're talking about; please open a new thread about this and try to elaborate on the issue and configuration you're using; Ideally if you had a test case that would be great.

              • 4. Why I got org.infinispan.tree.NodeKey serialization error in 5.0.0.cr when using distributed cache?
                yangju

                Sorry but I have to add some comments here to the infinispan developers about my recent experience with infinispan in our performance testing (two nodes, each nodes have 10 theads constanly hit the cache (read and update)). The result is frustrating. Because there so many issues found, it is hard to open one single jira. I documented here so that maybe the infinispan developers can tell me it is my config and design, not the framework's defects.

                 

                I have spent more than 6 weeks valuable developerment time fighting various issues that are related to distributed cache on 4.2.0 and 4.2.1 in the 2 nodes cluster. Those issues are mainly on timeout locking of cached entries, i.e. concurrency. Examples:

                NPE in informRehashOnJoin (ISPN-1066), Timeout in obtaining lock (even I have deadlock detection enabled), ISPN-434 creating a new distributed cache causing rehading to hang and many more. Some of the issues are said to be fixed already in 4.2.0 or 4.2.1 and I can still reproduce it.

                 

                The most annoying problem is that programmatically creating cache does not alway work as expected. If I call getCache(name) if the name does not exist, the cache will be created. This is fine. But the problem occurs when the node which have had the cache tries to lock an entry on the other node which does not yet have this cache. It hangs.

                 

                Another major issue is that I use jboss AS 6 and it has infinispan built in. If I configure my jpa to use infinispan as 2LC (distributed) and I have my programmatically created cache at the same time, I always got timeout locking issue, although I cannot tell why this happened from the log. I had to turn off infinispn for hibernate 2LC, then the problem is solved.

                 

                The third major issue is I could not find any document on how to upgrade infinispan in jboss 6. I need the query module which is only really avaliable (not a tech preview) in 5.0. But I tried many ways to upgrade to 5.0.0CR1, CR2 and CR3. On single node it was fine, but as soon as I have another node joined, I got the unmashalling error on NodeKey as described above. I know it must be some jboss jars need also upgrade, but what and how to get them ( I used Tattletale and it seems that jboss-marshalling-river jar not called at all).

                 

                I know that infinispan is an award winning software. So, I guess it could be my fault. But I have read the documentation (not many there) many times and I have spent lots of time trying different configuration.

                 

                Unless you could help me here (answering questions or provides more documentation/links), I would not waste more time on this software.

                • 5. Why I got org.infinispan.tree.NodeKey serialization error in 5.0.0.cr when using distributed cache?
                  yangju

                  Here is the stacktrace on "unable to obtain lock on programatically created cache":

                  14:43:24,421 WARN  [com.pearson.epen.tools.getWork.GetWorkThread] call() got exception:: org.infinispan.CacheException: Unable to acquire lock on cache with name ___defaultcache

                      at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:441) [:4.2.1.FINAL]

                      at org.jboss.ha.ispn.DefaultCacheContainer$SingletonCacheManager.getCache(DefaultCacheContainer.java:296) [:1.0.0.Final]

                      at org.jboss.ha.ispn.DefaultCacheContainer.getCache(DefaultCacheContainer.java:123) [:1.0.0.Final]

                      at org.jboss.ha.ispn.DefaultCacheContainer.getCache(DefaultCacheContainer.java:71) [:1.0.0.Final]

                      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [:1.6.0_24]

                      at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) [:1.6.0_24]

                      at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) [:1.6.0_24]

                      at java.lang.reflect.Method.invoke(Unknown Source) [:1.6.0_24]

                      at org.jboss.weld.bean.proxy.AbstractBeanInstance.invoke(AbstractBeanInstance.java:48) [:6.0.0.Final]

                      at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:125) [:6.0.0.Final]

                      at org.jboss.weld.proxies.org$jboss$weld$beanProducerMethod-com$pearson$epen$cdi$producer$InfinispanCacheManagerProducermethod_findCacheManager(com$pearson$epen$tools$JNDILookUpUtil,_org$slf4j$Logger)_$$_WeldClientProxy.getCache(org$jboss$weld$beanProducerMethod-com$pearson$epen$cdi$producer$InfinispanCacheManagerProducermethod_findCacheManager(com$pearson$epen$tools$JNDILookUpUtil,_org$slf4j$Logger)_$$_WeldClientProxy.java)

                      at com.pearson.epen.impl.service.cache.EpenInfinispanCacheServiceImpl.findCacheByName(EpenInfinispanCacheServiceImpl.java:87) [:]

                      at com.pearson.epen.impl.service.cache.EpenInfinispanCacheServiceImpl.findCacheByName(EpenInfinispanCacheServiceImpl.java:25) [:]

                      at com.pearson.epen.impl.service.cache.AbstractEpenAppCacheServiceImpl.findCacheByName(AbstractEpenAppCacheServiceImpl.java:49) [:]

                      at com.pearson.epen.impl.service.ResponseServiceImpl.getNextResponseRecord(ResponseServiceImpl.java:60) [:]

                      at com.pearson.epen.impl.service.ResponseServiceImpl.fetchNextResponse(ResponseServiceImpl.java:76) [:]

                      at com.pearson.epen.tools.getWork.GetWorkThread.call(GetWorkThread.java:50) [:]

                      at com.pearson.epen.tools.getWork.GetWorkThread.call(GetWorkThread.java:20) [:]

                      at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) [:1.6.0_24]

                      at java.util.concurrent.FutureTask.run(Unknown Source) [:1.6.0_24]

                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) [:1.6.0_24]

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [:1.6.0_24]

                      at java.lang.Thread.run(Unknown Source) [:1.6.0_24]

                   

                  Eventually:

                   

                   

                  used by: org.infinispan.util.concurrent.TimeoutException: Timed out after 120 seconds waiting for a response from xxxxxxxxxxx:1099

                      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:274) [:4.2.1.FINAL]

                      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:116) [:4.2.1.FINAL]

                      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:407) [:4.2.1.FINAL]

                      at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:102) [:4.2.1.FINAL]

                      at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:126) [:4.2.1.FINAL]

                      at org.infinispan.distribution.RehashTask$StateGrabber.call(RehashTask.java:139) [:4.2.1.FINAL]

                      at org.infinispan.distribution.RehashTask$StateGrabber.call(RehashTask.java:125) [:4.2.1.FINAL]

                      ... 5 more

                  • 6. Re: Why I got org.infinispan.tree.NodeKey serialization error in 5.0.0.cr when using distributed cache?
                    galder.zamarreno

                    First of all, Sanne has said it and I've said it in another thread too: You mix far too many questions/issues/worries/problems into a single thread which makes it very hard for us to reply to them. So, try to rationale your posts and we'll be able to help more easily. Let me answer one here:

                    yangju wrote:

                     

                    The most annoying problem is that programmatically creating cache does not alway work as expected. If I call getCache(name) if the name does not exist, the cache will be created. This is fine. But the problem occurs when the node which have had the cache tries to lock an entry on the other node which does not yet have this cache. It hangs.

                     

                    You have issues in this case cos asymmetric clusters are not supported (https://issues.jboss.org/browse/ISPN-658). If you create caches programmatically, do it at startup time so that when cache operations come through, all nodes in the cluster have these caches defines. In fact, you have even made a comment on https://issues.jboss.org/browse/ISPN-658 - so you're aware of the issue, and it's quite clear from the JIRA itself that this is not a simple problem to solve in our current architechture and there're clear, stablished workarounds for this type of scenarions.

                    • 7. Re: Why I got org.infinispan.tree.NodeKey serialization error in 5.0.0.cr when using distributed cache?
                      galder.zamarreno

                      For example, things to ask yourself are:

                       

                      Why do I need separate caches?

                       

                      If this is due to different configuration requirements, maybe you can have a set of pre-defined caches for the different config settings and distribute them to your clients accordingly.

                       

                      If it's to separate data from different users, you could tailor your keys to allow separation from different user data...etc.