1 2 3 Previous Next 30 Replies Latest reply on Aug 11, 2010 11:53 AM by manik Go to original post
      • 15. Re: Lock is not released on remote node after transaction commit
        manik

        It looks like your StringKey type cannot be marshalled by our marshalling layer.

         

         

         

        Caused by: org.infinispan.CacheException: Unable to marshall value TestKey{key=key2}
        at org.infinispan.marshall.MarshalledValue.serialize(MarshalledValue.java:101)
        at org.infinispan.marshall.MarshalledValue.getRaw(MarshalledValue.java:157)
        at org.infinispan.marshall.MarshalledValue$Externalizer.writeObject(MarshalledValue.java:244)
        at org.infinispan.marshall.jboss.ConstantObjectTable$ExternalizerAdapter.writeObject(ConstantObjectTable.java:271)
        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:147)
        at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:423)
        at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeObject(ReplicableCommandExternalizer.java:61)
        at org.infinispan.marshall.jboss.ConstantObjectTable$ExternalizerAdapter.writeObject(ConstantObjectTable.java:271)
        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:147)
        at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:423)
        at org.infinispan.marshall.jboss.GenericJBossMarshaller.objectToObjectStream(GenericJBossMarshaller.java:100)
        at org.infinispan.marshall.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:92)
        at org.infinispan.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:31)
        at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:22)
        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.marshallCall(CommandAwareRpcDispatcher.java:241)
        ... 39 more
        Caused by: java.io.InvalidObjectException: Unexpected illegal access exception
        at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
        at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1094)
        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:964)
        at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:423)
        at org.infinispan.marshall.jboss.GenericJBossMarshaller.objectToObjectStream(GenericJBossMarshaller.java:100)
        at org.infinispan.marshall.VersionAwareMarshaller.objectToObjectStream(VersionAwareMarshaller.java:146)
        at org.infinispan.marshall.MarshalledValue.serialize(MarshalledValue.java:92)
        ... 53 more
        Caused by: java.lang.IllegalAccessException: Class org.jboss.marshalling.river.RiverMarshaller can not access a member of class test.StringKey with modifiers "private"
        at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:65)

         

         

        Is this the same StringKey implementation you sent me earlier?

        • 16. Re: Lock is not released on remote node after transaction commit
          galder.zamarreno

          Re: marshall exception

           

          Manik, this has happened before, in fact a couple of days back. I saw it in hudson just a couple of days ago (hence why I was after sun.reflect.* classes on a Mac ):

           

          http://hudson.infinispan.org/hudson/job/Infinispan-trunk-JDK6-tcp/org.infinispan$infinispan-lucene-directory/56/testReport/junit/org.infinispan.lucene.locking/TransactionalLockManagerFunctionalTest/org_infinispan_lucene_locking_TransactionalLockManagerFunctionalTest_testLuceneIndexLocking/

           

          What JDK version, provider and plattform are you using?

          • 17. Re: Lock is not released on remote node after transaction commit
            galder.zamarreno

            Re: NPE

             

            java.lang.NullPointerException
                 at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
                 at org.infinispan.util.concurrent.locks.containers.OwnableReentrantPerEntryLockContainer.getLockFromMap(OwnableReentrantPerEntryLockContainer.java:38)
                 at org.infinispan.util.concurrent.locks.containers.OwnableReentrantPerEntryLockContainer.ownsLock(OwnableReentrantPerEntryLockContainer.java:28)
                 at org.infinispan.util.concurrent.locks.LockManagerImpl.ownsLock(LockManagerImpl.java:123)
                 at org.infinispan.util.concurrent.locks.DeadlockDetectingLockManager.localVsLocalDld(DeadlockDetectingLockManager.java:94)
                 at org.infinispan.util.concurrent.locks.DeadlockDetectingLockManager.lockAndRecord(DeadlockDetectingLockManager.java:78)
                 at org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:205)
            

             

            Hmmmm, seems like your keys are null? What version are you using? If the key was null and you're using a version CR1 or higher, the exception should be different (https://jira.jboss.org/browse/ISPN-520)

            • 18. Re: Lock is not released on remote node after transaction commit
              manik

              The NPE is probably the result of the marshalling issue.  Inability to marshall key, hence a nul key on the remote end.  That's what I suspect anyway, I don't have full logs.

              • 19. Re: Lock is not released on remote node after transaction commit
                pengyan

                the snapshot version, as the output I listed for my maven repository.

                Also those two exception happens just occasionally.

                • 20. Re: Lock is not released on remote node after transaction commit
                  manik

                  So the marshalling issue was a problem in JBoss Marshalling.  Reported as JBMAR-111 and already fixed in trunk.  Expect a new snapshot to be out with this fix soon.

                  • 21. Re: Lock is not released on remote node after transaction commit
                    pengyan

                    A bit confused... Is it a new snapshot for jboss marshalling or infinispan?

                    How could I configure my pom file to make it work?

                    • 22. Re: Lock is not released on remote node after transaction commit
                      galder.zamarreno

                      Expect a new snapshot of Infinispan. We'll let you know when ready.

                      • 23. Re: Lock is not released on remote node after transaction commit
                        galder.zamarreno

                        A new Infinispan snapshot has been uploaded which contains the fix for marshalling issue. For your own tracking, the snapshot is: 4.1.0-20100806.082729

                        • 24. Re: Lock is not released on remote node after transaction commit
                          manik

                          Thanks for releasing this, Galder.

                           

                          @Chun Li, could you please give this a try and let us know if this works for you.

                          • 25. Re: Lock is not released on remote node after transaction commit
                            pengyan

                            I could still see occasionally there's NullPointerException. Following is the stack trace.

                             

                            I can be sure that the key is not null.

                             

                            and line 103 for GenericInfinispanCache.java is

                              cache.getAdvancedCache().lock(key);

                             

                            and line 210 ~ 211 for ChunkAllocatorCache.java is

                             

                              ChunkKey<E> chunkKey = new ChunkKey<E>(key, chunkIdProvider.generateChunkId());
                              Integer availableResource = cache.findForUpdate(chunkKey);

                             

                            So I can be sure the key passed to lock method would never be null. Also I'm using infinispan-core-4.1.0-20100806.082729-7.jar.

                             

                             

                             

                            2010-08-09 05:36:58,360 ERROR [org.infinispan.interceptors.InvocationContextInterceptor](pool-184-thread-8) Execution error:
                            java.lang.NullPointerException
                                    at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
                                    at org.infinispan.util.concurrent.locks.containers.OwnableReentrantPerEntryLockContainer.getLockFromMap(OwnableReentrantPerEntryLockContainer.java:38)
                                    at org.infinispan.util.concurrent.locks.containers.OwnableReentrantPerEntryLockContainer.ownsLock(OwnableReentrantPerEntryLockContainer.java:28)
                                    at org.infinispan.util.concurrent.locks.LockManagerImpl.ownsLock(LockManagerImpl.java:123)
                                    at org.infinispan.util.concurrent.locks.DeadlockDetectingLockManager.localVsLocalDld(DeadlockDetectingLockManager.java:94)
                                    at org.infinispan.util.concurrent.locks.DeadlockDetectingLockManager.lockAndRecord(DeadlockDetectingLockManager.java:78)
                                    at org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:205)
                                    at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:148)
                                    at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:106)
                                    at org.infinispan.interceptors.LockingInterceptor.visitLockControlCommand(LockingInterceptor.java:146)
                                    at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:136)
                                    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
                                    at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
                                    at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:147)
                                    at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:136)
                                    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
                                    at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
                                    at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:147)
                                    at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:136)
                                    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
                                    at org.infinispan.interceptors.MarshalledValueInterceptor.visitLockControlCommand(MarshalledValueInterceptor.java:92)
                                    at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:136)
                                    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
                                    at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:171)
                                    at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:115)
                                    at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:136)
                                    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
                                    at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:58)
                                    at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:39)
                                    at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:147)
                                    at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:136)
                                    at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:271)
                                    at org.infinispan.CacheDelegate.lock(CacheDelegate.java:296)
                                    at org.infinispan.CacheDelegate.lock(CacheDelegate.java:289)
                                    at test.p.common.persistence.cache.GenericInfinispanCache.findForUpdate(GenericInfinispanCache.java:103)
                                    at test.p.common.persistence.cache.advanced.ChunkAllocatorCache.release(ChunkAllocatorCache.java:211)
                                    at test.p.odrm.resmanagement.cache.SOPServiceImpl.releaseBandwidth(SOPServiceImpl.java:41)
                                    at sun.reflect.GeneratedMethodAccessor611.invoke(Unknown Source)
                                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                    at java.lang.reflect.Method.invoke(Method.java:597)
                                    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
                                    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
                                    at $Proxy289.releaseBandwidth(Unknown Source)
                                    at test.p.odrm.resmanagement.ResourcePersistorImpl.releaseSessionResources(ResourcePersistorImpl.java:63)

                            • 26. Re: Lock is not released on remote node after transaction commit
                              manik

                              This happens occasionally?  Is it under specific load?  Do you have a unit/stress test for this?

                              • 27. Re: Lock is not released on remote node after transaction commit
                                pengyan

                                occasionally.

                                I could reproduce with 5 threads in attached testcase.

                                • 28. Re: Lock is not released on remote node after transaction commit
                                  manik

                                  To help isolate the problem, do you only see the occasional NPE when you use deadlock detection?  Or even without deadlock detection?

                                  • 29. Re: Lock is not released on remote node after transaction commit
                                    manik

                                    Ok, this is definitely a problem with deadlock detection, as I haven't been able to recreate it with deadlock detection disabled.  Only with it enabled.

                                     

                                    I've created a JIRA for this.  ISPN-594.