[manik]

It looks like your StringKey type cannot be marshalled by our marshalling layer.

Caused by: org.infinispan.CacheException: Unable to marshall value TestKey{key=key2}
at org.infinispan.marshall.MarshalledValue.serialize(MarshalledValue.java:101)
at org.infinispan.marshall.MarshalledValue.getRaw(MarshalledValue.java:157)
at org.infinispan.marshall.MarshalledValue$Externalizer.writeObject(MarshalledValue.java:244)
at org.infinispan.marshall.jboss.ConstantObjectTable$ExternalizerAdapter.writeObject(ConstantObjectTable.java:271)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:147)
at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:423)
at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeObject(ReplicableCommandExternalizer.java:61)
at org.infinispan.marshall.jboss.ConstantObjectTable$ExternalizerAdapter.writeObject(ConstantObjectTable.java:271)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:147)
at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:423)
at org.infinispan.marshall.jboss.GenericJBossMarshaller.objectToObjectStream(GenericJBossMarshaller.java:100)
at org.infinispan.marshall.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:92)
at org.infinispan.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:31)
at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:22)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.marshallCall(CommandAwareRpcDispatcher.java:241)
... 39 more
Caused by: java.io.InvalidObjectException: Unexpected illegal access exception
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1094)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:964)
at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:423)
at org.infinispan.marshall.jboss.GenericJBossMarshaller.objectToObjectStream(GenericJBossMarshaller.java:100)
at org.infinispan.marshall.VersionAwareMarshaller.objectToObjectStream(VersionAwareMarshaller.java:146)
at org.infinispan.marshall.MarshalledValue.serialize(MarshalledValue.java:92)
... 53 more
Caused by: java.lang.IllegalAccessException: Class org.jboss.marshalling.river.RiverMarshaller can not access a member of class test.StringKey with modifiers "private"
at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:65)

Is this the same StringKey implementation you sent me earlier?

[galder.zamarreno]

Re: marshall exception

Manik, this has happened before, in fact a couple of days back. I saw it in hudson just a couple of days ago (hence why I was after sun.reflect.* classes on a Mac ):

http://hudson.infinispan.org/hudson/job/Infinispan-trunk-JDK6-tcp/org.infinispan$infinispan-lucene-directory/56/testReport/junit/org.infinispan.lucene.locking/TransactionalLockManagerFunctionalTest/org_infinispan_lucene_locking_TransactionalLockManagerFunctionalTest_testLuceneIndexLocking/

What JDK version, provider and plattform are you using?

[galder.zamarreno]

Re: NPE

java.lang.NullPointerException
     at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
     at org.infinispan.util.concurrent.locks.containers.OwnableReentrantPerEntryLockContainer.getLockFromMap(OwnableReentrantPerEntryLockContainer.java:38)
     at org.infinispan.util.concurrent.locks.containers.OwnableReentrantPerEntryLockContainer.ownsLock(OwnableReentrantPerEntryLockContainer.java:28)
     at org.infinispan.util.concurrent.locks.LockManagerImpl.ownsLock(LockManagerImpl.java:123)
     at org.infinispan.util.concurrent.locks.DeadlockDetectingLockManager.localVsLocalDld(DeadlockDetectingLockManager.java:94)
     at org.infinispan.util.concurrent.locks.DeadlockDetectingLockManager.lockAndRecord(DeadlockDetectingLockManager.java:78)
     at org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:205)

Hmmmm, seems like your keys are null? What version are you using? If the key was null and you're using a version CR1 or higher, the exception should be different (https://jira.jboss.org/browse/ISPN-520)

[manik]

The NPE is probably the result of the marshalling issue.  Inability to marshall key, hence a nul key on the remote end.  That's what I suspect anyway, I don't have full logs.

[pengyan]

the snapshot version, as the output I listed for my maven repository.

Also those two exception happens just occasionally.

[manik]

So the marshalling issue was a problem in JBoss Marshalling.  Reported as JBMAR-111 and already fixed in trunk.  Expect a new snapshot to be out with this fix soon.

[pengyan]

A bit confused... Is it a new snapshot for jboss marshalling or infinispan?

How could I configure my pom file to make it work?

[galder.zamarreno]

Expect a new snapshot of Infinispan. We'll let you know when ready.

[galder.zamarreno]

A new Infinispan snapshot has been uploaded which contains the fix for marshalling issue. For your own tracking, the snapshot is: 4.1.0-20100806.082729

[manik]

Thanks for releasing this, Galder.

@Chun Li, could you please give this a try and let us know if this works for you.

[pengyan]

I could still see occasionally there's NullPointerException. Following is the stack trace.

I can be sure that the key is not null.

and line 103 for GenericInfinispanCache.java is

  cache.getAdvancedCache().lock(key);

and line 210 ~ 211 for ChunkAllocatorCache.java is

  ChunkKey<E> chunkKey = new ChunkKey<E>(key, chunkIdProvider.generateChunkId());
  Integer availableResource = cache.findForUpdate(chunkKey);

So I can be sure the key passed to lock method would never be null. Also I'm using infinispan-core-4.1.0-20100806.082729-7.jar.

2010-08-09 05:36:58,360 ERROR [org.infinispan.interceptors.InvocationContextInterceptor](pool-184-thread-8) Execution error:
java.lang.NullPointerException
        at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
        at org.infinispan.util.concurrent.locks.containers.OwnableReentrantPerEntryLockContainer.getLockFromMap(OwnableReentrantPerEntryLockContainer.java:38)
        at org.infinispan.util.concurrent.locks.containers.OwnableReentrantPerEntryLockContainer.ownsLock(OwnableReentrantPerEntryLockContainer.java:28)
        at org.infinispan.util.concurrent.locks.LockManagerImpl.ownsLock(LockManagerImpl.java:123)
        at org.infinispan.util.concurrent.locks.DeadlockDetectingLockManager.localVsLocalDld(DeadlockDetectingLockManager.java:94)
        at org.infinispan.util.concurrent.locks.DeadlockDetectingLockManager.lockAndRecord(DeadlockDetectingLockManager.java:78)
        at org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:205)
        at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:148)
        at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:106)
        at org.infinispan.interceptors.LockingInterceptor.visitLockControlCommand(LockingInterceptor.java:146)
        at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:136)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
        at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:147)
        at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:136)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
        at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:147)
        at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:136)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
        at org.infinispan.interceptors.MarshalledValueInterceptor.visitLockControlCommand(MarshalledValueInterceptor.java:92)
        at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:136)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
        at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:171)
        at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:115)
        at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:136)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
        at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:58)
        at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:39)
        at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:147)
        at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:136)
        at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:271)
        at org.infinispan.CacheDelegate.lock(CacheDelegate.java:296)
        at org.infinispan.CacheDelegate.lock(CacheDelegate.java:289)
        at test.p.common.persistence.cache.GenericInfinispanCache.findForUpdate(GenericInfinispanCache.java:103)
        at test.p.common.persistence.cache.advanced.ChunkAllocatorCache.release(ChunkAllocatorCache.java:211)
        at test.p.odrm.resmanagement.cache.SOPServiceImpl.releaseBandwidth(SOPServiceImpl.java:41)
        at sun.reflect.GeneratedMethodAccessor611.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
        at $Proxy289.releaseBandwidth(Unknown Source)
        at test.p.odrm.resmanagement.ResourcePersistorImpl.releaseSessionResources(ResourcePersistorImpl.java:63)

[manik]

This happens occasionally?  Is it under specific load?  Do you have a unit/stress test for this?

[pengyan]

occasionally.

I could reproduce with 5 threads in attached testcase.

[manik]

To help isolate the problem, do you only see the occasional NPE when you use deadlock detection?  Or even without deadlock detection?

[manik]

Ok, this is definitely a problem with deadlock detection, as I haven't been able to recreate it with deadlock detection disabled.  Only with it enabled.

I've created a JIRA for this.  ISPN-594.