Problems with defining node types in clustered environment

[rickcole3]

We are having a problem running Modeshape in a clustered environment. The issue arises when we attempt to define custom node types (via a .cnd file) in our .json repository configuration file. The first instance of Modeshape comes up fine. The problem is when the second instance comes up. We get a stack trace containing an NPE on both instances. When we remove the reference to the .cnd file from the repository .json file, everything starts up properly.

This may to be related to MODE-1703, which appears to have been fixed back about 24 days ago, but we are still seeing a problem . Any help on this would be greatly appreciated.

Here is the stack trace we are seeing on the first server we started:

java.lang.NullPointerException

          at org.infinispan.schematic.internal.delta.Operation.mutableParent(Operation.java:66)

          at org.infinispan.schematic.internal.delta.PutOperation.replay(PutOperation.java:88)

          at org.infinispan.schematic.internal.SchematicEntryDelta.merge(SchematicEntryDelta.java:62)

          at org.infinispan.commands.write.PutKeyValueCommand.perform(PutKeyValueCommand.java:94)

          at org.infinispan.interceptors.CallInterceptor.handleDefault(CallInterceptor.java:83)

          at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:61)

          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

          at org.infinispan.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:178)

          at org.infinispan.interceptors.ReplicationInterceptor.visitPutKeyValueCommand(ReplicationInterceptor.java:149)

          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

          at org.infinispan.interceptors.CacheStoreInterceptor.visitPutKeyValueCommand(CacheStoreInterceptor.java:209)

          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

          at org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:89)

          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

          at org.infinispan.interceptors.EntryWrappingInterceptor$EntryWrappingVisitor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:245)

          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

          at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:77)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:120)

          at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:105)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

          at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

          at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:106)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

          at org.infinispan.interceptors.StateTransferLockInterceptor.handleWithRetries(StateTransferLockInterceptor.java:207)

          at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:84)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:130)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:89)

          at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345)

          at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:127)

          at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:127)

          at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:136)

          at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithRetry(InboundInvocationHandlerImpl.java:162)

          at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:114)

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:217)

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:197)

          at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456)

          at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363)

          at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238)

          at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)

          at org.jgroups.JChannel.up(JChannel.java:716)

          at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)

          at org.jgroups.protocols.FRAG2.unfragment(FRAG2.java:302)

          at org.jgroups.protocols.FRAG2.up(FRAG2.java:162)

          at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)

          at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)

          at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)

          at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)

          at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:618)

          at org.jgroups.protocols.UNICAST.up(UNICAST.java:317)

          at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:595)

          at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:167)

          at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)

          at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)

          at org.jgroups.protocols.Discovery.up(Discovery.java:355)

          at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)

          at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)

          at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

          at java.lang.Thread.run(Thread.java:662)

Thanks,

Rick Cole

[hchiorean]

Hi,

Is this NPE the same on both nodes or are there different exceptions ? Also, could you pls post the full server startup log(s) ?

Thanks

[rickcole3]

It show up on both consoles. I have attached both log files:

Rick Cole

[rhauch]

Does this always happen when starting the second instance, or does it happen occasionally? It's obviously a bug, so can you please log a bug in our JIRA, and include all the same information that you mentioned above (including the attachments)?

Thanks in advance!

[rickcole3]

This happens every time at startup. I have even tried adding logic to create the node types programatically right after startup, but the same thing happens when the two instances attempt to sync up. I have created MODE-1733 and attached the two log files along with our repository and infinispan configuration files. Perhaps I have something configured wrong. Thanks!

[rhauch]

Thanks for logging that issue. We'll try to look at it soon and update the issue with what we find. It shouldn't be that difficult to do, so it's probably a bug.

[rickcole3]

I just noticed that I am seeing this whenever I do any type of update to a clustered instance. So, this is just not an issue with the node types. I currently don't have any node types defined in a .cnd file.

[rickcole3]

Randall:

I am curious if there is an ETA on a resolution to this issue? Thanks!

Rick

[rhauch]

I hope to look into it later today or tomorrow.