2 Replies Latest reply on Nov 20, 2018 10:31 AM by thomash

NullpointerException when starting Infinispan

thomash Nov 20, 2018 4:32 AM

I run into this problem at startup with wildfly-9.0.2.Final which contains Infinispan-7.2.3.Final.

Infinispan throws an nullpointerexception when putting someting into the cache:

Caused by: java.lang.NullPointerException
at org.infinispan.interceptors.locking.ClusteringDependentLogic$InvalidationLogic.localNodeIsPrimaryOwner(ClusteringDependentLogic.java:305) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.shouldLock(AbstractLockingInterceptor.java:154) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitPutMapCommand(NonTransactionalLockingInterceptor.java:60) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:55) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.commands.AbstractVisitor.visitPutMapCommand(AbstractVisitor.java:64) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:55) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:317) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:249) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.statetransfer.StateTransferInterceptor.visitPutMapCommand(StateTransferInterceptor.java:113) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:55) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutMapCommand(CacheMgmtInterceptor.java:160) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:55) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.commands.AbstractVisitor.visitPutMapCommand(AbstractVisitor.java:64) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:55) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1617) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.cache.impl.CacheImpl.putAllInternal(CacheImpl.java:1141) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.cache.impl.CacheImpl.putAll(CacheImpl.java:1135) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.cache.impl.CacheImpl.putAll(CacheImpl.java:348) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.cache.impl.AbstractDelegatingCache.putAll(AbstractDelegatingCache.java:302) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]

Before this, there are timeoutexceptions too, when retrieving the cache from the cache-container:

at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:171)
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:850) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:599) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:554) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:424) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:438) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:118)
at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:109)

Caused by: org.infinispan.util.concurrent.TimeoutException: Node 192.168.9.2 timed out
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:248) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:561) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.topology.LocalTopologyManagerImpl.executeOnCoordinator(LocalTopologyManagerImpl.java:518) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:117) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:110) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_181]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) [rt.jar:1.8.0_181]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) [rt.jar:1.8.0_181]
at java.lang.reflect.Method.invoke(Unknown Source) [rt.jar:1.8.0_181]
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
... 159 more
Caused by: org.jgroups.TimeoutException: timeout waiting for response from 192.168.9.2, request: org.jgroups.blocks.UnicastRequest@7ba98b59, req_id=3, mode=GET_ALL, target=192.168.9.2
at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:427) [jgroups-3.6.4.Final.jar:3.6.4.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:433) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:241) [infinispan-core-7.2.3.Final.jar:7.2.3.Final]
... 168 more

This never resolves automatically, I need to restart the complete server, and typically it just works then.

This happens when I start multiple nodes at the same time. I also face other exceptions which look a bit similar, but not entierly (cfr.https://developer.jboss.org/message/986508#986508)

The common part with both cases is that this happens when multiple nodes are starting at the same time.

What is causing this NPE, is there any way to prevent it and how come it never resolves?

This is the configuration

<cache-container name="interserver" default-cache="default" jndi-name="java:jboss/infinispan/interserver" >
<transport stack="udp-interserver" lock-timeout="60000" />
<replicated-cache name="default" mode="SYNC">
<locking isolation="READ_COMMITTED"/>
<transaction mode="BATCH"/>
<state-transfer enabled="false"/>
</replicated-cache>
<replicated-cache name="myCache"  mode="ASYNC">
<locking isolation="READ_COMMITTED"/>
<transaction mode="BATCH"/>
<state-transfer enabled="false"/>                    
</replicated-cache>
</cache-container>

Thanks for any info.

1. Re: NullpointerException when starting Infinispan

galder.zamarreno Nov 20, 2018 10:10 AM (in response to thomash)

Try a more recent combination? Also, you should use your own Infinispan libraries in your deployment, with your own config, rather than tap into what Wildfly provides out of the box.
Actions
2. Re: NullpointerException when starting Infinispan

thomash Nov 20, 2018 10:31 AM (in response to galder.zamarreno)

Thanks for the input, maybe I'll do that eventually. But first I need to understand why this happens. Is there a reason to assume it won't happen with another version of infinispan?
This is by the way not the default config that ships with wildfly. The default replicated cache is configured

Generally speaking, the cache works fine. It is only at startup (when the cluster is started entirely) that I see this. Starting a few nodes at a time, in a more sequential manner works fine too.
The problem I have is that it is not predictable or in which order the nodes will be started and how many at the same time. It should be robust in any combination.
Actions

Go to original post