Thanks for such a detailed response. We also can't afford a file-system as of now. We took the option 1 and made the changes as you suggested.
Performed a load test and still see the exceptions coming. I have also seen couple of other exceptions in addition to the "Unable to acquire lock"exceptions. Here they are
There are only two nodes in the cluster being load tested, each running on 8GB memory. I am also attaching the latest configuration with which the load test was ran. Please help.
01:22:39,777 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (transport-thread--p2-t24) ISPN000136: Execution error: org.infinispan.util.concurrent.Tim
eoutException: Node lxomavmtceap615-33617 timed out
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:174) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:536) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:290) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.query.indexmanager.RemoteIndexingBackend.sendCommand(RemoteIndexingBackend.java:116) [infinispan-query.jar:7.0.0.Final]
at org.infinispan.query.indexmanager.RemoteIndexingBackend.applyWork(RemoteIndexingBackend.java:64) [infinispan-query.jar:7.0.0.Final]
at org.infinispan.query.indexmanager.InfinispanBackendQueueProcessor.applyWork(InfinispanBackendQueueProcessor.java:80) [infinispan-query.jar:7.0.0.Final]
at org.hibernate.search.indexes.impl.DirectoryBasedIndexManager.performOperations(DirectoryBasedIndexManager.java:113) [hibernate-search-engine-5.0.0.Beta1.jar:
5.0.0.Beta1]
at org.hibernate.search.backend.impl.WorkQueuePerIndexSplitter.commitOperations(WorkQueuePerIndexSplitter.java:49) [hibernate-search-engine-5.0.0.Beta1.jar:5.0.
0.Beta1]
at org.hibernate.search.backend.impl.BatchedQueueingProcessor.performWorks(BatchedQueueingProcessor.java:82) [hibernate-search-engine-5.0.0.Beta1.jar:5.0.0.Beta
1]
at org.hibernate.search.backend.impl.TransactionalWorker.performWork(TransactionalWorker.java:86) [hibernate-search-engine-5.0.0.Beta1.jar:5.0.0.Beta1]
at org.infinispan.query.backend.QueryInterceptor.performSearchWorks(QueryInterceptor.java:235) [infinispan-query.jar:7.0.0.Final]
at org.infinispan.query.backend.QueryInterceptor.performSearchWork(QueryInterceptor.java:229) [infinispan-query.jar:7.0.0.Final]
at org.infinispan.query.backend.QueryInterceptor.updateIndexes(QueryInterceptor.java:223) [infinispan-query.jar:7.0.0.Final]
at org.infinispan.query.backend.QueryInterceptor.processPutMapCommand(QueryInterceptor.java:418) [infinispan-query.jar:7.0.0.Final]
at org.infinispan.query.backend.QueryInterceptor.visitPutMapCommand(QueryInterceptor.java:186) [infinispan-query.jar:7.0.0.Final]
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:47) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitPutMapCommand(NonTransactionalLockingInterceptor.java:67) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:47) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.commands.AbstractVisitor.visitPutMapCommand(AbstractVisitor.java:55) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:47) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.commands.AbstractVisitor.visitPutMapCommand(AbstractVisitor.java:55) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:47) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:172) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.statetransfer.StateTransferInterceptor.visitPutMapCommand(StateTransferInterceptor.java:100) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:47) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutMapCommand(CacheMgmtInterceptor.java:117) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:47) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.commands.AbstractVisitor.visitPutMapCommand(AbstractVisitor.java:55) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:47) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1576) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.cache.impl.CacheImpl.putAllInternal(CacheImpl.java:1098) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.cache.impl.CacheImpl.access$300(CacheImpl.java:121) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.cache.impl.CacheImpl$3.call(CacheImpl.java:1227) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.cache.impl.CacheImpl$3.call(CacheImpl.java:1222) [infinispan-core.jar:7.0.0.Final]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_65]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65]
Caused by: org.jgroups.TimeoutException: timeout waiting for response from lxomavmtceap615-33617, request: org.jgroups.blocks.UnicastRequest@1e1ece2f, req_id=524945, mode=GET_ALL, target=lxomavmtceap615-33617
at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:429) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:372) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:167) [infinispan-core.jar:7.0.0.Final]
... 48 more
01:56:14,089 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (Incoming-1,lxomavmtceap615-34710) ISPN000136: Execution error: java.lang.IllegalStateExce
ption: Could not get property value
at org.hibernate.search.util.impl.ReflectionHelper.getMemberValue(ReflectionHelper.java:82) [hibernate-search-engine-5.0.0.Beta1.jar:5.0.0.Beta1]
at org.hibernate.search.engine.spi.DocumentBuilderIndexedEntity.buildDocumentFields(DocumentBuilderIndexedEntity.java:409) [hibernate-search-engine-5.0.0.Beta1.
jar:5.0.0.Beta1]
at org.hibernate.search.engine.spi.DocumentBuilderIndexedEntity.buildDocumentFields(DocumentBuilderIndexedEntity.java:467) [hibernate-search-engine-5.0.0.Beta1.
jar:5.0.0.Beta1]
at org.hibernate.search.engine.spi.DocumentBuilderIndexedEntity.getDocument(DocumentBuilderIndexedEntity.java:359) [hibernate-search-engine-5.0.0.Beta1.jar:5.0.
0.Beta1]
at org.hibernate.search.engine.spi.DocumentBuilderIndexedEntity.createUpdateWork(DocumentBuilderIndexedEntity.java:288) [hibernate-search-engine-5.0.0.Beta1.jar
:5.0.0.Beta1]
at org.hibernate.search.engine.spi.DocumentBuilderIndexedEntity.addWorkToQueue(DocumentBuilderIndexedEntity.java:230) [hibernate-search-engine-5.0.0.Beta1.jar:5
.0.0.Beta1]
at org.hibernate.search.engine.impl.WorkPlan$PerEntityWork.enqueueLuceneWork(WorkPlan.java:486) [hibernate-search-engine-5.0.0.Beta1.jar:5.0.0.Beta1]
at org.hibernate.search.engine.impl.WorkPlan$PerClassWork.enqueueLuceneWork(WorkPlan.java:261) [hibernate-search-engine-5.0.0.Beta1.jar:5.0.0.Beta1]
at org.hibernate.search.engine.impl.WorkPlan.getPlannedLuceneWork(WorkPlan.java:147) [hibernate-search-engine-5.0.0.Beta1.jar:5.0.0.Beta1]
at org.hibernate.search.backend.impl.WorkQueue.prepareWorkPlan(WorkQueue.java:114) [hibernate-search-engine-5.0.0.Beta1.jar:5.0.0.Beta1]
at org.hibernate.search.backend.impl.BatchedQueueingProcessor.prepareWorks(BatchedQueueingProcessor.java:56) [hibernate-search-engine-5.0.0.Beta1.jar:5.0.0.Beta
1]
at org.hibernate.search.backend.impl.TransactionalWorker.performWork(TransactionalWorker.java:85) [hibernate-search-engine-5.0.0.Beta1.jar:5.0.0.Beta1]
at org.infinispan.query.backend.QueryInterceptor.performSearchWorks(QueryInterceptor.java:235) [infinispan-query.jar:7.0.0.Final]
at org.infinispan.query.backend.QueryInterceptor.performSearchWork(QueryInterceptor.java:229) [infinispan-query.jar:7.0.0.Final]
at org.infinispan.query.backend.QueryInterceptor.updateIndexes(QueryInterceptor.java:223) [infinispan-query.jar:7.0.0.Final]
at org.infinispan.query.backend.QueryInterceptor.processPutMapCommand(QueryInterceptor.java:418) [infinispan-query.jar:7.0.0.Final]
at org.infinispan.query.backend.QueryInterceptor.visitPutMapCommand(QueryInterceptor.java:186) [infinispan-query.jar:7.0.0.Final]
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:47) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitPutMapCommand(NonTransactionalLockingInterceptor.java:67) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:47) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.commands.AbstractVisitor.visitPutMapCommand(AbstractVisitor.java:55) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:47) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.commands.AbstractVisitor.visitPutMapCommand(AbstractVisitor.java:55) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:47) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:166) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.statetransfer.StateTransferInterceptor.visitPutMapCommand(StateTransferInterceptor.java:100) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:47) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutMapCommand(CacheMgmtInterceptor.java:117) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:47) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.commands.AbstractVisitor.visitPutMapCommand(AbstractVisitor.java:55) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:47) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:39) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:97) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:218) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:86) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:267) [infinispan-core.jar:7.0.0.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:211) [infinispan-core.jar:7.0.0.Final]
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:677) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.JChannel.up(JChannel.java:733) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.protocols.pbcast.StreamingStateTransfer.up(StreamingStateTransfer.java:231) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.protocols.RSVP.up(RSVP.java:201) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:505) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.protocols.FRAG2.up(FRAG2.java:182) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.protocols.FlowControl.up(FlowControl.java:447) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.protocols.FlowControl.up(FlowControl.java:447) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3.java:1087) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.protocols.UNICAST3.removeAndDeliver(UNICAST3.java:886) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.protocols.UNICAST3.handleBatchReceived(UNICAST3.java:867) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:517) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:674) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:213) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.protocols.TP.passBatchUp(TP.java:1605) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at org.jgroups.protocols.TP$BatchHandler.run(TP.java:1855) [jgroups-3.6.0.Final.jar:3.6.0.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_65]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65]
Caused by: java.lang.IllegalArgumentException: Invoking locationId on a null object
at org.hibernate.annotations.common.reflection.java.JavaXProperty.invoke(JavaXProperty.java:81) [hibernate-commons-annotations-4.0.1.Final.jar:4.0.1.Final]
at org.hibernate.search.util.impl.ReflectionHelper.getMemberValue(ReflectionHelper.java:79) [hibernate-search-engine-5.0.0.Beta1.jar:5.0.0.Beta1]
... 77 more
Caused by: java.lang.NullPointerException
at sun.reflect.GeneratedMethodAccessor384.invoke(Unknown Source) [:1.7.0_65]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65]
at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65]
at org.hibernate.annotations.common.reflection.java.JavaXProperty.invoke(JavaXProperty.java:74) [hibernate-commons-annotations-4.0.1.Final.jar:4.0.1.Final]
... 78 more