Unable to acquire lock exception| JBoss.org Content Archive (Read Only)

15. Re: Re: Unable to acquire lock exception

sannegrinovero Jan 28, 2015 6:34 AM (in response to jeetu)

You're having timeouts, which are caused either by some deadlock in your application or in our code - but we know of only one possible deadlock in Infinispan as I suggested above, and you can avoid it by tuning.

The other reason for which you could have timeouts is because it's all too slow and inefficient for the load you're putting on the system - which is also solvable by some tuning.

So you need to fix the overall efficiency. Upgrading version is a very reliable way to do that, since we just improved efficiency for this very same kind of configurations. Improvements were very significant (as in higher than two orders of magnitude in our tests), but I can't tell if it's enough for what you're doing.

And now you're posting a stacktrace with a NullPointerException on your model. That's also something you'll need to fix: if your system is going to throw thousands of exceptions per second, performance will be horrible and you'll have a good reason to expect Timeouts.

16. Re: Unable to acquire lock exception

jeetu Jan 28, 2015 7:46 AM (in response to sannegrinovero)

Thanks for the response Sanne.

Tuning is exactly the place where we need help with. We are also analyzing our application for any deadlock situations and dealing with it.

Our caching solution is proactive jobs loading the distributed indexed cache with inventory data obtained from a webservice interface and then query the cache. The loading of data is tracked by a lookup table (a distributed cache with keys and values) based framework that will record the loading progress and based on it (STARTED OR COMPLETED), the application will query the cache only if the loading is complete for a certain enterprise data. All of this is driven through ejb methods being called by jobs set through timerejb.

What i observed is that with the infinispan configuration that i have uploaded today, the putAll (sync) is taking quite a long time to complete. In this case around 2 hrs for 10k objects. Our custom framework that proactively loads things into cache tried to putAll the same records again (framework is written such that, it looks at the load status and if the load status is in STARTED state for too long, it tries to load things again), which is when we have the "Unable to aquire Lock" exceptions are being thrown.

Any things that you think can be tuned from the latest configuration, that can speed up the putAll??

Regards

Jithendra

17. Re: Unable to acquire lock exception

sannegrinovero Jan 28, 2015 8:21 AM (in response to jeetu)

I'm sorry I'd like to help but the problem is too vague. I can't guess what's wrong with your system, you need to at least be able to narrow down the problem.. for example verify if you have a network latency problem, that would need some platform level tuning (plenty of kernel options) or JGroups parameter tuning. Or you might have excessive locks in the JVM, in which case you'd need to identify those, and narrow down to a specific issue. Or you might have a heap configuration issue... Too many options for me guess from the stacktrace.

If you were able to describe what exactly is your problem, then we could suggest appropriate configuration changes.

The only thing I can say for sure is that you need to avoid those NullPointerException(s) and upgrade Infinispan.

18. Re: Unable to acquire lock exception

rvansa Jan 28, 2015 11:10 AM (in response to jeetu)

Haven't followed the full discussion, but if you're using non-transactional cache, I would strongly discourage use of putAll - the implementation is simply inefficient, resulting in quadratic (to number of nodes) amount of commands in the system. See https://www.mail-archive.com/infinispan-dev@lists.jboss.org/msg06008.html for closer description - nothing has improved since then. I'd recommend to use couple of putAsync calls instead.

19. Re: Unable to acquire lock exception

jeetu Jan 30, 2015 8:03 AM (in response to rvansa)

Thanks Radim for your input.

I read through your mail archive on the putAll performance. Is there a JIRA that i can follow to know in future, when this is resolved. This is exactly what we are doing in our app.

We use a non-transactional cache and we put a bunch of keys ( upto 30k at a time), which is taking forever to complete (though we have only two nodes in the distributed cluster).

You also said as a work around, to use couple of putAsyncs. How do you think, you propose a work around for our case?? create chunks of items and do multiple putAllAsynchs?? Please suggest.

Regards

Jithendra

20. Re: Unable to acquire lock exception

rvansa Jan 30, 2015 9:35 AM (in response to jeetu)

30k is really a lot. What I was suggesting is to execute cache.putAsync() commands, or cache.put() from threadpool - no putAll/putAllAsync (those differ only in the thread that is waiting). The system can handle usually a couple of hundred parallel operations per node.

Another workaround would be to find manually who are the owners (Infinispan exposes this through DistributionManager) and use DistributedExecutors to send those chunks to be inserted on given nodes manually with LOCAL_ONLY flag. However, this is not a straightforward solution (especially it's not resilient against topology change during the loading).

21. Re: Unable to acquire lock exception

jeetu Jan 31, 2015 1:59 AM (in response to rvansa)

We are going ahead with your first suggestion. I will let you know of the outcome. Thanks a lot for your response.

Regards

Jithendra

22. Re: Unable to acquire lock exception

jeetu Feb 5, 2015 7:56 AM (in response to rvansa)

Hi Radim,

With your suggestion, i created a thread pool with 35 threads and did a cache.put of the items. With this, i see a lot of these exceptions on our e2e environment. I am also attaching the infinispan-config and jgroups-tcp configs for your reference. Please let us know, what's happening. We are in a very critical state. Please help.

04:28:59,028 ERROR [stderr] (pool-23-thread-67) Exception in thread "pool-23-thread-67" org.infinispan.util.concurrent.TimeoutException: Node lxomavmtceap615-46391 time

d out

04:28:59,028 ERROR [stderr] (pool-23-thread-67) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispat

cher.java:174)

04:28:59,028 ERROR [stderr] (pool-23-thread-67) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:536)

04:28:59,028 ERROR [stderr] (pool-23-thread-67) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:290)

04:28:59,028 ERROR [stderr] (pool-23-thread-67) at org.infinispan.query.indexmanager.RemoteIndexingBackend.sendCommand(RemoteIndexingBackend.java:116)

04:28:59,028 ERROR [stderr] (pool-23-thread-67) at org.infinispan.query.indexmanager.RemoteIndexingBackend.applyWork(RemoteIndexingBackend.java:64)

04:28:59,029 ERROR [stderr] (pool-23-thread-67) at org.infinispan.query.indexmanager.InfinispanBackendQueueProcessor.applyWork(InfinispanBackendQueueProcessor.j

ava:80)

04:28:59,029 ERROR [stderr] (pool-23-thread-67) at org.hibernate.search.indexes.impl.DirectoryBasedIndexManager.performOperations(DirectoryBasedIndexManager.jav

a:113)

04:28:59,029 ERROR [stderr] (pool-23-thread-67) at org.hibernate.search.backend.impl.WorkQueuePerIndexSplitter.commitOperations(WorkQueuePerIndexSplitter.java:4

9)

04:28:59,029 ERROR [stderr] (pool-23-thread-67) at org.hibernate.search.backend.impl.BatchedQueueingProcessor.performWorks(BatchedQueueingProcessor.java:82)

04:28:59,029 ERROR [stderr] (pool-23-thread-67) at org.hibernate.search.backend.impl.TransactionalWorker.performWork(TransactionalWorker.java:86)

04:28:59,029 ERROR [stderr] (pool-23-thread-67) at org.infinispan.query.backend.QueryInterceptor.performSearchWorks(QueryInterceptor.java:235)

04:28:59,029 ERROR [stderr] (pool-23-thread-67) at org.infinispan.query.backend.QueryInterceptor.performSearchWork(QueryInterceptor.java:229)

04:28:59,029 ERROR [stderr] (pool-23-thread-67) at org.infinispan.query.backend.QueryInterceptor.updateIndexes(QueryInterceptor.java:223)

04:28:59,029 ERROR [stderr] (pool-23-thread-67) at org.infinispan.query.backend.QueryInterceptor.processPutKeyValueCommand(QueryInterceptor.java:446)

04:28:59,029 ERROR [stderr] (pool-23-thread-67) at org.infinispan.query.backend.QueryInterceptor.visitPutKeyValueCommand(QueryInterceptor.java:164)