How to prevent TimeoutException?

[nfilotto]

Hi,

We implemented our own CacheLoader (on JBC v 3.2.0 GA) and after our first test with a lot of concurrent writes (100 Threads), we got several TimeoutException (timeout as been set to 10 sec) which probably means that we have deadlocks. So my question is the following What are the best practices to prevent TimeoutException/deadlocks?

NB: I already increased the concurrency level, we got much less TimeoutException but we still get some.

Thank you in advance,
BR,
Nicolas Filotto

[galder.zamarreno]

As indicated in the documentation, JBoss Cache uses lock stripping by default and you can control the shared pool of locks by tweaking the concurrency level, as you already did. However, lock striping, under certain circumstances, can lead to deadlocks as indicated in https://jira.jboss.org/jira/browse/JBCACHE-1494. So, something good to try is to run your benchmark with:

<locking .... useLockStriping="false" />

More generally, debugging TimeoutExceptions involves generally producing some thread dumps when these are reported. The idea here is finding out which other thread might be holding that particular lock and see whether it's hanged or waiting for something else and see if that other resource can be tweaked accordingly.

Tools like JVisualVM (part of the jdk) can both generate and analyse thread dumps, otherwise you can use the rudimentary kill -3 or equivalent method and use standalone tools like to inspect them, i.e.:
- TDA (https://tda.dev.java.net/)
- or Samurai (http://yusuke.homeip.net/samurai/en/index.html)

Other tips to get around TimeoutExceptions include reducing the transactional scope of the operation that is holding the lock. That way, the time the lock is held is reduced and hence it's released earlier. Reducing the transactional scope can be achieved by either moving some operations out of the transaction or finding more efficient ways to do what is being done within the transaction.

Sometimes these TimeoutException can hide behind memory issues that produce long garbage collection periods. Turning on garbage collection logging (http://www.jboss.org/community/docs/DOC-12451) can help visualise whether there are noticeable stoppages. Turning on VM parameters can sometimes help reduce the stoppages which in turn increases speed of operation and consequently reduces TimeoutException occurrences.

[nfilotto]

Hi Galder,

Maybe the best way to get rid of TimeoutException is to avoid to use a CacheLoader behind a JBC instance, since database access and data replication create a latency that could cause TimeoutException. If I use JBC to replicate my data over my cluster and I manage myself the data storage (i.e. I use JBC without a CacheLoader), would it help to prevent the TimeoutException or for you TimeoutException are probably due to JBC misuse?

Thank you for your answer,

BR,

Nicolas

[mircea.markus]

Are you using transactions? If yes, one way that would make you avoid deadlocks is to order the fqns you access in a transaction (if possible). This might reduce the deadlocks drastically.

[nfilotto]

Actually we don't use explicit transaction up to now, we use invocation batching. Will it change something if we use explicitly a transaction?

[mircea.markus]

This approach (ordering of fqn) should be also helpful for invocation batching. If it is possible to order your fqn l(lexycographically, for e.g.) in the batch there should/might be much smaller number of deadlocks.

[nfilotto]

This is indeed a good approach to use "lock ordering" to avoid deadlocks, I definitively keep that in mind thanks. However, the code is quite complex so it is not always possible to apply this tip, so do you agree with me if I say that I should get less TimeoutException (deadlocks), if I get rid of the CacheLoader behind the JBC instance since I will have less latency so I will release the locks quicker which means that I reduce the window during which I can get deadlocks. Do you agree with that?

[galder.zamarreno]

Not necessarily. It all depends on the granularity of your batch/transaction.

If the batch/transaction encompasses Cache + JDBC Cache loader access

tx.begin(); // or cache.startBatch()
cache.put(); // cache configured with cache loader
tx.commit(); // or cache.endBatch -> at this point locks are released

If you use Cache + standard JDBC code, it depends how you structure the code. For example, you could do this:

tx.begin(); // or cache.startBatch()
cache.put(); // cache without cache loader
myJdbcAbstractionLayer.store(); // does the jdbc work
tx.commit(); // or cache.endBatch -> at this point locks are released

And in this case, this is hardly different to the option above. The locks are still released at commit/endBatch time. The only difference here is that the code in between cache and the cache loader that gets executed in the first option does not get executed in the second option.

The TimeoutExceptions could be reduced, as I said in the beginning, if you reduce the lock granularity, so you could do this:

tx.begin(); // or cache.startBatch()
cache.put(); // cache without cache loader
tx.commit(); // or cache.endBatch -> at this point locks are released
...
tx.begin(); // or cache.startBatch()

myJdbcAbstractionLayer.store(); // does the jdbc work
tx.commit(); // or cache.endBatch -> at this point locks are released

At this point yeah, the locks are held for a shorted period of time, so the chances of TE happening is reduced. However, the problem with this is that modifying the cache and the database are part of separate transactions, so you have to coordinate what to do with the cache if the jdbc code fails.

[mircea.markus]

ordering the keys avoid deadlocks, TE might still be there but the remaining one will not be caused by deadlocks.

E.g. no ordered fqns:

//Tx1

tm.beggin()

cache.put("/a", k,v);

cache.put("/b",k,v);

tm.commit();

//Tx2

tm.beggin()

cache.put("/b", k,v);

cache.put("/a",k,v);

tm.commit();

If both Tx1 and Tx2 will happen at the same time, a deadlock might happen, and for the entire dureation of locak acquisition timeout (AFAIK 10 sec by default) both "/a" and "/b" will be locked, so other threads might timeout as well (or at least be slowen down by this thread access).

Now, if both Tx1 and Tx2 would access the keys in the same lexycographical order, then no deadlock will happen.

//Tx1

tm.beggin()

cache.put("/a", k,v);

cache.put("/b",k,v);

tm.commit();

//Tx2

tm.beggin()

cache.put("/a", k,v);

cache.put("/b",k,v);

tm.commit();

You might also want to consider migrating to Infinispan, as it has an built-in deadlock detection mechanism for situations where key ordering is not possible/desirable.