write lock for / could not be acquired after 15000 ms

[jay_bee]

Hi,

We're using JBoss 4.2.2GA which incorporates JBoss Cache 1.4.1SP5 and problems are coming up during volume testing of our system. The system itself uses EJB3 and hibernate, JBoss Cache is used as a second level cache with optimistic locking

The exception below is thrown at above 500 concurrent users. I've had a google around and there looked like there was a similar problem happening in early versions of 1.4.1. We added an attribute to the config file to disable locking of a parent node on writing (as specified in the bug fix) but the same error came up.

Should we be using a later version of JBoss Cache?

Thanks

jay_bee

2008-04-18 17:32:42,580 ERROR
[org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/cs8].[default]]
Servlet.service() for servlet default threw exception
org.jboss.cache.lock.TimeoutException: write lock for / could not be
acquired after 15000 ms. Locks: Read lock owners: []
Write lock owner: GlobalTransaction:<192.168.100.13:33064>:245763
 (caller=GlobalTransaction:<192.168.100.13:33064>:245870, lock info:
write owner=GlobalTransaction:<192.168.100.13:33064>:245763
(org.jboss.cache.lock.LockStrategyReadCommitted@8a78b7))
 at org.jboss.cache.lock.IdentityLock.acquireWriteLock(IdentityLock.java:206)
 at org.jboss.cache.Node.acquireWriteLock(Node.java:529)
 at org.jboss.cache.Node.acquire(Node.java:476)
 at org.jboss.cache.interceptors.OptimisticLockingInterceptor.lockNodes(OptimisticLockingInterceptor.java:153)
 at org.jboss.cache.interceptors.OptimisticLockingInterceptor.invoke(OptimisticLockingInterceptor.java:80)
 at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
 at org.jboss.cache.interceptors.OptimisticReplicationInterceptor.invoke(OptimisticReplicationInterceptor.java:85)
 at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
 at org.jboss.cache.interceptors.TxInterceptor.runPreparePhase(TxInterceptor.java:873)
 at org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1156)
 at org.jboss.cache.interceptors.OrderedSynchronizationHandler.beforeCompletion(OrderedSynchronizationHandler.java:75)
 at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:114)

[jay_bee]

This was the bug I was referring to in the OP

http://jira.jboss.org/jira/browse/JBCACHE-955

[jay_bee]

Just had another look at it and I'm getting a bit confused with the locking reasons.

Why does the system try to (and obviously has done before) take a lock on the root node when the only nodes attached to the root are created at system startup time and consist of the two regions used by the hibernate 2nd level cache? Surely once these have been created there should be no need to take a lock on the root node again?

[khkachn]

I am running into this same issue but I have not come across a solution. If you do ever figure it out, please post the solution.

[manik]

Have a look at http://jira.jboss.org/jira/browse/JBCACHE-1228. I'd recommend upgrading to a newer SP.

[jay_bee]

Upgrading seemed to get rid of them for a bit but now we've cleared some other problems they've come back.

I'm a bit confused now though. We changed one of our entity beans to READ_ONLY and JBC complained that it couldn't get a write lock on it, even though it's only ever pulled from the DB and never modified.

The problem went away when we changed the isolation level, which I thought as we were using OPTIMISTIC locking should always be ignored. From what we can see the system is taking a write lock all the way up the tree from the target node to the root node whenever it wants to modify anything. Whereas we obviously only want it to take a lock on the node to be modified.

So....

1. We want the system to only take a write lock on the nodes it's actually changing. What should we set?

2. All transactions with our entity beans are managed by the JBoss transaction manager. If we change the isolation level to NONE will the transaction manager look after the integrity of the data for us?

[jay_bee]

After much testing and trawling of forums it looks like we're taking a 3rd option, which is unfortunately to stop using JBoss Cache. We'd rather not but as a solution to the problem can't be found we have no other option.

Hopefully at a future date we can reinstate it if a solution/explanation is found.

[manik]

Could you also confirm that you have set the LockParentForChildInsertRemove to it's default, which is false?

[jay_bee]

Yep. It was set to false, although in the link you provided:-

Controls whether inserting or removing a node requires a write lock on the node's parent (when pessimistic locking is used). The default value is false.

it seems to imply it should only be used on pessimistic locking.