[fungrim]

Thank you Brian. I think that answered my questions. I may have mroe detailed questions later, but will probably do them in another thread in such case.

Cheers
- Lars

[belaban]

While we're looking into this, a good workaround is to *not* create/remove nodes *directly* under root, because in this case we only need to acquire a read-lock for root, e.g.

/a/b/c, /x/y/z, but *not* /a, /b or /c

[akhopkar]

Hi guys,

I'm having a problem with the TreeCache that looks like it's being affected by the topic of this dicsussion. Could you guys please let me know if this is a related issue?

We have deployed JBoss 4.0.3 SP1 successfully about a month ago with 2 servers that are using the Tomcat Web clustering only (we are using an .ear file and calling local ejb3 from the .war files so the back end is not clustered).

For about a month, we had low load applications on this and it didn't have any problems.

About 2 days ago we migrated our high load applications to this cluster, and started noticing major problems. The applications are completely hanging since we are using synchronous replication, and they are basically timing out and we are having to constantly restart the servers and drop users by doing this, and after a period of time it happens again and we have to restart again.

We are using REPL_SYNC, and we do not have sticky session enabled so at any point any request can hit either box as directed by the load balancer, and we are relying on jboss clustering for the session replication entirely.

I'm noticing a session read timeout problem in the logs. Any input would be appreciated. Here's the exception that it keeps throwing in the log:

10:32:12,312 ERROR [IdentityLock] read lock for /JSESSION/localhost/ebol/gzJ2Abf3q7AcEOGdefAMFQ** could not be acquired by Thread[http-8081-Processor35,5,jboss] after 15000 ms. Locks: Read lock owners: []
Write lock owner: <myserver135:33077>:22862
, lock info: write owner=<myserver135:33077>:22862 (activeReaders=0, activeWriter=Thread[http-8081-Processor35,5,jboss], waitingReaders=0, waitingWriters=0, waitingUpgrader=0)
10:32:12,313 INFO [STDOUT] org.jboss.cache.lock.TimeoutException: read lock for /JSESSION/localhost/myapp/gzJ2Abf3q7AcEOGdefAMFQ** could not be acquired by Thread[http-8081-Processor35,5,jboss] after 15000 ms. Locks: Read lock owners: []
Write lock owner: <myserver135:33077>:22862
, lock info: write owner=<myserver135:33077>:22862 (activeReaders=0, activeWriter=Thread[http-8081-Processor35,5,jboss], waitingReaders=0, waitingWriters=0, waitingUpgrader=0)
10:32:12,313 INFO [STDOUT] at org.jboss.cache.lock.IdentityLock.acquireReadLock(IdentityLock.java:213)
10:32:12,313 INFO [STDOUT] at org.jboss.cache.Node.acquireReadLock(Node.java:501)
10:32:12,313 INFO [STDOUT] at org.jboss.cache.Node.acquire(Node.java:473)
10:32:12,313 INFO [STDOUT] at org.jboss.cache.interceptors.LockInterceptor.lock(LockInterceptor.java:245)
10:32:12,314 INFO [STDOUT] at org.jboss.cache.interceptors.LockInterceptor.invoke(LockInterceptor.java:159)
10:32:12,314 INFO [STDOUT] at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:41)
10:32:12,314 INFO [STDOUT] at org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:35)
10:32:12,314 INFO [STDOUT] at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:41)
10:32:12,314 INFO [STDOUT] at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:54)
10:32:12,314 INFO [STDOUT] at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:3116)
10:32:12,314 INFO [STDOUT] at org.jboss.cache.TreeCache.get(TreeCache.java:1571)
10:32:12,314 INFO [STDOUT] at org.jboss.cache.TreeCache.get(TreeCache.java:1555)
10:32:12,314 INFO [STDOUT] at sun.reflect.GeneratedMethodAccessor216.invoke(Unknown Source)
10:32:12,314 INFO [STDOUT] at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
10:32:12,314 INFO [STDOUT] at java.lang.reflect.Method.invoke(Unknown Source)
10:32:12,314 INFO [STDOUT] at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
10:32:12,314 INFO [STDOUT] at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
10:32:12,314 INFO [STDOUT] at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
10:32:12,314 INFO [STDOUT] at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:245)
10:32:12,314 INFO [STDOUT] at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
10:32:12,314 INFO [STDOUT] at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
10:32:12,314 INFO [STDOUT] at $Proxy100.get(Unknown Source)
10:32:12,314 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.JBossCacheService._get(JBossCacheService.java:402)
10:32:12,314 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.JBossCacheService.nodeDirty(JBossCacheService.java:679)
10:32:12,314 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.JBossCacheService.nodeModified(JBossCacheService.java:670)
10:32:12,314 INFO [STDOUT] at org.jboss.cache.TreeCache.notifyNodeModified(TreeCache.java:3187)
10:32:12,314 INFO [STDOUT] at org.jboss.cache.TreeCache._put(TreeCache.java:2420)
10:32:12,314 INFO [STDOUT] at sun.reflect.GeneratedMethodAccessor234.invoke(Unknown Source)
10:32:12,314 INFO [STDOUT] at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
10:32:12,314 INFO [STDOUT] at java.lang.reflect.Method.invoke(Unknown Source)
10:32:12,314 INFO [STDOUT] at org.jgroups.blocks.MethodCall.invoke(MethodCall.java:236)
10:32:12,314 INFO [STDOUT] at org.jboss.cache.interceptors.LockInterceptor.rollback(LockInterceptor.java:343)
10:32:12,315 INFO [STDOUT] at org.jboss.cache.interceptors.LockInterceptor.access$200(LockInterceptor.java:25)
10:32:12,315 INFO [STDOUT] at org.jboss.cache.interceptors.LockInterceptor$SynchronizationHandler.afterCompletion(LockInterceptor.java:425)
10:32:12,315 INFO [STDOUT] at org.jboss.cache.interceptors.OrderedSynchronizationHandler.afterCompletion(OrderedSynchronizationHandler.java:80)
10:32:12,315 INFO [STDOUT] at org.jboss.tm.TransactionImpl.doAfterCompletion(TransactionImpl.java:1508)
10:32:12,315 INFO [STDOUT] at org.jboss.tm.TransactionImpl.completeTransaction(TransactionImpl.java:1180)
10:32:12,315 INFO [STDOUT] at org.jboss.tm.TransactionImpl.commit(TransactionImpl.java:351)
10:32:12,315 INFO [STDOUT] at org.jboss.tm.TxManager.commit(TxManager.java:224)
10:32:12,315 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.JBossCacheManager.endTransaction(JBossCacheManager.java:636)
10:32:12,315 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.JBossCacheManager.processSessionRepl(JBossCacheManager.java:627)
10:32:12,315 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.JBossCacheManager.storeSession(JBossCacheManager.java:375)
10:32:12,315 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:38)
10:32:12,315 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:91)
10:32:12,315 INFO [STDOUT] at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:59)
10:32:12,315 INFO [STDOUT] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
10:32:12,315 INFO [STDOUT] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
10:32:12,315 INFO [STDOUT] at org.apache.catalina.valves.FastCommonAccessLogValve.invoke(FastCommonAccessLogValve.java:481)
10:32:12,315 INFO [STDOUT] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
10:32:12,315 INFO [STDOUT] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
10:32:12,315 INFO [STDOUT] at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
10:32:12,315 INFO [STDOUT] at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
10:32:12,315 INFO [STDOUT] at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
10:32:12,315 INFO [STDOUT] at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
10:32:12,315 INFO [STDOUT] at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
10:32:12,315 INFO [STDOUT] at java.lang.Thread.run(Unknown Source)

[ben.wang]

Actually your problem is not related to this topic but rather clustering forum.

My suggestion is try out 4.0.4CR2 release. We have made siginificant fixes there that may resolve your problem.

-Ben