JBoss 3.2.2 Clustering Issues After Servers are Running for
rcostanzo Jun 15, 2004 9:57 AMI am running JBoss 3.2.2 in production. Our system processes hundreds of thousands transactions a day. We are using clustering with cache invalidation with 3 instances across 3 machines. In general, everything works great. However, every other week or so, I will see the system "hang", with errors like these thrown:
2004-06-15 05:46:41,257 WARN [org.jboss.tm.TransactionImpl] Transaction TransactionImpl:XidImpl [FormatId=257, GlobalId=www7.mateso.com//55944547, BranchQual=] timed out. status=STATUS_ACTIVE
2004-06-15 05:46:41,266 WARN [org.jboss.ejb.plugins.AbstractInstanceCache] Unable to passivate due to ctx lock, id=5084257
2004-06-15 05:46:41,354 WARN [org.jboss.tm.TransactionImpl] Transaction TransactionImpl:XidImpl [FormatId=257, GlobalId=www7.mateso.com//55944555, BranchQual=] timed out. status=STATUS_ACTIVE
2004-06-15 05:47:18,543 WARN [org.jboss.tm.TransactionImpl] Transaction TransactionImpl:XidImpl [FormatId=257, GlobalId=www7.mateso.com//55944825, BranchQual=] timed out. status=STATUS_ACTIVE
2004-06-15 05:47:52,166 WARN [org.jboss.tm.TransactionImpl] Transaction TransactionImpl:XidImpl [FormatId=257, GlobalId=www7.mateso.com//55945145, BranchQual=] timed out. status=STATUS_ACTIVE
2004-06-15 05:47:52,168 ERROR [org.jboss.ejb.BeanLock] PoolThread-7Saw rolled back tx=TransactionImpl:XidImpl [FormatId=257, GlobalId=www7.mateso.com//55945145, BranchQual=] waiting for txLock
2004-06-15 05:47:52,168 WARN [org.jboss.tm.TransactionImpl] Lock contention, tx=TransactionImpl:XidImpl [FormatId=257, GlobalId=www7.mateso.com//55945145, BranchQual=]
2004-06-15 05:47:52,168 ERROR [org.jboss.ejb.plugins.LogInterceptor] RuntimeException:
java.lang.RuntimeException: Transaction marked for rollback, possibly a timeout
at org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock.waitForTx(QueuedPessimisticEJBLock.java(Compiled Code))
at org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock.doSchedule(QueuedPessimisticEJBLock.java(Compiled Code))
at org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock.schedule(QueuedPessimisticEJBLock.java(Compiled Code))
at org.jboss.ejb.plugins.EntityLockInterceptor.invoke(EntityLockInterceptor.java(Compiled Code))
at org.jboss.ejb.plugins.EntityCreationInterceptor.invoke(EntityCreationInterceptor.java(Compiled Code))
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java(Compiled Code))
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java(Compiled Code))
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java(Compiled Code))
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java(Compiled Code))
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java(Compiled Code))
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java(Compiled Code))
at org.jboss.ejb.EntityContainer.internalInvoke(EntityContainer.java(Compiled Code))
at org.jboss.ejb.Container.invoke(Container.java(Compiled Code))
at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java(Compiled Code))
at org.jboss.ejb.plugins.local.EntityProxy.invoke(EntityProxy.java(Compiled Code))
at $Proxy219.getConsumerValue(Unknown Source)
at mts.psp.consumer.ConsumerManagerEJB.getConsumerByEmail(ConsumerManagerEJB.java:248)
at sun.reflect.GeneratedMethodAccessor745.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java(Compiled Code))
at java.lang.reflect.Method.invoke(Method.java(Compiled Code))
at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java(Compiled Code))
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java(Compiled Code))
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java(Compiled Code))
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java(Compiled Code))
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java(Compiled Code))
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java(Compiled Code))
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java(Compiled Code))
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java(Compiled Code))
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java(Compiled Code))
at org.jboss.ejb.StatelessSessionContainer.internalInvoke(StatelessSessionContainer.java(Compiled Code))
at org.jboss.ejb.Container.invoke(Container.java(Compiled Code))
at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java(Compiled Code))
at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java(Compiled Code))
at $Proxy579.getConsumerByEmail(Unknown Source)
at mts.psp.consumer.ConsumerDelegate.getConsumerByEmail(ConsumerDelegate.java:290)
This occurs across all 3 instances at the same time, and the site becomes unaccessible. 9 times out of 10, the servers will snap out of this state within a few minutes, and everything will work fine again. However, sometimes it stays in this state for a longer period of time and I need to bounce the instances. I'm am pretty sure that there is no external program locking our database (which would cause similar behavior), especially since just an application server restart fixes the issue.
Another oddity (which may or may not be related), is I will see
-------------------------------------------------------
GMS: address is www7:32832
-------------------------------------------------------
Thrown out all of a sudden (www7 is one of the server names). Usually, you only see this on startup. After seeing this output, the other two members of the cluster will detect a dead member, and then detect a new member.
Any help or clues as to what is going on would be greatly appreciated.
A stack trace from one of the other servers looks pretty similar but is a little different:
2004-06-15 07:18:42,089 WARN [org.jboss.tm.TransactionImpl] Transaction TransactionImpl:XidImpl [FormatId=257, GlobalId=www4.mateso.com//50385479, BranchQual=] timed out. status=STATUS_ACTIVE
2004-06-15 07:18:42,120 ERROR [org.jboss.ejb.plugins.LogInterceptor] TransactionRolledbackLocalException in method: public abstract void mts.psp.consumer.ConsumerManagerLocal.updateSiteLoginDate(java.lang.Integer,java.lang.Integer), causedBy:
org.jboss.tm.JBossRollbackException: Unable to commit, tx=TransactionImpl:XidImpl [FormatId=257, GlobalId=www4.mateso.com//50385479, BranchQual=] status=STATUS_NO_TRANSACTION
at org.jboss.tm.TransactionImpl.commit(TransactionImpl.java(Compiled Code))
at org.jboss.ejb.plugins.TxInterceptorCMT.endTransaction(TxInterceptorCMT.java(Compiled Code))
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java(Compiled Code))
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java(Compiled Code))
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java(Compiled Code))
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java(Compiled Code))
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java(Compiled Code))
at org.jboss.ejb.StatelessSessionContainer.internalInvoke(StatelessSessionContainer.java(Compiled Code))
at org.jboss.ejb.Container.invoke(Container.java(Compiled Code))
at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java(Compiled Code))
at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java(Compiled Code))
at mts.psp.consumer.ConsumerDelegate.updateSiteLoginDate(ConsumerDelegate.java(Compiled Code))
at mts.psp.web.struts.util.WebSessionUtil.setLoggedIn(WebSessionUtil.java(Compiled Code))
at mts.psp.web.struts.auth.SessionFilter.handleCookiesAndMungeCode(SessionFilter.java(Compiled Code))
at mts.psp.web.struts.auth.SessionFilter.doFilter(SessionFilter.java(Compiled Code))
at org.mortbay.jetty.servlet.WebApplicationHandler$Chain.doFilter(WebApplicationHandler.java(Compiled Code))
at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java(Compiled Code))
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java(Compiled Code))
at org.mortbay.http.HttpContext.handle(HttpContext.java(Compiled Code))
at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java(Compiled Code))
at org.mortbay.http.HttpContext.handle(HttpContext.java(Compiled Code))
at org.mortbay.http.HttpServer.service(HttpServer.java(Compiled Code))
at org.jboss.jetty.Jetty.service(Jetty.java(Compiled Code))
at org.mortbay.http.HttpConnection.service(HttpConnection.java(Inlined Compiled Code))
at org.mortbay.http.ajp.AJP13Connection.handleNext(AJP13Connection.java(Compiled Code))
at org.mortbay.http.HttpConnection.handle(HttpConnection.java(Compiled Code))
at org.mortbay.http.ajp.AJP13Listener.handleConnection(AJP13Listener.java(Compiled Code))
at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java(Compiled Code))
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java(Compiled Code))