Deadlock JBoss 4.2.3 MySql 5.1.41
jboss4skatz Jun 15, 2010 1:21 PMWe are seeing a deadlock in our server (as reported by the stack trace below), but the odd thing is, the thread it claims is holding the contended lock does not seem to be holding that lock. Any idea on what might be causing this or what we can do to diagnose the problem would be appreciated.
We are using:
JBoss 4.2.3
MySQL 5.1.41 - Innodb tables
Connector/J: mysql-connector-java-5.1.11-bin.jar
NON JTA transaction factory
Java 1.6_17
Centos 5.4
Found one Java-level deadlock:
=============================
"http-0.0.0.0-8080-1500":
waiting to lock monitor 0x00002aab1f39aac0 (object 0x00002aaae13f5bc0, a com.xxx.businessobjects.Category),
which is held by "http-0.0.0.0-8080-486"
"http-0.0.0.0-8080-486":
waiting for ownable synchronizer 0x00002aaaf61202d0, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "http-0.0.0.0-8080-237"
"http-0.0.0.0-8080-237":
waiting to lock monitor 0x00002aab1f39aac0 (object 0x00002aaae13f5bc0, a com.xxx.businessobjects.Category),
which is held by "http-0.0.0.0-8080-486"
Java stack information for the threads listed above:
===================================================
"http-0.0.0.0-8080-1500":
at com.xxx.framework.facade.yyyFacade.loadAttributes(yyyFacade.java:251)
- waiting to lock <0x00002aaae13f5bc0> (a com.xxx.businessobjects.Category)
at com.xxx.framework.facade.CategoryManagementFacade.loadAttributes(CategoryManagementFacade.java:1605)
at com.xxx.yyy.ads.manager.AdJuggler.initCategory(AdJuggler.java:192)
at com.xxx.yyy.ads.manager.AdJuggler.initCategories(AdJuggler.java:215)
at com.xxx.yyy.ads.manager.AdJuggler.resetQueue(AdJuggler.java:542)
at com.xxx.yyy.ads.manager.AdJuggler.getCategories(AdJuggler.java:118)
at com.xxx.yyy.ads.manager.AdManager.getAd(AdManager.java:504)
at com.xxx.yyy.ads.manager.AdManager.loadAds(AdManager.java:199)
at com.xxx.yyy.admax.web.actions.AdServeAction.yyyExecute(AdServeAction.java:178)
at com.xxx.yyy.web.actions.base.yyyBaseAction.execute(yyyBaseAction.java:51)
at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:226)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:397)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.xxx.framework.servlet.filter.TrimResponseFilter.doFilter(TrimResponseFilter.java:79)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.xxx.framework.servlet.filter.AccessLogFilter.doFilter(AccessLogFilter.java:174)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.xxx.framework.servlet.filter.DeviceDetectFilter.doFilter(DeviceDetectFilter.java:181)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.xxx.framework.servlet.filter.WirelessOperatorDetectFilter.doFilter(WirelessOperatorDetectFilter.java:187)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.xxx.framework.servlet.filter.PortalDetectionFilter.doFilter(PortalDetectionFilter.java:122)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.xxx.framework.servlet.filter.AccessInitFilter.doFilter(AccessInitFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
at java.lang.Thread.run(Thread.java:619)
"http-0.0.0.0-8080-486":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaaf61202d0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.lock(BaseWrapperManagedConnection.java:254)
at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.tryLock(BaseWrapperManagedConnection.java:262)
at org.jboss.resource.adapter.jdbc.WrappedConnection.lock(WrappedConnection.java:79)
at org.jboss.resource.adapter.jdbc.WrappedStatement.lock(WrappedStatement.java:64)
at org.jboss.resource.adapter.jdbc.WrappedStatement.getMaxRows(WrappedStatement.java:372)
at net.sf.hibernate.impl.BatcherImpl.closeQueryStatement(BatcherImpl.java:147)
at net.sf.hibernate.impl.BatcherImpl.closeStatements(BatcherImpl.java:186)
at net.sf.hibernate.impl.SessionImpl.disconnect(SessionImpl.java:3348)
at net.sf.hibernate.impl.SessionImpl.close(SessionImpl.java:576)
at com.xxx.framework.servicelocator.HibernateServiceLocator.closeSession(HibernateServiceLocator.java:209)
at com.xxx.framework.services.yyyService.closeHibernateSession(yyyService.java:73)
at com.xxx.framework.services.yyyService.loadAttributes(yyyService.java:406)
at com.xxx.framework.facade.yyyFacade.loadAttributes(yyyFacade.java:257)
- locked <0x00002aaae13f5bc0> (a com.xxx.businessobjects.Category)
at com.xxx.framework.facade.CategoryManagementFacade.loadAttributes(CategoryManagementFacade.java:1605)
at com.xxx.yyy.ads.manager.AdJuggler.initCategory(AdJuggler.java:192)
at com.xxx.yyy.ads.manager.AdJuggler.initCategories(AdJuggler.java:215)
at com.xxx.yyy.ads.manager.AdJuggler.resetQueue(AdJuggler.java:542)
at com.xxx.yyy.ads.manager.AdJuggler.getCategories(AdJuggler.java:118)
at com.xxx.yyy.ads.manager.AdManager.getAd(AdManager.java:504)
at com.xxx.yyy.ads.manager.AdManager.loadAds(AdManager.java:199)
at com.xxx.yyy.admax.web.actions.AdServeAction.yyyExecute(AdServeAction.java:178)
at com.xxx.yyy.web.actions.base.yyyBaseAction.execute(yyyBaseAction.java:51)
at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:226)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:397)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.xxx.framework.servlet.filter.TrimResponseFilter.doFilter(TrimResponseFilter.java:79)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.xxx.framework.servlet.filter.AccessLogFilter.doFilter(AccessLogFilter.java:174)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.xxx.framework.servlet.filter.DeviceDetectFilter.doFilter(DeviceDetectFilter.java:181)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.xxx.framework.servlet.filter.WirelessOperatorDetectFilter.doFilter(WirelessOperatorDetectFilter.java:187)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.xxx.framework.servlet.filter.PortalDetectionFilter.doFilter(PortalDetectionFilter.java:122)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.xxx.framework.servlet.filter.AccessInitFilter.doFilter(AccessInitFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
at java.lang.Thread.run(Thread.java:619)
"http-0.0.0.0-8080-237":
at com.xxx.framework.facade.yyyFacade.loadAttributes(yyyFacade.java:251)
- waiting to lock <0x00002aaae13f5bc0> (a com.xxx.businessobjects.Category)
at com.xxx.framework.facade.CategoryManagementFacade.loadAttributes(CategoryManagementFacade.java:1605)
at com.xxx.yyy.ads.manager.AdJuggler.initCategory(AdJuggler.java:192)
at com.xxx.yyy.ads.manager.AdJuggler.initCategories(AdJuggler.java:215)
at com.xxx.yyy.ads.manager.AdJuggler.resetQueue(AdJuggler.java:542)
at com.xxx.yyy.ads.manager.AdJuggler.getCategories(AdJuggler.java:118)
at com.xxx.yyy.ads.manager.AdManager.getAd(AdManager.java:504)
at com.xxx.yyy.ads.manager.AdManager.loadAds(AdManager.java:199)
at com.xxx.yyy.admax.web.actions.AdServeAction.yyyExecute(AdServeAction.java:178)
at com.xxx.yyy.web.actions.base.yyyBaseAction.execute(yyyBaseAction.java:51)
at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:226)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:397)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.xxx.framework.servlet.filter.TrimResponseFilter.doFilter(TrimResponseFilter.java:79)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.xxx.framework.servlet.filter.AccessLogFilter.doFilter(AccessLogFilter.java:174)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.xxx.framework.servlet.filter.DeviceDetectFilter.doFilter(DeviceDetectFilter.java:181)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.xxx.framework.servlet.filter.WirelessOperatorDetectFilter.doFilter(WirelessOperatorDetectFilter.java:187)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.xxx.framework.servlet.filter.PortalDetectionFilter.doFilter(PortalDetectionFilter.java:122)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.xxx.framework.servlet.filter.AccessInitFilter.doFilter(AccessInitFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
at java.lang.Thread.run(Thread.java:619)