0 Replies Latest reply on Jun 15, 2010 1:21 PM by jboss4skatz

    Deadlock JBoss 4.2.3 MySql 5.1.41

    jboss4skatz
    jboss4skatz Jun 15, 2010 1:21 PM

      We are seeing a deadlock in our server (as reported by the stack trace below), but the odd thing is, the thread it claims is holding the contended lock does not seem to be holding that lock.  Any idea on what might be causing this or what we can do to diagnose the problem would be appreciated.

       

      We are using:

      JBoss 4.2.3

      MySQL 5.1.41 - Innodb tables

      Connector/J: mysql-connector-java-5.1.11-bin.jar

      NON JTA transaction factory

      Java 1.6_17

      Centos 5.4

       

      Found one Java-level deadlock:
      =============================
      "http-0.0.0.0-8080-1500":
        waiting to lock monitor 0x00002aab1f39aac0 (object 0x00002aaae13f5bc0, a com.xxx.businessobjects.Category),
        which is held by "http-0.0.0.0-8080-486"
      "http-0.0.0.0-8080-486":
        waiting for ownable synchronizer 0x00002aaaf61202d0, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
        which is held by "http-0.0.0.0-8080-237"
      "http-0.0.0.0-8080-237":
        waiting to lock monitor 0x00002aab1f39aac0 (object 0x00002aaae13f5bc0, a com.xxx.businessobjects.Category),
        which is held by "http-0.0.0.0-8080-486"

       

      Java stack information for the threads listed above:
      ===================================================
      "http-0.0.0.0-8080-1500":
              at com.xxx.framework.facade.yyyFacade.loadAttributes(yyyFacade.java:251)
              - waiting to lock <0x00002aaae13f5bc0> (a com.xxx.businessobjects.Category)
              at com.xxx.framework.facade.CategoryManagementFacade.loadAttributes(CategoryManagementFacade.java:1605)
              at com.xxx.yyy.ads.manager.AdJuggler.initCategory(AdJuggler.java:192)
              at com.xxx.yyy.ads.manager.AdJuggler.initCategories(AdJuggler.java:215)
              at com.xxx.yyy.ads.manager.AdJuggler.resetQueue(AdJuggler.java:542)
              at com.xxx.yyy.ads.manager.AdJuggler.getCategories(AdJuggler.java:118)
              at com.xxx.yyy.ads.manager.AdManager.getAd(AdManager.java:504)
              at com.xxx.yyy.ads.manager.AdManager.loadAds(AdManager.java:199)
              at com.xxx.yyy.admax.web.actions.AdServeAction.yyyExecute(AdServeAction.java:178)
              at com.xxx.yyy.web.actions.base.yyyBaseAction.execute(yyyBaseAction.java:51)
              at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421)
              at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:226)
              at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164)
              at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:397)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at com.xxx.framework.servlet.filter.TrimResponseFilter.doFilter(TrimResponseFilter.java:79)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at com.xxx.framework.servlet.filter.AccessLogFilter.doFilter(AccessLogFilter.java:174)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at com.xxx.framework.servlet.filter.DeviceDetectFilter.doFilter(DeviceDetectFilter.java:181)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at com.xxx.framework.servlet.filter.WirelessOperatorDetectFilter.doFilter(WirelessOperatorDetectFilter.java:187)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at com.xxx.framework.servlet.filter.PortalDetectionFilter.doFilter(PortalDetectionFilter.java:122)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at com.xxx.framework.servlet.filter.AccessInitFilter.doFilter(AccessInitFilter.java:52)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
              at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
              at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
              at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
              at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
              at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
              at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
              at java.lang.Thread.run(Thread.java:619)
      "http-0.0.0.0-8080-486":
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00002aaaf61202d0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
              at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
              at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
              at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.lock(BaseWrapperManagedConnection.java:254)
              at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.tryLock(BaseWrapperManagedConnection.java:262)
              at org.jboss.resource.adapter.jdbc.WrappedConnection.lock(WrappedConnection.java:79)
              at org.jboss.resource.adapter.jdbc.WrappedStatement.lock(WrappedStatement.java:64)
              at org.jboss.resource.adapter.jdbc.WrappedStatement.getMaxRows(WrappedStatement.java:372)
              at net.sf.hibernate.impl.BatcherImpl.closeQueryStatement(BatcherImpl.java:147)
              at net.sf.hibernate.impl.BatcherImpl.closeStatements(BatcherImpl.java:186)
              at net.sf.hibernate.impl.SessionImpl.disconnect(SessionImpl.java:3348)
              at net.sf.hibernate.impl.SessionImpl.close(SessionImpl.java:576)
              at com.xxx.framework.servicelocator.HibernateServiceLocator.closeSession(HibernateServiceLocator.java:209)
              at com.xxx.framework.services.yyyService.closeHibernateSession(yyyService.java:73)
              at com.xxx.framework.services.yyyService.loadAttributes(yyyService.java:406)
              at com.xxx.framework.facade.yyyFacade.loadAttributes(yyyFacade.java:257)
              - locked <0x00002aaae13f5bc0> (a com.xxx.businessobjects.Category)
              at com.xxx.framework.facade.CategoryManagementFacade.loadAttributes(CategoryManagementFacade.java:1605)
              at com.xxx.yyy.ads.manager.AdJuggler.initCategory(AdJuggler.java:192)
              at com.xxx.yyy.ads.manager.AdJuggler.initCategories(AdJuggler.java:215)
              at com.xxx.yyy.ads.manager.AdJuggler.resetQueue(AdJuggler.java:542)
              at com.xxx.yyy.ads.manager.AdJuggler.getCategories(AdJuggler.java:118)
              at com.xxx.yyy.ads.manager.AdManager.getAd(AdManager.java:504)
              at com.xxx.yyy.ads.manager.AdManager.loadAds(AdManager.java:199)
              at com.xxx.yyy.admax.web.actions.AdServeAction.yyyExecute(AdServeAction.java:178)
              at com.xxx.yyy.web.actions.base.yyyBaseAction.execute(yyyBaseAction.java:51)
              at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421)
              at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:226)
              at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164)
              at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:397)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at com.xxx.framework.servlet.filter.TrimResponseFilter.doFilter(TrimResponseFilter.java:79)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at com.xxx.framework.servlet.filter.AccessLogFilter.doFilter(AccessLogFilter.java:174)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at com.xxx.framework.servlet.filter.DeviceDetectFilter.doFilter(DeviceDetectFilter.java:181)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at com.xxx.framework.servlet.filter.WirelessOperatorDetectFilter.doFilter(WirelessOperatorDetectFilter.java:187)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at com.xxx.framework.servlet.filter.PortalDetectionFilter.doFilter(PortalDetectionFilter.java:122)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at com.xxx.framework.servlet.filter.AccessInitFilter.doFilter(AccessInitFilter.java:52)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
              at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
              at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
              at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
              at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
              at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
              at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
              at java.lang.Thread.run(Thread.java:619)
      "http-0.0.0.0-8080-237":
              at com.xxx.framework.facade.yyyFacade.loadAttributes(yyyFacade.java:251)
              - waiting to lock <0x00002aaae13f5bc0> (a com.xxx.businessobjects.Category)
              at com.xxx.framework.facade.CategoryManagementFacade.loadAttributes(CategoryManagementFacade.java:1605)
              at com.xxx.yyy.ads.manager.AdJuggler.initCategory(AdJuggler.java:192)
              at com.xxx.yyy.ads.manager.AdJuggler.initCategories(AdJuggler.java:215)
              at com.xxx.yyy.ads.manager.AdJuggler.resetQueue(AdJuggler.java:542)
              at com.xxx.yyy.ads.manager.AdJuggler.getCategories(AdJuggler.java:118)
              at com.xxx.yyy.ads.manager.AdManager.getAd(AdManager.java:504)
              at com.xxx.yyy.ads.manager.AdManager.loadAds(AdManager.java:199)
              at com.xxx.yyy.admax.web.actions.AdServeAction.yyyExecute(AdServeAction.java:178)
              at com.xxx.yyy.web.actions.base.yyyBaseAction.execute(yyyBaseAction.java:51)
              at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421)
              at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:226)
              at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164)
              at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:397)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at com.xxx.framework.servlet.filter.TrimResponseFilter.doFilter(TrimResponseFilter.java:79)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at com.xxx.framework.servlet.filter.AccessLogFilter.doFilter(AccessLogFilter.java:174)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at com.xxx.framework.servlet.filter.DeviceDetectFilter.doFilter(DeviceDetectFilter.java:181)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at com.xxx.framework.servlet.filter.WirelessOperatorDetectFilter.doFilter(WirelessOperatorDetectFilter.java:187)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at com.xxx.framework.servlet.filter.PortalDetectionFilter.doFilter(PortalDetectionFilter.java:122)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at com.xxx.framework.servlet.filter.AccessInitFilter.doFilter(AccessInitFilter.java:52)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
              at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
              at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
              at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
              at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
              at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
              at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
              at java.lang.Thread.run(Thread.java:619)

      • 2635 Views
      • Tags: