0 Replies Latest reply on Nov 9, 2012 2:43 AM by Lokesh Kumar Padhanvis

    Deadlock occurred .... plz help

    Lokesh Kumar Padhanvis Newbie

      Hi,

       

      One of our instances has gone into a deadlock state. I am Pasting the stack trace below.

      Looking at the stack trace, i feel this is related to MySQL connector/J, can anyone of you please let me know if there is any fix or workaround for this problem.

       

      NOTE: I am posting it here because, I am using JBOSS with Hibernate to manage the persistence using JPA. And there seems to be a similar problem with c3p0 that I have come across in the web.

      But did not find much information on this with respect to JBOSS and Hibernate.

       

      I have highlighted my application specific code.

       

      Found one Java-level deadlock:
      =============================
      "pool-34-thread-8":
        waiting to lock monitor 0x000000004c21d928 (object 0x00000007c25dcc70, a com.mysql.jdbc.JDBC4ResultSet),
        which is held by "message-outgoing queue processor"
      "message-outgoing queue processor":
        waiting to lock monitor 0x000000004c21e3a8 (object 0x00000007555cee90, a com.mysql.jdbc.JDBC4Connection),
        which is held by "pool-34-thread-8"
      
      Java stack information for the threads listed above:
      ===================================================
      "pool-34-thread-8":
          at com.mysql.jdbc.ResultSetImpl.realClose(ResultSetImpl.java:7477)
          - waiting to lock <0x00000007c25dcc70> (a com.mysql.jdbc.JDBC4ResultSet)
          at com.mysql.jdbc.ResultSetImpl.close(ResultSetImpl.java:907)
          at com.mysql.jdbc.StatementImpl.realClose(StatementImpl.java:2430)
          - locked <0x00000007c2e7b358> (a com.mysql.jdbc.JDBC4PreparedStatement)
          at com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:3079)
          - locked <0x00000007c2e7b358> (a com.mysql.jdbc.JDBC4PreparedStatement)
          at com.mysql.jdbc.ConnectionImpl.closeAllOpenStatements(ConnectionImpl.java:1585)
          at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4361)
          at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1557)
          - locked <0x00000007555cee90> (a com.mysql.jdbc.JDBC4Connection)
          at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.destroy(BaseWrapperManagedConnection.java:311)
          at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.doDestroy(InternalManagedConnectionPool.java:664)
          at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.returnConnection(InternalManagedConnectionPool.java:395)
          at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.returnConnection(JBossManagedConnectionPool.java:761)
          at org.jboss.resource.connectionmanager.BaseConnectionManager2.returnManagedConnection(BaseConnectionManager2.java:463)
          at org.jboss.resource.connectionmanager.BaseConnectionManager2$BaseConnectionEventListener.connectionErrorOccurred(BaseConnectionManager2.java:854)
          at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.connectionErrorOccurred(TxConnectionManager.java:792)
          at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.broadcastConnectionError(BaseWrapperManagedConnection.java:421)
          at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.connectionError(BaseWrapperManagedConnection.java:385)
          at org.jboss.resource.adapter.jdbc.WrappedConnection.checkException(WrappedConnection.java:865)
          at org.jboss.resource.adapter.jdbc.WrappedStatement.checkException(WrappedStatement.java:852)
          at org.jboss.resource.adapter.jdbc.WrappedResultSet.checkException(WrappedResultSet.java:1947)
          at org.jboss.resource.adapter.jdbc.WrappedResultSet.getString(WrappedResultSet.java:892)
          at org.hibernate.type.StringType.get(StringType.java:41)
          at org.hibernate.type.NullableType.nullSafeGet(NullableType.java:184)
          at org.hibernate.type.NullableType.nullSafeGet(NullableType.java:173)
          at org.hibernate.persister.collection.AbstractCollectionPersister.readKey(AbstractCollectionPersister.java:756)
          at org.hibernate.loader.Loader.readCollectionElement(Loader.java:998)
          at org.hibernate.loader.Loader.readCollectionElements(Loader.java:669)
          at org.hibernate.loader.Loader.getRowFromResultSet(Loader.java:614)
          at org.hibernate.loader.Loader.doQuery(Loader.java:724)
          at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
          at org.hibernate.loader.Loader.loadCollection(Loader.java:2019)
          at org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:59)
          at org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:587)
          at org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:83)
          at org.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:1744)
          at org.hibernate.collection.AbstractPersistentCollection.forceInitialization(AbstractPersistentCollection.java:476)
          at org.hibernate.engine.StatefulPersistenceContext.initializeNonLazyCollections(StatefulPersistenceContext.java:867)
          at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:264)
          at org.hibernate.loader.Loader.doList(Loader.java:2232)
          at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2129)
          at org.hibernate.loader.Loader.list(Loader.java:2124)
          at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:401)
          at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:363)
          at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196)
          at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1149)
          at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)
          at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:67)
          at com.zzz.ReadOnlyObjectSessionBean.getEndPointsByTimePeriod(ReadOnlyObjectSessionBean.java:3638)
          at sun.reflect.GeneratedMethodAccessor991.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeTarget(MethodInvocation.java:122)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
          at org.jboss.ejb3.EJBContainerInvocationWrapper.invokeNext(EJBContainerInvocationWrapper.java:69)
          at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.invoke(InterceptorSequencer.java:76)
          at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.aroundInvoke(InterceptorSequencer.java:62)
          at sun.reflect.GeneratedMethodAccessor463.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.jboss.aop.advice.PerJoinpointAdvice.invoke(PerJoinpointAdvice.java:174)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.fillMethod(InvocationContextInterceptor.java:72)
          at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_fillMethod_57850553.invoke(InvocationContextInterceptor_z_fillMethod_57850553.java)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.setup(InvocationContextInterceptor.java:88)
          at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_setup_57850553.invoke(InvocationContextInterceptor_z_setup_57850553.java)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:62)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:56)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:68)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
          at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:190)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptorv2.invoke(RoleBasedAuthorizationInterceptorv2.java:201)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.security.Ejb3AuthenticationInterceptorv2.invoke(Ejb3AuthenticationInterceptorv2.java:186)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:41)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.BlockContainerShutdownInterceptor.invoke(BlockContainerShutdownInterceptor.java:67)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.aspects.currentinvocation.CurrentInvocationInterceptor.invoke(CurrentInvocationInterceptor.java:67)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.session.SessionSpecContainer.invoke(SessionSpecContainer.java:180)
          at org.jboss.ejb3.session.SessionSpecContainer.invoke(SessionSpecContainer.java:238)
          at org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:188)
          at $Proxy230.getEndPointsByTimePeriod(Unknown Source)
          at com.zzz.AdapterManager$ScanTask.run(AdapterManager.java:2096)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
          at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
          at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:662)
      "message-outgoing queue processor":
          at com.mysql.jdbc.ConnectionImpl.getCharacterSetMetadata(ConnectionImpl.java:2799)
          - waiting to lock <0x00000007555cee90> (a com.mysql.jdbc.JDBC4Connection)
          at com.mysql.jdbc.Field.getStringFromBytes(Field.java:710)
          at com.mysql.jdbc.Field.getOriginalName(Field.java:652)
          at com.mysql.jdbc.ResultSetImpl.buildIndexMapping(ResultSetImpl.java:749)
          at com.mysql.jdbc.ResultSetImpl.findColumn(ResultSetImpl.java:1108)
          - locked <0x00000007c25dcc70> (a com.mysql.jdbc.JDBC4ResultSet)
          at com.mysql.jdbc.ResultSetImpl.getString(ResultSetImpl.java:5610)
          at org.jboss.resource.adapter.jdbc.WrappedResultSet.getString(WrappedResultSet.java:888)
          at org.hibernate.type.StringType.get(StringType.java:41)
          at org.hibernate.type.NullableType.nullSafeGet(NullableType.java:184)
          at org.hibernate.type.NullableType.nullSafeGet(NullableType.java:173)
          at org.hibernate.loader.Loader.getKeyFromResultSet(Loader.java:1121)
          at org.hibernate.loader.Loader.getRowFromResultSet(Loader.java:588)
          at org.hibernate.loader.Loader.doQuery(Loader.java:724)
          at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
          at org.hibernate.loader.Loader.doList(Loader.java:2232)
          at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2129)
          at org.hibernate.loader.Loader.list(Loader.java:2124)
          at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:401)
          at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:363)
          at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196)
          at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1149)
          at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)
          at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:67)
          at com.zzz.ReadOnlyObjectSessionBean.getObjectByName(ReadOnlyObjectSessionBean.java:206)
          at sun.reflect.GeneratedMethodAccessor1066.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeTarget(MethodInvocation.java:122)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
          at org.jboss.ejb3.EJBContainerInvocationWrapper.invokeNext(EJBContainerInvocationWrapper.java:69)
          at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.invoke(InterceptorSequencer.java:76)
          at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.aroundInvoke(InterceptorSequencer.java:62)
          at sun.reflect.GeneratedMethodAccessor463.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.jboss.aop.advice.PerJoinpointAdvice.invoke(PerJoinpointAdvice.java:174)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.fillMethod(InvocationContextInterceptor.java:72)
          at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_fillMethod_57850553.invoke(InvocationContextInterceptor_z_fillMethod_57850553.java)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.setup(InvocationContextInterceptor.java:88)
          at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_setup_57850553.invoke(InvocationContextInterceptor_z_setup_57850553.java)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:62)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:56)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:68)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
          at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:190)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptorv2.invoke(RoleBasedAuthorizationInterceptorv2.java:201)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.security.Ejb3AuthenticationInterceptorv2.invoke(Ejb3AuthenticationInterceptorv2.java:186)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:41)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.BlockContainerShutdownInterceptor.invoke(BlockContainerShutdownInterceptor.java:67)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.aspects.currentinvocation.CurrentInvocationInterceptor.invoke(CurrentInvocationInterceptor.java:67)
          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
          at org.jboss.ejb3.session.SessionSpecContainer.invoke(SessionSpecContainer.java:180)
          at org.jboss.ejb3.session.SessionSpecContainer.invoke(SessionSpecContainer.java:238)
          at org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:188)
          at $Proxy230.getObjectByName(Unknown Source)
          at com.zzz.SnmpDiscovery.start(SnmpDiscovery.java:83)
          at com.zzz.DiscoveryManager.DiscoveryTask(DiscoveryManager.java:325)
          at com.zzz.ServerManager.processMessage(ServerManager.java:2894)
          at com.zzz.ServerManager.access$700(ServerManager.java:90)
          at com.zzz.ServerManager$2.run(ServerManager.java:346)
          at java.lang.Thread.run(Thread.java:662)
      
      Found 1 deadlock.