Deadlock occurred .... plz help
Hi,
One of our instances has gone into a deadlock state. I am Pasting the stack trace below.
Looking at the stack trace, i feel this is related to MySQL connector/J, can anyone of you please let me know if there is any fix or workaround for this problem.
NOTE: I am posting it here because, I am using JBOSS with Hibernate to manage the persistence using JPA. And there seems to be a similar problem with c3p0 that I have come across in the web.
But did not find much information on this with respect to JBOSS and Hibernate.
I have highlighted my application specific code.
Found one Java-level deadlock: ============================= "pool-34-thread-8": waiting to lock monitor 0x000000004c21d928 (object 0x00000007c25dcc70, a com.mysql.jdbc.JDBC4ResultSet), which is held by "message-outgoing queue processor" "message-outgoing queue processor": waiting to lock monitor 0x000000004c21e3a8 (object 0x00000007555cee90, a com.mysql.jdbc.JDBC4Connection), which is held by "pool-34-thread-8" Java stack information for the threads listed above: =================================================== "pool-34-thread-8": at com.mysql.jdbc.ResultSetImpl.realClose(ResultSetImpl.java:7477) - waiting to lock <0x00000007c25dcc70> (a com.mysql.jdbc.JDBC4ResultSet) at com.mysql.jdbc.ResultSetImpl.close(ResultSetImpl.java:907) at com.mysql.jdbc.StatementImpl.realClose(StatementImpl.java:2430) - locked <0x00000007c2e7b358> (a com.mysql.jdbc.JDBC4PreparedStatement) at com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:3079) - locked <0x00000007c2e7b358> (a com.mysql.jdbc.JDBC4PreparedStatement) at com.mysql.jdbc.ConnectionImpl.closeAllOpenStatements(ConnectionImpl.java:1585) at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4361) at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1557) - locked <0x00000007555cee90> (a com.mysql.jdbc.JDBC4Connection) at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.destroy(BaseWrapperManagedConnection.java:311) at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.doDestroy(InternalManagedConnectionPool.java:664) at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.returnConnection(InternalManagedConnectionPool.java:395) at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.returnConnection(JBossManagedConnectionPool.java:761) at org.jboss.resource.connectionmanager.BaseConnectionManager2.returnManagedConnection(BaseConnectionManager2.java:463) at org.jboss.resource.connectionmanager.BaseConnectionManager2$BaseConnectionEventListener.connectionErrorOccurred(BaseConnectionManager2.java:854) at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.connectionErrorOccurred(TxConnectionManager.java:792) at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.broadcastConnectionError(BaseWrapperManagedConnection.java:421) at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.connectionError(BaseWrapperManagedConnection.java:385) at org.jboss.resource.adapter.jdbc.WrappedConnection.checkException(WrappedConnection.java:865) at org.jboss.resource.adapter.jdbc.WrappedStatement.checkException(WrappedStatement.java:852) at org.jboss.resource.adapter.jdbc.WrappedResultSet.checkException(WrappedResultSet.java:1947) at org.jboss.resource.adapter.jdbc.WrappedResultSet.getString(WrappedResultSet.java:892) at org.hibernate.type.StringType.get(StringType.java:41) at org.hibernate.type.NullableType.nullSafeGet(NullableType.java:184) at org.hibernate.type.NullableType.nullSafeGet(NullableType.java:173) at org.hibernate.persister.collection.AbstractCollectionPersister.readKey(AbstractCollectionPersister.java:756) at org.hibernate.loader.Loader.readCollectionElement(Loader.java:998) at org.hibernate.loader.Loader.readCollectionElements(Loader.java:669) at org.hibernate.loader.Loader.getRowFromResultSet(Loader.java:614) at org.hibernate.loader.Loader.doQuery(Loader.java:724) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259) at org.hibernate.loader.Loader.loadCollection(Loader.java:2019) at org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:59) at org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:587) at org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:83) at org.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:1744) at org.hibernate.collection.AbstractPersistentCollection.forceInitialization(AbstractPersistentCollection.java:476) at org.hibernate.engine.StatefulPersistenceContext.initializeNonLazyCollections(StatefulPersistenceContext.java:867) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:264) at org.hibernate.loader.Loader.doList(Loader.java:2232) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2129) at org.hibernate.loader.Loader.list(Loader.java:2124) at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:401) at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:363) at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196) at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1149) at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102) at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:67) at com.zzz.ReadOnlyObjectSessionBean.getEndPointsByTimePeriod(ReadOnlyObjectSessionBean.java:3638) at sun.reflect.GeneratedMethodAccessor991.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.aop.joinpoint.MethodInvocation.invokeTarget(MethodInvocation.java:122) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) at org.jboss.ejb3.EJBContainerInvocationWrapper.invokeNext(EJBContainerInvocationWrapper.java:69) at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.invoke(InterceptorSequencer.java:76) at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.aroundInvoke(InterceptorSequencer.java:62) at sun.reflect.GeneratedMethodAccessor463.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.aop.advice.PerJoinpointAdvice.invoke(PerJoinpointAdvice.java:174) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.fillMethod(InvocationContextInterceptor.java:72) at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_fillMethod_57850553.invoke(InvocationContextInterceptor_z_fillMethod_57850553.java) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.setup(InvocationContextInterceptor.java:88) at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_setup_57850553.invoke(InvocationContextInterceptor_z_setup_57850553.java) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:62) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:56) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:68) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79) at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:190) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptorv2.invoke(RoleBasedAuthorizationInterceptorv2.java:201) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.security.Ejb3AuthenticationInterceptorv2.invoke(Ejb3AuthenticationInterceptorv2.java:186) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:41) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.BlockContainerShutdownInterceptor.invoke(BlockContainerShutdownInterceptor.java:67) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.aspects.currentinvocation.CurrentInvocationInterceptor.invoke(CurrentInvocationInterceptor.java:67) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.session.SessionSpecContainer.invoke(SessionSpecContainer.java:180) at org.jboss.ejb3.session.SessionSpecContainer.invoke(SessionSpecContainer.java:238) at org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:188) at $Proxy230.getEndPointsByTimePeriod(Unknown Source) at com.zzz.AdapterManager$ScanTask.run(AdapterManager.java:2096) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "message-outgoing queue processor": at com.mysql.jdbc.ConnectionImpl.getCharacterSetMetadata(ConnectionImpl.java:2799) - waiting to lock <0x00000007555cee90> (a com.mysql.jdbc.JDBC4Connection) at com.mysql.jdbc.Field.getStringFromBytes(Field.java:710) at com.mysql.jdbc.Field.getOriginalName(Field.java:652) at com.mysql.jdbc.ResultSetImpl.buildIndexMapping(ResultSetImpl.java:749) at com.mysql.jdbc.ResultSetImpl.findColumn(ResultSetImpl.java:1108) - locked <0x00000007c25dcc70> (a com.mysql.jdbc.JDBC4ResultSet) at com.mysql.jdbc.ResultSetImpl.getString(ResultSetImpl.java:5610) at org.jboss.resource.adapter.jdbc.WrappedResultSet.getString(WrappedResultSet.java:888) at org.hibernate.type.StringType.get(StringType.java:41) at org.hibernate.type.NullableType.nullSafeGet(NullableType.java:184) at org.hibernate.type.NullableType.nullSafeGet(NullableType.java:173) at org.hibernate.loader.Loader.getKeyFromResultSet(Loader.java:1121) at org.hibernate.loader.Loader.getRowFromResultSet(Loader.java:588) at org.hibernate.loader.Loader.doQuery(Loader.java:724) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259) at org.hibernate.loader.Loader.doList(Loader.java:2232) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2129) at org.hibernate.loader.Loader.list(Loader.java:2124) at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:401) at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:363) at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196) at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1149) at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102) at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:67) at com.zzz.ReadOnlyObjectSessionBean.getObjectByName(ReadOnlyObjectSessionBean.java:206) at sun.reflect.GeneratedMethodAccessor1066.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.aop.joinpoint.MethodInvocation.invokeTarget(MethodInvocation.java:122) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) at org.jboss.ejb3.EJBContainerInvocationWrapper.invokeNext(EJBContainerInvocationWrapper.java:69) at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.invoke(InterceptorSequencer.java:76) at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.aroundInvoke(InterceptorSequencer.java:62) at sun.reflect.GeneratedMethodAccessor463.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.aop.advice.PerJoinpointAdvice.invoke(PerJoinpointAdvice.java:174) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.fillMethod(InvocationContextInterceptor.java:72) at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_fillMethod_57850553.invoke(InvocationContextInterceptor_z_fillMethod_57850553.java) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.setup(InvocationContextInterceptor.java:88) at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_setup_57850553.invoke(InvocationContextInterceptor_z_setup_57850553.java) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:62) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:56) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:68) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79) at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:190) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptorv2.invoke(RoleBasedAuthorizationInterceptorv2.java:201) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.security.Ejb3AuthenticationInterceptorv2.invoke(Ejb3AuthenticationInterceptorv2.java:186) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:41) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.BlockContainerShutdownInterceptor.invoke(BlockContainerShutdownInterceptor.java:67) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.aspects.currentinvocation.CurrentInvocationInterceptor.invoke(CurrentInvocationInterceptor.java:67) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.ejb3.session.SessionSpecContainer.invoke(SessionSpecContainer.java:180) at org.jboss.ejb3.session.SessionSpecContainer.invoke(SessionSpecContainer.java:238) at org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:188) at $Proxy230.getObjectByName(Unknown Source) at com.zzz.SnmpDiscovery.start(SnmpDiscovery.java:83) at com.zzz.DiscoveryManager.DiscoveryTask(DiscoveryManager.java:325) at com.zzz.ServerManager.processMessage(ServerManager.java:2894) at com.zzz.ServerManager.access$700(ServerManager.java:90) at com.zzz.ServerManager$2.run(ServerManager.java:346) at java.lang.Thread.run(Thread.java:662) Found 1 deadlock.