Bridge Status inconsistent when JDBC Connections are recreat

[rogerdudler]

We are using JBoss Messaging 1.4.0.GA on JBoss 4.0.5 with all the prerequisites needed by JBoss Messaging. Everything works fine, except some issues on the messaging bridge, when the underlying database connections became invalid.

The source destination of the bridge is located on a remote messaging server (also JBoss Messaging 1.4.0.GA, but on JBoss 4.2.2.GA) and the destination exists locally.

If a JDBC connection becomes invalid (in the following example, during the messaging reaper is running) and the pool creates a new one and the reaper updates properly, but after this, it seems that the Messaging Bridge has an inconsistent state. JMX Console shows that bridge is "Started", but it does not consume the remote queue anymore. After invoking "stop" and "start" on the bridge (JMX Console), the bridge begins to work as expected.

Any ideas?

.... before, there are connections which are unavailable, while reaper is running...

<->:<anonym>: 2007-12-05 09:23:33,937 [Timer-3] WARN jboss.messaging.core.impl.JDBCSupport - SQLException caught, SQLState null code:0- assuming deadlock detected, try:22
org.jboss.util.NestedSQLException: Could not create connection; - nested throwable: (com.ibm.db2.jcc.c.SqlException: DB2 SQL error: SQLCODE: -1035, SQLSTATE: 57019, SQLERRMC: null); - nested throwable: (org.jboss.resource.JBossResourceException: Could not create connection; - nested throwable: (com.ibm.db2.jcc.c.SqlException: DB2 SQL error: SQLCODE: -1035, SQLSTATE: 57019, SQLERRMC: null))
 at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:94)
 at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner.execute(JDBCSupport.java:383)
 at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner.executeWithRetry(JDBCSupport.java:407)
 at org.jboss.messaging.core.impl.JDBCPersistenceManager.reapUnreferencedMessages(JDBCPersistenceManager.java:2357)
 at org.jboss.messaging.core.impl.JDBCPersistenceManager.access$1000(JDBCPersistenceManager.java:78)
 at org.jboss.messaging.core.impl.JDBCPersistenceManager$Reaper.run(JDBCPersistenceManager.java:2381)
 at java.util.TimerThread.mainLoop(Timer.java:512)
 at java.util.TimerThread.run(Timer.java:462)
Caused by: org.jboss.resource.JBossResourceException: Could not create connection; - nested throwable: (com.ibm.db2.jcc.c.SqlException: DB2 SQL error: SQLCODE: -1035, SQLSTATE: 57019, SQLERRMC: null)
 at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:179)
 at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.createConnectionEventListener(InternalManagedConnectionPool.java:565)
 at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:250)
 at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:529)
 at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:341)
 at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:301)
 at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:396)
 at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
 at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:88)
 ... 7 more
Caused by: com.ibm.db2.jcc.c.SqlException: DB2 SQL error: SQLCODE: -1035, SQLSTATE: 57019, SQLERRMC: null
 at com.ibm.db2.jcc.c.p.a(p.java:3493)
 at com.ibm.db2.jcc.b.hb.q(hb.java:709)
 at com.ibm.db2.jcc.b.hb.p(hb.java:583)
 at com.ibm.db2.jcc.b.hb.l(hb.java:408)
 at com.ibm.db2.jcc.b.hb.d(hb.java:147)
 at com.ibm.db2.jcc.b.b.cc(b.java:1418)
 at com.ibm.db2.jcc.b.b.a(b.java:1342)
 at com.ibm.db2.jcc.b.b.s(b.java:1040)
 at com.ibm.db2.jcc.b.b.b(b.java:746)
 at com.ibm.db2.jcc.b.b.a(b.java:732)
 at com.ibm.db2.jcc.b.b.<init>(b.java:333)
 at com.ibm.db2.jcc.DB2Driver.connect(DB2Driver.java:165)
 at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:171)
 ... 15 more
<->:<anonym>: 2007-12-05 09:23:33,937 [Timer-3] WARN jboss.messaging.core.impl.JDBCSupport - Trying again after a pause
<->:<anonym>: 2007-12-05 09:23:34,583 [Timer-3] WARN jboss.messaging.core.impl.JDBCSupport - Update worked after retry

[goldi]

Using jboss-messaging-1.4.0.SP3 and jboss-4.2.1.GA I made a test forcing the connection to the database to be broken for a while and then restart the connection.
The result is that the bridge disconnects from remote server if the connection to the database is broken and then tries to reconnect. Thats fine. The problem is, that with each reconnect messages are consumed but not persisted anymore, even if the connection to the database is reastablished. The next thing is that the bridge does not stop to disconnect and reconnect after reastablishing the connection to the database.
I did the test twice, in the first test I was able to restart the bridge as rogerdudler described. In the second test I wasn't able to do that.

[goldi]

For Oracle10 I found a resolution to this problem. It is not a JBM problem, it is a datasource or better to say connectionpool problem. After the network to the database was broken JBM gets an invalid connection to the database. To solve this I added:

<!-- Uses the pingDatabase method to check a connection is still valid before handing it out from the pool -->
<valid-connection-checker-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleValidConnectionChecker</valid-connection-checker-class-name>

to my datasource.
Now I did the test with Oracle 9 and another Exception occurred:

2008-09-09 13:09:37,241 WARN [org.jboss.messaging.core.impl.JDBCSupport] SQLException caught, SQLState null code:17002- assuming deadlock detected, try:6
java.sql.SQLException: E/A-Exception: Connection reset by peer: socket write error
at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:180)
at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:222)
at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:335)
at oracle.jdbc.driver.OracleConnection.setAutoCommit(OracleConnection.java:1000)
at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.checkTransaction(BaseWrapperManagedConnection.java:429)
at org.jboss.resource.adapter.jdbc.WrappedConnection.checkTransaction(WrappedConnection.java:525)
at org.jboss.resource.adapter.jdbc.WrappedConnection.prepareStatement(WrappedConnection.java:184)
at org.jboss.messaging.core.impl.JDBCPersistenceManager$1AddReferenceRunner.doTransaction(JDBCPersistenceManager.java:1300)
at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner2.execute(JDBCSupport.java:465)
at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner2.executeWithRetry(JDBCSupport.java:503)
at org.jboss.messaging.core.impl.JDBCPersistenceManager.addReference(JDBCPersistenceManager.java:1362)
at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:226)
at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2203)
at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:741)
at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734)
at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
at org.jboss.remoting.Client.invoke(Client.java:1550)
at org.jboss.remoting.Client.invoke(Client.java:530)
at org.jboss.remoting.Client.invoke(Client.java:518)
at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:477)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:632)
at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:278)
at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect39.invoke(ProducerAspect39.java)
at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
at org.jboss.jms.server.bridge.Bridge.sendMessages(Bridge.java:1339)
at org.jboss.jms.server.bridge.Bridge.sendBatchNonTransacted(Bridge.java:1190)
at org.jboss.jms.server.bridge.Bridge.sendBatch(Bridge.java:1304)
at org.jboss.jms.server.bridge.Bridge.access$1800(Bridge.java:66)
at org.jboss.jms.server.bridge.Bridge$SourceListener.onMessage(Bridge.java:1632)
at org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:854)
at org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.java:99)
at org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:166)
at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:117)
at org.apache.activemq.thread.PooledTaskRunner.access$100(PooledTaskRunner.java:26)
at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:44)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690)
at java.lang.Thread.run(Thread.java:595)

but it shows the same behaviour as describe before. For this new problem my resolution does not work.

[pasanperera]

Warum's solution works for me with SQL server 2008.

Anyway for anyone who has an issue with dead connections refer http://docs.jboss.org/jbossas/6/JCA_Guide/en-US/html_single/index.html#d0e1768. Combination of idle connections & dead connections will give a solution. Try it and see.