3 JMS Bridges - 2 reconnect ok, 1 is dead ?
tpaterson0512 May 20, 2010 12:43 PMWe have several machines running 3 JMS Bridge Impls each..
these are all connecting to the same central JBoss 5/HornetQ server,
and collecting messages from Topic(s) across a WAN.
(all machines running same versions of JBoss/HornetQ)
we have noticed that if there are temporary connection issues,
or we restart Jboss on the central server - the bridges do not always successfully reconnect..
on one site - we have tried restarting the bridge concerned using the JMX console,
simply invoking stop() followed by start().
but this did not help...
upon examining the server.log - we noticed that the bridge concerned had stopped giving us any
log messages more than 24 hours ago (the other 2 bridges continued to give us messages).
and looking at the thread dump from the JMX console, we could see that this bridge seems to have only
1 active thread, whereas the 2 working bridges have 3 threads each..
Details for the failed bridge
$ grep pool-26-thread server.log
2010-05-19 11:46:52,031 WARNING [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-26-thread-1) Will retry after a pause of 50000 ms
2010-05-19 11:47:42,243 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:47:42,245 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:47:42,247 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-26-thread-1) Failed to set up JMS bridge connections. Most probably the source or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:48:32,568 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:48:32,570 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:48:32,572 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-26-thread-1) Failed to set up JMS bridge connections. Most probably the source or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:49:22,778 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:49:22,781 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:49:22,783 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-26-thread-1) Failed to set up JMS bridge connections. Most probably the source or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:50:12,981 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:50:12,983 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:50:12,985 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-26-thread-1) Failed to set up JMS bridge connections. Most probably the source or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:51:03,192 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:51:03,195 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:51:03,197 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-26-thread-1) Failed to set up JMS bridge connections. Most probably the source or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:51:53,410 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:51:53,413 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:51:53,415 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-26-thread-1) Failed to set up JMS bridge connections. Most probably the source or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:52:43,423 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-26-thread-1) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port: 1199, localAddr: null, localPort: 0, timeout: 0
output from ThreadDump via JMX console :
Thread: pool-26-thread-1 : priority:5, demon:false, threadId:7037, threadState:RUNNABLE
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0x1801c1e> (a java.io.BufferedInputStream)
java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2264)
java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2277)
java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2748)
java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:780)
java.io.ObjectInputStream.(ObjectInputStream.java:280)
org.jnp.interfaces.NamingContext.getServer(NamingContext.java:312)
org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1734)
org.jnp.interfaces.NamingContext.lookup(NamingContext.java:693)
org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686)
javax.naming.InitialContext.lookup(InitialContext.java:392)
org.hornetq.jms.bridge.impl.JNDIFactorySupport.createObject(JNDIFactorySupport.java:58)
org.hornetq.jms.bridge.impl.JNDIDestinationFactory.createDestination(JNDIDestinationFactory.java:40)
org.hornetq.jms.bridge.impl.JMSBridgeImpl.setupJMSObjects(JMSBridgeImpl.java:1082)
org.hornetq.jms.bridge.impl.JMSBridgeImpl.setupJMSObjectsWithRetry(JMSBridgeImpl.java:1342)
org.hornetq.jms.bridge.impl.JMSBridgeImpl.access$2100(JMSBridgeImpl.java:68)
org.hornetq.jms.bridge.impl.JMSBridgeImpl$FailureHandler.run(JMSBridgeImpl.java:1858)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:619)
Locked synchronizers :
- locked <0x1b62480> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
Details for the working bridges
$ grep pool-5-thread server.log
2010-05-19 11:45:19,733 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:45:19,756 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:45:19,761 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:46:10,136 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:46:10,146 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:46:10,148 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:47:00,156 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-5-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
1199, localAddr: null, localPort: 0, timeout: 0
2010-05-19 11:47:00,589 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-5-thread-3) createSocket, hostAddr: pinpro/192.168.1.10, port: 1099, localAddr: n
ull, localPort: 0, timeout: 0
2010-05-19 11:47:00,665 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:47:00,667 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:47:00,669 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:47:50,880 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:47:50,882 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:47:50,885 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:48:41,107 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:48:41,109 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:48:41,111 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:49:31,324 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:49:31,326 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:49:31,328 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:50:21,540 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:50:21,542 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:50:21,544 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:51:11,771 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:51:11,774 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:51:11,776 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:52:01,989 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:52:01,990 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:52:01,993 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:52:51,997 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-5-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
1199, localAddr: null, localPort: 0, timeout: 0
2010-05-19 11:52:52,473 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-5-thread-3) createSocket, hostAddr: pinpro/192.168.1.10, port: 1099, localAddr: n
ull, localPort: 0, timeout: 0
2010-05-19 11:52:52,617 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:52:52,619 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:52:52,622 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:53:42,933 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:53:42,935 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:53:42,937 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:54:33,488 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:54:33,491 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:54:33,493 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
<snip>
2010-05-19 15:00:13,866 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-2) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 15:01:03,964 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-2) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 15:01:53,971 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-5-thread-2) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
1199, localAddr: null, localPort: 0, timeout: 0
2010-05-19 15:01:54,087 DEBUG [org.jnp.interfaces.NamingContext] (pool-5-thread-2) Failed to connect to pinnet.actual-systems.com:1199
2010-05-19 15:01:59,091 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-2) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 15:02:49,123 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-5-thread-2) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
1199, localAddr: null, localPort: 0, timeout: 0
2010-05-19 15:02:50,947 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-5-thread-2) createSocket, hostAddr: pinpro/192.168.1.10, port: 1099, localAddr: n
ull, localPort: 0, timeout: 0
2010-05-19 15:02:51,137 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-2) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 15:02:51,463 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-2) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 15:03:51,314 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-2) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 15:04:42,023 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-2) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 15:04:42,540 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-2) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 15:04:43,021 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-2) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 15:05:33,091 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-5-thread-2) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
1199, localAddr: null, localPort: 0, timeout: 0
2010-05-19 15:05:34,779 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-5-thread-2) createSocket, hostAddr: pinpro/192.168.1.10, port: 1099, localAddr: n
ull, localPort: 0, timeout: 0
2010-05-19 15:05:34,973 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-2) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 15:05:35,163 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-2) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 15:05:35,840 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-2) Succeeded in reconnecting to servers
2010-05-19 15:12:22,350 ERROR [org.hornetq.core.client.impl.ClientSessionImpl] (pool-5-thread-3) Caught jmsexecptione
2010-05-19 15:12:22,350 WARN [com.arjuna.ats.jta.logging.loggerI18N] (pool-5-thread-3) [com.arjuna.ats.internal.jta.transaction.arjunacore.delistresource] [
com.arjuna.ats.internal.jta.transaction.arjunacore.delistresource] TransactionImple.delistResource - caught exception during delist : XAException.XAER_RMERR
2010-05-19 15:12:22,351 ERROR [org.hornetq.core.client.impl.ClientSessionImpl] (pool-5-thread-3) Caught jmsexecptione
2010-05-19 15:12:22,351 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (pool-5-thread-3) [com.arjuna.ats.arjuna.coordinator.BasicAction_54] - Top-level
abort of action -3f57fef6:acf1:4bf1815e:3d0b7 received TwoPhaseOutcome.FINISH_ERROR from <ClassName:RecordType.JTA_RECORD>
2010-05-19 15:12:22,352 WARNING [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Will retry after a pause of 50000 ms
2010-05-19 15:13:12,357 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-5-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
1199, localAddr: null, localPort: 0, timeout: 0
2010-05-19 15:13:13,290 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 15:14:03,841 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-5-thread-3) createSocket, hostAddr: pinpro/192.168.1.10, port: 1099, localAddr: n
ull, localPort: 0, timeout: 0
2010-05-19 15:14:04,519 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 15:14:04,732 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 15:14:05,396 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Succeeded in reconnecting to servers
thread info from jmx console :
Thread: pool-5-thread-1 : priority:5, demon:false, threadId:83, threadState:TIMED_WAITING
- waiting on <0xc7b3f4> (a org.hornetq.core.client.impl.ClientConsumerImpl)
java.lang.Object.wait(Native Method)
org.hornetq.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:238)
org.hornetq.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:322)
org.hornetq.jms.client.HornetQMessageConsumer.getMessage(HornetQMessageConsumer.java:224)
org.hornetq.jms.client.HornetQMessageConsumer.receive(HornetQMessageConsumer.java:133)
org.hornetq.jms.bridge.impl.JMSBridgeImpl$SourceReceiver.run(JMSBridgeImpl.java:1734)
- locked <0x1794a14> (a java.lang.Object)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:619)
Locked synchronizers :
- locked <0x7dfa7b> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
Thread: pool-5-thread-2 : priority:5, demon:false, threadId:159, threadState:WAITING
- waiting on <0xc4cce> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
java.lang.Thread.run(Thread.java:619)
Thread: pool-5-thread-3 : priority:5, demon:false, threadId:201, threadState:WAITING
- waiting on <0xc4cce> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
java.lang.Thread.run(Thread.java:619)
$ grep pool-3-thread server.log
2010-05-19 11:45:42,020 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:45:42,022 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:45:42,027 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:46:32,276 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:46:32,278 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:46:32,280 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:47:22,517 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:47:22,519 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:47:22,521 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:48:12,777 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:48:12,779 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:48:12,782 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:49:03,427 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:49:03,429 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:49:03,431 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:49:53,470 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
1199, localAddr: null, localPort: 0, timeout: 0
2010-05-19 11:49:53,884 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinpro/192.168.1.10, port: 1099, localAddr: n
ull, localPort: 0, timeout: 0
2010-05-19 11:49:53,972 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:49:53,974 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:49:53,977 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:50:44,299 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:50:44,301 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:50:44,303 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:51:34,640 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:51:34,642 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:51:34,644 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:52:25,026 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:52:25,046 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:52:25,048 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:53:15,302 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:53:15,304 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:53:15,307 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:54:05,566 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:54:05,568 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:54:05,570 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:54:55,807 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:54:55,810 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 11:54:55,812 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 11:55:45,857 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
1199, localAddr: null, localPort: 0, timeout: 0
2010-05-19 11:55:46,277 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinpro/192.168.1.10, port: 1099, localAddr: n
ull, localPort: 0, timeout: 0
<snip>
2010-05-19 12:05:11,877 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 12:05:11,879 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 12:28:49,821 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 12:29:39,887 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
1199, localAddr: null, localPort: 0, timeout: 0
2010-05-19 12:29:41,522 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinpro/192.168.1.10, port: 1099, localAddr: n
ull, localPort: 0, timeout: 0
2010-05-19 12:29:41,639 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 12:29:41,781 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 12:29:42,240 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Succeeded in reconnecting to servers
2010-05-19 14:57:36,821 ERROR [org.hornetq.core.client.impl.ClientSessionImpl] (pool-3-thread-3) Caught jmsexecptione
2010-05-19 14:57:36,821 WARN [com.arjuna.ats.jta.logging.loggerI18N] (pool-3-thread-3) [com.arjuna.ats.internal.jta.transaction.arjunacore.delistresource] [
com.arjuna.ats.internal.jta.transaction.arjunacore.delistresource] TransactionImple.delistResource - caught exception during delist : XAException.XAER_RMERR
2010-05-19 14:57:36,821 ERROR [org.hornetq.core.client.impl.ClientSessionImpl] (pool-3-thread-3) Caught jmsexecptione
2010-05-19 14:57:36,822 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (pool-3-thread-3) [com.arjuna.ats.arjuna.coordinator.BasicAction_54] - Top-level
abort of action -3f57fef6:acf1:4bf1815e:36354 received TwoPhaseOutcome.FINISH_ERROR from <ClassName:RecordType.JTA_RECORD>
2010-05-19 14:57:36,823 WARNING [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Will retry after a pause of 50000 ms
2010-05-19 14:58:26,828 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
1199, localAddr: null, localPort: 0, timeout: 0
2010-05-19 14:58:26,948 DEBUG [org.jnp.interfaces.NamingContext] (pool-3-thread-3) Failed to connect to pinnet.actual-systems.com:1199
2010-05-19 14:58:31,952 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 14:59:21,960 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
1199, localAddr: null, localPort: 0, timeout: 0
2010-05-19 14:59:23,117 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 15:00:13,122 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
1199, localAddr: null, localPort: 0, timeout: 0
2010-05-19 15:00:13,773 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 15:01:03,929 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 15:01:53,932 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
1199, localAddr: null, localPort: 0, timeout: 0
2010-05-19 15:01:54,048 DEBUG [org.jnp.interfaces.NamingContext] (pool-3-thread-3) Failed to connect to pinnet.actual-systems.com:1199
2010-05-19 15:01:59,049 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 15:02:49,123 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
1199, localAddr: null, localPort: 0, timeout: 0
2010-05-19 15:02:51,263 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 15:02:51,510 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 15:02:53,660 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Succeeded in reconnecting to servers
2010-05-19 15:12:22,877 ERROR [org.hornetq.core.client.impl.ClientSessionImpl] (pool-3-thread-3) Caught jmsexecptione
2010-05-19 15:12:22,878 WARN [com.arjuna.ats.jta.logging.loggerI18N] (pool-3-thread-3) [com.arjuna.ats.internal.jta.transaction.arjunacore.delistresource] [
com.arjuna.ats.internal.jta.transaction.arjunacore.delistresource] TransactionImple.delistResource - caught exception during delist : XAException.XAER_RMERR
2010-05-19 15:12:22,878 ERROR [org.hornetq.core.client.impl.ClientSessionImpl] (pool-3-thread-3) Caught jmsexecptione
2010-05-19 15:12:22,878 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (pool-3-thread-3) [com.arjuna.ats.arjuna.coordinator.BasicAction_54] - Top-level
abort of action -3f57fef6:acf1:4bf1815e:3d02e received TwoPhaseOutcome.FINISH_ERROR from <ClassName:RecordType.JTA_RECORD>
2010-05-19 15:12:22,880 WARNING [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Will retry after a pause of 50000 ms
2010-05-19 15:13:12,889 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
1199, localAddr: null, localPort: 0, timeout: 0
2010-05-19 15:13:13,464 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
or target servers are unavailable. Will retry after a pause of 50000 ms
2010-05-19 15:14:04,162 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 15:14:04,364 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
2010-05-19 15:14:05,067 INFO [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Succeeded in reconnecting to servers
thread info from jmx console :
Thread: pool-3-thread-1 : priority:5, demon:false, threadId:69, threadState:TIMED_WAITING
- waiting on <0x81f841> (a java.lang.Object)
java.lang.Object.wait(Native Method)
org.hornetq.jms.bridge.impl.JMSBridgeImpl$BatchTimeChecker.run(JMSBridgeImpl.java:1956)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:619)
Locked synchronizers :
- locked <0x9f1a23> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
Thread: pool-3-thread-2 : priority:5, demon:false, threadId:71, threadState:TIMED_WAITING
- waiting on <0xe1e4f2> (a org.hornetq.core.client.impl.ClientConsumerImpl)
java.lang.Object.wait(Native Method)
org.hornetq.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:238)
org.hornetq.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:322)
org.hornetq.jms.client.HornetQMessageConsumer.getMessage(HornetQMessageConsumer.java:224)
org.hornetq.jms.client.HornetQMessageConsumer.receive(HornetQMessageConsumer.java:133)
org.hornetq.jms.bridge.impl.JMSBridgeImpl$SourceReceiver.run(JMSBridgeImpl.java:1734)
- locked <0x81f841> (a java.lang.Object)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:619)
Locked synchronizers :
- locked <0x16cf754> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
Thread: pool-3-thread-3 : priority:5, demon:false, threadId:155, threadState:WAITING
- waiting on <0x967733> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
java.lang.Thread.run(Thread.java:619)
any thoughts ?
all of the above using a build from TRUNK
-
jms-bridge-jboss-beans.xml 12.9 KB
-
HtmlAdaptor.htm.zip 9.6 KB
-
server.log.2010-05-19.gz 2.3 MB