9 Replies Latest reply on May 21, 2010 12:06 PM by artp

    3 JMS Bridges - 2 reconnect ok, 1 is dead ?

      We have several machines running 3 JMS Bridge Impls each..

       

      these are all connecting to the same central JBoss 5/HornetQ server,

       

      and collecting messages from Topic(s) across a WAN.

       

      (all machines running same versions of JBoss/HornetQ)

       

       

      we have noticed that if there are temporary connection issues,
      or we restart Jboss on the central server - the bridges do not always successfully reconnect..

       


      on one site - we have tried restarting the bridge concerned using the JMX console,
      simply invoking stop() followed by start().

       

      but this did not help...

       


      upon examining the server.log - we noticed that the bridge concerned had stopped giving us any
      log messages more than 24 hours ago (the other 2 bridges continued to give us messages).

       

      and looking at the thread dump from the JMX console, we could see that this bridge seems to have only
      1 active thread, whereas the 2 working bridges have 3 threads each..

       

       

      Details for the failed bridge


      $ grep pool-26-thread server.log



      2010-05-19 11:46:52,031 WARNING [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-26-thread-1) Will retry after a pause of 50000 ms
      2010-05-19 11:47:42,243 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:47:42,245 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:47:42,247 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-26-thread-1) Failed to set up JMS bridge connections. Most probably the source or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:48:32,568 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:48:32,570 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:48:32,572 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-26-thread-1) Failed to set up JMS bridge connections. Most probably the source or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:49:22,778 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:49:22,781 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:49:22,783 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-26-thread-1) Failed to set up JMS bridge connections. Most probably the source or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:50:12,981 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:50:12,983 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:50:12,985 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-26-thread-1) Failed to set up JMS bridge connections. Most probably the source or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:51:03,192 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:51:03,195 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:51:03,197 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-26-thread-1) Failed to set up JMS bridge connections. Most probably the source or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:51:53,410 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:51:53,413 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-26-thread-1) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:51:53,415 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-26-thread-1) Failed to set up JMS bridge connections. Most probably the source or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:52:43,423 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-26-thread-1) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port: 1199, localAddr: null, localPort: 0, timeout: 0

       


      output from ThreadDump via JMX console :

       


      Thread: pool-26-thread-1 : priority:5, demon:false, threadId:7037, threadState:RUNNABLE

       

          java.net.SocketInputStream.socketRead0(Native Method)
          java.net.SocketInputStream.read(SocketInputStream.java:129)
          java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
          java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
          java.io.BufferedInputStream.read(BufferedInputStream.java:317)
          - locked <0x1801c1e> (a java.io.BufferedInputStream)
          java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2264)
          java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2277)
          java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2748)
          java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:780)
          java.io.ObjectInputStream.(ObjectInputStream.java:280)
          org.jnp.interfaces.NamingContext.getServer(NamingContext.java:312)
          org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1734)
          org.jnp.interfaces.NamingContext.lookup(NamingContext.java:693)
          org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686)
          javax.naming.InitialContext.lookup(InitialContext.java:392)
          org.hornetq.jms.bridge.impl.JNDIFactorySupport.createObject(JNDIFactorySupport.java:58)
          org.hornetq.jms.bridge.impl.JNDIDestinationFactory.createDestination(JNDIDestinationFactory.java:40)
          org.hornetq.jms.bridge.impl.JMSBridgeImpl.setupJMSObjects(JMSBridgeImpl.java:1082)
          org.hornetq.jms.bridge.impl.JMSBridgeImpl.setupJMSObjectsWithRetry(JMSBridgeImpl.java:1342)
          org.hornetq.jms.bridge.impl.JMSBridgeImpl.access$2100(JMSBridgeImpl.java:68)
          org.hornetq.jms.bridge.impl.JMSBridgeImpl$FailureHandler.run(JMSBridgeImpl.java:1858)
          java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
          java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
          java.lang.Thread.run(Thread.java:619)

       

          Locked synchronizers :
          - locked <0x1b62480> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

       

       

       

      Details  for the working bridges


      $ grep pool-5-thread server.log

       

      2010-05-19 11:45:19,733 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:45:19,756 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:45:19,761 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:46:10,136 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:46:10,146 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:46:10,148 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:47:00,156 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-5-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
      1199, localAddr: null, localPort: 0, timeout: 0
      2010-05-19 11:47:00,589 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-5-thread-3) createSocket, hostAddr: pinpro/192.168.1.10, port: 1099, localAddr: n
      ull, localPort: 0, timeout: 0
      2010-05-19 11:47:00,665 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:47:00,667 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:47:00,669 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:47:50,880 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:47:50,882 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:47:50,885 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:48:41,107 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:48:41,109 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:48:41,111 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:49:31,324 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:49:31,326 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:49:31,328 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:50:21,540 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:50:21,542 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:50:21,544 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:51:11,771 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:51:11,774 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:51:11,776 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:52:01,989 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:52:01,990 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:52:01,993 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:52:51,997 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-5-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
      1199, localAddr: null, localPort: 0, timeout: 0
      2010-05-19 11:52:52,473 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-5-thread-3) createSocket, hostAddr: pinpro/192.168.1.10, port: 1099, localAddr: n
      ull, localPort: 0, timeout: 0
      2010-05-19 11:52:52,617 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:52:52,619 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:52:52,622 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:53:42,933 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:53:42,935 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:53:42,937 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:54:33,488 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:54:33,491 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:54:33,493 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms


      <snip>


      2010-05-19 15:00:13,866 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-2) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 15:01:03,964 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-2) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 15:01:53,971 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-5-thread-2) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
      1199, localAddr: null, localPort: 0, timeout: 0
      2010-05-19 15:01:54,087 DEBUG [org.jnp.interfaces.NamingContext] (pool-5-thread-2) Failed to connect to pinnet.actual-systems.com:1199
      2010-05-19 15:01:59,091 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-2) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 15:02:49,123 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-5-thread-2) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
      1199, localAddr: null, localPort: 0, timeout: 0
      2010-05-19 15:02:50,947 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-5-thread-2) createSocket, hostAddr: pinpro/192.168.1.10, port: 1099, localAddr: n
      ull, localPort: 0, timeout: 0
      2010-05-19 15:02:51,137 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-2) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 15:02:51,463 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-2) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 15:03:51,314 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-2) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 15:04:42,023 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-2) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 15:04:42,540 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-2) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 15:04:43,021 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-2) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 15:05:33,091 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-5-thread-2) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
      1199, localAddr: null, localPort: 0, timeout: 0
      2010-05-19 15:05:34,779 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-5-thread-2) createSocket, hostAddr: pinpro/192.168.1.10, port: 1099, localAddr: n
      ull, localPort: 0, timeout: 0
      2010-05-19 15:05:34,973 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-2) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 15:05:35,163 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-2) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 15:05:35,840 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-2) Succeeded in reconnecting to servers
      2010-05-19 15:12:22,350 ERROR [org.hornetq.core.client.impl.ClientSessionImpl] (pool-5-thread-3) Caught jmsexecptione
      2010-05-19 15:12:22,350 WARN  [com.arjuna.ats.jta.logging.loggerI18N] (pool-5-thread-3) [com.arjuna.ats.internal.jta.transaction.arjunacore.delistresource] [
      com.arjuna.ats.internal.jta.transaction.arjunacore.delistresource] TransactionImple.delistResource - caught exception during delist : XAException.XAER_RMERR
      2010-05-19 15:12:22,351 ERROR [org.hornetq.core.client.impl.ClientSessionImpl] (pool-5-thread-3) Caught jmsexecptione
      2010-05-19 15:12:22,351 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (pool-5-thread-3) [com.arjuna.ats.arjuna.coordinator.BasicAction_54] - Top-level
      abort of action -3f57fef6:acf1:4bf1815e:3d0b7 received TwoPhaseOutcome.FINISH_ERROR from <ClassName:RecordType.JTA_RECORD>
      2010-05-19 15:12:22,352 WARNING [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Will retry after a pause of 50000 ms
      2010-05-19 15:13:12,357 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-5-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
      1199, localAddr: null, localPort: 0, timeout: 0
      2010-05-19 15:13:13,290 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 15:14:03,841 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-5-thread-3) createSocket, hostAddr: pinpro/192.168.1.10, port: 1099, localAddr: n
      ull, localPort: 0, timeout: 0
      2010-05-19 15:14:04,519 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 15:14:04,732 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-5-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 15:14:05,396 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-5-thread-3) Succeeded in reconnecting to servers



      thread info from jmx console :



      Thread: pool-5-thread-1 : priority:5, demon:false, threadId:83, threadState:TIMED_WAITING


          - waiting on <0xc7b3f4> (a org.hornetq.core.client.impl.ClientConsumerImpl)
          java.lang.Object.wait(Native Method)
          org.hornetq.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:238)
          org.hornetq.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:322)
          org.hornetq.jms.client.HornetQMessageConsumer.getMessage(HornetQMessageConsumer.java:224)
          org.hornetq.jms.client.HornetQMessageConsumer.receive(HornetQMessageConsumer.java:133)
          org.hornetq.jms.bridge.impl.JMSBridgeImpl$SourceReceiver.run(JMSBridgeImpl.java:1734)
          - locked <0x1794a14> (a java.lang.Object)
          java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
          java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
          java.lang.Thread.run(Thread.java:619)


          Locked synchronizers :
          - locked <0x7dfa7b> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)



      Thread: pool-5-thread-2 : priority:5, demon:false, threadId:159, threadState:WAITING


          - waiting on <0xc4cce> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
          sun.misc.Unsafe.park(Native Method)
          java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
          java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
          java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
          java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
          java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
          java.lang.Thread.run(Thread.java:619)



      Thread: pool-5-thread-3 : priority:5, demon:false, threadId:201, threadState:WAITING


          - waiting on <0xc4cce> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
          sun.misc.Unsafe.park(Native Method)
          java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
          java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
          java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
          java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
          java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
          java.lang.Thread.run(Thread.java:619)






      $ grep pool-3-thread server.log


      2010-05-19 11:45:42,020 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:45:42,022 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:45:42,027 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:46:32,276 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:46:32,278 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:46:32,280 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:47:22,517 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:47:22,519 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:47:22,521 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:48:12,777 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:48:12,779 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:48:12,782 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:49:03,427 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:49:03,429 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:49:03,431 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:49:53,470 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
      1199, localAddr: null, localPort: 0, timeout: 0
      2010-05-19 11:49:53,884 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinpro/192.168.1.10, port: 1099, localAddr: n
      ull, localPort: 0, timeout: 0
      2010-05-19 11:49:53,972 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:49:53,974 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:49:53,977 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:50:44,299 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:50:44,301 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:50:44,303 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:51:34,640 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:51:34,642 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:51:34,644 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:52:25,026 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:52:25,046 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:52:25,048 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:53:15,302 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:53:15,304 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:53:15,307 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:54:05,566 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:54:05,568 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:54:05,570 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:54:55,807 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:54:55,810 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 11:54:55,812 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 11:55:45,857 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
      1199, localAddr: null, localPort: 0, timeout: 0
      2010-05-19 11:55:46,277 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinpro/192.168.1.10, port: 1099, localAddr: n
      ull, localPort: 0, timeout: 0


      <snip>


      2010-05-19 12:05:11,877 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 12:05:11,879 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 12:28:49,821 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 12:29:39,887 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
      1199, localAddr: null, localPort: 0, timeout: 0
      2010-05-19 12:29:41,522 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinpro/192.168.1.10, port: 1099, localAddr: n
      ull, localPort: 0, timeout: 0
      2010-05-19 12:29:41,639 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 12:29:41,781 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 12:29:42,240 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Succeeded in reconnecting to servers
      2010-05-19 14:57:36,821 ERROR [org.hornetq.core.client.impl.ClientSessionImpl] (pool-3-thread-3) Caught jmsexecptione
      2010-05-19 14:57:36,821 WARN  [com.arjuna.ats.jta.logging.loggerI18N] (pool-3-thread-3) [com.arjuna.ats.internal.jta.transaction.arjunacore.delistresource] [
      com.arjuna.ats.internal.jta.transaction.arjunacore.delistresource] TransactionImple.delistResource - caught exception during delist : XAException.XAER_RMERR
      2010-05-19 14:57:36,821 ERROR [org.hornetq.core.client.impl.ClientSessionImpl] (pool-3-thread-3) Caught jmsexecptione
      2010-05-19 14:57:36,822 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (pool-3-thread-3) [com.arjuna.ats.arjuna.coordinator.BasicAction_54] - Top-level
      abort of action -3f57fef6:acf1:4bf1815e:36354 received TwoPhaseOutcome.FINISH_ERROR from <ClassName:RecordType.JTA_RECORD>
      2010-05-19 14:57:36,823 WARNING [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Will retry after a pause of 50000 ms
      2010-05-19 14:58:26,828 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
      1199, localAddr: null, localPort: 0, timeout: 0
      2010-05-19 14:58:26,948 DEBUG [org.jnp.interfaces.NamingContext] (pool-3-thread-3) Failed to connect to pinnet.actual-systems.com:1199
      2010-05-19 14:58:31,952 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 14:59:21,960 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
      1199, localAddr: null, localPort: 0, timeout: 0
      2010-05-19 14:59:23,117 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 15:00:13,122 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
      1199, localAddr: null, localPort: 0, timeout: 0
      2010-05-19 15:00:13,773 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 15:01:03,929 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 15:01:53,932 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
      1199, localAddr: null, localPort: 0, timeout: 0
      2010-05-19 15:01:54,048 DEBUG [org.jnp.interfaces.NamingContext] (pool-3-thread-3) Failed to connect to pinnet.actual-systems.com:1199
      2010-05-19 15:01:59,049 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 15:02:49,123 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
      1199, localAddr: null, localPort: 0, timeout: 0
      2010-05-19 15:02:51,263 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 15:02:51,510 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 15:02:53,660 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Succeeded in reconnecting to servers
      2010-05-19 15:12:22,877 ERROR [org.hornetq.core.client.impl.ClientSessionImpl] (pool-3-thread-3) Caught jmsexecptione
      2010-05-19 15:12:22,878 WARN  [com.arjuna.ats.jta.logging.loggerI18N] (pool-3-thread-3) [com.arjuna.ats.internal.jta.transaction.arjunacore.delistresource] [
      com.arjuna.ats.internal.jta.transaction.arjunacore.delistresource] TransactionImple.delistResource - caught exception during delist : XAException.XAER_RMERR
      2010-05-19 15:12:22,878 ERROR [org.hornetq.core.client.impl.ClientSessionImpl] (pool-3-thread-3) Caught jmsexecptione
      2010-05-19 15:12:22,878 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (pool-3-thread-3) [com.arjuna.ats.arjuna.coordinator.BasicAction_54] - Top-level
      abort of action -3f57fef6:acf1:4bf1815e:3d02e received TwoPhaseOutcome.FINISH_ERROR from <ClassName:RecordType.JTA_RECORD>
      2010-05-19 15:12:22,880 WARNING [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Will retry after a pause of 50000 ms
      2010-05-19 15:13:12,889 DEBUG [org.jnp.interfaces.TimedSocketFactory] (pool-3-thread-3) createSocket, hostAddr: pinnet.actual-systems.com/81.86.230.24, port:
      1199, localAddr: null, localPort: 0, timeout: 0
      2010-05-19 15:13:13,464 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Failed to set up JMS bridge connections. Most probably the source
      or target servers are unavailable. Will retry after a pause of 50000 ms
      2010-05-19 15:14:04,162 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 15:14:04,364 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (pool-3-thread-3) Started Netty Connector version 3.2.0.BETA1-r2215
      2010-05-19 15:14:05,067 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-3) Succeeded in reconnecting to servers

       


      thread info from jmx console :

       


      Thread: pool-3-thread-1 : priority:5, demon:false, threadId:69, threadState:TIMED_WAITING

       

          - waiting on <0x81f841> (a java.lang.Object)
          java.lang.Object.wait(Native Method)
          org.hornetq.jms.bridge.impl.JMSBridgeImpl$BatchTimeChecker.run(JMSBridgeImpl.java:1956)
          java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
          java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
          java.lang.Thread.run(Thread.java:619)

       

          Locked synchronizers :
          - locked <0x9f1a23> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

       

      Thread: pool-3-thread-2 : priority:5, demon:false, threadId:71, threadState:TIMED_WAITING

       

          - waiting on <0xe1e4f2> (a org.hornetq.core.client.impl.ClientConsumerImpl)
          java.lang.Object.wait(Native Method)
          org.hornetq.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:238)
          org.hornetq.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:322)
          org.hornetq.jms.client.HornetQMessageConsumer.getMessage(HornetQMessageConsumer.java:224)
          org.hornetq.jms.client.HornetQMessageConsumer.receive(HornetQMessageConsumer.java:133)
          org.hornetq.jms.bridge.impl.JMSBridgeImpl$SourceReceiver.run(JMSBridgeImpl.java:1734)
          - locked <0x81f841> (a java.lang.Object)
          java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
          java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
          java.lang.Thread.run(Thread.java:619)

       

          Locked synchronizers :
          - locked <0x16cf754> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

       

      Thread: pool-3-thread-3 : priority:5, demon:false, threadId:155, threadState:WAITING

       

          - waiting on <0x967733> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
          sun.misc.Unsafe.park(Native Method)
          java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
          java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
          java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
          java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
          java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
          java.lang.Thread.run(Thread.java:619)

       

       

      any thoughts ?

       

      all of the above using a build from TRUNK

        • 1. Re: 3 JMS Bridges - 2 reconnect ok, 1 is dead ?
          artp

          I noticed the JMS bridge being broken in 2.0.0GA and BETA5 in certain situations on startup if the target server of the JMSBridge wasn't available. The StartUpFailureHandler is broken because it succeeds but doesn't initialize threads to receive messages. I modified the StartUpFailureHandler to initialize threads to start receiving from the source(ie BatchTimeCheck and the checkerThread). If you look at the how start() sets up the threads, you need to do the same thing in when the startupFailureHandler is in the succeeded() method.

          1 of 1 people found this helpful
          • 2. Re: 3 JMS Bridges - 2 reconnect ok, 1 is dead ?
            artp

            Here's what I did...

            • 3. Re: 3 JMS Bridges - 2 reconnect ok, 1 is dead ?
              timfox

              Looks like your bridge is hung on doing a JNDI lookup.

               

              This is hard for me to tell why, as it's not HornetQ code.

              • 4. Re: 3 JMS Bridges - 2 reconnect ok, 1 is dead ?
                timfox

                Probably what happened is your bridge was doing a JNDI lookup, the connection to the server was created ok, then the network went down before the Object was downloaded fully from JNDI.

                 

                In this case since TCP is a reliable protocol, it will just hang there in the hope that the rest of the data will come eventually, which it never will.

                 

                The JNDI implementation should really deal with this, and timeout - like I say, this the JNDI impl is not part of HornetQ.

                 

                What we could do is do our own timeout on the HQ side - if a JNDI look up takes too long, we could interrupt the thread and try again. This will cater for the case where the JNDI impl is not well behaved.

                 

                You could add a JIRA feature request for this if you like.

                • 5. Re: 3 JMS Bridges - 2 reconnect ok, 1 is dead ?

                  Thanks for the replies guys,

                   

                  The problem seems to be happening regularly on quite a few of our consumers.

                   

                   

                  Tim - I'm no expert here at all,

                   

                  but should the SocketInputSteam.read() not throw some kind of SocketException if the remote connection has gone away ?

                   

                   

                  we did restart the server more than once - which would explain the disconnection and attempt to reconnect by the bridge,

                  but of course we would hope that the bridge would recover gracefully.

                   

                   

                  Certainly quite happy to create a JIRA issue, but this does seem like it might be a problem elsewhere ?

                   

                   

                  Thanks

                  • 6. Re: 3 JMS Bridges - 2 reconnect ok, 1 is dead ?

                    Hi Art,

                     

                    thanks for your contribution - but it does appear that you amended an earlier version of the code than is currently in TRUNK ?,

                    and the code has since been changed to use a thread pool / executor with a fixed size of 3 threads,

                    so I'm not 100% sure if these changes are still applicable ?

                    • 7. Re: 3 JMS Bridges - 2 reconnect ok, 1 is dead ?
                      timfox

                      Terry Paterson wrote:

                       

                      Thanks for the replies guys,

                       

                      The problem seems to be happening regularly on quite a few of our consumers.

                       

                       

                      Tim - I'm no expert here at all,

                       

                      but should the SocketInputSteam.read() not throw some kind of SocketException if the remote connection has gone away ?

                       

                       


                      It depends.

                       

                      If the connection has gone away because the target server has closed it, or the process has crashed or has been killed (in which case the OS closes the connection for you), then the client will indeed receive an exception.

                       

                      However if the connection has gone away because of an interruption on the network connection, e.g. someone has pulled the cable out, or the network is temporarily down, then the TCP client has no way of distinguishing this from just losing packet. After all the TCP protocol is designed to cope with lost packets, it will simply resend them until it has received acks - try it yourself, open a TCP connection from one server to another, pull out your network cable and plug it in 5 mins later, the connection will resume.

                       

                      So, in the case the network has disappeared, the client will just sit there waiting for it to come back. There is a timeout somewhere but the default is very long IIRC.

                      1 of 1 people found this helpful
                      • 8. Re: 3 JMS Bridges - 2 reconnect ok, 1 is dead ?

                        Thanks Tim

                         

                        we have added a new JIRA issue :

                         

                         

                        https://jira.jboss.org/browse/HORNETQ-393

                        • 9. Re: 3 JMS Bridges - 2 reconnect ok, 1 is dead ?
                          artp

                          It was from 2.0.0GA....I wasn't sure if it had already been fixed in trunk. I was just trying to let you know about an issue I saw when I was setting up the jms bridge a while back.