2 Replies Latest reply on Jan 17, 2012 9:06 AM by dgyimesi

    Failed to handle failover

    yo_
    yo_ May 9, 2011 7:56 PM

      Hello - I've been running HQ 2.1.2/JBoss 4.2.3 for several months but when recently upgrading to HQ 2.2 I've noticed the following appearing repeatedly.

       

      I don't think it is a network issue because I'm able to telnet to xxx.3 5445 from xxx.140 and get a connection.

       

      I read in another thread that the fail-over implementation has changed in 2.1.2 -> 2.2 -- could this be related? Has anyone seen errors like this?

       

       

       

      011-05-09 03:32:28,363 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (Thread-22827 (group:HornetQ-client-global-threads-95625177):) Connection failure has been detected: Did not receive data from server for org.hornetq.core.remoting.impl.netty.NettyConnection@2b17908[local= /xx.xxx.xx.140:45902, remote=/xx.xxx.xx.3:5445] [code=3]

      2011-05-09 03:32:28,367 ERROR [org.hornetq.core.client.impl.ClientSessionImpl] (Thread-22641 (group:HornetQ-client-global-threads-95625177):) Failed to handle failover

      HornetQException[errorCode=3 message=Timed out waiting for response when sending packet 32]

              at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:276)

              at org.hornetq.core.client.impl.ClientSessionImpl.handleFailover(ClientSessionImpl.java:920)

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:846)

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:573)

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:467)

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$700(ClientSessionFactoryImpl.java:74)

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1268)

              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:528)

              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:298)

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl$PingRunnable$1.run(ClientSessionFactoryImpl.java:1326)

              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

              at java.lang.Thread.run(Thread.java:619)

       

       

      ....

       

      2011-05-09 03:32:42,157 ERROR [org.hornetq.core.client.impl.ClientSessionImpl] (Thread-22792 (group:HornetQ-client-global-threads-95625177):) Failed to handle failover

      HornetQException[errorCode=3 message=Timed out waiting for response when sending packet 32]

              at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:276)

              at org.hornetq.core.client.impl.ClientSessionImpl.handleFailover(ClientSessionImpl.java:920)

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:846)

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:573)

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:467)

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$700(ClientSessionFactoryImpl.java:74)

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1268)

              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:528)

              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:298)

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl$PingRunnable$1.run(ClientSessionFactoryImpl.java:1326)

              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

              at java.lang.Thread.run(Thread.java:619)

      2011-05-09 03:32:42,157 ERROR [org.hornetq.core.server.cluster.impl.BridgeImpl] (Thread-22671 (group:HornetQ-client-global-threads-95625177):) Failed to cancel refs

      HornetQException[errorCode=3 message=Timed out waiting for response when sending packet 34]

              at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:276)

              at org.hornetq.core.client.impl.ClientSessionImpl.internalCreateQueue(ClientSessionImpl.java:1748)

              at org.hornetq.core.client.impl.ClientSessionImpl.createQueue(ClientSessionImpl.java:303)

              at org.hornetq.core.client.impl.DelegatingSession.createQueue(DelegatingSession.java:284)

              at org.hornetq.core.server.cluster.impl.ClusterConnectionBridge.setupNotificationConsumer(ClusterConnectionBridge.java:198)

              at org.hornetq.core.server.cluster.impl.ClusterConnectionBridge.afterConnect(ClusterConnectionBridge.java:223)

              at org.hornetq.core.server.cluster.impl.BridgeImpl.fail(BridgeImpl.java:477)

              at org.hornetq.core.server.cluster.impl.BridgeImpl.connectionFailed(BridgeImpl.java:422)

              at org.hornetq.core.server.cluster.impl.ClusterConnectionBridge.connectionFailed(ClusterConnectionBridge.java:249)

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:797)

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:594)

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:467)

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$700(ClientSessionFactoryImpl.java:74)

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1268)

              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:528)

              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:298)

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl$PingRunnable$1.run(ClientSessionFactoryImpl.java:1326)

              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

              at java.lang.Thread.run(Thread.java:619)

        • 1. Failed to handle failover
          clebert.suconic
          clebert.suconic May 9, 2011 9:47 PM (in response to yo_)

          The integration on JBoss 4.2 was not made by us.. it was just a community contribution...

           

          I would say it's not tested config.

           

           

          I'm seriously considering removing it from the distribution.

           

           

          But from what I have seen at your exceptions, it seems you don't have a backup node configured.

            • Actions
          • 2. Re: Failed to handle failover
            dgyimesi
            dgyimesi Jan 17, 2012 9:06 AM (in response to yo_)

            I encountered the same error. I have a homogeneous cluster where HornetQ 2.1.2's are bridged to bring failover.

            However, when one node fails I get the "[BridgeImpl]  Failed to cancel refs" message.

             

            Did you able to fix this?

              • Actions