In Active-Backup mode, existing JMS clients fail to re-connect after failover

[aengineer]

This test was conducted using source from the HQ trunk dated 12.06.2010

Conside the scenario that we have a HQ active-backup configuration. Server1 is active, Server2 is backup. JMS clients use a JNDI provider URL of "jnp://server1:port1,server2:port2"

Consider the following sequence of events:
a) start active server
b) start backup server
c) Start a JMS queue producer and a JMS queue receiver. Both are able to send and receive messages respectively.
c) kill active server
    - backup server now becomes active

At this point, the JMS clients are unable to reconnect.

The Q consumer is hung, and the Q producer keep throwing this exception:
2010-12-15 16:59:48,137 ERROR (QueueSendLoop.java:104) - Caught exception #[1]. Message:Timed out waiting for response when sending packet 71
javax.jms.JMSException: Timed out waiting for response when sending packet 71
    at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:277)
    at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:285)
    at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:140)
    at org.hornetq.jms.client.HornetQMessageProducer.doSend(HornetQMessageProducer.java:451)
    at org.hornetq.jms.client.HornetQMessageProducer.send(HornetQMessageProducer.java:199)
    at com.putnam.jboss.queue.QueueSendLoop.send(QueueSendLoop.java:67)
    at com.putnam.jboss.queue.QueueSendLoop.run(QueueSendLoop.java:91)
    at com.putnam.jboss.queue.QueueSendLoop.main(QueueSendLoop.java:78)
Caused by: HornetQException[errorCode=3 message=Timed out waiting for response when sending packet 71]
    ... 8 more
   
   
If I terminate both clients and restart them, they are able to connect and send/receive messages.

My HQ configuration files are attached.

[aengineer]

https://issues.jboss.org/browse/HORNETQ-592

Also, I am using a consumer window size of 0. There were issues related to a 0 consumer window size and reconnects, which I believe have been fixed.  Also see: https://issues.jboss.org/browse/HORNETQ-522

[aengineer]

I get the same (failed) behavior with a non-zero consumer window size.

[gaohoward]

I think https://issues.jboss.org/browse/HORNETQ-522 has been fixed before. But recently we have done some important HA improvements and probably broke it. I'll take a look at it. Thanks!

Howard

[gaohoward]

Hi Just had a look at your configuration, it seems they are not updated to the new HA added in trunk. You can have a look at

jms example 'non-transaction-failover' or some other failover examples in HornetQ.

For example the 'clustered' now is required to be 'true' for a live-backup pair to work.

[aengineer]

I get the same behavior with clustered = true.

Also tried using the latest code from the trunk, dated 12.16.2010, and the behavior remains unchanged. Using latest trunk code, the exception thrown by the producer is:

2010-12-16 10:44:09,246 ERROR (QueueSendLoop.java:104) - Caught exception #[3]. Message:Timed out waiting for response when sending packet 71
javax.jms.JMSException: Timed out waiting for response when sending packet 71
    at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:277)
    at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:286)
    at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:141)
    at org.hornetq.jms.client.HornetQMessageProducer.doSend(HornetQMessageProducer.java:451)
    at org.hornetq.jms.client.HornetQMessageProducer.send(HornetQMessageProducer.java:199)
    at com.putnam.jboss.queue.QueueSendLoop.send(QueueSendLoop.java:67)
    at com.putnam.jboss.queue.QueueSendLoop.run(QueueSendLoop.java:91)
    at com.putnam.jboss.queue.QueueSendLoop.main(QueueSendLoop.java:78)
Caused by: HornetQException[errorCode=3 message=Timed out waiting for response when sending packet 71]
    ... 8 more

[ataylor]

take a look at the new failover examples, one thing you need to do is configure yopu factories to support HA.

also are the 2 servers using the same journal, its hard to tell

[aengineer]

Is there a specific HA example that one should be looking at? I previous used "transaction-failover" as a template, but the config files for this example are completely different than the config files for "non-transaction-failover". "transaction-failover" config files do not set "clustered=true", they dont define any discovery or broadcast  groups, and the default values for "shared-store" and "backup" are  used (which are both false).

Is "transaction-failover" a valid example?

[ataylor]

some of the examples still fixing, any multiple failover ones should be fine.

[aengineer]

Ignore this post - configuration issue