Live/Backup automatic client fail-over issue

[mofarn]

Hi all,

I have two standalone servers(live and backup) on the same machine. (Hornetq 2.2.14)

I want automatic client automatic fail-over mechanisem by instantiating connection factory (HornetQJMSConnectionFactory) directly (without jndi). I used jmsTemplate for sending messages!

My applicationContext.xml:

<bean name="connectionFactory" class="org.hornetq.jms.client.HornetQJMSConnectionFactory">
                    <constructor-arg name="ha" value="true" />
                    <constructor-arg>
                              <bean class="org.hornetq.api.core.DiscoveryGroupConfiguration">
                                        <constructor-arg value="231.7.7.7" />
                                        <constructor-arg value="9876" />
                              </bean>
                    </constructor-arg>
                    <property name="clientFailureCheckPeriod" value="30000" /> <!-- default -->
                    <property name="failoverOnInitialConnection" value="true" />
                    <property name="initialConnectAttempts" value="15" />
                    <property name="reconnectAttempts" value="5" />
                    <property name="retryInterval" value="1000" />
</bean>

<bean name="messageSender" class="messaging.producer.MessageSender2">
      <property name="jmsTemplate" ref="jmsTemplate" />
</bean>

<bean name="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
      <property name="connectionFactory" ref="connectionFactory" />
</bean>

My code for senidng message:

public class MessageSender2 
{
          private JmsTemplate jmsTemplate;

          public void setJmsTemplate(JmsTemplate jmsTemplate) {
                    this.jmsTemplate = jmsTemplate;
          }

          public void send(final Serializable obj, final String queueName)
          {
                    Queue queue = HornetQJMSClient.createQueue(queueName);
                    try
                    {
                              jmsTemplate.send(queue, new MessageCreator() 
                              { 
                                        @Override
                                        public Message createMessage(Session session) throws JMSException 
                                        {
                                                  ObjectMessage message = session.createObjectMessage(obj);
                                                  return message;
                                        }
                              });
                              System.out.println("message sent"); 
                    }
                    catch(JmsException e)
                    {
                              e.printStackTrace();
                    }
          }
}

Consider this scenario:

1. Start live and backup server
2. Start client sending 1000 messages with send method (messages sent succesfully)
3. Shutdown the live server (Backup server is live now)

When client want resume sending messages, it can't fail-over and gets this error:

WARNING: Tried 15 times to connect. Now giving up on reconnecting it.
org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Failed to create session factory; nested exception is HornetQException[errorCode=2 message=Cannot connect to server(s). Tried with all available servers.]
          at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
          at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:168)
          at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:469)
          at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:534)
          at messaging.producer.MessageSender2.send(MessageSender2.java:29)
          at test.Test.test1(Test.java:31)
          at test.Test.main(Test.java:50)
Caused by: javax.jms.JMSException: Failed to create session factory
          at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:605)
          at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:119)
          at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:114)
          at org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:184)
          at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:456)
          ... 4 more
Caused by: HornetQException[errorCode=2 message=Cannot connect to server(s). Tried with all available servers.]
          at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:774)
          at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:601)
          ... 8 more
128- Nov 28, 2012 12:16:51 PM org.hornetq.core.logging.impl.JULLogDelegate warn
WARNING: Tried 15 times to connect. Now giving up on reconnecting it.
org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Failed to create session factory; nested exception is HornetQException[errorCode=2 message=Cannot connect to server(s). Tried with all available servers.]
          at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
          at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:168)
          at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:469)
          at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:534)
          at messaging.producer.MessageSender2.send(MessageSender2.java:29)
          at test.Test.test1(Test.java:31)
          at test.Test.main(Test.java:50)
Caused by: javax.jms.JMSException: Failed to create session factory
          at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:605)
          at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:119)
          at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:114)
          at org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:184)
          at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:456)
          ... 4 more
Caused by: HornetQException[errorCode=2 message=Cannot connect to server(s). Tried with all available servers.]
          at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:774)
          at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:601)
          ... 8 more

I have no idea why it happens (or am i doing it right??); Please reply back, thanks in advance.

[ataylor]

first thing i would say is that Spring is creating a new connection for every message sent so is not actually failing over but failing on its initial connection after the live server has crashed. I think you can use a cached connection with Spring but i wouldn't know how to configur ethis, consult the spring docs

Also you have initial connect attempts to 15 (5 on reconnect) is this enough, is your backup server coming online ok.

Also check that the backup is actually starting when the live crashes.

I would start by using one of the failover examples and remove spring completely, it just confuses matters. Once you are happy that your env is working ok, i.e. you may need a loop back address, then you can add spring.

[mofarn]

Andy Taylor wrote:

Also you have initial connect attempts to 15 (5 on reconnect) is this enough, is your backup server coming online ok.

Also check that the backup is actually starting when the live crashes.

I would start by using one of the failover examples and remove spring completely, it just confuses matters. Once you are happy that your env is working ok, i.e. you may need a loop back address, then you can add spring.

Backup server comming online ok:

* [Activation for server HornetQServerImpl::serverUUID=0220a790-3182-11e2-ac17-37a9b1c86abd] 28-Nov 13:49:30,805 INFO [HornetQServerImpl]  Backup Server is now live

Andy Taylor wrote:

I would start by using one of the failover examples and remove spring completely, it just confuses matters. Once you are happy that your env is working ok, i.e. you may need a loop back address, then you can add spring.

I re-initialized connectionFactory in catch clause, and the reconnection problem solved (??) :

private JmsTemplate jmsTemplate;
private HornetQJMSConnectionFactory connectionFactory;
private static DiscoveryGroupConfiguration groupConfiguration = new DiscoveryGroupConfiguration("231.7.7.7", 9876);

public MessageSender3() 
{
                    connectionFactory = (HornetQJMSConnectionFactory) HornetQJMSClient.createConnectionFactoryWithHA(groupConfiguration, JMSFactoryType.CF);
                    connectionFactory.setInitialConnectAttempts(15);
                    connectionFactory.setReconnectAttempts(15);
                    connectionFactory.setRetryInterval(1000);
                    jmsTemplate = new JmsTemplate();
                    jmsTemplate.setConnectionFactory(connectionFactory);
}

public void send(Serializable obj, String queueName) throws JMSException
{
                    try
                    {
                              Queue queue = HornetQJMSClient.createQueue(queueName);
                              jmsTemplate.send(queue,new MessageCreator()
                              {
                                        @Override
                                        public Message createMessage(Session session) throws JMSException
                                        {
                                                  TextMessage message = session.createTextMessage("salam");
                                                  return message;
                                        }
                              });
                              System.out.println("message sent");
                    }
                    catch (Exception e)
                    {
                              e.printStackTrace();
                              connectionFactory = (HornetQJMSConnectionFactory) HornetQJMSClient.createConnectionFactoryWithHA(groupConfiguration, JMSFactoryType.CF);
                              jmsTemplate.setConnectionFactory(connectionFactory);
                    }
}

Why the client can't fail-over automatically??

[ataylor]

I re-initialized connectionFactory in catch clause, and the reconnection problem solved (??) :

No, if you configure it correctly you dont have to recreate it, you ned to configure spring to cache the connections created by this factory.

Why the client can't fail-over automatically??

I don't know, like i say, make sure the failover examples work on your machine, then add spring to the equation

[mofarn]

Andy Taylor wrote:

I re-initialized connectionFactory in catch clause, and the reconnection problem solved (??) :

No, if you configure it correctly you dont have to recreate it, you ned to configure spring to cache the connections created by this factory.

In my last replay, i didn't use spring configs!

[ataylor]

have you managed to run the failover examples?

[mofarn]

Andy Taylor wrote:

have you managed to run the failover examples?

Yes, they worked fine!

[ataylor]

Yes, they worked fine!

Then its a spring config isssue, make sure spring is caching its connections and the connection factories are configured correctly and all should work.

[mofarn]

Then its a spring config isssue, make sure spring is caching its connections and the connection factories are configured correctly and all should work.

In MessageSender3 (Reply number 3), i didn't use spring, but i should re-instantiate the connection factory in catch clause

[ataylor]

you dont need to do anything, simply configure the connection factory to create its conections to reconnect and use caching on the spring side

[angela429]

Hi,Andy

I have problem like this.And I've run the transactional-faliover examples,and it worked well.Then I changed the <failover-on-shutdown> into true,live is on port 5445,and backup on 5446.I test the two standalone nodes on win7, The backup announced normally.When I close the live using Ctrl + C ,the backup print the last log

* [Activation for server HornetQServerImpl::serverUUID=e095c29e-4289-11e2-9883-e
730fbe3bdb1] 11-Dec 9:30:45,847 INFO [HornetQServerImpl]  HornetQ Backup Server
version 2.2.14.Final (HQ_2_2_14_FINAL, 122) [e095c29e-4289-11e2-9883-e730fbe3bdb
1] started, waiting live to fail before it gets active

* [Thread-0 (HornetQ-server-HornetQServerImpl::serverUUID=e095c29e-4289-11e2-988
3-e730fbe3bdb1-17666385)] 11-Dec 9:30:47,31 INFO [ClusterConnectionImpl]  backup
 announced

* [Thread-1 (HornetQ-client-global-threads-30397362)] 11-Dec 9:31:6,786 WARNING
[RemotingConnectionImpl]  Connection failure has been detected: The connection w
as disconnected because of server shutdown [code=4]

The backup didn't changed to live.I didn't changed anything except for the <failover-on-shutdown> configuration.

Do you have any suggestions for me.

My second problem is that I have at least 10thousand+ messages to be sent , and  all are alarm messages,so the order can't change.The distribution queue in cluster can only dispatcher the messages and can not assure the order received by consumer,Am I right?

Do you have any intention to meet the order requirement in cluster next version?

As for the caching connection said by moji,I was configured like this

<bean id="messageTopic" class="org.hornetq.api.jms.HornetQJMSClient" factory-method="createQueue">


      <constructor-arg value="exampleQueue" />
</bean>




<bean id="transportConfiguration" class="org.hornetq.api.core.DiscoveryGroupConfiguration">
 <constructor-arg name="groupAddress" value="231.7.7.7"/>
 <constructor-arg name="groupPort" value="9876"></constructor-arg>
</bean>


<bean id="cachedConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory" ref="connectionFactory"></property>
<property name="sessionCacheSize" value="10"></property>
<property name="reconnectOnException" value="true"></property>
</bean>


<bean id="connectionFactory" class="org.hornetq.api.jms.HornetQJMSClient" factory-method="createConnectionFactoryWithHA">
<constructor-arg type="org.hornetq.api.jms.JMSFactoryType" value="CF"/>
<constructor-arg>
<ref bean="transportConfiguration"/>
</constructor-arg>
</bean>


<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory" ref="cachedConnectionFactory" />
<property name="pubSubDomain" value="true" />
</bean>

[ataylor]

The backup didn't changed to live.I didn't changed anything except for the <failover-on-shutdown> configuration.

Do you have any suggestions for me.

As long as its true it shouldnt matter how you stop the server failover should occur

My second problem is that I have at least 10thousand+ messages to be sent , and  all are alarm messages,so the order can't change.The distribution queue in cluster can only dispatcher the messages and can not assure the order received by consumer,Am I right?

Do you have any intention to meet the order requirement in cluster next version?

The order requirment between a cluster is already adhered too, order is gauranteed between a single producer and a single consumer. The caveat here is redelivery and redistribution.