11 Replies Latest reply on Dec 11, 2012 6:05 AM by Andy Taylor

    Live/Backup automatic client fail-over issue

    m f Newbie

      Hi all,

       

      I have two standalone servers(live and backup) on the same machine. (Hornetq 2.2.14)

      I want automatic client automatic fail-over mechanisem by instantiating connection factory (HornetQJMSConnectionFactory) directly (without jndi). I used jmsTemplate for sending messages!

      My applicationContext.xml:

      <bean name="connectionFactory" class="org.hornetq.jms.client.HornetQJMSConnectionFactory">
                          <constructor-arg name="ha" value="true" />
                          <constructor-arg>
                                    <bean class="org.hornetq.api.core.DiscoveryGroupConfiguration">
                                              <constructor-arg value="231.7.7.7" />
                                              <constructor-arg value="9876" />
                                    </bean>
                          </constructor-arg>
                          <property name="clientFailureCheckPeriod" value="30000" /> <!-- default -->
                          <property name="failoverOnInitialConnection" value="true" />
                          <property name="initialConnectAttempts" value="15" />
                          <property name="reconnectAttempts" value="5" />
                          <property name="retryInterval" value="1000" />
      </bean>
      
      <bean name="messageSender" class="messaging.producer.MessageSender2">
            <property name="jmsTemplate" ref="jmsTemplate" />
      </bean>
      
      <bean name="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
            <property name="connectionFactory" ref="connectionFactory" />
      </bean>
      

       

      My code for senidng message:

      public class MessageSender2 
      {
                private JmsTemplate jmsTemplate;
      
                public void setJmsTemplate(JmsTemplate jmsTemplate) {
                          this.jmsTemplate = jmsTemplate;
                }
      
                public void send(final Serializable obj, final String queueName)
                {
                          Queue queue = HornetQJMSClient.createQueue(queueName);
                          try
                          {
                                    jmsTemplate.send(queue, new MessageCreator() 
                                    { 
                                              @Override
                                              public Message createMessage(Session session) throws JMSException 
                                              {
                                                        ObjectMessage message = session.createObjectMessage(obj);
                                                        return message;
                                              }
                                    });
                                    System.out.println("message sent"); 
                          }
                          catch(JmsException e)
                          {
                                    e.printStackTrace();
                          }
                }
      }
      

       

      Consider this scenario:

      1. Start live and backup server
      2. Start client sending 1000 messages with send method (messages sent succesfully)
      3. Shutdown the live server (Backup server is live now)

       

      When client want resume sending messages, it can't fail-over and gets this error:

      WARNING: Tried 15 times to connect. Now giving up on reconnecting it.
      org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Failed to create session factory; nested exception is HornetQException[errorCode=2 message=Cannot connect to server(s). Tried with all available servers.]
                at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
                at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:168)
                at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:469)
                at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:534)
                at messaging.producer.MessageSender2.send(MessageSender2.java:29)
                at test.Test.test1(Test.java:31)
                at test.Test.main(Test.java:50)
      Caused by: javax.jms.JMSException: Failed to create session factory
                at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:605)
                at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:119)
                at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:114)
                at org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:184)
                at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:456)
                ... 4 more
      Caused by: HornetQException[errorCode=2 message=Cannot connect to server(s). Tried with all available servers.]
                at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:774)
                at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:601)
                ... 8 more
      128- Nov 28, 2012 12:16:51 PM org.hornetq.core.logging.impl.JULLogDelegate warn
      WARNING: Tried 15 times to connect. Now giving up on reconnecting it.
      org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Failed to create session factory; nested exception is HornetQException[errorCode=2 message=Cannot connect to server(s). Tried with all available servers.]
                at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
                at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:168)
                at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:469)
                at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:534)
                at messaging.producer.MessageSender2.send(MessageSender2.java:29)
                at test.Test.test1(Test.java:31)
                at test.Test.main(Test.java:50)
      Caused by: javax.jms.JMSException: Failed to create session factory
                at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:605)
                at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:119)
                at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:114)
                at org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:184)
                at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:456)
                ... 4 more
      Caused by: HornetQException[errorCode=2 message=Cannot connect to server(s). Tried with all available servers.]
                at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:774)
                at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:601)
                ... 8 more
      
      

       

      I have no idea why it happens (or am i doing it right??); Please reply back, thanks in advance.

        • 1. Re: Live/Backup automatic client fail-over issue
          Andy Taylor Master

          first thing i would say is that Spring is creating a new connection for every message sent so is not actually failing over but failing on its initial connection after the live server has crashed. I think you can use a cached connection with Spring but i wouldn't know how to configur ethis, consult the spring docs

           

          Also you have initial connect attempts to 15 (5 on reconnect) is this enough, is your backup server coming online ok.

           

          Also check that the backup is actually starting when the live crashes.

           

          I would start by using one of the failover examples and remove spring completely, it just confuses matters. Once you are happy that your env is working ok, i.e. you may need a loop back address, then you can add spring.

          • 2. Re: Live/Backup automatic client fail-over issue
            m f Newbie

            Andy Taylor wrote:


            Also you have initial connect attempts to 15 (5 on reconnect) is this enough, is your backup server coming online ok.

             

            Also check that the backup is actually starting when the live crashes.

             

            I would start by using one of the failover examples and remove spring completely, it just confuses matters. Once you are happy that your env is working ok, i.e. you may need a loop back address, then you can add spring.

            Backup server comming online ok:

            * [Activation for server HornetQServerImpl::serverUUID=0220a790-3182-11e2-ac17-37a9b1c86abd] 28-Nov 13:49:30,805 INFO [HornetQServerImpl]  Backup Server is now live

             

            Andy Taylor wrote:

            I would start by using one of the failover examples and remove spring completely, it just confuses matters. Once you are happy that your env is working ok, i.e. you may need a loop back address, then you can add spring.

            I re-initialized connectionFactory in catch clause, and the reconnection problem solved (??) :

            private JmsTemplate jmsTemplate;
            private HornetQJMSConnectionFactory connectionFactory;
            private static DiscoveryGroupConfiguration groupConfiguration = new DiscoveryGroupConfiguration("231.7.7.7", 9876);
            
            public MessageSender3() 
            {
                                connectionFactory = (HornetQJMSConnectionFactory) HornetQJMSClient.createConnectionFactoryWithHA(groupConfiguration, JMSFactoryType.CF);
                                connectionFactory.setInitialConnectAttempts(15);
                                connectionFactory.setReconnectAttempts(15);
                                connectionFactory.setRetryInterval(1000);
                                jmsTemplate = new JmsTemplate();
                                jmsTemplate.setConnectionFactory(connectionFactory);
            }
            
            public void send(Serializable obj, String queueName) throws JMSException
            {
                                try
                                {
                                          Queue queue = HornetQJMSClient.createQueue(queueName);
                                          jmsTemplate.send(queue,new MessageCreator()
                                          {
                                                    @Override
                                                    public Message createMessage(Session session) throws JMSException
                                                    {
                                                              TextMessage message = session.createTextMessage("salam");
                                                              return message;
                                                    }
                                          });
                                          System.out.println("message sent");
                                }
                                catch (Exception e)
                                {
                                          e.printStackTrace();
                                          connectionFactory = (HornetQJMSConnectionFactory) HornetQJMSClient.createConnectionFactoryWithHA(groupConfiguration, JMSFactoryType.CF);
                                          jmsTemplate.setConnectionFactory(connectionFactory);
                                }
            }
            

             

            Why the client can't fail-over automatically??

            • 3. Re: Live/Backup automatic client fail-over issue
              Andy Taylor Master

              I re-initialized connectionFactory in catch clause, and the reconnection problem solved (??) :

              No, if you configure it correctly you dont have to recreate it, you ned to configure spring to cache the connections created by this factory.

              Why the client can't fail-over automatically??

              I don't know, like i say, make sure the failover examples work on your machine, then add spring to the equation

              • 4. Re: Live/Backup automatic client fail-over issue
                m f Newbie

                Andy Taylor wrote:

                I re-initialized connectionFactory in catch clause, and the reconnection problem solved (??) :

                No, if you configure it correctly you dont have to recreate it, you ned to configure spring to cache the connections created by this factory.

                In my last replay, i didn't use spring configs!

                • 5. Re: Live/Backup automatic client fail-over issue
                  Andy Taylor Master

                  have you managed to run the failover examples?

                  • 6. Re: Live/Backup automatic client fail-over issue
                    m f Newbie

                    Andy Taylor wrote:

                    have you managed to run the failover examples?

                    Yes, they worked fine!

                    • 7. Re: Live/Backup automatic client fail-over issue
                      Andy Taylor Master

                      Yes, they worked fine!

                      Then its a spring config isssue, make sure spring is caching its connections and the connection factories are configured correctly and all should work.

                      • 8. Re: Live/Backup automatic client fail-over issue
                        m f Newbie

                        Then its a spring config isssue, make sure spring is caching its connections and the connection factories are configured correctly and all should work.

                        In MessageSender3 (Reply number 3), i didn't use spring, but i should re-instantiate the connection factory in catch clause

                        • 9. Re: Live/Backup automatic client fail-over issue
                          Andy Taylor Master

                          you dont need to do anything, simply configure the connection factory to create its conections to reconnect and use caching on the spring side

                          • 10. Re: Live/Backup automatic client fail-over issue
                            Fang Fang Newbie

                            Hi,Andy

                            I have problem like this.And I've run the transactional-faliover examples,and it worked well.Then I changed the <failover-on-shutdown> into true,live is on port 5445,and backup on 5446.I test the two standalone nodes on win7, The backup announced normally.When I close the live using Ctrl + C ,the backup print the last log

                             

                            * [Activation for server HornetQServerImpl::serverUUID=e095c29e-4289-11e2-9883-e
                            730fbe3bdb1] 11-Dec 9:30:45,847 INFO [HornetQServerImpl]  HornetQ Backup Server
                            version 2.2.14.Final (HQ_2_2_14_FINAL, 122) [e095c29e-4289-11e2-9883-e730fbe3bdb
                            1] started, waiting live to fail before it gets active
                            
                            * [Thread-0 (HornetQ-server-HornetQServerImpl::serverUUID=e095c29e-4289-11e2-988
                            3-e730fbe3bdb1-17666385)] 11-Dec 9:30:47,31 INFO [ClusterConnectionImpl]  backup
                             announced
                            
                            * [Thread-1 (HornetQ-client-global-threads-30397362)] 11-Dec 9:31:6,786 WARNING
                            [RemotingConnectionImpl]  Connection failure has been detected: The connection w
                            as disconnected because of server shutdown [code=4]
                            
                            

                             

                            The backup didn't changed to live.I didn't changed anything except for the <failover-on-shutdown> configuration.

                            Do you have any suggestions for me.

                             

                            My second problem is that I have at least 10thousand+ messages to be sent , and  all are alarm messages,so the order can't change.The distribution queue in cluster can only dispatcher the messages and can not assure the order received by consumer,Am I right?

                            Do you have any intention to meet the order requirement in cluster next version?

                             

                             

                            As for the caching connection said by moji,I was configured like this

                             

                            <bean id="messageTopic" class="org.hornetq.api.jms.HornetQJMSClient" factory-method="createQueue">
                            
                            
                                  <constructor-arg value="exampleQueue" />
                            </bean>
                            
                            
                            
                            
                            <bean id="transportConfiguration" class="org.hornetq.api.core.DiscoveryGroupConfiguration">
                             <constructor-arg name="groupAddress" value="231.7.7.7"/>
                             <constructor-arg name="groupPort" value="9876"></constructor-arg>
                            </bean>
                            
                            
                            <bean id="cachedConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
                            <property name="targetConnectionFactory" ref="connectionFactory"></property>
                            <property name="sessionCacheSize" value="10"></property>
                            <property name="reconnectOnException" value="true"></property>
                            </bean>
                            
                            
                            <bean id="connectionFactory" class="org.hornetq.api.jms.HornetQJMSClient" factory-method="createConnectionFactoryWithHA">
                            <constructor-arg type="org.hornetq.api.jms.JMSFactoryType" value="CF"/>
                            <constructor-arg>
                            <ref bean="transportConfiguration"/>
                            </constructor-arg>
                            </bean>
                            
                            
                            <bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
                            <property name="connectionFactory" ref="cachedConnectionFactory" />
                            <property name="pubSubDomain" value="true" />
                            </bean>
                            
                            
                            
                            • 11. Re: Live/Backup automatic client fail-over issue
                              Andy Taylor Master

                              The backup didn't changed to live.I didn't changed anything except for the <failover-on-shutdown> configuration.

                              Do you have any suggestions for me.

                              As long as its true it shouldnt matter how you stop the server failover should occur

                               

                               

                              My second problem is that I have at least 10thousand+ messages to be sent , and  all are alarm messages,so the order can't change.The distribution queue in cluster can only dispatcher the messages and can not assure the order received by consumer,Am I right?

                              Do you have any intention to meet the order requirement in cluster next version?

                              The order requirment between a cluster is already adhered too, order is gauranteed between a single producer and a single consumer. The caveat here is redelivery and redistribution.