1 Reply Latest reply on Jan 16, 2015 5:15 AM by ikaychef

    can't send messages to HornetQ server after failover-failback

    ikaychef Newbie

      Hello,

       

      I'm setting up WildFly in Domain Mode and trying to configure it with a colocated replication with HornetQ. On a server crash hornetq is failing over and the backup server on the surviving node is starting and all messages are consumed by the left node.

      If I restart the crashed server the backup hornetq server is failing back, but now I have two issues if I try to send new messages:

       

      1. I get the following exception if I try to send new messages after failover-failback:

       

      2015-01-14 15:04:56 ERROR root:73 - javax.jms.JMSException: Failed to create session factory

      javax.jms.JMSException: Failed to create session factory

        at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:673)

        at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:112)

        at jms.ClusteredQueueTest.runTest(ClusteredQueueTest.java:50)

        at jms.ClusteredQueueTest.main(ClusteredQueueTest.java:26)

      Caused by: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s). Tried with all available servers.]

        at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:905)

        at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:669)

        ... 3 more

       

      2. If I send messages directly to the server which restarted, the messages are not loadbalanced between the servers anymore.

       

      I've tested a similiar configuration with JBoss EAP 6.3 and had no issues after multiple failover-failback tests. Is there a proper way to configure wildfly in domain mode and colocated hornetq replication? The messaging subsystem is:

        <hornetq-server name="live">

                          <persistence-enabled>true</persistence-enabled>

                          <cluster-password>${jboss.messaging.cluster.password:123123}</cluster-password>

                          <backup>false</backup>

                          <allow-failback>true</allow-failback>

                          <failover-on-shutdown>true</failover-on-shutdown>

                          <shared-store>false</shared-store>

                          <journal-file-size>102400</journal-file-size>

                          <check-for-live-server>true</check-for-live-server>

                          <backup-group-name>${hornetq.live.group}</backup-group-name>

                          <paging-directory path="data/jms/${hornetq.live.group}/paging"/>

                          <bindings-directory path="data/jms/${hornetq.live.group}/bindings"/>

                          <journal-directory path="data/jms/${hornetq.live.group}/journal"/>

                          <large-messages-directory path="data/jms/${hornetq.live.group}/large-messages"/>

       

                          <connectors>

                              <http-connector name="http-connector" socket-binding="http">

                                  <param key="http-upgrade-endpoint" value="http-acceptor"/>

                              </http-connector>

          <http-connector name="http-connector-throughput" socket-binding="http">

                                  <param key="http-upgrade-endpoint" value="http-acceptor-throughput"/>

                                  <param key="batch-delay" value="50"/>

                              </http-connector>

                              <in-vm-connector name="in-vm" server-id="0"/>

                          </connectors>

       

                          <acceptors>

                              <http-acceptor http-listener="default" name="http-acceptor"/>

        <http-acceptor http-listener="default" name="http-acceptor-throughput">

                                  <param key="batch-delay" value="50"/>

                                  <param key="direct-deliver" value="false"/>

                              </http-acceptor>

                              <in-vm-acceptor name="in-vm" server-id="0"/>

                          </acceptors>

       

                          <broadcast-groups>

                              <broadcast-group name="bg-group1">

                                  <socket-binding>messaging-group</socket-binding>

                                  <connector-ref>

                                      http-connector

                                  </connector-ref>

                              </broadcast-group>

                          </broadcast-groups>

       

                          <discovery-groups>

                              <discovery-group name="dg-group1">

                                  <socket-binding>messaging-group</socket-binding>

                                  <refresh-timeout>15000</refresh-timeout>

                              </discovery-group>

                          </discovery-groups>

       

                          <cluster-connections>

                              <cluster-connection name="my-cluster">

                                  <address>jms</address>

                                  <connector-ref>http-connector</connector-ref>

                                  <check-period>10000</check-period>

                                  <connection-ttl>20000</connection-ttl>

                                  <call-timeout>10000</call-timeout>

                                  <reconnect-attempts>1</reconnect-attempts>

                                  <use-duplicate-detection>true</use-duplicate-detection>

                                  <max-hops>1</max-hops>

                                  <discovery-group-ref discovery-group-name="dg-group1"/>

                              </cluster-connection>

                          </cluster-connections>

       

                          <security-settings>

                              <security-setting match="#">

                                  <permission type="send" roles="guest"/>

                                  <permission type="consume" roles="guest"/>

                                  <permission type="createNonDurableQueue" roles="guest"/>

                                  <permission type="deleteNonDurableQueue" roles="guest"/>

                              </security-setting>

                          </security-settings>

       

                          <address-settings>

                              <address-setting match="#">

                                  <dead-letter-address>jms.queue.DLQ</dead-letter-address>

                                  <expiry-address>jms.queue.ExpiryQueue</expiry-address>

                                  <max-size-bytes>10485760</max-size-bytes>

                                  <page-size-bytes>2097152</page-size-bytes>

                                  <message-counter-history-day-limit>10</message-counter-history-day-limit>

                                  <redistribution-delay>1000</redistribution-delay>

                              </address-setting>

                          </address-settings>

       

                          <jms-connection-factories>

                              <connection-factory name="InVmConnectionFactory">

                                  <connectors>

                                      <connector-ref connector-name="in-vm"/>

                                  </connectors>

                                  <entries>

                                      <entry name="java:/InVmConnectionFactory"/>

                                  </entries>

                                  <connection-ttl>20000</connection-ttl>

                                  <call-timeout>10000</call-timeout>

                                  <retry-interval>1000</retry-interval>

                                  <retry-interval-multiplier>1.0</retry-interval-multiplier>

                                  <max-retry-interval>1000</max-retry-interval>

                              </connection-factory>

                              <connection-factory name="RemoteConnectionFactory">

                                  <connectors>

                                      <connector-ref connector-name="http-connector"/>

                                  </connectors>

                                  <entries>

                                      <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>

                                  </entries>

                                  <ha>true</ha>

                                  <connection-ttl>20000</connection-ttl>

                                  <call-timeout>10000</call-timeout>

                                  <retry-interval>1000</retry-interval>

                                  <retry-interval-multiplier>1.0</retry-interval-multiplier>

                                  <max-retry-interval>1000</max-retry-interval>

                              </connection-factory>

                              <pooled-connection-factory name="hornetq-ra">

                                  <transaction mode="xa"/>

                                  <connectors>

                                      <connector-ref connector-name="in-vm"/>

                                  </connectors>

                                  <entries>

                                      <entry name="java:jboss/DefaultJMSConnectionFactory"/>

                                      <entry name="java:/ConnectionFactory"/>

                                      <entry name="java:/JmsXA"/>

                                  </entries>

                                  <ha>true</ha>

                                  <block-on-acknowledge>true</block-on-acknowledge>

                                  <reconnect-attempts>1</reconnect-attempts>

                              </pooled-connection-factory>

                          </jms-connection-factories>

       

                          <jms-destinations>

                              <jms-queue name="ExpiryQueue">

                                  <entry name="java:/jms/queue/ExpiryQueue"/>

                              </jms-queue>

                              <jms-queue name="DLQ">

                                  <entry name="java:/jms/queue/DLQ"/>

                              </jms-queue>

                              <jms-queue name="jmsQueue">

                                  <entry name="java:jboss/exported/jms/queue/jmsQueue"/>

                                  <durable>true</durable>

                              </jms-queue>

                              <jms-queue name="CMTQueue">

                                  <entry name="java:/queue/CMTQueue"/>

                                  <durable>true</durable>

                              </jms-queue>

                              <jms-topic name="jmsTopic">

                                  <entry name="java:jboss/exported/jms/topic/jmsTopic"/>

                              </jms-topic>

                          </jms-destinations>

                      </hornetq-server>

                      <hornetq-server name="backup">

                          <persistence-enabled>true</persistence-enabled>

                          <cluster-password>${jboss.messaging.cluster.password:123123}</cluster-password>

                       <backup>true</backup>

                          <allow-failback>true</allow-failback>

                          <shared-store>false</shared-store>

                          <journal-file-size>102400</journal-file-size>

                          <backup-group-name>${hornetq.backup.group}</backup-group-name>

                          <paging-directory path="data/jms/${hornetq.backup.group}/paging"/>

                          <bindings-directory path="data/jms/${hornetq.backup.group}/bindings"/>

                          <journal-directory path="data/jms/${hornetq.backup.group}/journal"/>

                          <large-messages-directory path="data/jms/${hornetq.backup.group}/large-messages"/>

       

                          <connectors>

                              <http-connector name="http-backup-connector" socket-binding="http">

                                  <param key="http-upgrade-endpoint" value="http-backup-acceptor"/>

                              </http-connector>

        <http-connector name="http-backup-connector-throughput" socket-binding="http">

                                  <param key="http-upgrade-endpoint" value="http-backup-acceptor-throughput"/>

                                  <param key="batch-delay" value="50"/>

                              </http-connector>

                              <in-vm-connector name="in-vm" server-id="1"/>

                          </connectors>

       

                          <acceptors>

                              <http-acceptor http-listener="default" name="http-backup-acceptor"/>

                                 <http-acceptor http-listener="default" name="http-backup-acceptor-throughput">

                                  <param key="batch-delay" value="50"/>

                                  <param key="direct-deliver" value="false"/>

                              </http-acceptor>

                              <in-vm-acceptor name="in-vm" server-id="1"/>

                          </acceptors>

       

                          <broadcast-groups>

                              <broadcast-group name="bg-group1">

                                  <socket-binding>messaging-group</socket-binding>

                                  <connector-ref>

                                      http-backup-connector

                                  </connector-ref>

                              </broadcast-group>

                          </broadcast-groups>

       

       

                          <discovery-groups>

                              <discovery-group name="dg-group1">

                                  <socket-binding>messaging-group</socket-binding>

                                  <refresh-timeout>15000</refresh-timeout>

                              </discovery-group>

                          </discovery-groups>

       

                          <cluster-connections>

                              <cluster-connection name="my-cluster">

                                  <address>jms</address>

                                  <connector-ref>http-backup-connector</connector-ref>

                                  <check-period>10000</check-period>

                                  <connection-ttl>20000</connection-ttl>

                                  <call-timeout>10000</call-timeout>

                                  <reconnect-attempts>1</reconnect-attempts>

                                  <use-duplicate-detection>true</use-duplicate-detection>

                                  <max-hops>1</max-hops>

                                  <discovery-group-ref discovery-group-name="dg-group1"/>

                              </cluster-connection>

                          </cluster-connections>

       

                          <security-settings>

                              <security-setting match="#">

                                  <permission type="send" roles="guest"/>

                                  <permission type="consume" roles="guest"/>

                                  <permission type="createNonDurableQueue" roles="guest"/>

                                  <permission type="deleteNonDurableQueue" roles="guest"/>

                              </security-setting>

                          </security-settings>

       

                          <address-settings>

                              <address-setting match="#">

                                  <max-size-bytes>10485760</max-size-bytes>

                                  <page-size-bytes>2097152</page-size-bytes>

                                  <message-counter-history-day-limit>10</message-counter-history-day-limit>

                                  <redistribution-delay>1000</redistribution-delay>

                              </address-setting>

                          </address-settings>

        </hornetq-server>

       

      ${hornetq.live.group} and ${hornetq.backup.group} are provided through parameters on both nodes "-Dhornetq.live.group=group1 -Dhornetq.backup.group=group2" and on the other node "-Dhornetq.live.group=group2 -Dhornetq.backup.group=group1"