2 Replies Latest reply on Aug 4, 2014 9:31 AM by jbertram

    Exception reported during cluster fail-over using HQ 2.3

    ejb3workshop
    ejb3workshop Jul 30, 2014 7:15 AM

      When we stop our main server and fail over to the backup server we see an exception reported in the log. The same exception is raised when we fall back to the main server. We have two servers, one active and the other one as backup. They are distributed across separate servers and configured for the same port.

       

      Jul 30, 2014 11:05:06 AM org.hornetq.integration.bootstrap.HornetQBootstrapServer main

      INFO: HQ101000: Starting HornetQ Server

      Jul 30, 2014 11:05:08 AM org.hornetq.core.deployers.impl.FileConfigurationParser parseMainConfig

      INFO: HQ221038: Configuration option clustered is deprecated. Consult the manual for details.

      Jul 30, 2014 11:05:08 AM org.hornetq.core.server.impl.HornetQServerImpl start

      INFO: HQ221000: backup server is starting with configuration HornetQ Configuration (clustered=true,backup=true,sharedStore=true,journalDirectory=/mnt/hCOMMSDEVTINY3_server1/journal,bindingsDirectory=/mnt/hornetq/COMMSDEVTINY3_server1/bindings,largeMessagesDirectory=/mnt/hornetq/COMMSDEVTINY3_server1/lssages,pagingDirectory=/mnt/hornetq/COMMSDEVTINY3_server1/paging)

      Jul 30, 2014 11:05:08 AM org.hornetq.core.server.impl.FileLockNodeManager startBackup

      INFO: HQ221032: Waiting to become backup node

      Jul 30, 2014 11:05:08 AM org.hornetq.core.server.impl.FileLockNodeManager startBackup

      INFO: HQ221033: ** got backup lock

      Jul 30, 2014 11:05:08 AM org.hornetq.core.persistence.impl.journal.JournalStorageManager <init>

      INFO: HQ221012: Using AIO Journal

      Jul 30, 2014 11:05:09 AM org.hornetq.core.server.impl.HornetQServerImpl$SharedStoreBackupActivation run

      INFO: HQ221109: HornetQ Backup Server version 2.3.0.SNAPSHOT (colonizer, 123) [4699c2d8-17bc-11e4-9db3-4b1e63521921] started, waiting live to fail befoets active

      Jul 30, 2014 11:05:09 AM org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$2 run

      INFO: HQ221031: backup announced

      Jul 30, 2014 11:05:15 AM org.hornetq.core.protocol.core.impl.RemotingConnectionImpl fail

      WARN: HQ212037: Connection failure has been detected: HQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]

      Jul 30, 2014 11:05:24 AM org.hornetq.core.server.impl.HornetQServerImpl deployQueue

      INFO: HQ221003: trying to deploy queue jms.queue.DLQ

      Jul 30, 2014 11:05:24 AM org.hornetq.core.server.impl.HornetQServerImpl deployQueue

      INFO: HQ221003: trying to deploy queue jms.queue.ExpiryQueue

      Jul 30, 2014 11:05:24 AM org.hornetq.core.server.impl.HornetQServerImpl deployQueue

      INFO: HQ221003: trying to deploy queue jms.queue.ExampleQueue

      Jul 30, 2014 11:05:24 AM org.hornetq.core.server.impl.HornetQServerImpl deployQueue

      INFO: HQ221003: trying to deploy queue jms.queue.eventQueue

      Jul 30, 2014 11:05:24 AM org.hornetq.core.server.impl.HornetQServerImpl deployQueue

      INFO: HQ221003: trying to deploy queue jms.queue.eventEmailQueue

      Jul 30, 2014 11:05:24 AM org.hornetq.core.server.impl.HornetQServerImpl deployQueue

      INFO: HQ221003: trying to deploy queue jms.queue.eventEmailQueue2

      Jul 30, 2014 11:05:24 AM org.hornetq.core.server.impl.HornetQServerImpl deployQueue

      INFO: HQ221003: trying to deploy queue jms.queue.eventTenancyQueue

      Jul 30, 2014 11:05:24 AM org.hornetq.core.server.impl.HornetQServerImpl deployQueue

      INFO: HQ221003: trying to deploy queue jms.queue.eventTenancyReplyQueue

      Jul 30, 2014 11:05:24 AM org.hornetq.core.server.impl.HornetQServerImpl deployQueue

      INFO: HQ221003: trying to deploy queue jms.queue.futureWebServiceQueue

      Jul 30, 2014 11:05:24 AM org.hornetq.core.server.impl.HornetQServerImpl deployQueue

      INFO: HQ221003: trying to deploy queue jms.topic.ExampleTopic

      Jul 30, 2014 11:05:24 AM org.hornetq.core.server.impl.HornetQServerImpl deployQueue

      INFO: HQ221003: trying to deploy queue jms.topic.workspaceTopic

      Jul 30, 2014 11:05:24 AM org.hornetq.core.remoting.impl.netty.NettyAcceptor start

      INFO: HQ221020: Started Netty Acceptor version 3.6.2.Final-c0d783c 10.92.14.196:5455 for CORE protocol

      Jul 30, 2014 11:05:24 AM org.hornetq.core.remoting.impl.netty.NettyAcceptor start

      INFO: HQ221020: Started Netty Acceptor version 3.6.2.Final-c0d783c 10.92.14.196:5445 for CORE protocol

      Jul 30, 2014 11:05:24 AM org.hornetq.core.client.impl.ServerLocatorImpl$StaticConnector$1 connectionFailed

      WARN: HQ212028: error starting server locator

      HornetQException[errorType=ILLEGAL_STATE message=null]

              at org.hornetq.core.client.impl.ServerLocatorImpl.initialise(ServerLocatorImpl.java:371)

              at org.hornetq.core.client.impl.ServerLocatorImpl.start(ServerLocatorImpl.java:566)

              at org.hornetq.core.client.impl.ServerLocatorImpl$StaticConnector$1.connectionFailed(ServerLocatorImpl.java:1773)

              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:570)

              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:341)

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1631)

              at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:106)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

              at java.lang.Thread.run(Thread.java:745)

       

      Jul 30, 2014 11:05:24 AM org.hornetq.core.server.impl.HornetQServerImpl$SharedStoreBackupActivation run

      INFO: HQ221010: Backup Server is now live

       

      I went over my configuration and compared it with the static-connector example included with 2.3, but can't figure out what the problem is.

      Server 1 Config:

       

      <configuration xmlns="urn:hornetq"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">
   <clustered>true</clustered>
   <cluster-user>hornetQClusterUser</cluster-user>
   <cluster-password>XXXX</cluster-password>
   <shared-store>true</shared-store>
   <backup>${backup:false}</backup>
   <allow-failback>true</allow-failback>
   <failover-on-shutdown>true</failover-on-shutdown>
   <paging-directory>${data.dir:../data}/paging</paging-directory>
   <bindings-directory>${data.dir:../data}/bindings</bindings-directory>
   <journal-directory>${data.dir:../data}/journal</journal-directory>
   <journal-min-files>10</journal-min-files>
   <large-messages-directory>${data.dir:../data}/large-messages</large-messages-directory>
   <!-- Connectors -->
   <connectors>
    <connector name="netty">
      <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
      <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>
      <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
    </connector>
    <connector name="server2-connector">
      <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
      <param key="host"  value="10.92.14.196"/>
      <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
    </connector>
    <connector name="netty-throughput">
      <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
      <param key="host" value="${hornetq.remoting.netty.host:localhost}"/>
      <param key="port" value="${hornetq.remoting.netty.batch.port:5455}"/>
      <param key="batch-delay" value="50"/>
    </connector>
  </connectors>
  <!-- Acceptors -->
  <acceptors>
    <acceptor name="netty">
      <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
      <param key="host" value="${hornetq.remoting.netty.host:localhost}"/>
      <param key="port" value="${hornetq.remoting.netty.port:5445}"/>
    </acceptor>
    <acceptor name="netty-throughput">
      <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
      <param key="host" value="${hornetq.remoting.netty.host:localhost}"/>
      <param key="port" value="${hornetq.remoting.netty.batch.port:5455}"/>
      <param key="batch-delay" value="50"/>
      <param key="direct-deliver" value="false"/>
    </acceptor>
  </acceptors>
  <cluster-connections>
    <cluster-connection name="my-cluster">
      <address>jms</address>
      <connector-ref>netty</connector-ref>
      <retry-interval>500</retry-interval>
      <use-duplicate-detection>true</use-duplicate-detection>
      <forward-when-no-consumers>false</forward-when-no-consumers>
      <static-connectors>
        <connector-ref>server2-connector</connector-ref>
      </static-connectors>
    </cluster-connection>
  </cluster-connections>
  <security-settings>
    <security-setting match="#">
      <permission type="createNonDurableQueue" roles="guest"/>
      <permission type="deleteNonDurableQueue" roles="guest"/>
      <permission type="consume" roles="guest"/>
      <permission type="send" roles="guest"/>
    </security-setting>
  </security-settings>
  <address-settings>
      <!--default for catch all-->
    <address-setting match="#">
      <dead-letter-address>jms.queue.DLQ</dead-letter-address>
      <expiry-address>jms.queue.ExpiryQueue</expiry-address>
      <redistribution-delay>0</redistribution-delay>
      <redelivery-delay>0</redelivery-delay>
      <max-size-bytes>1048576000</max-size-bytes>
      <message-counter-history-day-limit>10</message-counter-history-day-limit>
      <address-full-policy>BLOCK</address-full-policy>
    </address-setting>
    <address-setting match="jms.#">
      <redistribution-delay>0</redistribution-delay>
    </address-setting>
  </address-settings>
</configuration>

       

      Server 2 Config:

       

      <configuration xmlns="urn:hornetq"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">
   <clustered>true</clustered>
   <cluster-user>hornetQClusterUser</cluster-user>
   <cluster-password>XXXX</cluster-password>
   <shared-store>true</shared-store>
   <backup>${backup:false}</backup>
   <allow-failback>true</allow-failback>
   <failover-on-shutdown>true</failover-on-shutdown>
   <paging-directory>${data.dir:../data}/paging</paging-directory>
   <bindings-directory>${data.dir:../data}/bindings</bindings-directory>
   <journal-directory>${data.dir:../data}/journal</journal-directory>
   <journal-min-files>10</journal-min-files>
   <large-messages-directory>${data.dir:../data}/large-messages</large-messages-directory>
   <!-- Connectors -->
   <connectors>
    <connector name="netty">
      <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
      <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>
      <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
    </connector>
    <connector name="server1-connector">
      <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
      <param key="host"  value="10.92.14.194"/>
      <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
    </connector>

    <connector name="netty-throughput">
      <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
      <param key="host" value="${hornetq.remoting.netty.host:localhost}"/>
      <param key="port" value="${hornetq.remoting.netty.batch.port:5455}"/>
      <param key="batch-delay" value="50"/>
    </connector>
  </connectors>
  <!-- Acceptors -->
  <acceptors>
    <acceptor name="netty">
      <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
      <param key="host" value="${hornetq.remoting.netty.host:localhost}"/>
      <param key="port" value="${hornetq.remoting.netty.port:5445}"/>
    </acceptor>
    <acceptor name="netty-throughput">
      <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
      <param key="host" value="${hornetq.remoting.netty.host:localhost}"/>
      <param key="port" value="${hornetq.remoting.netty.batch.port:5455}"/>
      <param key="batch-delay" value="50"/>
      <param key="direct-deliver" value="false"/>
    </acceptor>
  </acceptors>
  <cluster-connections>
    <cluster-connection name="my-cluster">
      <address>jms</address>
      <connector-ref>netty</connector-ref>
      <retry-interval>500</retry-interval>
      <use-duplicate-detection>true</use-duplicate-detection>
      <forward-when-no-consumers>false</forward-when-no-consumers>
      <static-connectors>
        <connector-ref>server1-connector</connector-ref>
      </static-connectors>
    </cluster-connection>
  </cluster-connections>
  <security-settings>
    <security-setting match="#">
      <permission type="createNonDurableQueue" roles="guest"/>
      <permission type="deleteNonDurableQueue" roles="guest"/>
      <permission type="consume" roles="guest"/>
      <permission type="send" roles="guest"/>
    </security-setting>
  </security-settings>
  <address-settings>
      <!--default for catch all-->
    <address-setting match="#">
      <dead-letter-address>jms.queue.DLQ</dead-letter-address>
      <expiry-address>jms.queue.ExpiryQueue</expiry-address>
      <redistribution-delay>0</redistribution-delay>
      <redelivery-delay>0</redelivery-delay>
      <max-size-bytes>1048576000</max-size-bytes>
      <message-counter-history-day-limit>10</message-counter-history-day-limit>
      <address-full-policy>BLOCK</address-full-policy>
    </address-setting>
    <address-setting match="jms.#">
      <redistribution-delay>0</redistribution-delay>
    </address-setting>
  </address-settings>
</configuration>