Live server constantly tries and fails to connect to backup

[tomasz.lewandowski]

I have a HornetQ 2.2.14 standalone live/backup static tcp configuration. My OS is Centos 6. Java is Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)

After start (both Live and Backup) I see constant messages in Live log:

2014-03-27 09:21:13,008 Thread-24 (HornetQ-server-HornetQServerImpl::serverUUID=69f8ea44-b346-11e3-a530-dd4c353e1ec9-1951982241) DEBUG [	impl.netty.NettyConnector] [	debug] [ 91] : Started Netty Connector version 3.2.5.Final-a96d88c
2014-03-27 09:21:13,008 Thread-24 (HornetQ-server-HornetQServerImpl::serverUUID=69f8ea44-b346-11e3-a530-dd4c353e1ec9-1951982241) DEBUG [	client.impl.ClientSessionFactoryImpl] [	debug] [ 91] : Trying to connect at the main server using connector :org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=192.168.1.2
2014-03-27 09:21:13,009 Thread-24 (HornetQ-server-HornetQServerImpl::serverUUID=69f8ea44-b346-11e3-a530-dd4c353e1ec9-1951982241) DEBUG [	client.impl.ClientSessionFactoryImpl] [	debug] [ 91] : Main server is not up. Hopefully there's a backup configured now!
2014-03-27 09:21:13,009 Thread-24 (HornetQ-server-HornetQServerImpl::serverUUID=69f8ea44-b346-11e3-a530-dd4c353e1ec9-1951982241) DEBUG [	client.impl.ClientSessionFactoryImpl] [	debug] [ 91] : Trying to connect towards ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl (identity=(main-ClusterConnection::HornetQServerImpl::serverUUID=69f8ea44-b346-11e3-a530-dd4c353e1ec9)) [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=192.168.1.2], discoveryGroupConfiguration=null], connectorConfig=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=192.168.1.2, backupConfig=null]
2014-03-27 09:21:13,009 Thread-24 (HornetQ-server-HornetQServerImpl::serverUUID=69f8ea44-b346-11e3-a530-dd4c353e1ec9-1951982241) DEBUG [	client.impl.ServerLocatorImpl] [	debug] [ 96] : Connector [initialConnector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=192.168.1.2]::Exception on establish connector initial connection

HornetQException[errorCode=2 message=Unable to connect to server using configuration org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=192.168.1.2]

	at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:234)
	at org.hornetq.core.client.impl.ServerLocatorImpl$StaticConnector$Connector.tryConnect(ServerLocatorImpl.java:1777)
	at org.hornetq.core.client.impl.ServerLocatorImpl$StaticConnector.connect(ServerLocatorImpl.java:1614)
	at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:587)
	at org.hornetq.core.client.impl.ServerLocatorImpl$3.run(ServerLocatorImpl.java:554)
	at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

I observed that backup server doesn't listen on port 5445, until it becomes Live.

Is this correct behaviour or have I misconfigured something? This is my config:

Master (LIVE):

<configuration xmlns="urn:hornetq" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">

        <paging-directory>${shared.data.dir}/paging</paging-directory>
        <bindings-directory>${shared.data.dir}/bindings</bindings-directory>
        <journal-directory>${shared.data.dir}/journal</journal-directory>
        <journal-min-files>10</journal-min-files>
        <large-messages-directory>${shared.data.dir}/large-messages</large-messages-directory>

        <shared-store>true</shared-store>
        <clustered>true</clustered>

        <cluster-user>HORNETQ.CLUSTER.ADMIN.USER</cluster-user>
        <cluster-password>abc</cluster-password>
        <failover-on-shutdown>true</failover-on-shutdown>

        <acceptors>
                <acceptor name="netty-acceptor">
                        <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
                        <param key="port" value="${hornetq.remoting.netty.port.master}" />
                        <param key="host" value="${hornetq.remoting.netty.host.master}" />
                </acceptor>
        </acceptors>

        <connectors>
                <connector name="netty-connector-master">
                        <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
                        <param key="port" value="${hornetq.remoting.netty.port.master}" />
                        <param key="host" value="${hornetq.remoting.netty.host.master}" />
                </connector>
                <connector name="netty-connector-slave">
                        <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
                        <param key="port" value="${hornetq.remoting.netty.port.slave}" />
                        <param key="host" value="${hornetq.remoting.netty.host.slave}" />
                </connector>
        </connectors>

        <cluster-connections>
                <cluster-connection name="ik-cluster">
                        <address>jms</address>
                        <connector-ref>netty-connector-master</connector-ref>
                        <retry-interval>500</retry-interval>
                        <static-connectors>
                                <connector-ref>netty-connector-slave</connector-ref>
                        </static-connectors>
                </cluster-connection>
        </cluster-connections>

</configuration>

Slave (BACKUP):

<configuration xmlns="urn:hornetq" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">

        <paging-directory>${shared.data.dir}/paging</paging-directory>
        <bindings-directory>${shared.data.dir}/bindings</bindings-directory>
        <journal-directory>${shared.data.dir}/journal</journal-directory>
        <journal-min-files>10</journal-min-files>
        <large-messages-directory>${shared.data.dir}/large-messages</large-messages-directory>

        <shared-store>true</shared-store>
        <clustered>true</clustered>
        <backup>true</backup>
        
        <failover-on-shutdown>true</failover-on-shutdown>

        <cluster-user>HORNETQ.CLUSTER.ADMIN.USER</cluster-user>
        <cluster-password>abc</cluster-password>

        <acceptors>
                <acceptor name="netty-acceptor">
                        <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
                        <param key="port" value="${hornetq.remoting.netty.port.slave}" />
                        <param key="host" value="${hornetq.remoting.netty.host.slave}" />
                </acceptor>
        </acceptors>

        <connectors>
                <connector name="netty-connector-master">
                        <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
                        <param key="port" value="${hornetq.remoting.netty.port.master}" />
                        <param key="host" value="${hornetq.remoting.netty.host.master}" />
                </connector>
                <connector name="netty-connector-slave">
                        <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
                        <param key="port" value="${hornetq.remoting.netty.port.slave}" />
                        <param key="host" value="${hornetq.remoting.netty.host.slave}" />
                </connector>
        </connectors>

        <cluster-connections>
                <cluster-connection name="ik-cluster">
                        <address>jms</address>
                        <connector-ref>netty-connector-slave</connector-ref>
                        <retry-interval>500</retry-interval>
                        <static-connectors>
                                <connector-ref>netty-connector-master</connector-ref>
                        </static-connectors>
                </cluster-connection>
        </cluster-connections>

</configuration>

[jbertram]

Couple questions...

• What system properties are you setting on each server?
• Are all these log messages at DEBUG level?
• Have you noticed any functional impact on your application?
• Have you thought about upgrading to a more recent version (e.g. 2.4.0.Final)?

[ataylor]

yes, these are just debug messages we use for debugging. they are not errors.

[tomasz.lewandowski]

ad What system properties are you setting on each server?)

On both master(live) and slave (backup) I set:

-Dshared.data.dir=/opt/shared/hornetq-data

-Dhornetq.remoting.netty.host.master=192.168.1.1

-Dhornetq.remoting.netty.port.master=5445

-Dhornetq.remoting.netty.host.slave=192.168.1.2

-Dhornetq.remoting.netty.port.slave=5445

ad Are all these log messages at DEBUG level?)

Yes, thay are all DEBUG messages, but still I dont feel comfortable with it, as it repeats very often (probably due to retry-interval set to 500)

ad Have you noticed any functional impact on your application?)

I had some problems (not diagnozed fully yet) that clients temporarly couldn't connect to HornetQ, and I just want to investigate any suspicious logs.

HornetQ seems to work fine (also tested failover) but still dont feel comfortable with it.

ad Have you thought about upgrading to a more recent version (e.g. 2.4.0.Final)?

Upgrading would have bigger impact on my system so I don't consider it for now (we have a number of JBoss 7.1 as clients of this standalone HQ, so I expect compatibility problems).