6 Replies Latest reply on Jul 7, 2016 2:36 PM by lendebaets

Getting exception in log during failback and no message is processed - Remote queue binding has already been bound in the post office

veenaonnet Apr 25, 2014 8:40 AM

Hi,

I am using HornetQ 2.4 with JBoss with collocated topology with replication in paging mode.

Scenario is -

Node 1: Started - HornetQ live started

Node 2: Started - HornetQ live started

Node 2:Stopped

Node1 - HornetQ backup started

Node 2 - Started - Attempts failback

At this point

Node 2 shows error

[org.hornetq.core.server] (Thread-1 (hornetq-netty-threads-564058766)) HQ224018: Failed to create session: HornetQSessionCreationException[errorType=SESSION_CREATION_REJECTED message=HQ119034: Server not started]
at org.hornetq.core.protocol.core.impl.HornetQPacketHandler.handleCreateSession(HornetQPacketHandler.java:122) [hornetq-server-2.4.0-SNAPSHOT.jar:]
at org.hornetq.core.protocol.core.impl.HornetQPacketHandler.handlePacket(HornetQPacketHandler.java:78) [hornetq-server-2.4.0-SNAPSHOT.jar:]
at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:631) [hornetq-core-client-2.4.0-SNAPSHOT.jar:]
at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:547) [hornetq-core-client-2.4.0-SNAPSHOT.jar:]
at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:523) [hornetq-core-client-2.4.0-SNAPSHOT.jar:]
at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:629) [hornetq-server-2.4.0-SNAPSHOT.jar:]
at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.channelRead(HornetQChannelHandler.java:74) [hornetq-core-client-2.4.0-SNAPSHOT.jar:]
at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338) [netty-all-4.0.12.Final.jar:]
at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324) [netty-all-4.0.12.Final.jar:]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:153) [netty-all-4.0.12.Final.jar:]
at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338) [netty-all-4.0.12.Final.jar:]
at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324) [netty-all-4.0.12.Final.jar:]
at io.netty.handler.codec.ByteToMessageDecoder.handlerRemoved(ByteToMessageDecoder.java:110) [netty-all-4.0.12.Final.jar:]

However HornetQ server gets started after that

2014-04-25 01:57:26,108 INFO [org.hornetq.core.server] (MSC service thread 1-27) HQ221007: Server is now live
2014-04-25 01:57:26,108 INFO [org.hornetq.core.server] (MSC service thread 1-27) HQ221001: HornetQ Server version 2.4.0.SNAPSHOT (Andromedian Fly, 123) [991862ea-cc57-11e3-899d-217962e95a1d]
2014-04-25 01:57:26,112 INFO [org.hornetq.core.server] (ServerService Thread Pool -- 66) HQ221003: trying to deploy queue jms.queue.FileMoveJobs
2014-04-25 01:57:26,170 INFO [org.jboss.as.messaging] (ServerService Thread Pool -- 66) JBAS011601: Bound messaging object to jndi name java:/queue/FileMoveJobs
2014-04-25 01:57:26,171 INFO [org.hornetq.core.server] (ServerService Thread Pool -- 88) HQ221003: trying to deploy queue jms.topic.Notification
2014-04-25 01:57:26,211 INFO [org.jboss.as.messaging] (ServerService Thread Pool -- 88) JBAS011601: Bound messaging object to jndi name java:/topic/Notification
2014-04-25 01:57:26,211 INFO [org.hornetq.core.server] (ServerService Thread Pool -- 87) HQ221003: trying to deploy queue jms.queue.UpdateClosedCaptionStructureJobs
2014-04-25 01:57:26,233 INFO [org.jboss.as.messaging] (ServerService Thread Pool -- 87) JBAS011601: Bound messaging object to jndi name java:/queue/UpdateClosedCaptionStructureJobs
2014-04-25 01:57:26,233 INFO [org.hornetq.core.server] (ServerService Thread Pool -- 61) HQ221003: trying to deploy queue jms.queue.SynchronizeFolderJobs
2014-04-25 01:57:26,240 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-10) JBAS010406: Registered connection factory java:/JmsXA
2014-04-25 01:57:26,249 INFO [org.hornetq.core.server] (Thread-10 (HornetQ-server-HornetQServerImpl::serverUUID=991862ea-cc57-11e3-899d-217962e95a1d-1200323700)) HQ221027: Bridge ClusterConnectionBridge@57d9b2a4 [name=sf.msf-cluster-10.4.167.99.7389f807-cc48-11e3-9d7d-4daadff738e7, queue=QueueImpl[name=sf.msf-cluster-10.4.167.99.7389f807-cc48-11e3-9d7d-4daadff738e7, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=991862ea-cc57-11e3-899d-217962e95a1d]]@4a9e9396 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@57d9b2a4 [name=sf.msf-cluster-10.4.167.99.7389f807-cc48-11e3-9d7d-4daadff738e7, queue=QueueImpl[name=sf.msf-cluster-10.4.167.99.7389f807-cc48-11e3-9d7d-4daadff738e7, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=991862ea-cc57-11e3-899d-217962e95a1d]]@4a9e9396 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-27], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1472723337[nodeUUID=991862ea-cc57-11e3-899d-217962e95a1d, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-28, address=jms, server=HornetQServerImpl::serverUUID=991862ea-cc57-11e3-899d-217962e95a1d])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-27], discoveryGroupConfiguration=null]] is connected

At this time it seems that Node1 is also trying to start the bridge due to the first error on Node 2

Node 1:

2014-04-25 01:57:26,116 WARN [org.hornetq.core.server] (Thread-23 (HornetQ-server-HornetQServerImpl::serverUUID=null-172395014)) HQ222098: Server is starting, retry to create the session for bridge sf.msf-cluster-10.4.167.99.a595a484-cb70-11e3-aae2-af4ddcb5b420

2014-04-25 01:57:26,178 INFO [org.hornetq.core.server] (Thread-28 (HornetQ-server-HornetQServerImpl::serverUUID=null-172395014)) HQ221027: Bridge ClusterConnectionBridge@3fe12713 [name=sf.msf-cluster-10.4.167.99.991862ea-cc57-11e3-899d-217962e95a1d, queue=QueueImpl[name=sf.msf-cluster-10.4.167.99.991862ea-cc57-11e3-899d-217962e95a1d, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=7389f807-cc48-11e3-9d7d-4daadff738e7]]@56cc7128 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@3fe12713 [name=sf.msf-cluster-10.4.167.99.991862ea-cc57-11e3-899d-217962e95a1d, queue=QueueImpl[name=sf.msf-cluster-10.4.167.99.991862ea-cc57-11e3-899d-217962e95a1d, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=7389f807-cc48-11e3-9d7d-4daadff738e7]]@56cc7128 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-28], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@277207633[nodeUUID=7389f807-cc48-11e3-9d7d-4daadff738e7, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-27, address=jms, server=HornetQServerImpl::serverUUID=7389f807-cc48-11e3-9d7d-4daadff738e7])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-28], discoveryGroupConfiguration=null]] is connected

2014-04-25 01:57:27,137 INFO [org.hornetq.core.server] (Thread-21 (HornetQ-server-HornetQServerImpl::serverUUID=null-172395014)) HQ221027: Bridge ClusterConnectionBridge@6e98d7c1 [name=sf.msf-cluster-10.4.167.99.a595a484-cb70-11e3-aae2-af4ddcb5b420, queue=QueueImpl[name=sf.msf-cluster-10.4.167.99.a595a484-cb70-11e3-aae2-af4ddcb5b420, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=7389f807-cc48-11e3-9d7d-4daadff738e7]]@747cf8a0 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@6e98d7c1 [name=sf.msf-cluster-10.4.167.99.a595a484-cb70-11e3-aae2-af4ddcb5b420, queue=QueueImpl[name=sf.msf-cluster-10.4.167.99.a595a484-cb70-11e3-aae2-af4ddcb5b420, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=7389f807-cc48-11e3-9d7d-4daadff738e7]]@747cf8a0 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-28], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@277207633[nodeUUID=7389f807-cc48-11e3-9d7d-4daadff738e7, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-27, address=jms, server=HornetQServerImpl::serverUUID=7389f807-cc48-11e3-9d7d-4daadff738e7])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-28], discoveryGroupConfiguration=null]] is connected

2014-04-25 01:57:33,062 WARN [org.hornetq.core.server] (Thread-52 (HornetQ-client-global-threads-107449287)) HQ222139: MessageFlowRecordImpl [nodeID=a595a484-cb70-11e3-aae2-af4ddcb5b420, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-28, queueName=sf.msf-cluster-10.4.167.99.a595a484-cb70-11e3-aae2-af4ddcb5b420, queue=QueueImpl[name=sf.msf-cluster-10.4.167.99.a595a484-cb70-11e3-aae2-af4ddcb5b420, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=7389f807-cc48-11e3-9d7d-4daadff738e7]]@747cf8a0, isClosed=false, firstReset=true]::Remote queue binding 7a9c07e8-e1d1-4f59-b6a0-927fb8b61544991862ea-cc57-11e3-899d-217962e95a1d has already been bound in the post office. Most likely cause for this is you have a loop in your cluster due to cluster max-hops being too large or you have multiple cluster connections to the same nodes using overlapping addresses

and then it shows errors for all the messages

2014-04-25 01:57:33,063 ERROR [org.hornetq.core.server] (Thread-52 (HornetQ-client-global-threads-107449287)) HQ224037: cluster connection Failed to handle message: java.lang.IllegalStateException: Cannot find binding for 1dadda13-b432-4004-b1df-e180d77245a5991862ea-cc57-11e3-899d-217962e95a1d on ClusterConnectionImpl@277207633[nodeUUID=7389f807-cc48-11e3-9d7d-4daadff738e7, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-27, address=jms, server=HornetQServerImpl::serverUUID=7389f807-cc48-11e3-9d7d-4daadff738e7]
        at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.doConsumerCreated(ClusterConnectionImpl.java:1396) [hornetq-server-2.4.0-SNAPSHOT.jar:]
        at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.onMessage(ClusterConnectionImpl.java:1143) [hornetq-server-2.4.0-SNAPSHOT.jar:]
        at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1117) [hornetq-core-client-2.4.0-SNAPSHOT.jar:]
        at org.hornetq.core.client.impl.ClientConsumerImpl.access$500(ClientConsumerImpl.java:57) [hornetq-core-client-2.4.0-SNAPSHOT.jar:]
        at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1252) [hornetq-core-client-2.4.0-SNAPSHOT.jar:]

Is it because bridge is started by 2 different threads(2 different nodes at the same time) with same addresses? Is it some race condition?

Laos, is there any reason because of which HornetQ server did not start first time on Node2 after failback?

Please note that failback is working fine normally.This issue is not seen always.

Please let me know how to solve it?

Regards,

Veena

1. Re: Getting exception in log during failback and no message is processed - Remote queue binding has already been bound in the post office

gaohoward Apr 25, 2014 8:53 AM (in response to veenaonnet)

In your scenario:

Node 1: Started - HornetQ live started
Node 2: Started - HornetQ live started
Node 2:Stopped
Node1 - HornetQ backup started
Node 2 - Started - Attempts failback

When Node 2 stopped, shouldn't Node 2 backup started ? Why Node 1 backup started? That seems strange.
Actions
2. Re: Getting exception in log during failback and no message is processed - Remote queue binding has already been bound in the post office

veenaonnet Apr 28, 2014 12:14 AM (in response to gaohoward)

Hi,

Sorry for the confusion. I am using collocated topology hence backup of Node 2 is on Node1. That is what I meant. So yes you are correct, backup of Node2 HornetQ which was on Node1 started.

Node 1: Started - HornetQ live started
Node 2: Started - HornetQ live started
Node 2:Stopped
HornetQ backup of Node2 on Node1 started
Node 2 - Started - Attempts failback - Live on Node2 started
Actions
3. Re: Getting exception in log during failback and no message is processed - Remote queue binding has already been bound in the post office

gaohoward Apr 28, 2014 9:15 AM (in response to veenaonnet)

Thanks. Do you mind uploading your configuration files?
Actions
4. Re: Getting exception in log during failback and no message is processed - Remote queue binding has already been bound in the post office

veenaonnet Apr 29, 2014 2:27 AM (in response to gaohoward)

Hi,

Please find below HornetQ configuration of both the nodes.
Node1:
<subsystem xmlns="urn:jboss:domain:messaging:1.3">
   <hornetq-server>
    <clustered>true</clustered>
    <persistence-enabled>true</persistence-enabled>
    <security-enabled>true</security-enabled>
    <cluster-user>xyz</cluster-user>
    <cluster-password>xyz</cluster-password>
    <failover-on-shutdown>true</failover-on-shutdown>
    <shared-store>false</shared-store>
    <journal-type>ASYNCIO</journal-type>
    <journal-file-size>10485760</journal-file-size>
    <journal-min-files>2</journal-min-files>
    <check-for-live-server>true</check-for-live-server>
    <backup-group-name>x.x.x.1</backup-group-name>
    <paging-directory path="x.x.x.1/paging"/>
    <bindings-directory path="x.x.x.1/bindings"/>
    <journal-directory path="x.x.x.1/journal"/>
    <large-messages-directory path="x.x.x.1/large-messages"/>
    <max-saved-replicated-journals-size>40
    </max-saved-replicated-journals-size>
    <connectors>
     <connector name="netty">
      <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory
      </factory-class>
      <param key="host" value="x.x.x.1"/>
      <param key="port" value="5445"/>
     </connector>
     <netty-connector name="netty-throughput" socket-binding="messaging-throughput">
      <param key="batch-delay" value="50"/>
     </netty-connector>
     <in-vm-connector name="in-vm" server-id="0"/>
    </connectors>
    <acceptors>
     <acceptor name="netty">
                        <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
                        <param key="host" value="0.0.0.0"/>
                        <param key="port" value="5445"/>
                    </acceptor>
     <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">
      <param key="batch-delay" value="50"/>
      <param key="direct-deliver" value="false"/>
     </netty-acceptor>
     <in-vm-acceptor name="in-vm" server-id="0"/>
    </acceptors>
    <broadcast-groups>
     <broadcast-group name="bg-group">
      <group-address>228.x.x.99</group-address>
      <group-port>9876</group-port>
      <broadcast-period>5000</broadcast-period>
      <connector-ref>netty</connector-ref>
     </broadcast-group>
    </broadcast-groups>
    <discovery-groups>
     <discovery-group name="dg-group">
      <group-address>228.x.x.99</group-address>
      <group-port>9876</group-port>
      <refresh-timeout>10000</refresh-timeout>
     </discovery-group>
    </discovery-groups>
    <cluster-connections>
     <cluster-connection name="msf-cluster-x.x.x.99">
      <address>jms</address>
      <connector-ref>netty</connector-ref>
      <discovery-group-ref discovery-group-name="dg-group"/>
      <retry-interval>1000</retry-interval>
      <use-duplicate-detection>true</use-duplicate-detection>
      <forward-when-no-consumers>false</forward-when-no-consumers>
      <max-hops>1</max-hops>
     </cluster-connection>
    </cluster-connections>
    <security-settings>
     <security-setting match="#">
      <permission roles="guest" type="send"/>
      <permission roles="guest" type="consume"/>
      <permission roles="guest" type="createDurableQueue"/>
      <permission roles="guest" type="deleteDurableQueue"/>
      <permission roles="guest" type="createNonDurableQueue"/>
      <permission roles="guest" type="deleteNonDurableQueue"/>
      <permission roles="guest" type="manage"/>
     </security-setting>
    </security-settings>
    <address-settings>

     <address-setting match="#">
      <dead-letter-address>jms.queue.DLQ</dead-letter-address>
      <expiry-address>jms.queue.ExpiryQueue</expiry-address>
      <redelivery-delay>0</redelivery-delay>
      <redistribution-delay>0</redistribution-delay>
      <page-size-bytes>10485760</page-size-bytes>
      <max-size-bytes>104857600</max-size-bytes>
      <address-full-policy>PAGE</address-full-policy>
      <message-counter-history-day-limit>1
      </message-counter-history-day-limit>
     </address-setting>
    </address-settings>
    <jms-connection-factories>
     <connection-factory name="InVmConnectionFactory">
      <connectors>
       <connector-ref connector-name="in-vm"/>
      </connectors>
      <entries>
       <entry name="java:/ConnectionFactory"/>
      </entries>
      <ha>true</ha>
               <block-on-acknowledge>true</block-on-acknowledge>
                        <retry-interval>1000</retry-interval>
               <retry-interval-multiplier>1.0</retry-interval-multiplier>
                        <reconnect-attempts>3</reconnect-attempts>
      <client-failure-check-period>-1</client-failure-check-period>
                        <connection-ttl>-1</connection-ttl>
                        <confirmation-window-size>1000000</confirmation-window-size>
      <call-timeout>180000</call-timeout>
     </connection-factory>
     <connection-factory name="RemoteConnectionFactory">
      <connectors>
       <connector-ref connector-name="netty"/>
      </connectors>
      <entries>
       <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
      </entries>
      <ha>true</ha>
      <block-on-acknowledge>true</block-on-acknowledge>
      <retry-interval>1000</retry-interval>
      <retry-interval-multiplier>1.0</retry-interval-multiplier>
      <reconnect-attempts>3</reconnect-attempts>
      <client-failure-check-period>60000</client-failure-check-period>
                        <connection-ttl>600000</connection-ttl>
      <confirmation-window-size>1000000</confirmation-window-size>
      <call-timeout>180000</call-timeout>
     </connection-factory>
     <pooled-connection-factory name="hornetq-ra">
      <transaction mode="xa"/>
      <connectors>
       <connector-ref connector-name="in-vm"/>
      </connectors>
      <entries>
       <entry name="java:/JmsXA"/>
      </entries>
     </pooled-connection-factory>
    </jms-connection-factories>
    <jms-destinations>
     <jms-queue name="Jobs">
      <entry name="queue/Jobs"/>
      <durable>true</durable>
     </jms-queue>
     <jms-topic name="Notification">
      <entry name="topic/Notification"/>
     </jms-topic>
    </jms-destinations>
   </hornetq-server>
   <hornetq-server name="backup">
    <clustered>true</clustered>
    <backup>true</backup>
    <persistence-enabled>true</persistence-enabled>
    <security-enabled>true</security-enabled>
    <cluster-user>xyz</cluster-user>
    <cluster-password>xyz</cluster-password>
    <allow-failback>true</allow-failback>
    <failover-on-shutdown>true</failover-on-shutdown>
    <shared-store>false</shared-store>
    <journal-type>ASYNCIO</journal-type>
    <journal-file-size>102400</journal-file-size>
    <journal-min-files>2</journal-min-files>
    <check-for-live-server>true</check-for-live-server>
    <backup-group-name>x.x.x.2
    </backup-group-name>
    <paging-directory path="x.x.x.2/paging"/>
    <bindings-directory path="x.x.x.2/bindings"/>
    <journal-directory path="x.x.x.2/journal"/>
    <large-messages-directory path="x.x.x.2/large-messages"/>
    <max-saved-replicated-journals-size>40
    </max-saved-replicated-journals-size>
    <live-connector-ref>netty</live-connector-ref>
    <connectors>
     <connector name="netty-backup">
      <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory
      </factory-class>
      <param key="host" value="x.x.x.1"/>
      <param key="port" value="5446"/>
     </connector>
     <in-vm-connector name="in-vm" server-id="1"/>
    </connectors>
    <acceptors>
     <in-vm-acceptor name="in-vm" server-id="1"/>
                    <acceptor name="netty-backup">
                        <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
                        <param key="host" value="0.0.0.0"/>
                        <param key="port" value="5446"/>
                    </acceptor>
    </acceptors>
    <broadcast-groups>
     <broadcast-group name="bg-group-backup">
      <group-address>228.x.x.99</group-address>
      <group-port>9876</group-port>
      <broadcast-period>5000</broadcast-period>
      <connector-ref>netty-backup</connector-ref>
     </broadcast-group>
    </broadcast-groups>
    <discovery-groups>
     <discovery-group name="dg-group-backup">
      <group-address>228.x.x.99</group-address>
      <group-port>9876</group-port>
      <refresh-timeout>10000</refresh-timeout>
     </discovery-group>
    </discovery-groups>
    <cluster-connections>
     <cluster-connection name="msf-cluster-x.x.x.99">
      <address>jms</address>
      <connector-ref>netty-backup</connector-ref>
      <discovery-group-ref discovery-group-name="dg-group-backup"/>
      <retry-interval>1000</retry-interval>
      <use-duplicate-detection>true</use-duplicate-detection>
      <forward-when-no-consumers>false</forward-when-no-consumers>
      <max-hops>1</max-hops>
     </cluster-connection>
    </cluster-connections>
    <security-settings>
     <security-setting match="#">
      <permission roles="guest" type="send"/>
      <permission roles="guest" type="consume"/>
      <permission roles="guest" type="createDurableQueue"/>
      <permission roles="guest" type="deleteDurableQueue"/>
      <permission roles="guest" type="createNonDurableQueue"/>
      <permission roles="guest" type="deleteNonDurableQueue"/>
      <permission roles="guest" type="manage"/>
     </security-setting>
    </security-settings>
    <address-settings>
     <address-setting match="#">
      <dead-letter-address>jms.queue.DLQ</dead-letter-address>
      <expiry-address>jms.queue.ExpiryQueue</expiry-address>
      <redelivery-delay>0</redelivery-delay>
      <redistribution-delay>0</redistribution-delay>
      <page-size-bytes>10485760</page-size-bytes>
      <max-size-bytes>104857600</max-size-bytes>
      <address-full-policy>PAGE</address-full-policy>
      <message-counter-history-day-limit>1
      </message-counter-history-day-limit>
     </address-setting>
    </address-settings>
   </hornetq-server>
</subsystem>
Node2:
<subsystem xmlns="urn:jboss:domain:messaging:1.3">
   <hornetq-server>
    <clustered>true</clustered>
    <persistence-enabled>true</persistence-enabled>
    <security-enabled>true</security-enabled>
    <cluster-user>xyz</cluster-user>
    <cluster-password>xyz</cluster-password>
    <failover-on-shutdown>true</failover-on-shutdown>
    <shared-store>false</shared-store>
    <journal-type>ASYNCIO</journal-type>
    <journal-file-size>10485760</journal-file-size>
    <journal-min-files>2</journal-min-files>
    <check-for-live-server>true</check-for-live-server>
    <backup-group-name>x.x.x.2</backup-group-name>
    <paging-directory path="x.x.x.2/paging"/>
    <bindings-directory path="x.x.x.2/bindings"/>
    <journal-directory path="x.x.x.2/journal"/>
    <large-messages-directory path="x.x.x.2/large-messages"/>
    <max-saved-replicated-journals-size>40
    </max-saved-replicated-journals-size>
    <connectors>
     <connector name="netty">
      <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory
      </factory-class>
      <param key="host" value="x.x.x.2"/>
      <param key="port" value="5445"/>
     </connector>
     <netty-connector name="netty-throughput" socket-binding="messaging-throughput">
      <param key="batch-delay" value="50"/>
     </netty-connector>
     <in-vm-connector name="in-vm" server-id="0"/>
    </connectors>
    <acceptors>
     <acceptor name="netty">
                        <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
                        <param key="host" value="0.0.0.0"/>
                        <param key="port" value="5445"/>
                    </acceptor>
     <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">
      <param key="batch-delay" value="50"/>
      <param key="direct-deliver" value="false"/>
     </netty-acceptor>
     <in-vm-acceptor name="in-vm" server-id="0"/>
    </acceptors>
    <broadcast-groups>
     <broadcast-group name="bg-group">
      <group-address>228.x.x.99</group-address>
      <group-port>9876</group-port>
      <broadcast-period>5000</broadcast-period>
      <connector-ref>netty</connector-ref>
     </broadcast-group>
    </broadcast-groups>
    <discovery-groups>
     <discovery-group name="dg-group">
      <group-address>228.x.x.99</group-address>
      <group-port>9876</group-port>
      <refresh-timeout>10000</refresh-timeout>
     </discovery-group>
    </discovery-groups>
    <cluster-connections>
     <cluster-connection name="msf-cluster-x.x.x.99">
      <address>jms</address>
      <connector-ref>netty</connector-ref>
      <discovery-group-ref discovery-group-name="dg-group"/>
      <retry-interval>1000</retry-interval>
      <use-duplicate-detection>true</use-duplicate-detection>
      <forward-when-no-consumers>false</forward-when-no-consumers>
      <max-hops>1</max-hops>
     </cluster-connection>
    </cluster-connections>
    <security-settings>
     <security-setting match="#">
      <permission roles="guest" type="send"/>
      <permission roles="guest" type="consume"/>
      <permission roles="guest" type="createDurableQueue"/>
      <permission roles="guest" type="deleteDurableQueue"/>
      <permission roles="guest" type="createNonDurableQueue"/>
      <permission roles="guest" type="deleteNonDurableQueue"/>
      <permission roles="guest" type="manage"/>
     </security-setting>
    </security-settings>
    <address-settings>

     <address-setting match="#">
      <dead-letter-address>jms.queue.DLQ</dead-letter-address>
      <expiry-address>jms.queue.ExpiryQueue</expiry-address>
      <redelivery-delay>0</redelivery-delay>
      <redistribution-delay>0</redistribution-delay>
      <page-size-bytes>10485760</page-size-bytes>
      <max-size-bytes>104857600</max-size-bytes>
      <address-full-policy>PAGE</address-full-policy>
      <message-counter-history-day-limit>1
      </message-counter-history-day-limit>
     </address-setting>
    </address-settings>
    <jms-connection-factories>
     <connection-factory name="InVmConnectionFactory">
      <connectors>
       <connector-ref connector-name="in-vm"/>
      </connectors>
      <entries>
       <entry name="java:/ConnectionFactory"/>
      </entries>
      <ha>true</ha>
               <block-on-acknowledge>true</block-on-acknowledge>
                        <retry-interval>1000</retry-interval>
               <retry-interval-multiplier>1.0</retry-interval-multiplier>
                        <reconnect-attempts>3</reconnect-attempts>
      <client-failure-check-period>-1</client-failure-check-period>
                        <connection-ttl>-1</connection-ttl>
                        <confirmation-window-size>1000000</confirmation-window-size>
      <call-timeout>180000</call-timeout>
     </connection-factory>
     <connection-factory name="RemoteConnectionFactory">
      <connectors>
       <connector-ref connector-name="netty"/>
      </connectors>
      <entries>
       <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
      </entries>
      <ha>true</ha>
      <block-on-acknowledge>true</block-on-acknowledge>
      <retry-interval>1000</retry-interval>
      <retry-interval-multiplier>1.0</retry-interval-multiplier>
      <reconnect-attempts>3</reconnect-attempts>
      <client-failure-check-period>60000</client-failure-check-period>
                        <connection-ttl>600000</connection-ttl>
      <confirmation-window-size>1000000</confirmation-window-size>
      <call-timeout>180000</call-timeout>
     </connection-factory>
     <pooled-connection-factory name="hornetq-ra">
      <transaction mode="xa"/>
      <connectors>
       <connector-ref connector-name="in-vm"/>
      </connectors>
      <entries>
       <entry name="java:/JmsXA"/>
      </entries>
     </pooled-connection-factory>
    </jms-connection-factories>
    <jms-destinations>
     <jms-queue name="Jobs">
      <entry name="queue/ProxyJobs"/>
      <durable>true</durable>
     </jms-queue>
     <jms-topic name="Notification">
      <entry name="topic/Notification"/>
     </jms-topic>
    </jms-destinations>
   </hornetq-server>
   <hornetq-server name="backup">
    <clustered>true</clustered>
    <backup>true</backup>
    <persistence-enabled>true</persistence-enabled>
    <security-enabled>true</security-enabled>
    <cluster-user>xyz</cluster-user>
    <cluster-password>xyz</cluster-password>
    <allow-failback>true</allow-failback>
    <failover-on-shutdown>true</failover-on-shutdown>
    <shared-store>false</shared-store>
    <journal-type>ASYNCIO</journal-type>
    <journal-file-size>102400</journal-file-size>
    <journal-min-files>2</journal-min-files>
    <check-for-live-server>true</check-for-live-server>
    <backup-group-name>x.x.x.1
    </backup-group-name>
    <paging-directory path="x.x.x.1/paging"/>
    <bindings-directory path="x.x.x.1/bindings"/>
    <journal-directory path="x.x.x.1/journal"/>
    <large-messages-directory path="x.x.x.1/large-messages"/>
    <max-saved-replicated-journals-size>40
    </max-saved-replicated-journals-size>
    <live-connector-ref>netty</live-connector-ref>
    <connectors>
     <connector name="netty-backup">
      <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory
      </factory-class>
      <param key="host" value="x.x.x.2"/>
      <param key="port" value="5446"/>
     </connector>
     <in-vm-connector name="in-vm" server-id="1"/>
    </connectors>
    <acceptors>
     <in-vm-acceptor name="in-vm" server-id="1"/>
                    <acceptor name="netty-backup">
                        <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
                        <param key="host" value="0.0.0.0"/>
                        <param key="port" value="5446"/>
                    </acceptor>
    </acceptors>
    <broadcast-groups>
     <broadcast-group name="bg-group-backup">
      <group-address>228.x.x.99</group-address>
      <group-port>9876</group-port>
      <broadcast-period>5000</broadcast-period>
      <connector-ref>netty-backup</connector-ref>
     </broadcast-group>
    </broadcast-groups>
    <discovery-groups>
     <discovery-group name="dg-group-backup">
      <group-address>228.x.x.99</group-address>
      <group-port>9876</group-port>
      <refresh-timeout>10000</refresh-timeout>
     </discovery-group>
    </discovery-groups>
    <cluster-connections>
     <cluster-connection name="msf-cluster-x.x.x.99">
      <address>jms</address>
      <connector-ref>netty-backup</connector-ref>
      <discovery-group-ref discovery-group-name="dg-group-backup"/>
      <retry-interval>1000</retry-interval>
      <use-duplicate-detection>true</use-duplicate-detection>
      <forward-when-no-consumers>false</forward-when-no-consumers>
      <max-hops>1</max-hops>
     </cluster-connection>
    </cluster-connections>
    <security-settings>
     <security-setting match="#">
      <permission roles="guest" type="send"/>
      <permission roles="guest" type="consume"/>
      <permission roles="guest" type="createDurableQueue"/>
      <permission roles="guest" type="deleteDurableQueue"/>
      <permission roles="guest" type="createNonDurableQueue"/>
      <permission roles="guest" type="deleteNonDurableQueue"/>
      <permission roles="guest" type="manage"/>
     </security-setting>
    </security-settings>
    <address-settings>
     <address-setting match="#">
      <dead-letter-address>jms.queue.DLQ</dead-letter-address>
      <expiry-address>jms.queue.ExpiryQueue</expiry-address>
      <redelivery-delay>0</redelivery-delay>
      <redistribution-delay>0</redistribution-delay>
      <page-size-bytes>10485760</page-size-bytes>
      <max-size-bytes>104857600</max-size-bytes>
      <address-full-policy>PAGE</address-full-policy>
      <message-counter-history-day-limit>1
      </message-counter-history-day-limit>
     </address-setting>
    </address-settings>
   </hornetq-server>
</subsystem>

Regards,
Veena
Actions
5. Re: Getting exception in log during failback and no message is processed - Remote queue binding has already been bound in the post office

thoemic Dec 10, 2015 11:42 AM (in response to veenaonnet)

Hi Veena,

Did you solve your problem? If so, i would appreciate if you could share any insights, since i'm experiencing the same problem.

Best,
Thomas
Actions
6. Re: Getting exception in log during failback and no message is processed - Remote queue binding has already been bound in the post office

lendebaets Jul 7, 2016 2:36 PM (in response to veenaonnet)

Was there any solution? I saw the same log in a High Availability Cluster environment with Docker Containers. They come out approximately 20 seconds after the topic subscribers subscribe to the topics.
Actions

Go to original post