6 Replies Latest reply on Jul 7, 2016 2:36 PM by Leonard De Baets

    Getting exception in log during failback and no message is processed - Remote queue binding has already been bound in the post office

    veenaonnet Newbie

      Hi,

       

      I am using HornetQ 2.4 with JBoss with collocated topology with replication in paging mode.

      Scenario is -

      Node 1: Started - HornetQ live started

      Node 2: Started - HornetQ live started

      Node 2:Stopped

      Node1  - HornetQ backup started

      Node 2 - Started - Attempts failback

       

      At this point

      Node 2 shows error

       

      [org.hornetq.core.server] (Thread-1 (hornetq-netty-threads-564058766)) HQ224018: Failed to create session: HornetQSessionCreationException[errorType=SESSION_CREATION_REJECTED message=HQ119034: Server not started]
      at org.hornetq.core.protocol.core.impl.HornetQPacketHandler.handleCreateSession(HornetQPacketHandler.java:122) [hornetq-server-2.4.0-SNAPSHOT.jar:]
      at org.hornetq.core.protocol.core.impl.HornetQPacketHandler.handlePacket(HornetQPacketHandler.java:78) [hornetq-server-2.4.0-SNAPSHOT.jar:]
      at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:631) [hornetq-core-client-2.4.0-SNAPSHOT.jar:]
      at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:547) [hornetq-core-client-2.4.0-SNAPSHOT.jar:]
      at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:523) [hornetq-core-client-2.4.0-SNAPSHOT.jar:]
      at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:629) [hornetq-server-2.4.0-SNAPSHOT.jar:]
      at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.channelRead(HornetQChannelHandler.java:74) [hornetq-core-client-2.4.0-SNAPSHOT.jar:]
      at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338) [netty-all-4.0.12.Final.jar:]
      at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324) [netty-all-4.0.12.Final.jar:]
      at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:153) [netty-all-4.0.12.Final.jar:]
      at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338) [netty-all-4.0.12.Final.jar:]
      at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324) [netty-all-4.0.12.Final.jar:]
      at io.netty.handler.codec.ByteToMessageDecoder.handlerRemoved(ByteToMessageDecoder.java:110) [netty-all-4.0.12.Final.jar:]

       

      However HornetQ server gets started after that

      2014-04-25 01:57:26,108 INFO [org.hornetq.core.server] (MSC service thread 1-27) HQ221007: Server is now live
      2014-04-25 01:57:26,108 INFO [org.hornetq.core.server] (MSC service thread 1-27) HQ221001: HornetQ Server version 2.4.0.SNAPSHOT (Andromedian Fly, 123) [991862ea-cc57-11e3-899d-217962e95a1d]
      2014-04-25 01:57:26,112 INFO [org.hornetq.core.server] (ServerService Thread Pool -- 66) HQ221003: trying to deploy queue jms.queue.FileMoveJobs
      2014-04-25 01:57:26,170 INFO [org.jboss.as.messaging] (ServerService Thread Pool -- 66) JBAS011601: Bound messaging object to jndi name java:/queue/FileMoveJobs
      2014-04-25 01:57:26,171 INFO [org.hornetq.core.server] (ServerService Thread Pool -- 88) HQ221003: trying to deploy queue jms.topic.Notification
      2014-04-25 01:57:26,211 INFO [org.jboss.as.messaging] (ServerService Thread Pool -- 88) JBAS011601: Bound messaging object to jndi name java:/topic/Notification
      2014-04-25 01:57:26,211 INFO [org.hornetq.core.server] (ServerService Thread Pool -- 87) HQ221003: trying to deploy queue jms.queue.UpdateClosedCaptionStructureJobs
      2014-04-25 01:57:26,233 INFO [org.jboss.as.messaging] (ServerService Thread Pool -- 87) JBAS011601: Bound messaging object to jndi name java:/queue/UpdateClosedCaptionStructureJobs
      2014-04-25 01:57:26,233 INFO [org.hornetq.core.server] (ServerService Thread Pool -- 61) HQ221003: trying to deploy queue jms.queue.SynchronizeFolderJobs
      2014-04-25 01:57:26,240 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-10) JBAS010406: Registered connection factory java:/JmsXA
      2014-04-25 01:57:26,249 INFO [org.hornetq.core.server] (Thread-10 (HornetQ-server-HornetQServerImpl::serverUUID=991862ea-cc57-11e3-899d-217962e95a1d-1200323700)) HQ221027: Bridge ClusterConnectionBridge@57d9b2a4 [name=sf.msf-cluster-10.4.167.99.7389f807-cc48-11e3-9d7d-4daadff738e7, queue=QueueImpl[name=sf.msf-cluster-10.4.167.99.7389f807-cc48-11e3-9d7d-4daadff738e7, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=991862ea-cc57-11e3-899d-217962e95a1d]]@4a9e9396 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@57d9b2a4 [name=sf.msf-cluster-10.4.167.99.7389f807-cc48-11e3-9d7d-4daadff738e7, queue=QueueImpl[name=sf.msf-cluster-10.4.167.99.7389f807-cc48-11e3-9d7d-4daadff738e7, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=991862ea-cc57-11e3-899d-217962e95a1d]]@4a9e9396 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-27], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1472723337[nodeUUID=991862ea-cc57-11e3-899d-217962e95a1d, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-28, address=jms, server=HornetQServerImpl::serverUUID=991862ea-cc57-11e3-899d-217962e95a1d])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-27], discoveryGroupConfiguration=null]] is connected

      At this time it seems that Node1 is also trying to start the bridge due to the first error on Node 2

      Node 1:

       

      2014-04-25 01:57:26,116 WARN  [org.hornetq.core.server] (Thread-23 (HornetQ-server-HornetQServerImpl::serverUUID=null-172395014)) HQ222098: Server is starting, retry to create the session for bridge sf.msf-cluster-10.4.167.99.a595a484-cb70-11e3-aae2-af4ddcb5b420


      2014-04-25 01:57:26,178 INFO  [org.hornetq.core.server] (Thread-28 (HornetQ-server-HornetQServerImpl::serverUUID=null-172395014)) HQ221027: Bridge ClusterConnectionBridge@3fe12713 [name=sf.msf-cluster-10.4.167.99.991862ea-cc57-11e3-899d-217962e95a1d, queue=QueueImpl[name=sf.msf-cluster-10.4.167.99.991862ea-cc57-11e3-899d-217962e95a1d, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=7389f807-cc48-11e3-9d7d-4daadff738e7]]@56cc7128 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@3fe12713 [name=sf.msf-cluster-10.4.167.99.991862ea-cc57-11e3-899d-217962e95a1d, queue=QueueImpl[name=sf.msf-cluster-10.4.167.99.991862ea-cc57-11e3-899d-217962e95a1d, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=7389f807-cc48-11e3-9d7d-4daadff738e7]]@56cc7128 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-28], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@277207633[nodeUUID=7389f807-cc48-11e3-9d7d-4daadff738e7, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-27, address=jms, server=HornetQServerImpl::serverUUID=7389f807-cc48-11e3-9d7d-4daadff738e7])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-28], discoveryGroupConfiguration=null]] is connected

      2014-04-25 01:57:27,137 INFO  [org.hornetq.core.server] (Thread-21 (HornetQ-server-HornetQServerImpl::serverUUID=null-172395014)) HQ221027: Bridge ClusterConnectionBridge@6e98d7c1 [name=sf.msf-cluster-10.4.167.99.a595a484-cb70-11e3-aae2-af4ddcb5b420, queue=QueueImpl[name=sf.msf-cluster-10.4.167.99.a595a484-cb70-11e3-aae2-af4ddcb5b420, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=7389f807-cc48-11e3-9d7d-4daadff738e7]]@747cf8a0 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@6e98d7c1 [name=sf.msf-cluster-10.4.167.99.a595a484-cb70-11e3-aae2-af4ddcb5b420, queue=QueueImpl[name=sf.msf-cluster-10.4.167.99.a595a484-cb70-11e3-aae2-af4ddcb5b420, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=7389f807-cc48-11e3-9d7d-4daadff738e7]]@747cf8a0 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-28], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@277207633[nodeUUID=7389f807-cc48-11e3-9d7d-4daadff738e7, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-27, address=jms, server=HornetQServerImpl::serverUUID=7389f807-cc48-11e3-9d7d-4daadff738e7])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-28], discoveryGroupConfiguration=null]] is connected

      2014-04-25 01:57:33,062 WARN  [org.hornetq.core.server] (Thread-52 (HornetQ-client-global-threads-107449287)) HQ222139: MessageFlowRecordImpl [nodeID=a595a484-cb70-11e3-aae2-af4ddcb5b420, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-28, queueName=sf.msf-cluster-10.4.167.99.a595a484-cb70-11e3-aae2-af4ddcb5b420, queue=QueueImpl[name=sf.msf-cluster-10.4.167.99.a595a484-cb70-11e3-aae2-af4ddcb5b420, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=7389f807-cc48-11e3-9d7d-4daadff738e7]]@747cf8a0, isClosed=false, firstReset=true]::Remote queue binding 7a9c07e8-e1d1-4f59-b6a0-927fb8b61544991862ea-cc57-11e3-899d-217962e95a1d has already been bound in the post office. Most likely cause for this is you have a loop in your cluster due to cluster max-hops being too large or you have multiple cluster connections to the same nodes using overlapping addresses

       

      and then it shows errors for all the messages

      2014-04-25 01:57:33,063 ERROR [org.hornetq.core.server] (Thread-52 (HornetQ-client-global-threads-107449287)) HQ224037: cluster connection Failed to handle message: java.lang.IllegalStateException: Cannot find binding for 1dadda13-b432-4004-b1df-e180d77245a5991862ea-cc57-11e3-899d-217962e95a1d on ClusterConnectionImpl@277207633[nodeUUID=7389f807-cc48-11e3-9d7d-4daadff738e7, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-27, address=jms, server=HornetQServerImpl::serverUUID=7389f807-cc48-11e3-9d7d-4daadff738e7]
              at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.doConsumerCreated(ClusterConnectionImpl.java:1396) [hornetq-server-2.4.0-SNAPSHOT.jar:]
              at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.onMessage(ClusterConnectionImpl.java:1143) [hornetq-server-2.4.0-SNAPSHOT.jar:]
              at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1117) [hornetq-core-client-2.4.0-SNAPSHOT.jar:]
              at org.hornetq.core.client.impl.ClientConsumerImpl.access$500(ClientConsumerImpl.java:57) [hornetq-core-client-2.4.0-SNAPSHOT.jar:]
              at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1252) [hornetq-core-client-2.4.0-SNAPSHOT.jar:]

       

       

      Is it because bridge is started by 2 different threads(2 different nodes at the same time) with same addresses? Is it some race condition?

      Laos, is there any reason because of which HornetQ server did not start first time on Node2 after failback?

      Please note that failback is working fine normally.This issue is not seen always.

       

      Please let me know how to solve it?

       

      Regards,

      Veena

        • 1. Re: Getting exception in log during failback and no message is processed - Remote queue binding has already been bound in the post office
          Yong Hao Gao Master

          In your scenario:

           

          Node 1: Started - HornetQ live started

          Node 2: Started - HornetQ live started

          Node 2:Stopped

          Node1  - HornetQ backup started

          Node 2 - Started - Attempts failback

           

          When Node 2 stopped, shouldn't Node 2 backup started ? Why Node 1 backup started? That seems strange.

          • 2. Re: Getting exception in log during failback and no message is processed - Remote queue binding has already been bound in the post office
            veenaonnet Newbie

            Hi,

             

            Sorry for the confusion. I am using collocated topology hence backup of Node 2 is on Node1. That is what I meant. So yes you are correct, backup of Node2 HornetQ which was on Node1 started.

             

            Node 1: Started - HornetQ live started

            Node 2: Started - HornetQ live started

            Node 2:Stopped

            HornetQ backup of Node2 on Node1 started

            Node 2 - Started - Attempts failback - Live on Node2 started

            • 4. Re: Getting exception in log during failback and no message is processed - Remote queue binding has already been bound in the post office
              veenaonnet Newbie

              Hi,

               

              Please find below HornetQ configuration of both the nodes.

              Node1:

              <subsystem xmlns="urn:jboss:domain:messaging:1.3">
                 <hornetq-server>
                  <clustered>true</clustered>
                  <persistence-enabled>true</persistence-enabled>
                  <security-enabled>true</security-enabled>
                  <cluster-user>xyz</cluster-user>
                  <cluster-password>xyz</cluster-password>
                  <failover-on-shutdown>true</failover-on-shutdown>
                  <shared-store>false</shared-store>
                  <journal-type>ASYNCIO</journal-type>
                  <journal-file-size>10485760</journal-file-size>
                  <journal-min-files>2</journal-min-files>
                  <check-for-live-server>true</check-for-live-server>
                  <backup-group-name>x.x.x.1</backup-group-name>
                  <paging-directory path="x.x.x.1/paging"/>
                  <bindings-directory path="x.x.x.1/bindings"/>
                  <journal-directory path="x.x.x.1/journal"/>
                  <large-messages-directory path="x.x.x.1/large-messages"/>
                  <max-saved-replicated-journals-size>40
                  </max-saved-replicated-journals-size>
                  <connectors>
                   <connector name="netty">
                    <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory
                    </factory-class>
                    <param key="host" value="x.x.x.1"/>
                    <param key="port" value="5445"/>
                   </connector>
                   <netty-connector name="netty-throughput" socket-binding="messaging-throughput">
                    <param key="batch-delay" value="50"/>
                   </netty-connector>
                   <in-vm-connector name="in-vm" server-id="0"/>
                  </connectors>

                  <acceptors>
                   <acceptor name="netty">
                                      <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
                                      <param key="host" value="0.0.0.0"/>
                                      <param key="port" value="5445"/>
                                  </acceptor>
                   <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">
                    <param key="batch-delay" value="50"/>
                    <param key="direct-deliver" value="false"/>
                   </netty-acceptor>
                   <in-vm-acceptor name="in-vm" server-id="0"/>
                  </acceptors>

                  <broadcast-groups>
                   <broadcast-group name="bg-group">
                    <group-address>228.x.x.99</group-address>
                    <group-port>9876</group-port>
                    <broadcast-period>5000</broadcast-period>
                    <connector-ref>netty</connector-ref>
                   </broadcast-group>
                  </broadcast-groups>
                  <discovery-groups>
                   <discovery-group name="dg-group">
                    <group-address>228.x.x.99</group-address>
                    <group-port>9876</group-port>
                    <refresh-timeout>10000</refresh-timeout>
                   </discovery-group>
                  </discovery-groups>
                  <cluster-connections>
                   <cluster-connection name="msf-cluster-x.x.x.99">
                    <address>jms</address>
                    <connector-ref>netty</connector-ref>
                    <discovery-group-ref discovery-group-name="dg-group"/>
                    <retry-interval>1000</retry-interval>
                    <use-duplicate-detection>true</use-duplicate-detection>
                    <forward-when-no-consumers>false</forward-when-no-consumers>
                    <max-hops>1</max-hops>
                   </cluster-connection>
                  </cluster-connections>
                  <security-settings>
                   <security-setting match="#">
                    <permission roles="guest" type="send"/>
                    <permission roles="guest" type="consume"/>
                    <permission roles="guest" type="createDurableQueue"/>
                    <permission roles="guest" type="deleteDurableQueue"/>
                    <permission roles="guest" type="createNonDurableQueue"/>
                    <permission roles="guest" type="deleteNonDurableQueue"/>
                    <permission roles="guest" type="manage"/>
                   </security-setting>
                  </security-settings>
                  <address-settings>
                  
                   <address-setting match="#">
                    <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                    <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                    <redelivery-delay>0</redelivery-delay>
                    <redistribution-delay>0</redistribution-delay>
                    <page-size-bytes>10485760</page-size-bytes>
                    <max-size-bytes>104857600</max-size-bytes>
                    <address-full-policy>PAGE</address-full-policy>
                    <message-counter-history-day-limit>1
                    </message-counter-history-day-limit>
                   </address-setting>
                  </address-settings>
                  <jms-connection-factories>
                   <connection-factory name="InVmConnectionFactory">
                    <connectors>
                     <connector-ref connector-name="in-vm"/>
                    </connectors>
                    <entries>
                     <entry name="java:/ConnectionFactory"/>
                    </entries>
                    <ha>true</ha>
                             <block-on-acknowledge>true</block-on-acknowledge>
                                      <retry-interval>1000</retry-interval>
                             <retry-interval-multiplier>1.0</retry-interval-multiplier>
                                      <reconnect-attempts>3</reconnect-attempts>
                    <client-failure-check-period>-1</client-failure-check-period>
                                      <connection-ttl>-1</connection-ttl>
                                      <confirmation-window-size>1000000</confirmation-window-size>
                    <call-timeout>180000</call-timeout>
                   </connection-factory>
                   <connection-factory name="RemoteConnectionFactory">
                    <connectors>
                     <connector-ref connector-name="netty"/>
                    </connectors>
                    <entries>
                     <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
                    </entries>
                    <ha>true</ha>
                    <block-on-acknowledge>true</block-on-acknowledge>
                    <retry-interval>1000</retry-interval>
                    <retry-interval-multiplier>1.0</retry-interval-multiplier>
                    <reconnect-attempts>3</reconnect-attempts>
                    <client-failure-check-period>60000</client-failure-check-period>
                                      <connection-ttl>600000</connection-ttl>
                    <confirmation-window-size>1000000</confirmation-window-size>
                    <call-timeout>180000</call-timeout>
                   </connection-factory>
                   <pooled-connection-factory name="hornetq-ra">
                    <transaction mode="xa"/>
                    <connectors>
                     <connector-ref connector-name="in-vm"/>
                    </connectors>
                    <entries>
                     <entry name="java:/JmsXA"/>
                    </entries>
                   </pooled-connection-factory>
                  </jms-connection-factories>
                  <jms-destinations>
                   <jms-queue name="Jobs">
                    <entry name="queue/Jobs"/>
                    <durable>true</durable>
                   </jms-queue>
                   <jms-topic name="Notification">
                    <entry name="topic/Notification"/>
                   </jms-topic>
                  </jms-destinations>
                 </hornetq-server>
                 <hornetq-server name="backup">
                  <clustered>true</clustered>
                  <backup>true</backup>
                  <persistence-enabled>true</persistence-enabled>
                  <security-enabled>true</security-enabled>
                  <cluster-user>xyz</cluster-user>
                  <cluster-password>xyz</cluster-password>
                  <allow-failback>true</allow-failback>
                  <failover-on-shutdown>true</failover-on-shutdown>
                  <shared-store>false</shared-store>
                  <journal-type>ASYNCIO</journal-type>
                  <journal-file-size>102400</journal-file-size>
                  <journal-min-files>2</journal-min-files>
                  <check-for-live-server>true</check-for-live-server>
                  <backup-group-name>x.x.x.2
                  </backup-group-name>
                  <paging-directory path="x.x.x.2/paging"/>
                  <bindings-directory path="x.x.x.2/bindings"/>
                  <journal-directory path="x.x.x.2/journal"/>
                  <large-messages-directory path="x.x.x.2/large-messages"/>
                  <max-saved-replicated-journals-size>40
                  </max-saved-replicated-journals-size>
                  <live-connector-ref>netty</live-connector-ref>
                  <connectors>
                   <connector name="netty-backup">
                    <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory
                    </factory-class>
                    <param key="host" value="x.x.x.1"/>
                    <param key="port" value="5446"/>
                   </connector>
                   <in-vm-connector name="in-vm" server-id="1"/>
                  </connectors>

                  <acceptors>
                   <in-vm-acceptor name="in-vm" server-id="1"/>
                                  <acceptor name="netty-backup">
                                      <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
                                      <param key="host" value="0.0.0.0"/>
                                      <param key="port" value="5446"/>
                                  </acceptor>
                  </acceptors>

                  <broadcast-groups>
                   <broadcast-group name="bg-group-backup">
                    <group-address>228.x.x.99</group-address>
                    <group-port>9876</group-port>
                    <broadcast-period>5000</broadcast-period>
                    <connector-ref>netty-backup</connector-ref>
                   </broadcast-group>
                  </broadcast-groups>
                  <discovery-groups>
                   <discovery-group name="dg-group-backup">
                    <group-address>228.x.x.99</group-address>
                    <group-port>9876</group-port>
                    <refresh-timeout>10000</refresh-timeout>
                   </discovery-group>
                  </discovery-groups>
                  <cluster-connections>
                   <cluster-connection name="msf-cluster-x.x.x.99">
                    <address>jms</address>
                    <connector-ref>netty-backup</connector-ref>
                    <discovery-group-ref discovery-group-name="dg-group-backup"/>
                    <retry-interval>1000</retry-interval>
                    <use-duplicate-detection>true</use-duplicate-detection>
                    <forward-when-no-consumers>false</forward-when-no-consumers>
                    <max-hops>1</max-hops>
                   </cluster-connection>
                  </cluster-connections>
                  <security-settings>
                   <security-setting match="#">
                    <permission roles="guest" type="send"/>
                    <permission roles="guest" type="consume"/>
                    <permission roles="guest" type="createDurableQueue"/>
                    <permission roles="guest" type="deleteDurableQueue"/>
                    <permission roles="guest" type="createNonDurableQueue"/>
                    <permission roles="guest" type="deleteNonDurableQueue"/>
                    <permission roles="guest" type="manage"/>
                   </security-setting>
                  </security-settings>
                  <address-settings>
                   <address-setting match="#">
                    <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                    <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                    <redelivery-delay>0</redelivery-delay>
                    <redistribution-delay>0</redistribution-delay>
                    <page-size-bytes>10485760</page-size-bytes>
                    <max-size-bytes>104857600</max-size-bytes>
                    <address-full-policy>PAGE</address-full-policy>
                    <message-counter-history-day-limit>1
                    </message-counter-history-day-limit>
                   </address-setting>
                  </address-settings>
                 </hornetq-server>
                </subsystem>

              Node2:

              <subsystem xmlns="urn:jboss:domain:messaging:1.3">
                 <hornetq-server>
                  <clustered>true</clustered>
                  <persistence-enabled>true</persistence-enabled>
                  <security-enabled>true</security-enabled>
                  <cluster-user>xyz</cluster-user>
                  <cluster-password>xyz</cluster-password>
                  <failover-on-shutdown>true</failover-on-shutdown>
                  <shared-store>false</shared-store>
                  <journal-type>ASYNCIO</journal-type>
                  <journal-file-size>10485760</journal-file-size>
                  <journal-min-files>2</journal-min-files>
                  <check-for-live-server>true</check-for-live-server>
                  <backup-group-name>x.x.x.2</backup-group-name>
                  <paging-directory path="x.x.x.2/paging"/>
                  <bindings-directory path="x.x.x.2/bindings"/>
                  <journal-directory path="x.x.x.2/journal"/>
                  <large-messages-directory path="x.x.x.2/large-messages"/>
                  <max-saved-replicated-journals-size>40
                  </max-saved-replicated-journals-size>
                  <connectors>
                   <connector name="netty">
                    <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory
                    </factory-class>
                    <param key="host" value="x.x.x.2"/>
                    <param key="port" value="5445"/>
                   </connector>
                   <netty-connector name="netty-throughput" socket-binding="messaging-throughput">
                    <param key="batch-delay" value="50"/>
                   </netty-connector>
                   <in-vm-connector name="in-vm" server-id="0"/>
                  </connectors>

                  <acceptors>
                   <acceptor name="netty">
                                      <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
                                      <param key="host" value="0.0.0.0"/>
                                      <param key="port" value="5445"/>
                                  </acceptor>
                   <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">
                    <param key="batch-delay" value="50"/>
                    <param key="direct-deliver" value="false"/>
                   </netty-acceptor>
                   <in-vm-acceptor name="in-vm" server-id="0"/>
                  </acceptors>

                  <broadcast-groups>
                   <broadcast-group name="bg-group">
                    <group-address>228.x.x.99</group-address>
                    <group-port>9876</group-port>
                    <broadcast-period>5000</broadcast-period>
                    <connector-ref>netty</connector-ref>
                   </broadcast-group>
                  </broadcast-groups>
                  <discovery-groups>
                   <discovery-group name="dg-group">
                    <group-address>228.x.x.99</group-address>
                    <group-port>9876</group-port>
                    <refresh-timeout>10000</refresh-timeout>
                   </discovery-group>
                  </discovery-groups>
                  <cluster-connections>
                   <cluster-connection name="msf-cluster-x.x.x.99">
                    <address>jms</address>
                    <connector-ref>netty</connector-ref>
                    <discovery-group-ref discovery-group-name="dg-group"/>
                    <retry-interval>1000</retry-interval>
                    <use-duplicate-detection>true</use-duplicate-detection>
                    <forward-when-no-consumers>false</forward-when-no-consumers>
                    <max-hops>1</max-hops>
                   </cluster-connection>
                  </cluster-connections>
                  <security-settings>
                   <security-setting match="#">
                    <permission roles="guest" type="send"/>
                    <permission roles="guest" type="consume"/>
                    <permission roles="guest" type="createDurableQueue"/>
                    <permission roles="guest" type="deleteDurableQueue"/>
                    <permission roles="guest" type="createNonDurableQueue"/>
                    <permission roles="guest" type="deleteNonDurableQueue"/>
                    <permission roles="guest" type="manage"/>
                   </security-setting>
                  </security-settings>
                  <address-settings>
                  
                   <address-setting match="#">
                    <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                    <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                    <redelivery-delay>0</redelivery-delay>
                    <redistribution-delay>0</redistribution-delay>
                    <page-size-bytes>10485760</page-size-bytes>
                    <max-size-bytes>104857600</max-size-bytes>
                    <address-full-policy>PAGE</address-full-policy>
                    <message-counter-history-day-limit>1
                    </message-counter-history-day-limit>
                   </address-setting>
                  </address-settings>
                  <jms-connection-factories>
                   <connection-factory name="InVmConnectionFactory">
                    <connectors>
                     <connector-ref connector-name="in-vm"/>
                    </connectors>
                    <entries>
                     <entry name="java:/ConnectionFactory"/>
                    </entries>
                    <ha>true</ha>
                             <block-on-acknowledge>true</block-on-acknowledge>
                                      <retry-interval>1000</retry-interval>
                             <retry-interval-multiplier>1.0</retry-interval-multiplier>
                                      <reconnect-attempts>3</reconnect-attempts>
                    <client-failure-check-period>-1</client-failure-check-period>
                                      <connection-ttl>-1</connection-ttl>
                                      <confirmation-window-size>1000000</confirmation-window-size>
                    <call-timeout>180000</call-timeout>
                   </connection-factory>
                   <connection-factory name="RemoteConnectionFactory">
                    <connectors>
                     <connector-ref connector-name="netty"/>
                    </connectors>
                    <entries>
                     <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
                    </entries>
                    <ha>true</ha>
                    <block-on-acknowledge>true</block-on-acknowledge>
                    <retry-interval>1000</retry-interval>
                    <retry-interval-multiplier>1.0</retry-interval-multiplier>
                    <reconnect-attempts>3</reconnect-attempts>
                    <client-failure-check-period>60000</client-failure-check-period>
                                      <connection-ttl>600000</connection-ttl>
                    <confirmation-window-size>1000000</confirmation-window-size>
                    <call-timeout>180000</call-timeout>
                   </connection-factory>
                   <pooled-connection-factory name="hornetq-ra">
                    <transaction mode="xa"/>
                    <connectors>
                     <connector-ref connector-name="in-vm"/>
                    </connectors>
                    <entries>
                     <entry name="java:/JmsXA"/>
                    </entries>
                   </pooled-connection-factory>
                  </jms-connection-factories>
                  <jms-destinations>
                   <jms-queue name="Jobs">
                    <entry name="queue/ProxyJobs"/>
                    <durable>true</durable>
                   </jms-queue>
                   <jms-topic name="Notification">
                    <entry name="topic/Notification"/>
                   </jms-topic>
                  </jms-destinations>
                 </hornetq-server>
                 <hornetq-server name="backup">
                  <clustered>true</clustered>
                  <backup>true</backup>
                  <persistence-enabled>true</persistence-enabled>
                  <security-enabled>true</security-enabled>
                  <cluster-user>xyz</cluster-user>
                  <cluster-password>xyz</cluster-password>
                  <allow-failback>true</allow-failback>
                  <failover-on-shutdown>true</failover-on-shutdown>
                  <shared-store>false</shared-store>
                  <journal-type>ASYNCIO</journal-type>
                  <journal-file-size>102400</journal-file-size>
                  <journal-min-files>2</journal-min-files>
                  <check-for-live-server>true</check-for-live-server>
                  <backup-group-name>x.x.x.1
                  </backup-group-name>
                  <paging-directory path="x.x.x.1/paging"/>
                  <bindings-directory path="x.x.x.1/bindings"/>
                  <journal-directory path="x.x.x.1/journal"/>
                  <large-messages-directory path="x.x.x.1/large-messages"/>
                  <max-saved-replicated-journals-size>40
                  </max-saved-replicated-journals-size>
                  <live-connector-ref>netty</live-connector-ref>
                  <connectors>
                   <connector name="netty-backup">
                    <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory
                    </factory-class>
                    <param key="host" value="x.x.x.2"/>
                    <param key="port" value="5446"/>
                   </connector>
                   <in-vm-connector name="in-vm" server-id="1"/>
                  </connectors>

                  <acceptors>
                   <in-vm-acceptor name="in-vm" server-id="1"/>
                                  <acceptor name="netty-backup">
                                      <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
                                      <param key="host" value="0.0.0.0"/>
                                      <param key="port" value="5446"/>
                                  </acceptor>
                  </acceptors>

                  <broadcast-groups>
                   <broadcast-group name="bg-group-backup">
                    <group-address>228.x.x.99</group-address>
                    <group-port>9876</group-port>
                    <broadcast-period>5000</broadcast-period>
                    <connector-ref>netty-backup</connector-ref>
                   </broadcast-group>
                  </broadcast-groups>
                  <discovery-groups>
                   <discovery-group name="dg-group-backup">
                    <group-address>228.x.x.99</group-address>
                    <group-port>9876</group-port>
                    <refresh-timeout>10000</refresh-timeout>
                   </discovery-group>
                  </discovery-groups>
                  <cluster-connections>
                   <cluster-connection name="msf-cluster-x.x.x.99">
                    <address>jms</address>
                    <connector-ref>netty-backup</connector-ref>
                    <discovery-group-ref discovery-group-name="dg-group-backup"/>
                    <retry-interval>1000</retry-interval>
                    <use-duplicate-detection>true</use-duplicate-detection>
                    <forward-when-no-consumers>false</forward-when-no-consumers>
                    <max-hops>1</max-hops>
                   </cluster-connection>
                  </cluster-connections>
                  <security-settings>
                   <security-setting match="#">
                    <permission roles="guest" type="send"/>
                    <permission roles="guest" type="consume"/>
                    <permission roles="guest" type="createDurableQueue"/>
                    <permission roles="guest" type="deleteDurableQueue"/>
                    <permission roles="guest" type="createNonDurableQueue"/>
                    <permission roles="guest" type="deleteNonDurableQueue"/>
                    <permission roles="guest" type="manage"/>
                   </security-setting>
                  </security-settings>
                  <address-settings>
                   <address-setting match="#">
                    <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                    <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                    <redelivery-delay>0</redelivery-delay>
                    <redistribution-delay>0</redistribution-delay>
                    <page-size-bytes>10485760</page-size-bytes>
                    <max-size-bytes>104857600</max-size-bytes>
                    <address-full-policy>PAGE</address-full-policy>
                    <message-counter-history-day-limit>1
                    </message-counter-history-day-limit>
                   </address-setting>
                  </address-settings>
                 </hornetq-server>
                </subsystem>

              Regards,

              Veena

              • 5. Re: Getting exception in log during failback and no message is processed - Remote queue binding has already been bound in the post office
                Thomas Siegrist Newbie

                Hi Veena,

                 

                Did you solve your problem? If so, i would appreciate if you could share any insights, since i'm experiencing the same problem.

                 

                Best,

                Thomas

                • 6. Re: Getting exception in log during failback and no message is processed - Remote queue binding has already been bound in the post office
                  Leonard De Baets Newbie

                  Was there any solution?  I saw the same log in a High Availability Cluster environment with Docker Containers.  They come out approximately 20 seconds after the topic subscribers subscribe to the topics.