HornetQ 2.4 getting exception cluster connection Failed to handle message: java.lang.IllegalStateException: Cannot find binding
Hi,
I am using HornetQ 2.4 with Jboss7.2 with HA/clustered collocated topology. I am getting below messages in server.log on failover/failback.
What could be the reasons for these messages? How will it impact?
Is there anything wrong in my configuration?
WARN [org.hornetq.core.server] (Thread-441 (HornetQ-client-global-threads-2030641453)) HQ222139: MessageFlowRecordImpl [nodeID=28724b18-d414-11e3-8a48-bbd1fcea1f52, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-27, queueName=sf.msf-cluster-10.4.167.99.28724b18-d414-11e3-8a48-bbd1fcea1f52, queue=QueueImpl[name=sf.msf-cluster-10.4.167.99.28724b18-d414-11e3-8a48-bbd1fcea1f52, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=fbb5116e-d4df-11e3-973b-d3182c8b4acb]]@84a7f11, isClosed=false, firstReset=true]::Remote queue binding ad0fdf27-d62d-49a6-b6dd-558c324eae51e771683a-d517-11e3-920d-1520ea3f5321 has already been bound in the post office. Most likely cause for this is you have a loop in your cluster due to cluster max-hops being too large or you have multiple cluster connections to the same nodes using overlapping addresses
HQ224037: cluster connection Failed to handle message: java.lang.IllegalStateException: Cannot find binding for jms.queue.HPFileChangeNotificationJobs991862ea-cc57-11e3-899d-217962e95a1d on ClusterConnectionImpl@277207633[nodeUUID=7389f807-cc48-11e3-9d7d-4daadff738e7, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-4-167-27, address=jms, server=HornetQServerImpl::serverUUID=7389f807-cc48-11e3-9d7d-4daadff738e7]
Configuration:
| Node1 | Node2 |
|---|---|
<profile> <subsystem xmlns="urn:jboss:domain:messaging:1.3"> <hornetq-server> <clustered>true</clustered> <persistence-enabled>true</persistence-enabled> <security-enabled>true</security-enabled> <cluster-user>xyz</cluster-user> <cluster-password>xyz</cluster-password> <failover-on-shutdown>true</failover-on-shutdown> <shared-store>false</shared-store> <journal-type>ASYNCIO</journal-type> <journal-file-size>10485760</journal-file-size> <journal-min-files>2</journal-min-files> <check-for-live-server>true</check-for-live-server> <backup-group-name>x.x.x.1</backup-group-name> <paging-directory path="x.x.x.1/paging" /> <bindings-directory path="x.x.x.1/bindings" /> <journal-directory path="x.x.x.1/journal" /> <large-messages-directory path="x.x.x.1/large-messages" /> <max-saved-replicated-journals-size>40</max-saved-replicated-journals-size> <connectors> <connector name="netty"> <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class> <param key="host" value="x.x.x.1" /> <param key="port" value="5445" /> </connector> <netty-connector name="netty-throughput" socket-binding="messaging-throughput"> <param key="batch-delay" value="50" /> </netty-connector> <in-vm-connector name="in-vm" server-id="0" /> </connectors> <acceptors> <acceptor name="netty"> <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class> <param key="host" value="0.0.0.0" /> <param key="port" value="5445" /> </acceptor> <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput"> <param key="batch-delay" value="50" /> <param key="direct-deliver" value="false" /> </netty-acceptor> <in-vm-acceptor name="in-vm" server-id="0" /> </acceptors> <broadcast-groups> <broadcast-group name="bg-group"> <group-address>228.x.x.99</group-address> <group-port>9876</group-port> <broadcast-period>5000</broadcast-period> <connector-ref>netty</connector-ref> </broadcast-group> </broadcast-groups> <discovery-groups> <discovery-group name="dg-group"> <group-address>228.x.x.99</group-address> <group-port>9876</group-port> <refresh-timeout>10000</refresh-timeout> </discovery-group> </discovery-groups> <cluster-connections> <cluster-connection name="msf-cluster-x.x.x.99"> <address>jms</address> <connector-ref>netty</connector-ref> <discovery-group-ref discovery-group-name="dg-group" /> <retry-interval>1000</retry-interval> <use-duplicate-detection>true</use-duplicate-detection> <forward-when-no-consumers>false</forward-when-no-consumers> <max-hops>1</max-hops> </cluster-connection> </cluster-connections> <security-settings> <security-setting match="#"> <permission roles="guest" type="send" /> <permission roles="guest" type="consume" /> <permission roles="guest" type="createDurableQueue" /> <permission roles="guest" type="deleteDurableQueue" /> <permission roles="guest" type="createNonDurableQueue" /> <permission roles="guest" type="deleteNonDurableQueue" /> <permission roles="guest" type="manage" /> </security-setting> </security-settings> <address-settings> <address-setting match="#"> <dead-letter-address>jms.queue.DLQ</dead-letter-address> <expiry-address>jms.queue.ExpiryQueue</expiry-address> <redelivery-delay>0</redelivery-delay> <redistribution-delay>0</redistribution-delay> <page-size-bytes>10485760</page-size-bytes> <max-size-bytes>104857600</max-size-bytes> <address-full-policy>PAGE</address-full-policy> <message-counter-history-day-limit>1</message-counter-history-day-limit> </address-setting> </address-settings> <jms-connection-factories> <connection-factory name="InVmConnectionFactory"> <connectors> <connector-ref connector-name="in-vm" /> </connectors> <entries> <entry name="java:/ConnectionFactory" /> </entries> <ha>true</ha> <block-on-acknowledge>true</block-on-acknowledge> <retry-interval>1000</retry-interval> <retry-interval-multiplier>1.0</retry-interval-multiplier> <reconnect-attempts>3</reconnect-attempts> <client-failure-check-period>-1</client-failure-check-period> <connection-ttl>-1</connection-ttl> <confirmation-window-size>1000000</confirmation-window-size> <call-timeout>180000</call-timeout> </connection-factory> <connection-factory name="RemoteConnectionFactory"> <connectors> <connector-ref connector-name="netty" /> </connectors> <entries> <entry name="java:jboss/exported/jms/RemoteConnectionFactory" /> </entries> <ha>true</ha> <block-on-acknowledge>true</block-on-acknowledge> <retry-interval>1000</retry-interval> <retry-interval-multiplier>1.0</retry-interval-multiplier> <reconnect-attempts>3</reconnect-attempts> <client-failure-check-period>60000</client-failure-check-period> <connection-ttl>600000</connection-ttl> <confirmation-window-size>1000000</confirmation-window-size> <call-timeout>180000</call-timeout> </connection-factory> <pooled-connection-factory name="hornetq-ra"> <transaction mode="xa" /> <connectors> <connector-ref connector-name="in-vm" /> </connectors> <entries> <entry name="java:/JmsXA" /> </entries> </pooled-connection-factory> </jms-connection-factories> <jms-destinations> <jms-queue name="Jobs"> <entry name="queue/Jobs" /> <durable>true</durable> </jms-queue> <jms-topic name="Notification"> <entry name="topic/Notification" /> </jms-topic> </jms-destinations> </hornetq-server> <hornetq-server name="backup"> <clustered>true</clustered> <backup>true</backup> <persistence-enabled>true</persistence-enabled> <security-enabled>true</security-enabled> <cluster-user>xyz</cluster-user> <cluster-password>xyz</cluster-password> <allow-failback>true</allow-failback> <failover-on-shutdown>true</failover-on-shutdown> <shared-store>false</shared-store> <journal-type>ASYNCIO</journal-type> <journal-file-size>102400</journal-file-size> <journal-min-files>2</journal-min-files> <check-for-live-server>true</check-for-live-server> <backup-group-name>x.x.x.2</backup-group-name> <paging-directory path="x.x.x.2/paging" /> <bindings-directory path="x.x.x.2/bindings" /> <journal-directory path="x.x.x.2/journal" /> <large-messages-directory path="x.x.x.2/large-messages" /> <max-saved-replicated-journals-size>40</max-saved-replicated-journals-size> <live-connector-ref>netty</live-connector-ref> <connectors> <connector name="netty-backup"> <factory-class> org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class> <param key="host" value="x.x.x.1" /> <param key="port" value="5446" /> </connector> <in-vm-connector name="in-vm" server-id="1" /> </connectors> <acceptors> <in-vm-acceptor name="in-vm" server-id="1" /> <acceptor name="netty-backup"> <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class> <param key="host" value="0.0.0.0" /> <param key="port" value="5446" /> </acceptor> </acceptors> <broadcast-groups> <broadcast-group name="bg-group-backup"> <group-address>228.x.x.99</group-address> <group-port>9876</group-port> <broadcast-period>5000</broadcast-period> <connector-ref>netty-backup</connector-ref> </broadcast-group> </broadcast-groups> <discovery-groups> <discovery-group name="dg-group-backup"> <group-address>228.x.x.99</group-address> <group-port>9876</group-port> <refresh-timeout>10000</refresh-timeout> </discovery-group> </discovery-groups> <cluster-connections> <cluster-connection name="msf-cluster-x.x.x.99"> <address>jms</address> <connector-ref>netty-backup</connector-ref> <discovery-group-ref discovery-group-name="dg-group-backup" /> <retry-interval>1000</retry-interval> <use-duplicate-detection>true</use-duplicate-detection> <forward-when-no-consumers>false</forward-when-no-consumers> <max-hops>1</max-hops> </cluster-connection> </cluster-connections> <security-settings> <security-setting match="#"> <permission roles="guest" type="send" /> <permission roles="guest" type="consume" /> <permission roles="guest" type="createDurableQueue" /> <permission roles="guest" type="deleteDurableQueue" /> <permission roles="guest" type="createNonDurableQueue" /> <permission roles="guest" type="deleteNonDurableQueue" /> <permission roles="guest" type="manage" /> </security-setting> </security-settings> <address-settings> <address-setting match="#"> <dead-letter-address>jms.queue.DLQ</dead-letter-address> <expiry-address>jms.queue.ExpiryQueue</expiry-address> <redelivery-delay>0</redelivery-delay> <redistribution-delay>0</redistribution-delay> <page-size-bytes>10485760</page-size-bytes> <max-size-bytes>104857600</max-size-bytes> <address-full-policy>PAGE</address-full-policy> <message-counter-history-day-limit>1</message-counter-history-day-limit> </address-setting> </address-settings> </hornetq-server> </subsystem> </profile> <socket-binding-group default-interface="public" name="standard-sockets" port-offset="${jboss.socket.binding.port-offset:0}"> <socket-binding interface="management" name="management-native" port="${jboss.management.native.port:9999}" /> <socket-binding interface="management" name="management-http" port="${jboss.management.http.port:9990}" /> <socket-binding interface="management" name="management-https" port="${jboss.management.https.port:9443}" /> <socket-binding name="ajp" port="8009" /> <socket-binding name="http" port="80" /> <socket-binding name="https" port="443" /> <socket-binding interface="unsecure" name="jacorb" port="3528" /> <socket-binding interface="unsecure" name="jacorb-ssl" port="3529" /> <socket-binding multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700" name="jgroups-mping" port="0" /> <socket-binding name="jgroups-tcp" port="7600" /> <socket-binding name="jgroups-tcp-fd" port="57600" /> <socket-binding multicast-address="228.x.x.99" multicast-port="45688" name="jgroups-udp" port="55200" /> <socket-binding name="jgroups-udp-fd" port="54200" /> <socket-binding name="messaging" port="5445" /> <socket-binding name="messaging-throughput" port="5455" /> <socket-binding multicast-address="228.x.x.99" multicast-port="23364" name="modcluster" port="0" /> <socket-binding interface="management" name="osgi-http" port="8090" /> <socket-binding name="remoting" port="4447" /> <socket-binding name="txn-recovery-environment" port="4712" /> <socket-binding name="txn-status-manager" port="4713" /> <socket-binding name="messaging-backup" port="5446" /> <outbound-socket-binding name="mail-smtp"> <remote-destination host="localhost" port="25" /> </outbound-socket-binding> </socket-binding-group> | <profile> <subsystem xmlns="urn:jboss:domain:messaging:1.3"> <hornetq-server> <clustered>true</clustered> <persistence-enabled>true</persistence-enabled> <security-enabled>true</security-enabled> <cluster-user>xyz</cluster-user> <cluster-password>xyz</cluster-password> <failover-on-shutdown>true</failover-on-shutdown> <shared-store>false</shared-store> <journal-type>ASYNCIO</journal-type> <journal-file-size>10485760</journal-file-size> <journal-min-files>2</journal-min-files> <check-for-live-server>true</check-for-live-server> <backup-group-name>x.x.x.2</backup-group-name> <paging-directory path="x.x.x.2/paging" /> <bindings-directory path="x.x.x.2/bindings" /> <journal-directory path="x.x.x.2/journal" /> <large-messages-directory path="x.x.x.2/large-messages" /> <max-saved-replicated-journals-size>40</max-saved-replicated-journals-size> <connectors> <connector name="netty"> <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class> <param key="host" value="x.x.x.2" /> <param key="port" value="5445" /> </connector> <netty-connector name="netty-throughput" socket-binding="messaging-throughput"> <param key="batch-delay" value="50" /> </netty-connector> <in-vm-connector name="in-vm" server-id="0" /> </connectors> <acceptors> <acceptor name="netty"> <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class> <param key="host" value="0.0.0.0" /> <param key="port" value="5445" /> </acceptor> <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput"> <param key="batch-delay" value="50" /> <param key="direct-deliver" value="false" /> </netty-acceptor> <in-vm-acceptor name="in-vm" server-id="0" /> </acceptors> <broadcast-groups> <broadcast-group name="bg-group"> <group-address>228.x.x.99</group-address> <group-port>9876</group-port> <broadcast-period>5000</broadcast-period> <connector-ref>netty</connector-ref> </broadcast-group> </broadcast-groups> <discovery-groups> <discovery-group name="dg-group"> <group-address>228.x.x.99</group-address> <group-port>9876</group-port> <refresh-timeout>10000</refresh-timeout> </discovery-group> </discovery-groups> <cluster-connections> <cluster-connection name="msf-cluster-x.x.x.99"> <address>jms</address> <connector-ref>netty</connector-ref> <discovery-group-ref discovery-group-name="dg-group" /> <retry-interval>1000</retry-interval> <use-duplicate-detection>true</use-duplicate-detection> <forward-when-no-consumers>false</forward-when-no-consumers> <max-hops>1</max-hops> </cluster-connection> </cluster-connections> <security-settings> <security-setting match="#"> <permission roles="guest" type="send" /> <permission roles="guest" type="consume" /> <permission roles="guest" type="createDurableQueue" /> <permission roles="guest" type="deleteDurableQueue" /> <permission roles="guest" type="createNonDurableQueue" /> <permission roles="guest" type="deleteNonDurableQueue" /> <permission roles="guest" type="manage" /> </security-setting> </security-settings> <address-settings> <address-setting match="#"> <dead-letter-address>jms.queue.DLQ</dead-letter-address> <expiry-address>jms.queue.ExpiryQueue</expiry-address> <redelivery-delay>0</redelivery-delay> <redistribution-delay>0</redistribution-delay> <page-size-bytes>10485760</page-size-bytes> <max-size-bytes>104857600</max-size-bytes> <address-full-policy>PAGE</address-full-policy> <message-counter-history-day-limit>1</message-counter-history-day-limit> </address-setting> </address-settings> <jms-connection-factories> <connection-factory name="InVmConnectionFactory"> <connectors> <connector-ref connector-name="in-vm" /> </connectors> <entries> <entry name="java:/ConnectionFactory" /> </entries> <ha>true</ha> <block-on-acknowledge>true</block-on-acknowledge> <retry-interval>1000</retry-interval> <retry-interval-multiplier>1.0</retry-interval-multiplier> <reconnect-attempts>3</reconnect-attempts> <client-failure-check-period>-1</client-failure-check-period> <connection-ttl>-1</connection-ttl> <confirmation-window-size>1000000</confirmation-window-size> <call-timeout>180000</call-timeout> </connection-factory> <connection-factory name="RemoteConnectionFactory"> <connectors> <connector-ref connector-name="netty" /> </connectors> <entries> <entry name="java:jboss/exported/jms/RemoteConnectionFactory" /> </entries> <ha>true</ha> <block-on-acknowledge>true</block-on-acknowledge> <retry-interval>1000</retry-interval> <retry-interval-multiplier>1.0</retry-interval-multiplier> <reconnect-attempts>3</reconnect-attempts> <client-failure-check-period>60000</client-failure-check-period> <connection-ttl>600000</connection-ttl> <confirmation-window-size>1000000</confirmation-window-size> <call-timeout>180000</call-timeout> </connection-factory> <pooled-connection-factory name="hornetq-ra"> <transaction mode="xa" /> <connectors> <connector-ref connector-name="in-vm" /> </connectors> <entries> <entry name="java:/JmsXA" /> </entries> </pooled-connection-factory> </jms-connection-factories> <jms-destinations> <jms-queue name="Jobs"> <entry name="queue/Jobs" /> <durable>true</durable> </jms-queue> <jms-topic name="Notification"> <entry name="topic/Notification" /> </jms-topic> </jms-destinations> </hornetq-server> <hornetq-server name="backup"> <clustered>true</clustered> <backup>true</backup> <persistence-enabled>true</persistence-enabled> <security-enabled>true</security-enabled> <cluster-user>xyz</cluster-user> <cluster-password>xyz</cluster-password> <allow-failback>true</allow-failback> <failover-on-shutdown>true</failover-on-shutdown> <shared-store>false</shared-store> <journal-type>ASYNCIO</journal-type> <journal-file-size>102400</journal-file-size> <journal-min-files>2</journal-min-files> <check-for-live-server>true</check-for-live-server> <backup-group-name>x.x.x.1</backup-group-name> <paging-directory path="x.x.x.1/paging" /> <bindings-directory path="x.x.x.1/bindings" /> <journal-directory path="x.x.x.1/journal" /> <large-messages-directory path="x.x.x.1/large-messages" /> <max-saved-replicated-journals-size>40</max-saved-replicated-journals-size> <live-connector-ref>netty</live-connector-ref> <connectors> <connector name="netty-backup"> <factory-class> org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class> <param key="host" value="x.x.x.2" /> <param key="port" value="5446" /> </connector> <in-vm-connector name="in-vm" server-id="1" /> </connectors> <acceptors> <in-vm-acceptor name="in-vm" server-id="1" /> <acceptor name="netty-backup"> <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class> <param key="host" value="0.0.0.0" /> <param key="port" value="5446" /> </acceptor> </acceptors> <broadcast-groups> <broadcast-group name="bg-group-backup"> <group-address>228.x.x.99</group-address> <group-port>9876</group-port> <broadcast-period>5000</broadcast-period> <connector-ref>netty-backup</connector-ref> </broadcast-group> </broadcast-groups> <discovery-groups> <discovery-group name="dg-group-backup"> <group-address>228.x.x.99</group-address> <group-port>9876</group-port> <refresh-timeout>10000</refresh-timeout> </discovery-group> </discovery-groups> <cluster-connections> <cluster-connection name="msf-cluster-x.x.x.99"> <address>jms</address> <connector-ref>netty-backup</connector-ref> <discovery-group-ref discovery-group-name="dg-group-backup" /> <retry-interval>1000</retry-interval> <use-duplicate-detection>true</use-duplicate-detection> <forward-when-no-consumers>false</forward-when-no-consumers> <max-hops>1</max-hops> </cluster-connection> </cluster-connections> <security-settings> <security-setting match="#"> <permission roles="guest" type="send" /> <permission roles="guest" type="consume" /> <permission roles="guest" type="createDurableQueue" /> <permission roles="guest" type="deleteDurableQueue" /> <permission roles="guest" type="createNonDurableQueue" /> <permission roles="guest" type="deleteNonDurableQueue" /> <permission roles="guest" type="manage" /> </security-setting> </security-settings> <address-settings> <address-setting match="#"> <dead-letter-address>jms.queue.DLQ</dead-letter-address> <expiry-address>jms.queue.ExpiryQueue</expiry-address> <redelivery-delay>0</redelivery-delay> <redistribution-delay>0</redistribution-delay> <page-size-bytes>10485760</page-size-bytes> <max-size-bytes>104857600</max-size-bytes> <address-full-policy>PAGE</address-full-policy> <message-counter-history-day-limit>1</message-counter-history-day-limit> </address-setting> </address-settings> </hornetq-server> </subsystem> </profile> <socket-binding-group default-interface="public" name="standard-sockets" port-offset="${jboss.socket.binding.port-offset:0}"> <socket-binding interface="management" name="management-native" port="${jboss.management.native.port:9999}" /> <socket-binding interface="management" name="management-http" port="${jboss.management.http.port:9990}" /> <socket-binding interface="management" name="management-https" port="${jboss.management.https.port:9443}" /> <socket-binding name="ajp" port="8009" /> <socket-binding name="http" port="80" /> <socket-binding name="https" port="443" /> <socket-binding interface="unsecure" name="jacorb" port="3528" /> <socket-binding interface="unsecure" name="jacorb-ssl" port="3529" /> <socket-binding multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700" name="jgroups-mping" port="0" /> <socket-binding name="jgroups-tcp" port="7600" /> <socket-binding name="jgroups-tcp-fd" port="57600" /> <socket-binding multicast-address="228.x.x.99" multicast-port="45688" name="jgroups-udp" port="55200" /> <socket-binding name="jgroups-udp-fd" port="54200" /> <socket-binding name="messaging" port="5445" /> <socket-binding name="messaging-throughput" port="5455" /> <socket-binding multicast-address="228.x.x.99" multicast-port="23364" name="modcluster" port="0" /> <socket-binding interface="management" name="osgi-http" port="8090" /> <socket-binding name="remoting" port="4447" /> <socket-binding name="txn-recovery-environment" port="4712" /> <socket-binding name="txn-status-manager" port="4713" /> <socket-binding name="messaging-backup" port="5446" /> <outbound-socket-binding name="mail-smtp"> <remote-destination host="localhost" port="25" /> </outbound-socket-binding> </socket-binding-group> |
Regards,
Veena