can't send messages to HornetQ server after failover-failback
Hello,
I'm setting up WildFly in Domain Mode and trying to configure it with a colocated replication with HornetQ. On a server crash hornetq is failing over and the backup server on the surviving node is starting and all messages are consumed by the left node.
If I restart the crashed server the backup hornetq server is failing back, but now I have two issues if I try to send new messages:
1. I get the following exception if I try to send new messages after failover-failback:
2015-01-14 15:04:56 ERROR root:73 - javax.jms.JMSException: Failed to create session factory
javax.jms.JMSException: Failed to create session factory
at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:673)
at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:112)
at jms.ClusteredQueueTest.runTest(ClusteredQueueTest.java:50)
at jms.ClusteredQueueTest.main(ClusteredQueueTest.java:26)
Caused by: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s). Tried with all available servers.]
at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:905)
at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:669)
... 3 more
2. If I send messages directly to the server which restarted, the messages are not loadbalanced between the servers anymore.
I've tested a similiar configuration with JBoss EAP 6.3 and had no issues after multiple failover-failback tests. Is there a proper way to configure wildfly in domain mode and colocated hornetq replication? The messaging subsystem is:
<hornetq-server name="live">
<persistence-enabled>true</persistence-enabled>
<cluster-password>${jboss.messaging.cluster.password:123123}</cluster-password>
<backup>false</backup>
<allow-failback>true</allow-failback>
<failover-on-shutdown>true</failover-on-shutdown>
<shared-store>false</shared-store>
<journal-file-size>102400</journal-file-size>
<check-for-live-server>true</check-for-live-server>
<backup-group-name>${hornetq.live.group}</backup-group-name>
<paging-directory path="data/jms/${hornetq.live.group}/paging"/>
<bindings-directory path="data/jms/${hornetq.live.group}/bindings"/>
<journal-directory path="data/jms/${hornetq.live.group}/journal"/>
<large-messages-directory path="data/jms/${hornetq.live.group}/large-messages"/>
<connectors>
<http-connector name="http-connector" socket-binding="http">
<param key="http-upgrade-endpoint" value="http-acceptor"/>
</http-connector>
<http-connector name="http-connector-throughput" socket-binding="http">
<param key="http-upgrade-endpoint" value="http-acceptor-throughput"/>
<param key="batch-delay" value="50"/>
</http-connector>
<in-vm-connector name="in-vm" server-id="0"/>
</connectors>
<acceptors>
<http-acceptor http-listener="default" name="http-acceptor"/>
<http-acceptor http-listener="default" name="http-acceptor-throughput">
<param key="batch-delay" value="50"/>
<param key="direct-deliver" value="false"/>
</http-acceptor>
<in-vm-acceptor name="in-vm" server-id="0"/>
</acceptors>
<broadcast-groups>
<broadcast-group name="bg-group1">
<socket-binding>messaging-group</socket-binding>
<connector-ref>
http-connector
</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="dg-group1">
<socket-binding>messaging-group</socket-binding>
<refresh-timeout>15000</refresh-timeout>
</discovery-group>
</discovery-groups>
<cluster-connections>
<cluster-connection name="my-cluster">
<address>jms</address>
<connector-ref>http-connector</connector-ref>
<check-period>10000</check-period>
<connection-ttl>20000</connection-ttl>
<call-timeout>10000</call-timeout>
<reconnect-attempts>1</reconnect-attempts>
<use-duplicate-detection>true</use-duplicate-detection>
<max-hops>1</max-hops>
<discovery-group-ref discovery-group-name="dg-group1"/>
</cluster-connection>
</cluster-connections>
<security-settings>
<security-setting match="#">
<permission type="send" roles="guest"/>
<permission type="consume" roles="guest"/>
<permission type="createNonDurableQueue" roles="guest"/>
<permission type="deleteNonDurableQueue" roles="guest"/>
</security-setting>
</security-settings>
<address-settings>
<address-setting match="#">
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<max-size-bytes>10485760</max-size-bytes>
<page-size-bytes>2097152</page-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<redistribution-delay>1000</redistribution-delay>
</address-setting>
</address-settings>
<jms-connection-factories>
<connection-factory name="InVmConnectionFactory">
<connectors>
<connector-ref connector-name="in-vm"/>
</connectors>
<entries>
<entry name="java:/InVmConnectionFactory"/>
</entries>
<connection-ttl>20000</connection-ttl>
<call-timeout>10000</call-timeout>
<retry-interval>1000</retry-interval>
<retry-interval-multiplier>1.0</retry-interval-multiplier>
<max-retry-interval>1000</max-retry-interval>
</connection-factory>
<connection-factory name="RemoteConnectionFactory">
<connectors>
<connector-ref connector-name="http-connector"/>
</connectors>
<entries>
<entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
</entries>
<ha>true</ha>
<connection-ttl>20000</connection-ttl>
<call-timeout>10000</call-timeout>
<retry-interval>1000</retry-interval>
<retry-interval-multiplier>1.0</retry-interval-multiplier>
<max-retry-interval>1000</max-retry-interval>
</connection-factory>
<pooled-connection-factory name="hornetq-ra">
<transaction mode="xa"/>
<connectors>
<connector-ref connector-name="in-vm"/>
</connectors>
<entries>
<entry name="java:jboss/DefaultJMSConnectionFactory"/>
<entry name="java:/ConnectionFactory"/>
<entry name="java:/JmsXA"/>
</entries>
<ha>true</ha>
<block-on-acknowledge>true</block-on-acknowledge>
<reconnect-attempts>1</reconnect-attempts>
</pooled-connection-factory>
</jms-connection-factories>
<jms-destinations>
<jms-queue name="ExpiryQueue">
<entry name="java:/jms/queue/ExpiryQueue"/>
</jms-queue>
<jms-queue name="DLQ">
<entry name="java:/jms/queue/DLQ"/>
</jms-queue>
<jms-queue name="jmsQueue">
<entry name="java:jboss/exported/jms/queue/jmsQueue"/>
<durable>true</durable>
</jms-queue>
<jms-queue name="CMTQueue">
<entry name="java:/queue/CMTQueue"/>
<durable>true</durable>
</jms-queue>
<jms-topic name="jmsTopic">
<entry name="java:jboss/exported/jms/topic/jmsTopic"/>
</jms-topic>
</jms-destinations>
</hornetq-server>
<hornetq-server name="backup">
<persistence-enabled>true</persistence-enabled>
<cluster-password>${jboss.messaging.cluster.password:123123}</cluster-password>
<backup>true</backup>
<allow-failback>true</allow-failback>
<shared-store>false</shared-store>
<journal-file-size>102400</journal-file-size>
<backup-group-name>${hornetq.backup.group}</backup-group-name>
<paging-directory path="data/jms/${hornetq.backup.group}/paging"/>
<bindings-directory path="data/jms/${hornetq.backup.group}/bindings"/>
<journal-directory path="data/jms/${hornetq.backup.group}/journal"/>
<large-messages-directory path="data/jms/${hornetq.backup.group}/large-messages"/>
<connectors>
<http-connector name="http-backup-connector" socket-binding="http">
<param key="http-upgrade-endpoint" value="http-backup-acceptor"/>
</http-connector>
<http-connector name="http-backup-connector-throughput" socket-binding="http">
<param key="http-upgrade-endpoint" value="http-backup-acceptor-throughput"/>
<param key="batch-delay" value="50"/>
</http-connector>
<in-vm-connector name="in-vm" server-id="1"/>
</connectors>
<acceptors>
<http-acceptor http-listener="default" name="http-backup-acceptor"/>
<http-acceptor http-listener="default" name="http-backup-acceptor-throughput">
<param key="batch-delay" value="50"/>
<param key="direct-deliver" value="false"/>
</http-acceptor>
<in-vm-acceptor name="in-vm" server-id="1"/>
</acceptors>
<broadcast-groups>
<broadcast-group name="bg-group1">
<socket-binding>messaging-group</socket-binding>
<connector-ref>
http-backup-connector
</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="dg-group1">
<socket-binding>messaging-group</socket-binding>
<refresh-timeout>15000</refresh-timeout>
</discovery-group>
</discovery-groups>
<cluster-connections>
<cluster-connection name="my-cluster">
<address>jms</address>
<connector-ref>http-backup-connector</connector-ref>
<check-period>10000</check-period>
<connection-ttl>20000</connection-ttl>
<call-timeout>10000</call-timeout>
<reconnect-attempts>1</reconnect-attempts>
<use-duplicate-detection>true</use-duplicate-detection>
<max-hops>1</max-hops>
<discovery-group-ref discovery-group-name="dg-group1"/>
</cluster-connection>
</cluster-connections>
<security-settings>
<security-setting match="#">
<permission type="send" roles="guest"/>
<permission type="consume" roles="guest"/>
<permission type="createNonDurableQueue" roles="guest"/>
<permission type="deleteNonDurableQueue" roles="guest"/>
</security-setting>
</security-settings>
<address-settings>
<address-setting match="#">
<max-size-bytes>10485760</max-size-bytes>
<page-size-bytes>2097152</page-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<redistribution-delay>1000</redistribution-delay>
</address-setting>
</address-settings>
</hornetq-server>
${hornetq.live.group} and ${hornetq.backup.group} are provided through parameters on both nodes "-Dhornetq.live.group=group1 -Dhornetq.backup.group=group2" and on the other node "-Dhornetq.live.group=group2 -Dhornetq.backup.group=group1"