Failover - Reconnection of the client not working
gabrielp Jan 24, 2012 1:58 PMHi all, I'm having an issue with the failover mechanism of HornetQ related to the client losing the session and not being able to continue inserting messages in the queue.
The client is a jmeter test that uses a Java Request to insert messages in a queue. It connects directly to the endpoint (url:1499) and inserts messages there. The tests we are doing for the failover consists in killing or shutting down the live server while we are sending 2000 messages.
We see the backup server becoming live
2012-01-24 09:15:00,772 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.MessagingPartition] [AsynchViewChangeHandler Thread] Dead members: 1 ([63.131.131.231:1499])
2012-01-24 09:15:00,772 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.MessagingPartition] [AsynchViewChangeHandler Thread] New Members : 0 ([])
2012-01-24 09:15:00,772 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.MessagingPartition] [AsynchViewChangeHandler Thread] All Members : 1 ([63.131.131.231:1299])
...
INFO: Backup Server is now live
but the rest of the messages are rejected with the following exception:
24/01/2012 14:58:39 org.hornetq.core.logging.impl.JULLogDelegate warn
ADVERTENCIA: Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
javax.jms.JMSException: Connection failure detected. Unblocking a blocking call that will never get a response
at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:286)
at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:287)
at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:142)
at org.hornetq.jms.client.HornetQMessageProducer.doSend(HornetQMessageProducer.java:451)
at org.hornetq.jms.client.HornetQMessageProducer.send(HornetQMessageProducer.java:199)
at com.xxx.testing.sampler.XDtoPublisherSampler.runTest(XDtoPublisherSampler.java:106)
at org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSampler.java:161)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:411)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:297)
at java.lang.Thread.run(Thread.java:619)
Caused by: HornetQException[errorCode=5 message=Connection failure detected. Unblocking a blocking call that will never get a response]
... 10 more
24/01/2012 14:58:39 org.hornetq.core.logging.impl.JULLogDelegate warn
ADVERTENCIA: Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
javax.jms.IllegalStateException: Session is closed
at org.hornetq.jms.client.HornetQSession.checkClosed(HornetQSession.java:1008)
at org.hornetq.jms.client.HornetQSession.createObjectMessage(HornetQSession.java:165)
at com.xxx.testing.sampler.XDtoPublisherSampler.createObjectMessage(XDtoPublisherSampler.java:82)
at com.xxx.testing.sampler.XDtoPublisherSampler.runTest(PlayphoneDtoPublisherSampler.java:104)
at org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSampler.java:161)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:411)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:297)
at java.lang.Thread.run(Thread.java:619)
The live hornetq instance has the following config:
hornetq-configuration.xml
<clustered>true</clustered>
<failover-on-shutdown>true</failover-on-shutdown>
<shared-store>true</shared-store>
<allow-failback>true</allow-failback>
<journal-directory>${jboss.hornetq.file.dir}/journal</journal-directory>
<connectors>
<connector name="netty">
<factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
<param key="host" value="${jboss.bind.address:localhost}"/>
<param key="port" value="${hornetq.remoting.netty.port:5445}"/>
</connector>
<connector name="backup">
<factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
<param key="host" value="${hornetq.backup.host.address:localhost}"/>
<param key="port" value="${hornetq.remoting.netty.backup.port:5445}"/>
</connector>
</connectors>
<acceptors>
<acceptor name="netty">
<factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
<param key="host" value="${jboss.bind.address:localhost}"/>
<param key="port" value="${hornetq.remoting.netty.port:5445}"/>
</acceptor>
</acceptors>
<cluster-connections>
<cluster-connection name="prep-hornetq-cluster">
<address>jms</address>
<connector-ref>netty</connector-ref>
<retry-interval>500</retry-interval>
<use-duplicate-detection>true</use-duplicate-detection>
<forward-when-no-consumers>true</forward-when-no-consumers>
<max-hops>1</max-hops>
<static-connectors>
<connector-ref>backup</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
hornetq-jms.xml
<connection-factory name="NettyConnectionFactory">
<xa>false</xa>
<connectors>
<connector-ref connector-name="netty"/>
</connectors>
<entries>
<entry name="/ConnectionFactory"/>
</entries>
<reconnect-attempts>-1</reconnect-attempts>
</connection-factory>
The script for starting the hornetq is the following
/run.sh -c all-with-hornetq-live -b ux2.host.net -Dserver.host.name=ux2.host.net -Dpp.web.port=8404 -Djboss.service.binding.set=ports-04 -g MessagingPartition -Djboss.messaging.ServerPeerID=0 -Djboss.server.log.dir=/usr/software/runtime/hornetql/logs -Dpp.app.folder=/usr/software/runtime/hornetql/app -Djboss.hornetq.file.dir=/usr/software/runtime/hornetq -Dhornetq.server-id=0 -Dhornetq.remoting.netty.port=5445 -Dhornetq.remoting.netty.batch.port=5455 -Dhornetq.backup.host.address=ux2.host.net -Dhornetq.remoting.netty.backup.port=5446 &
And the backup server is configured like this:
hornetq-configuration.xml
<clustered>true</clustered>
<failover-on-shutdown>true</failover-on-shutdown>
<shared-store>true</shared-store>
<allow-failback>true</allow-failback>
<backup>true</backup>
<journal-directory>${jboss.hornetq.file.dir}/journal</journal-directory>
...
<connector name="netty">
<factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
<param key="host" value="${jboss.bind.address:localhost}"/>
<param key="port" value="${hornetq.remoting.netty.port:5446}"/>
</connector>
<connector name="backup">
<factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
<param key="host" value="${hornetq.backup.host.address:localhost}"/>
<param key="port" value="${hornetq.remoting.netty.backup.port:5445}"/>
</connector>
...
<acceptor name="netty">
<factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
<param key="host" value="${jboss.bind.address:localhost}"/>
<param key="port" value="${hornetq.remoting.netty.port:5446}"/>
</acceptor>
...
<cluster-connections>
<cluster-connection name="prep-hornetq-cluster">
<address>jms</address>
<connector-ref>netty</connector-ref>
<retry-interval>500</retry-interval>
<use-duplicate-detection>true</use-duplicate-detection>
<forward-when-no-consumers>true</forward-when-no-consumers>
<max-hops>1</max-hops>
<static-connectors>
<connector-ref>backup</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
hornetq-jms.xml
<connection-factory name="NettyConnectionFactory">
<xa>false</xa>
<connectors>
<connector-ref connector-name="netty"/>
</connectors>
<entries>
<entry name="/ConnectionFactory"/>
</entries>
</connection-factory>
The script
./run.sh -c all-with-hornetq-backup -b ux2.host.playp.net -Dserver.host.name=ux2.host.net -Dpp.web.port=8204 -Djboss.service.binding.set=ports-02 -g MessagingPartition -Djboss.messaging.ServerPeerID=1 -Djboss.server.log.dir=/usr/software/runtime/hornetqb/logs -Dpp.app.folder=/usr/software/runtime/hornetqb/app -Djboss.hornetq.file.dir=/usr/software/runtime/hornetq -Dhornetq.server-id=1 -Dhornetq.remoting.netty.port=5446 -Dhornetq.remoting.netty.batch.port=5456 -Dhornetq.backup.host.address=ux2.host.playp.net -Dhornetq.remoting.netty.backup.port=5445 &
Thanks for your help
-
hornetq-jms(live).xml 2.4 KB
-
hornetq-jms(backup).xml 2.3 KB