Connection failed with failedOver=false: HornetQInternalErrorException in clustured node

[valsaraj007]

Hi,

I have set up a vertical scalable cluster with two WildFly-8.2.0 instances located on the same machine. I have disconnected one node which processed the previous request. Then I got the following warning in another node. The application that I used for testing is very simple application: https://github.com/liweinan/cluster-demo.

2015-03-12 17:25:16,055 WARN  [org.hornetq.core.server] (Thread-11 (HornetQ-client-global-threads-859399932)) HQ222095: Connection failed with failedOver=false: HornetQInternalErrorException[errorType=INTERNAL_ERROR message=HQ119005: Exception in Netty transport]

  at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.exceptionCaught(HornetQChannelHandler.java:101) [hornetq-core-client-2.4.5.Final.jar:]

  at io.netty.channel.DefaultChannelHandlerContext.invokeExceptionCaught(DefaultChannelHandlerContext.java:276) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at io.netty.channel.DefaultChannelHandlerContext.fireExceptionCaught(DefaultChannelHandlerContext.java:254) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at io.netty.channel.DefaultChannelHandlerContext.invokeExceptionCaught(DefaultChannelHandlerContext.java:276) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at io.netty.channel.DefaultChannelHandlerContext.fireExceptionCaught(DefaultChannelHandlerContext.java:254) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:773) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:95) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:157) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:485) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:452) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:346) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_07]

Caused by: java.io.IOException: An existing connection was forcibly closed by the remote host

  at sun.nio.ch.SocketDispatcher.read0(Native Method) [rt.jar:1.7.0_07]

  at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43) [rt.jar:1.7.0_07]

  at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218) [rt.jar:1.7.0_07]

  at sun.nio.ch.IOUtil.read(IOUtil.java:186) [rt.jar:1.7.0_07]

  at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359) [rt.jar:1.7.0_07]

  at io.netty.buffer.UnpooledUnsafeDirectByteBuf.setBytes(UnpooledUnsafeDirectByteBuf.java:446) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:871) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:208) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:124) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  ... 5 more

Please let me know what's wrong.

Thanks!

[jbertram]

It looks to me like you're server's are forming a HornetQ cluster and when you drop one node the other node throws an exception related to its cluster bridge to the node you dropped.

[valsaraj007]

Thanks jbertram!

Is this warning harmful? any way to avoid this?

[jbertram]

I don't believe it's harmful.  It's simply alerting you to the fact that the cluster bridge was closed unexpectedly.

There are a couple of (obvious) ways to avoid it:

1. Don't kill a node in the cluster.
2. Don't cluster the HornetQ instances.

[valsaraj007]

Thanks jbertram!

1. I killed one node to test failover.

2. If we do not cluster HornetQ, will failover work? I have used full-ha to start nodes. Didn't do anything in configuration to cluster the HornetQ instances. Is cluster configured by default?

[jbertram]

Are you testing HornetQ fail-over or fail-over for some other technology (e.g. web)?

Despite the file's name, the standalone-full-ha.xml configuration is not sufficient for HornetQ HA (i.e. fail-over) functionality.  To be clear, it is sufficient for HornetQ load-balancing.  See the HornetQ documentation on HA for more details on configuring fail-over.

[valsaraj007]

Thanks jbertram!

I am testing fail-over of the JavaEE application.

[valsaraj007]

Hi jbertram,

Our application using queues very much. So when we placed a message in queue and that nodes crashes, will the second node will continue from that state? We are using full HA profile and didn't make any change in HornetQ section.

Thanks!

[jbertram]

Our application using queues very much. So when we placed a message in queue and that nodes crashes, will the second node will continue from that state? We are using full HA profile and didn't make any change in HornetQ section.

If you didn't change the HornetQ configuration from the default standalone-full-ha.xml then, no, you will not get fail-over or HA functionality for JMS messages.

See my previous comment for more details.