HornetQ - Wildfly 8 broker reconnection problems with ArtemisQ - Wildfly 11 client
sunnyrays Apr 20, 2018 12:34 AMHi all,
I have a JMS client and a remote JMS broker, and they are both hosted on Wildfly 8.2.0 server. The JMS broker has a topic, and the JMS client listens to this remote topic and consumes the messages. Both of them use HornetQ that comes with Wildfly 8.2.0, so the inter-connectivity works seamlessly. I have to upgrade the JMS client to Wildfly 11.0.0, which uses ArtemisQ. The remote JMS broker will have to stay on Wildfly 8.2.0 for now. ArtemisQ provides support for HornetQ protocol. Using this support, I have managed to get the inter-connectivity to work. The client on Wildlfy 11.0.0 is able to publish messages to the remote topic on Wildfly 8.2.0, and also able to listen to it and consume messages.
The remote JMS broker on Wildfly 8.2.0 has a standard netty acceptor:
<netty-acceptor name="broker-messaging-acceptor" socket-binding="broker-messaging">
<param key="batch-delay" value="50"/>
<param key="direct-deliver" value="false"/>
<param key="host" value="0.0.0.0"/>
</netty-acceptor>
with socket binding:
<socket-binding name="broker-messaging" port="15455"/>
The topic is:
<jms-topic name="Product">
<entry name="topic/Product"/>
<entry name="java:jboss/exported/jms/topic/Product"/>
</jms-topic>
On the Wildfly 11.0.0 JMS client, I created a corresponding netty connector:
<connector name="remote-hornetq-connector" socket-binding="broker-socket" factory-class="org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory"/>
with socket binding:
<outbound-socket-binding name="broker-socket">
<remote-destination host="abc.xyz" port="15455"/>
</outbound-socket-binding>
and then a pooled connection factory using HornetQClientProtocolManagerFactory that uses this netty connector:
<pooled-connection-factory name="pooled-remote-connection-factory" entries="java:/RemoteJmsXA" connectors="remote-hornetq-connector" protocol-manager-factory="org.apache.activemq.artemis.core.protocol.hornetq.client.HornetQClientProtocolManagerFactory" transaction="xa" user="test" password="test"/>
The listener MDB class on the JMS client is:
@MessageDriven(activationConfig = {
@ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Topic"),
@ActivationConfigProperty(propertyName = "destination", propertyValue = "topic/Product"),
@ActivationConfigProperty(propertyName = "subscriptionDurability", propertyValue = "Durable"),
@ActivationConfigProperty(propertyName = "subscriptionName", propertyValue = "ProductTopicListener"),
@ActivationConfigProperty(propertyName = "reconnectAttempts", propertyValue = "-1"),
@ActivationConfigProperty(propertyName = "setupAttempts", propertyValue = "-1"),
@ActivationConfigProperty(propertyName = "maxSession", propertyValue = "5") })
@ResourceAdapter("pooled-remote-connection-factory")
@TransactionAttribute(TransactionAttributeType.REQUIRED)
public class ProductTopicListener implements MessageListener {
Using the above configurations, when I start the Wildfly 11.0.0 client, it comes up and starts looking for the broker, which is normal:
2018-04-19 22:10:25,295 INFO [org.apache.activemq.artemis.ra] (default-threads - 1) AMQ151005: awaiting server availability
2018-04-19 22:10:27,288 INFO [org.apache.activemq.artemis.ra] (default-threads - 3) AMQ151001: Attempting to reconnect org.apache.activemq.artemis.ra.inflow.ActiveMQActivationSpec(ra=org.wildfly.extension.messaging.activemq.ActiveMQResourceAdapter@c3be5320 destination=topic/Product destinationType=javax.jms.Topic ack=Auto-acknowledge durable=true clientID=null subscription=ProductTopicListener user=null maxSession=5)
As soon as I start the Wildfly 8.2.0 broker, the client connects with it successfully:
2018-04-19 22:11:25,809 INFO [org.apache.activemq.artemis.ra] (default-threads - 4) AMQ151002: Reconnected with broker
I am able to publish messages to the remote topic and then listen to it correctly from the client. All the changes that I made were on the client. No changes were required on the broker. As long as the broker is up, I can shut down the client and start it cleanly, and it reconnects with the broker automatically. When the broker is shut down, I get a bunch of WARN logs on the client stating that the connection no longer exists, which is again fine:
2018-04-19 22:11:36,575 WARN [org.apache.activemq.artemis.core.client] (Thread-2 (ActiveMQ-client-global-threads)) AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
Now, with the Wildfly 11.0.0 client still up, when I try to start the Wildfly 8.2.0 broker back up, it blows up with a ton of these exceptions:
2018-04-19 22:12:57,424 ERROR [org.hornetq.core.client] (Thread-10 (hornetq-netty-threads-1812662695)) HQ214013: Failed to decode packet: java.lang.IllegalArgumentException: HQ119032: Invalid type: -4
at org.hornetq.core.protocol.core.impl.PacketDecoder.decode(PacketDecoder.java:447) [hornetq-core-client-2.4.5.Final.jar:]
at org.hornetq.core.protocol.ServerPacketDecoder.decode(ServerPacketDecoder.java:172) [hornetq-server-2.4.5.Final.jar:]
at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:494) [hornetq-core-client-2.4.5.Final.jar:]
at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:658) [hornetq-server-2.4.5.Final.jar:]
at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.channelRead(HornetQChannelHandler.java:73) [hornetq-core-client-2.4.5.Final.jar:]
at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338) [netty-all-4.0.15.Final.jar:4.0.15.Final]
at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324) [netty-all-4.0.15.Final.jar:4.0.15.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:153) [netty-all-4.0.15.Final.jar:4.0.15.Final]
at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338) [netty-all-4.0.15.Final.jar:4.0.15.Final]
at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324) [netty-all-4.0.15.Final.jar:4.0.15.Final]
at io.netty.handler.codec.ByteToMessageDecoder.handlerRemoved(ByteToMessageDecoder.java:110) [netty-all-4.0.15.Final.jar:4.0.15.Final]
at io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:524) [netty-all-4.0.15.Final.jar:4.0.15.Final]
at io.netty.channel.DefaultChannelPipeline.callHandlerRemoved(DefaultChannelPipeline.java:518) [netty-all-4.0.15.Final.jar:4.0.15.Final]
at io.netty.channel.DefaultChannelPipeline.remove0(DefaultChannelPipeline.java:348) [netty-all-4.0.15.Final.jar:4.0.15.Final]
at io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:319) [netty-all-4.0.15.Final.jar:4.0.15.Final]
at io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:296) [netty-all-4.0.15.Final.jar:4.0.15.Final]
at org.hornetq.core.protocol.ProtocolHandler$ProtocolDecoder.decode(ProtocolHandler.java:168) [hornetq-server-2.4.5.Final.jar:]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:226) [netty-all-4.0.15.Final.jar:4.0.15.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:139) [netty-all-4.0.15.Final.jar:4.0.15.Final]
at org.hornetq.core.protocol.ProtocolHandler$ProtocolDecoder.channelRead(ProtocolHandler.java:111) [hornetq-server-2.4.5.Final.jar:]
at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338) [netty-all-4.0.15.Final.jar:4.0.15.Final]
at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324) [netty-all-4.0.15.Final.jar:4.0.15.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785) [netty-all-4.0.15.Final.jar:4.0.15.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:132) [netty-all-4.0.15.Final.jar:4.0.15.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:485) [netty-all-4.0.15.Final.jar:4.0.15.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:452) [netty-all-4.0.15.Final.jar:4.0.15.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:346) [netty-all-4.0.15.Final.jar:4.0.15.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101) [netty-all-4.0.15.Final.jar:4.0.15.Final]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_161]
The HornetQ broker is trying to decode a packet of type '-4' and throwing these exceptions constantly. No matter how many times I try to bring up the broker, I get the same exceptions. The only way out is to shut down the client and then start the broker again. I should also mention that when the broker is down, the client is extremely sluggish to shut down. I have to kill the client process. Ideally, the broker should just start up, and the client should reconnect with the broker, like it does when the servers are started for the first time.
Any ideas as to what is happening to the broker here, and what should be done to fix this problem? Any suggestions would greatly help!