10 Replies Latest reply on May 8, 2018 9:07 PM by sunnyrays

    HornetQ - Wildfly 8 broker reconnection problems with ArtemisQ - Wildfly 11 client

    sunnyrays

      Hi all,

       

      I have a JMS client and a remote JMS broker, and they are both hosted on Wildfly 8.2.0 server. The JMS broker has a topic, and the JMS client listens to this remote topic and consumes the messages. Both of them use HornetQ that comes with Wildfly 8.2.0, so the inter-connectivity works seamlessly. I have to upgrade the JMS client to Wildfly 11.0.0, which uses ArtemisQ. The remote JMS broker will have to stay on Wildfly 8.2.0 for now. ArtemisQ provides support for HornetQ protocol. Using this support, I have managed to get the inter-connectivity to work. The client on Wildlfy 11.0.0 is able to publish messages to the remote topic on Wildfly 8.2.0, and also able to listen to it and consume messages.

       

      The remote JMS broker on Wildfly 8.2.0 has a standard netty acceptor:
      <netty-acceptor name="broker-messaging-acceptor" socket-binding="broker-messaging">
          <param key="batch-delay" value="50"/>
          <param key="direct-deliver" value="false"/>
          <param key="host" value="0.0.0.0"/>
      </netty-acceptor>

       

      with socket binding:
      <socket-binding name="broker-messaging" port="15455"/>

       

      The topic is:
      <jms-topic name="Product">
          <entry name="topic/Product"/>
          <entry name="java:jboss/exported/jms/topic/Product"/>
      </jms-topic>

       

      On the Wildfly 11.0.0 JMS client, I created a corresponding netty connector:
      <connector name="remote-hornetq-connector" socket-binding="broker-socket" factory-class="org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory"/>

       

      with socket binding:
      <outbound-socket-binding name="broker-socket">
          <remote-destination host="abc.xyz" port="15455"/>
      </outbound-socket-binding>

       

      and then a pooled connection factory using HornetQClientProtocolManagerFactory that uses this netty connector:
      <pooled-connection-factory name="pooled-remote-connection-factory" entries="java:/RemoteJmsXA" connectors="remote-hornetq-connector" protocol-manager-factory="org.apache.activemq.artemis.core.protocol.hornetq.client.HornetQClientProtocolManagerFactory" transaction="xa" user="test" password="test"/>

       

      The listener MDB class on the JMS client is:

      @MessageDriven(activationConfig = {
      @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Topic"),
          @ActivationConfigProperty(propertyName = "destination", propertyValue = "topic/Product"),
          @ActivationConfigProperty(propertyName = "subscriptionDurability", propertyValue = "Durable"),
          @ActivationConfigProperty(propertyName = "subscriptionName", propertyValue = "ProductTopicListener"),
          @ActivationConfigProperty(propertyName = "reconnectAttempts", propertyValue = "-1"),
          @ActivationConfigProperty(propertyName = "setupAttempts", propertyValue = "-1"),
          @ActivationConfigProperty(propertyName = "maxSession", propertyValue = "5") })
      @ResourceAdapter("pooled-remote-connection-factory")
      @TransactionAttribute(TransactionAttributeType.REQUIRED)
      public class ProductTopicListener implements MessageListener {

       

      Using the above configurations, when I start the Wildfly 11.0.0 client, it comes up and starts looking for the broker, which is normal:
      2018-04-19 22:10:25,295 INFO  [org.apache.activemq.artemis.ra] (default-threads - 1) AMQ151005: awaiting server availability
      2018-04-19 22:10:27,288 INFO  [org.apache.activemq.artemis.ra] (default-threads - 3) AMQ151001: Attempting to reconnect org.apache.activemq.artemis.ra.inflow.ActiveMQActivationSpec(ra=org.wildfly.extension.messaging.activemq.ActiveMQResourceAdapter@c3be5320 destination=topic/Product destinationType=javax.jms.Topic ack=Auto-acknowledge durable=true clientID=null subscription=ProductTopicListener user=null maxSession=5)

       

      As soon as I start the Wildfly 8.2.0 broker, the client connects with it successfully:
      2018-04-19 22:11:25,809 INFO  [org.apache.activemq.artemis.ra] (default-threads - 4) AMQ151002: Reconnected with broker

       

      I am able to publish messages to the remote topic and then listen to it correctly from the client. All the changes that I made were on the client. No changes were required on the broker. As long as the broker is up, I can shut down the client and start it cleanly, and it reconnects with the broker automatically. When the broker is shut down, I get a bunch of WARN logs on the client stating that the connection no longer exists, which is again fine:
      2018-04-19 22:11:36,575 WARN  [org.apache.activemq.artemis.core.client] (Thread-2 (ActiveMQ-client-global-threads)) AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]

       

      Now, with the Wildfly 11.0.0 client still up, when I try to start the Wildfly 8.2.0 broker back up, it blows up with a ton of these exceptions:
      2018-04-19 22:12:57,424 ERROR [org.hornetq.core.client] (Thread-10 (hornetq-netty-threads-1812662695)) HQ214013: Failed to decode packet: java.lang.IllegalArgumentException: HQ119032: Invalid type: -4
      at org.hornetq.core.protocol.core.impl.PacketDecoder.decode(PacketDecoder.java:447) [hornetq-core-client-2.4.5.Final.jar:]
      at org.hornetq.core.protocol.ServerPacketDecoder.decode(ServerPacketDecoder.java:172) [hornetq-server-2.4.5.Final.jar:]
      at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:494) [hornetq-core-client-2.4.5.Final.jar:]
      at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:658) [hornetq-server-2.4.5.Final.jar:]
      at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.channelRead(HornetQChannelHandler.java:73) [hornetq-core-client-2.4.5.Final.jar:]
      at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338) [netty-all-4.0.15.Final.jar:4.0.15.Final]
      at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324) [netty-all-4.0.15.Final.jar:4.0.15.Final]
      at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:153) [netty-all-4.0.15.Final.jar:4.0.15.Final]
      at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338) [netty-all-4.0.15.Final.jar:4.0.15.Final]
      at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324) [netty-all-4.0.15.Final.jar:4.0.15.Final]
      at io.netty.handler.codec.ByteToMessageDecoder.handlerRemoved(ByteToMessageDecoder.java:110) [netty-all-4.0.15.Final.jar:4.0.15.Final]
      at io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:524) [netty-all-4.0.15.Final.jar:4.0.15.Final]
      at io.netty.channel.DefaultChannelPipeline.callHandlerRemoved(DefaultChannelPipeline.java:518) [netty-all-4.0.15.Final.jar:4.0.15.Final]
      at io.netty.channel.DefaultChannelPipeline.remove0(DefaultChannelPipeline.java:348) [netty-all-4.0.15.Final.jar:4.0.15.Final]
      at io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:319) [netty-all-4.0.15.Final.jar:4.0.15.Final]
      at io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:296) [netty-all-4.0.15.Final.jar:4.0.15.Final]
      at org.hornetq.core.protocol.ProtocolHandler$ProtocolDecoder.decode(ProtocolHandler.java:168) [hornetq-server-2.4.5.Final.jar:]
      at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:226) [netty-all-4.0.15.Final.jar:4.0.15.Final]
      at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:139) [netty-all-4.0.15.Final.jar:4.0.15.Final]
      at org.hornetq.core.protocol.ProtocolHandler$ProtocolDecoder.channelRead(ProtocolHandler.java:111) [hornetq-server-2.4.5.Final.jar:]
      at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338) [netty-all-4.0.15.Final.jar:4.0.15.Final]
      at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324) [netty-all-4.0.15.Final.jar:4.0.15.Final]
      at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785) [netty-all-4.0.15.Final.jar:4.0.15.Final]
      at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:132) [netty-all-4.0.15.Final.jar:4.0.15.Final]
      at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:485) [netty-all-4.0.15.Final.jar:4.0.15.Final]
      at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:452) [netty-all-4.0.15.Final.jar:4.0.15.Final]
      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:346) [netty-all-4.0.15.Final.jar:4.0.15.Final]
      at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101) [netty-all-4.0.15.Final.jar:4.0.15.Final]
      at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_161]

       

      The HornetQ broker is trying to decode a packet of type '-4' and throwing these exceptions constantly. No matter how many times I try to bring up the broker, I get the same exceptions. The only way out is to shut down the client and then start the broker again. I should also mention that when the broker is down, the client is extremely sluggish to shut down. I have to kill the client process. Ideally, the broker should just start up, and the client should reconnect with the broker, like it does when the servers are started for the first time.

      Any ideas as to what is happening to the broker here, and what should be done to fix this problem? Any suggestions would greatly help!

        • 1. Re: HornetQ - Wildfly 8 broker reconnection problems with ArtemisQ - Wildfly 11 client
          sunnyrays

          I think I found the reason why this is happening. I enabled TRACE for org.apache.activemq.artemis on the Wildfly 11.0.0 client to see what is going on under the covers. Here is what I found:

           

          2018-04-20 14:21:17,584 TRACE [org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl] (Thread-4 (ActiveMQ-client-global-scheduled-threads)) Writing buffer for channelID=0

          2018-04-20 14:21:17,585 DEBUG [org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl] (Thread-3 (ActiveMQ-client-global-threads)) Invocation of interceptor org.apache.activemq.artemis.core.protocol.hornetq.HQPropertiesConversionInterceptor on PACKET(CheckFailoverMessage)[type=-4, channelID=0, packetObject=CheckFailoverMessage, nodeID=0252b6e4-4414-11e8-8790-eb5efeec11e2] returned true

          2018-04-20 14:21:17,586 TRACE [org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl] (Thread-3 (ActiveMQ-client-global-threads)) Sending blocking PACKET(CheckFailoverMessage)[type=-4, channelID=1, packetObject=CheckFailoverMessage, nodeID=0252b6e4-4414-11e8-8790-eb5efeec11e2]

          2018-04-20 14:21:17,614 TRACE [org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl] (Thread-8 (ActiveMQ-client-netty-threads)) handling packet PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=60000]

          2018-04-20 14:21:17,615 DEBUG [org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl] (Thread-8 (ActiveMQ-client-netty-threads)) Invocation of interceptor org.apache.activemq.artemis.core.protocol.hornetq.HQPropertiesConversionInterceptor on PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=60000] returned true

           

          When the Wildfly 8.2.0 broker starts up, ArtemisQ on the Wildfly 11.0.0 client sends a number of these packets - PACKET(CheckFailoverMessage)[type=-4, channelID=1, packetObject=CheckFailoverMessage, nodeID=0252b6e4-4414-11e8-8790-eb5efeec11e2]. The problem here is that there is no packet of type CheckFailoverMessage (type=-4) in HornetQ, which is why I get the exception:

          2018-04-20 14:21:17,592 ERROR [org.hornetq.core.client] (Thread-10 (hornetq-netty-threads-1812662695)) HQ214013: Failed to decode packet: java.lang.IllegalArgumentException: HQ119032: Invalid type: -4

          at org.hornetq.core.protocol.core.impl.PacketDecoder.decode(PacketDecoder.java:447) [hornetq-core-client-2.4.5.Final.jar:]

          at org.hornetq.core.protocol.ServerPacketDecoder.decode(ServerPacketDecoder.java:172) [hornetq-server-2.4.5.Final.jar:]

           

          PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=60000] is a valid packet for both ArtemisQ and HornetQ, so this is decoded correctly, but not the CheckFailoverMessage (type=-4) packet. The PacketDecoder class in HornetQ does not have the below case, which is present in PacketDecoder for ArtemisQ:

             case CHECK_FOR_FAILOVER: {

                  packet = new CheckFailoverMessage();

                  break;

              }

           

          Is there a way to prevent ArtemisQ from sending the CheckFailoverMessage packet to HornetQ? Or, is there a way we can convert the CheckFailoverMessage to an acceptable packet for HornetQ? Or, is it a bug that requires a fix for the HornetQ support in Artemis? Please advise!

          • 2. Re: HornetQ - Wildfly 8 broker reconnection problems with ArtemisQ - Wildfly 11 client
            mnovak

            I'm afraid that this a bug. It's not possible to stop sending this packet somehow in configuration of client. Only what can help is to write interceptor for WF11 client which checks all packet types and filter out all unwanted packets. Check Artemis 1.5 docs how to do it.

            • 3. Re: HornetQ - Wildfly 8 broker reconnection problems with ArtemisQ - Wildfly 11 client
              sunnyrays

              Thanks for your response, mnovak. Yes, I already tried using an Interceptor that implements org.apache.activemq.artemis.api.core.Interceptor. If the packet is of type CheckFailoverMessage, I am setting the intercept to return false. This fixed the above start-up problems in HornetQ as the CheckFailoverMessage was no longer sent, but re-connection between ArtemisQ and HornetQ was not successful. Looks like something else also needs to be handled if this packet is not sent. I was not able to publish to the remote topic and listen to it anymore.

               

              I did a little more digging and came across this - https://issues.apache.org/jira/browse/ARTEMIS-1639. It appears that the Artemis team has fixed this problem, but it will be available in Artemis 2.5. As per https://issues.jboss.org/browse/JBEAP-13856, Artemis 2.5 will not be available before Wildfly 13, which is not released yet. Wildfly 11.0.0 comes bundled with Artemis 1.5.5 (artemis-*-1.5.5.jbossorg-008 jars). I looked at the commits for this - https://github.com/apache/activemq-artemis/pull/1819. It appears that the 4 modified class files are in artemis-core-client-*.jar and artemis-hqclient-protocol-*.jar. Is there a newer version of these two jars containing this fix, that I can use with Wildfly 11.0.0? I think that would solve my problem. Please let me know.

               

              Thanks!

              • 4. Re: HornetQ - Wildfly 8 broker reconnection problems with ArtemisQ - Wildfly 11 client
                mnovak

                You did a great progress and very good digging around this issue. This fix needs to be back-ported to Artemis 1.5.5.008. I've cherry picked it to 1.5.5.008 tag and somehow fixed the conflicts. I've attached modified jars. Could you try it? We'll see if it does the job. I did not have time to try it so there might be other issue.

                • 5. Re: HornetQ - Wildfly 8 broker reconnection problems with ArtemisQ - Wildfly 11 client
                  sunnyrays

                  Perfect! I tried these jars and now the connectivity works correctly. The ‘CheckFailoverMessage’ packet is no longer sent to HornetQ on startup, and once the ‘Ping’ packet gets a valid response, the two servers re-connect correctly. I am able to publish and listen to the remote topic again. I am testing this with some other scenarios as well, but it looks good so far!

                   

                  Just an FYI on the things that I tried myself - I looked at the changes made in the four files - TransportConfiguration.java, ClientSessionFactoryImpl.java, ClientProtocolManager.java, and HornetQClientProtocolManager.java - as mentioned in https://github.com/apache/activemq-artemis/pull/1819/files. I created these files on my own using all the necessary dependencies, compiled the required class files after making the given changes, and re-packaged new jars - artemis-core-client-1.5.5.jbossorg-008-updated.jar and artemis-hqclient-protocol-1.5.5.jbossorg-008-updated.jar. These jars worked correctly as well! I am attaching them here for your reference.

                   

                  But, now that you have provided these two jars, I will be using these. These are much safer!

                   

                  Thank you so much, mnovak !

                  • 6. Re: HornetQ - Wildfly 8 broker reconnection problems with ArtemisQ - Wildfly 11 client
                    mnovak

                    You're welcome! Backward compatibility things can be tricky. It's good to see that you test it with other scenarios. If you plan to put it to production then all lifecycle and crash scenarios should be tried.

                    • 7. Re: HornetQ - Wildfly 8 broker reconnection problems with ArtemisQ - Wildfly 11 client
                      sunnyrays

                      mnovak I have tested the jars locally and they seem to be working correctly so far. Will you be adding the final jars to a standard repository location from where we can download them? If so, then please let me know the location. Also, will these two Artemis jars be bundled with future downloads of Wildfly 10/11/12? Thanks again!

                      • 8. Re: HornetQ - Wildfly 8 broker reconnection problems with ArtemisQ - Wildfly 11 client
                        mnovak

                        I don't have rights to deploy it to some maven repo. Also I did the backport in the rough way. There were some changes in server jars where I did not bother to make it right (I don't know how). Could you create ARTEMIS upstream jira to backport it to Artemis 1.5.x, please? It will be much better if Artemis dev will do it.

                        • 10. Re: HornetQ - Wildfly 8 broker reconnection problems with ArtemisQ - Wildfly 11 client
                          sunnyrays

                          mnovak - as per the conversation in ARTEMIS-1849, I checked out the '1.5.5.jbossorg-008' tag from 'jboss-activemq-artemis' project, applied the required fixes manually, and built the entire project with necessary jars for Wildfly 11.0.0 myself. Probably that's what you did too. The new jars are working correctly now.