Communication between Hornetq 2.4.0.Final and Hornetq 2.2.5.Final

[natarajanram]

Hi,

We have a use case where we are trying to establish communication between Hornetq 2.4.0.Final and Hornetq 2.2.5.Final - i.e. message comes from HQ2.4.0 into HQ2.2.5 and then get re-directed back to HQ2.4.0 for consumption.

Looking at suggestions from Justin at: HornetQ 2.3.1 compatibility with HornetQ 2.2.5 we were able to do this using JMS bridge which uses JTA transaction manager (as also mentioned in this post: HornetQ standalone - JMS Bridge? ).

To avoid using the JTA transaction manager jars and for performance reasons, we wanted to see if the Core-Bridge can be used (instead of JMS bridge). The thought is to create the Bridge in HQ2.2.5 server and its purpose is to pick up a queue message from HQ2.4.0 and redirect it back to a different queue in HQ2.4.0. Both are running on localhost (HQ2.4 netty connector on 5449 and HQ2.2.5 on default 5445).

My hornetq-configuration in 2.2.5 is as follows:

<connectors>

<snip>

      <!-- Connector to the other node -->

      <connector name="remote-connector">

         <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

         <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>

         <param key="port"  value="${hornetq.remoting.netty.batch.port:5449}"/>

      </connector>  

  </connectors>

   <queues>  

      <queue name="jms.queue.bridgeExampleQueue">

         <address>jms.queue.bridgeExampleQueue</address>

      </queue>

  </queues>

   <!-- We set-up a bridge that forwards from a queue on this node to an address on another node.

   We specify a filter with the bridge, and a transformer too. The filter and transformer are optional -->

   <bridges>

      <bridge name="my-bridge">

          <queue-name>jms.queue.bridgeExampleQueue</queue-name>

          <forwarding-address>jms.queue.targetExampleQueue</forwarding-address>

          <reconnect-attempts>-1</reconnect-attempts>

          <static-connectors>

             <connector-ref>remote-connector</connector-ref>

          </static-connectors>

       </bridge>

   </bridges> 

As mentioned, bridgeExampleQueue and targetExampleQueue are in HQ2.4.0 and i am trying to use the bridge in HQ2.2.5 to re-direct the message from bridgeExampleQueue to targetExampleQueue.

I take it that with the above configuration it recognizes that targetExampleQueue is on another HQ instance and looks for it at: localhost:5449.

The issue is that the bridge does not know that "bridgeExampleQueue" (the source queue) is also on remote location (i.e. localhost on 5449). I do not see a way to specify that the source (<queue-name>) is on different HQ instance. Is this possible?

Thanks in advance

Ram

[jbertram]

The functionality you're looking for is not supported by the core bridge. The core bridge just forwards messages from a *local* queue to a different address. The address to which the messages are forwarded can be on a the local server or a remote server, but the originating queue must be local.

You could set up a bridge on the remote HornetQ instance to forward messages back or you could just use the JMS bridge as you already have been doing.

[natarajanram]

Thank you for your response. If core bridge is setup on 2 different HornetQ instances as you suggested, how will the performance be compared to setting up JMS bridge on 1 server?

[jbertram]

It's impossible to comment authoritatively on performance given all the unknown details, but I would expect performance of 2 core bridges to generally be higher than a single JMS bridge using ONCE_AND_ONLY_ONCE QoS.

[natarajanram]

Hi Justin,

Thanks a lot for your input. We tried setting up the core bridge and were successful in setting one in Hornetq 2.2.5 with the forwarding address queue in Hornetq 2.4.0.

But we are having issues with the reverse test case – setting up the bridge in Hornetq 2.4.0 and forwarding queue in Hornetq 2.2.5.

The configuration snippets in Hornetq2.4.0 is as follows:

Hornetq-configuration.xml (2.4.0)

   <connectors>

      <connector name="netty">

<factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

         <param key="host" value="${hornetq.remoting.netty.host:localhost}"/>

         <param key="port" value="${hornetq.remoting.netty.port:5449}"/>

      </connector>

      <connector name="netty-throughput">

<factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

         <param key="host" value="${hornetq.remoting.netty.host:localhost}"/>

         <param key="port" value="${hornetq.remoting.netty.batch.port:5459}"/>

         <param key="batch-delay" value="50"/>

      </connector>

      <!-- Connector to the other node -->

      <connector name="remote-connector">

<factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

         <param key="port" value="5445"/>

      </connector>

   </connectors>

   <acceptors>

      <acceptor name="netty">

<factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>

         <param key="host" value="${hornetq.remoting.netty.host:localhost}"/>

         <param key="port" value="${hornetq.remoting.netty.port:5449}"/>

      </acceptor>

      <acceptor name="netty-throughput">

         <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>

         <param key="host" value="${hornetq.remoting.netty.host:localhost}"/>

         <param key="port" value="${hornetq.remoting.netty.batch.port:5459}"/>

         <param key="batch-delay" value="50"/>

         <param key="direct-deliver" value="false"/>

      </acceptor>

   </acceptors>

      <!-- We need to create a core queue for the JMS queue explicitly because the bridge will be deployed

   before the JMS queue is deployed, so the first time, it otherwise won't find the queue -->

   <queues>  

      <queue name="jms.queue.sausage-factory">

<address>jms.queue.sausage-factory</address>

      </queue>

  </queues>

   <!-- We set-up a bridge that forwards from a queue on this node to an address on another node.

   We specify a filter with the bridge, and a transformer too. The filter and transformer are optional -->

   <bridges>

      <bridge name="my-bridge">

<queue-name>jms.queue.sausage-factory</queue-name>

<forwarding-address>jms.queue.mincing-machine</forwarding-address>

<reconnect-attempts>10</reconnect-attempts>

          <static-connectors>

<connector-ref>remote-connector</connector-ref>

</static-connectors>

       </bridge>

   </bridges>

Hornetq-jms.xml (2.4.0)

   <connection-factory name="NettyConnectionFactory">

      <xa>false</xa>

      <connectors>

         <connector-ref connector-name="netty"/>

      </connectors>

      <entries>

         <entry name="/ConnectionFactory"/>

      </entries>

   </connection-factory>

   <connection-factory name="NettyThroughputConnectionFactory">

      <xa>true</xa>

      <connectors>

         <connector-ref connector-name="netty-throughput"/>

      </connectors>

      <entries>

         <entry name="/XAThroughputConnectionFactory"/>

      </entries>

   </connection-factory>

   <connection-factory name="NettyThroughputConnectionFactory">

      <xa>false</xa>

      <connectors>

         <connector-ref connector-name="netty-throughput"/>

      </connectors>

      <entries>

         <entry name="/ThroughputConnectionFactory"/>

      </entries>

   </connection-factory>

   <connection-factory name="ClientConnectionFactory">

      <connectors>

         <connector-ref connector-name="netty"/>

      </connectors>

      <entries>

         <entry name="/client/ConnectionFactory"/>

      </entries>

   </connection-factory>

   <!--the connection factory used by the example-->

   <connection-factory name="ConnectionFactory">

      <connectors>

         <connector-ref connector-name="netty"/>

      </connectors>

      <entries>

         <entry name="ConnectionFactory1"/>

      </entries>

   </connection-factory>

   <queue name="DLQ">

      <entry name="/queue/DLQ"/>

   </queue>

   <queue name="ExpiryQueue">

      <entry name="/queue/ExpiryQueue"/>

   </queue>

      <!--the queue used by the example-->

   <queue name="sausage-factory">

      <entry name="/queue/sausage-factory"/>

   </queue>

Similarly in 2.2.5, the configurations are:

Hornetq-configurations.xml

   <connectors>

      <connector name="netty">

<factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

         <param key="host" value="${hornetq.remoting.netty.host:localhost}"/>

         <param key="port" value="${hornetq.remoting.netty.port:5445}"/>

      </connector>

      <connector name="netty-throughput">

         <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

         <param key="host" value="${hornetq.remoting.netty.host:localhost}"/>

         <param key="port" value="${hornetq.remoting.netty.batch.port:5455}"/>

         <param key="batch-delay" value="50"/>

      </connector>

   </connectors>

   <acceptors>

      <acceptor name="netty">

<factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>

         <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>

         <param key="port" value="${hornetq.remoting.netty.port:5445}"/>

      </acceptor>

      <acceptor name="netty-throughput">

<factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>

         <param key="host" value="${hornetq.remoting.netty.host:localhost}"/>

         <param key="port" value="${hornetq.remoting.netty.batch.port:5455}"/>

         <param key="batch-delay" value="50"/>

         <param key="direct-deliver" value="false"/>

      </acceptor>

   </acceptors>

Hornet-jms.xml (2.2.5)

   <connection-factory name="NettyConnectionFactory">

      <xa>true</xa>

      <connectors>

         <connector-ref connector-name="netty"/>

      </connectors>

      <entries>

         <entry name="/XAConnectionFactory"/>

      </entries>

   </connection-factory>

   <connection-factory name="NettyConnectionFactory">

      <xa>false</xa>

      <connectors>

         <connector-ref connector-name="netty"/>

      </connectors>

      <entries>

         <entry name="/ConnectionFactory"/>

      </entries>

   </connection-factory>

   <connection-factory name="NettyThroughputConnectionFactory">

      <xa>true</xa>

      <connectors>

         <connector-ref connector-name="netty-throughput"/>

      </connectors>

      <entries>

         <entry name="/XAThroughputConnectionFactory"/>

      </entries>

   </connection-factory>

   <connection-factory name="NettyThroughputConnectionFactory">

      <xa>false</xa>

      <connectors>

         <connector-ref connector-name="netty-throughput"/>

      </connectors>

      <entries>

         <entry name="/ThroughputConnectionFactory"/>

      </entries>

   </connection-factory>

   <!--the queue used by the example-->

   <queue name="mincing-machine">

      <entry name="/queue/mincing-machine"/>

   </queue>

I am able to start both servers. After a while, I see this message in the hornet-2.4.0 console:

INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.queue.sausage-factory

INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.queue.DLQ

INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.queue.ExpiryQueue

INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.queue.sausage-factory

INFO  [org.hornetq.core.server] HQ221020: Started Netty Acceptor version 4.0.13.Final localhost:5459

INFO  [org.hornetq.core.server] HQ221020: Started Netty Acceptor version 4.0.13.Final localhost:5449

WARN  [org.hornetq.core.server] HQ222171: Bridge my-bridge forwarding address jms.queue.mincing-machine could not be resolved on address-settings configuration

INFO  [org.hornetq.core.server] HQ221007: Server is now live

INFO  [org.hornetq.core.server] HQ221001: HornetQ Server version 2.5.0.SNAPSHOT (Wild Hornet, 124) [fac29d9c-23bd-11e9-aac4-8955a5d2568b]

WARNING [io.netty.channel.AbstractChannel] Can't invoke task later as EventLoop rejected it: java.util.concurrent.RejectedExecutionException: event executor terminated

               at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:703) [netty.jar:4.0.13.Final]

               at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:296) [netty.jar:4.0.13.Final]

               at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:688) [netty.jar:4.0.13.Final]

               at io.netty.channel.AbstractChannel$AbstractUnsafe.invokeLater(AbstractChannel.java:727) [netty.jar:4.0.13.Final]

               at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:557) [netty.jar:4.0.13.Final]

               at io.netty.channel.DefaultChannelPipeline$HeadHandler.close(DefaultChannelPipeline.java:1018) [netty.jar:4.0.13.Final]

I came across: https://issues.jboss.org/browse/HORNETQ-1312 - which seems to indicate that the issue happens during stress to the system. I got them even before attempting to send any messages. I also made sure to kill any hornetq processes before repeating the test. Can you please provide your inputs on why the above could have happened?

Similarly, after HornetQ 2.2.5 starts up, I see this in the console:

INFO [org.hornetq.core.server.impl.HornetQServerImpl]  trying to deploy queue jms.queue.DLQ

INFO [org.hornetq.core.server.impl.HornetQServerImpl]  trying to deploy queue jms.queue.ExpiryQueue

INFO [org.hornetq.core.server.impl.HornetQServerImpl]  trying to deploy queue jms.queue.mincing-machine

INFO [org.hornetq.core.remoting.impl.netty.NettyAcceptor]  Started Netty Acceptor version 3.2.3.Final-r${buildNumber} localhost:5445 for CORE protocol

INFO [org.hornetq.core.remoting.impl.netty.NettyAcceptor]  Started Netty Acceptor version 3.2.3.Final-r${buildNumber} localhost:5455 for CORE protocol

INFO [org.hornetq.core.server.impl.HornetQServerImpl]  Server is now live

INFO [org.hornetq.core.server.impl.HornetQServerImpl]  HornetQ Server version 2.2.5.Final (HQ_2_2_5_FINAL_AS7, 121) [cff16942-2053-11e9-b690-fa163e83fad6] started

WARNING [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]  Connection failure has been detected: Did not receive data from /127.0.0.1:33927. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]

WARNING [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]  Connection failure has been detected: Did not receive data from /127.0.0.1:33934. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]

I am not sure if the above is due to stress to the system, please let me know what could have caused it.

After the above, I tried to run a modified version of 2.4.0 jms/bridge example. I commented out the steps to start the servers, since I had already done so. So just ran the portions that look up the connection, queues and produce the message, send and then consume the message. This failed while trying to create Connection in Hornetq 2.2.5 with the following exception:

INFO: using jnp://localhost:1100 for jndi

ConnectionFactory0 - which is HQ2.4.0 - HornetQConnectionFactory [serverLocator=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5449&host=localhost], discoveryGroupConfiguration=null], clientID=null, consumerWindowSize = 1048576, dupsOKBatchSize=1048576, transactionBatchSize=1048576, readOnly=false]

• org.hornetq.common.example.HornetQExample getContext

INFO: using jnp://localhost:1099 for jndi

** Sent message: MESSAGE: Test message sent at 1549370916428  node 0

ConnectionFactory1 - which is HQ2.2.5 - HornetQConnectionFactory [serverLocator=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=localhost], discoveryGroupConfiguration=null], clientID=null, consumerWindowSize = 1048576, dupsOKBatchSize=1048576, transactionBatchSize=1048576, readOnly=false]

1. javax.jms.JMSException: Failed to create session factory

               at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:676)

               at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:115)

               at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:110)

               at org.hornetq.jms.example.BridgeExample.runExample(BridgeExample.java:103)

               at org.hornetq.common.example.HornetQExample.run(HornetQExample.java:70)

               <snip>

Caused by: java.lang.NullPointerException

               at org.hornetq.core.client.impl.ServerLocatorImpl.assertOpen(ServerLocatorImpl.java:2035)

               at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:831)

               at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:672)

               ... 31 more         

I am not sure if the above is expected after Hornetq 2.2.5 console had this message:

WARNING [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]  Connection failure has been detected: Did not receive data from /127.0.0.1:33934. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]

Please let me know if I have missed anything in config or potentially if it’s a code issue.

Thanks

Ram

[jbertram]

It could be that the client libraries from HornetQ 2.4.0 are not compatible with 2.2.5, although that would be a surprise to me because we've always tried to maintain compatibility with older brokers. Bugs are always possible.

In order to investigate this further I'd need to set up instances of HornetQ 2.2.5 & 2.4.0, etc., but ultimately I'm just not willing to invest more time on this thread on versions of the software which are so old. The HornetQ code-base was donated to the Apache ActiveMQ community over 3 years ago and it lives on as the ActiveMQ Artemis broker. We've had 23 releases since then. I recommend you keep your software updated in order to get all the latest bug fixes and new features and also to receive support. Using versions which are so old means it's unlikely you'll receive much community support.

At this point I'd recommend you keep your JMS bridge solution in place. You might even deploy a second bridge with the same configuration to see if that might increase message throughput.

[natarajanram]

Thanks for the suggestions.