performance of cluster over WAN

[nmichalakis]

Hi everyone,

I'm evaluating HornetQ to possibly replace our existing solution. We have 2 datacenters, one in the US, one in Europe. I was reading the docs, but couldn't find many details on what exactly the underlying protocol is for sending messages between cluster nodes is. I 've seen that HornetQ relies  on Netty connectors on default configs, but is there more specific information on whether there are particular choices that would perform better over a high latency and possibly lossy links (we observed a 1-10% packet loss sometimes)? More particularly I'm looking to find out whether the connections happen over TCP or UDP, single or multiple connections between nodes and what serialization protocol is used and any reconnection policies.

I appreciate the help,

Nikos

[timfox]

For message distribution across unreliable connections like WAN, I recommend you use a core bridge (see user manual for more info)

[nmichalakis]

Hi Tim,

Thanks for the pointer.

I've looked at the core bridge docs at http://hornetq.sourceforge.net/docs/hornetq-2.0.0.GA/user-manual/en/html/core-bridges.html and here are some of my answers to my own questions. Hopefully you can verify or fill in the gaps:

What kind of transport is there between HornetQ nodes? TCP? UDP?

The core bridge can be set up with different connectors, but couldn't find much information about what connection factories are the best choices for a WAN and why? I'd imagine a TCP connection factory would be best. Are there any more documents on that?

Are there single or multiple connections open between nodes for redundancy?

(couldn't find info on that)

What happens when a TCP connection fails? Do nodes detect that event and try to reestablish the connection automatically?

A core bridge has configurable timeout and retry mechanisms that can deal with that as well as detect duplicates.

How does a node detect a connection failed? Does it rely on TCP or it's own ack messages?

(couldn't find info on that)

Do connections rely on some mechanism to deal with link packet loss?

I would guess you rely on TCP for that, but not sure if you have anything on top of that (say you see that suddenly a TCP connection slows down, you may open multiple connections to the target and send a message multiple times).

How are messages serialized over the transport? Do you use RMI or something else?

(couldn't find info on that)

I appreciate any help,

Nikos

[timfox]

Nikos, AFAICT all your questions are answered in the user manual, chaperts on core bridges, configuring transport etc:

http://hornetq.sourceforge.net/docs/hornetq-2.1.0.CR1/user-manual/en/html/configuring-transports.html

http://hornetq.sourceforge.net/docs/hornetq-2.1.0.CR1/user-manual/en/html/connection-ttl.html

http://hornetq.sourceforge.net/docs/hornetq-2.1.0.CR1/user-manual/en/html/send-guarantees.html

http://hornetq.sourceforge.net/docs/hornetq-2.1.0.CR1/user-manual/en/html/client-reconnection.html

http://hornetq.sourceforge.net/docs/hornetq-2.1.0.CR1/user-manual/en/html/duplicate-detection.html

etc

[nmichalakis]

Hi Tim,

I've read those docs, but they didn't really find what I was looking for.

I'd like to know what the Netty Acceptor and Netty Connector do under the cover. Are they using UDP or TCP for example? Is there a way to configure them to UDP or TCP explicitly to test the different configurations?

Also, how is a message serialized over the network? Do you use a standard or you have your own serialization protocols?

thanks

[clebert.suconic]

Nikos,

My advice is, read the documentation. You can answer most of these questions if you read the whole doc yourself.

I know it's easier to ask, but we won't scale up if we answer questions that can be answered by reading the docs.

In addition, look at this  blog post written by Jeff: http://hornetq.blogspot.com/2009/10/understanding-connectors-acceptors.html

Also, Serialization on HornetQ is used only to serialize ObjectMessages.. Which there is a word about it on the chapter about performance.

A messaging system (any messaging system out there.. I know most of them) is responsible for sending messages which will consist for message properties and a payload of a byteArray. that's pretty much it on the server side.

[nmichalakis]

I apologize for any misunderstanding. Your documentation is great (far better than the other broker products I'm evaluating) so it was very easy for me to setup a cluster using server discovery using the 2.0.0.GA version.
I tried to read it as thoroughly as I could. However, when it came down to tweaking the configuration to test different options I was stuck trying to figure out if UDP is an option that we could test as well as how are messages serialized
and whether there was an option to tweak there (our current solution uses RMI and we suspect it is hurting our performance). Regarding serialization, Clebert, thank you for the answer. Regarding configuring the transport
your docs point to Netty. The netty docs mention that they support both UDP and TCP transports transparently, so I wanted to know how HornetQ uses Netty underneath.

So I searched for UDP on your doc:

http://hornetq.sourceforge.net/docs/hornetq-2.0.0.GA/user-manual/en/html_single/index.html

Here is what I found:
1.
30.6 Message Counters
udpateTimestamp (a typo I presume)

2.
38.2. Server discovery
Server discovery uses UDP  multicast to broadcast server connection settings. If UDP is disabled on your network... (related to server discovery, not the transport between cluster nodes or bridges...)

3.
38.2.1. Broadcast Groups
...It also defines the UDP address and port settings... (also related to server discovery, not transport on bridges...)

group-port. This is the UDP port number used for broadcasting. This parameter is mandatory. (same here...)

4.
38.2.3. Defining Discovery Groups on the Server
group-port. This is the UDP port of the multicast group. (also about server discovery...)

5.
38.5. Specifying Members of a Cluster Explicitly
Sometimes UDP is not enabled on a network so it's not possible to use UDP server discovery for clients to discover...(server discovery...)

6.
38.7.1. Symmetric cluster
Typically the cluster connection will use server discovery in order to know what other servers in the cluster it should connect to,
although it is possible to explicitly define each target server too in the cluster connection if, for example, UDP is not available on your network. (server discovery...)

Also I had read the blog post before you mentioned it (which is a great overview of acceptors and connectors):
http://hornetq.blogspot.com/2009/10/understanding-connectors-acceptors.html

But a search on "UDP" produces no matches.

There is a thread http://community.jboss.org/message/541291 that discusses UDP also related to server discovery.

Finally, a grep on the hornetq-2.0.0.GA/ directory after I untar hornetq returns "UDP" only in the context of server discovery or connection factory discovery.

Then I decided to register to JBoss and post my questions. I hope I have convinced you that I do take your documentation seriously.

You explicitly mention TCP, HTTP, SSL, Servlet but no UDP. My logical conclusion was that you don't have a UDP transport, but I just wanted to make sure by asking you guys. It would have been
nice to have this stated explicitly since Netty does support UDP (thus my confusion).

So is my assumption correct of you not supporting UDP transports?

I hope you take my feedback in a positive way and hopefully can help you improve the already great documentation you have.

cheers,
nikos

[clebert.suconic]

Thanks, you sound much more concise now.. and it is easier to understand what are your questions now.

Maybe it was just my impression before.

So, we don't use UDP at the moment, just TCP as you noticed on the doc... UDP is used just for discovery but not for transport. UDP is used to establish membership and establishing bridges between the clusters. There is a JIRA somewhere though to distribute message topics on the cluster through UDP. (Maybe JGroups? It wasn't decided the implementation yet)

Regarding Serialization on the server... Serialization is slow.. We just marshal the messages ourselves and send the byte arrays as fast as possible. Serialization as I said is only used at the client o marshal/unmarshal the body on ObjectMessage, but the message is always sent as a CoreMessage underneath. At the ServerMessage, you will realize the body of the message is a byte array exposed as a Buffer through the API.

We wouldn't get near the performance we see with HornetQ if we were using any reflection or extensive meta-data and classloading at the server's side (read it serialization).

[nmichalakis]

Thanks for the answers. That's what i was looking for. I'm glad we synced up

It's great to hear. Using NIO and avoiding RMI are great design choices for performance.