Sending large files over ActiveMQ

[brett_brettl]

I am working on a project where I will probably have two to three ActiveMQ brokers setup in a chain where I will need to be publishing files that can be anywhere from a few MB to over 70 MB in size. I have already been able to setup some topic forwarding across the brokers and that all seems fine. The problem right now is how do I transfer the files efficiently. What I want is to be able to send a set of messages that contain portions of the file and then just have the receiver put them back together. From what I have looked into I think there are two ways of doing this.

It seems that with ActiveMQ you can use ActiveMQ Stream objects where you just write to one stream from the producer side and read from another stream on the subscribers side. I had found an example like this and was able to get it working. The problem is that when I send a message I want it to propagate through the brokers to the client immediately. In my test it looked like the first broker in the chain had to receive at least 64 KB before it passed the message on to the next broker. If I wrote more than 64 KB in one write to the stream it looked like the ActiveMQ broker made it into two messages. While this does seem to work I need more finite control over the size of the messages so unless there is some way to change the 64 KB enforced message size that I saw then I might need to find another way.

The second option I tried was of just sending multiple ByteMessage objects. In this case I was able to control the size of the messages sent and they seemed to propagate immediately across the brokers.

So my question is what is considered the best way for transferring large files over ActiveMQ while having complete control over the size of the messages sent? Is sending multiple ByteMessages the best solution? Is there something about the ActiveMQ Stream messages that I overlooked that could of given me greater control over the message size? Thank you for any help in this.

Edited by: brett on Nov 1, 2010 3:11 PM

[dejanb_dejan]

Hi,

the best way to move large messages is to use Blob messages (http://activemq.apache.org/blob-messages.html). In this scenario the content of the message is transferred through some external server (usually http or ftp) and the rest goes through the broker. This gives the best balance as the large content goes through the infrastructure made to deal with it and doesn't impact the broker.

Hope this helps,

Dejan

Edited by: dejanb on Dec 9, 2010 4:13 PM

[rnewcomb_rnewcomb]

I would favor an out-of-band mechanism for delivering the payload, such as the Blob message described by Dejan, or a simple Text message containing a URL for retrieving the data.  Note that the ActiveMQ Blob message is not a standard JMS type, and is therefore non-portable.

If you really want to send the actual payload though the broker network, I recommend using the Stream message instead of programmatically marshalling / chunking the content into separate Bytes messages. The programmatic marshalling mechanism would require a custom unmarshaller on the receiving side - which is undesirable.

The 64KB message chunking size you are observing in the Stream message handling across brokers is likely driven by the TCP transport socket buffer.  You may want to tune the TCP configuration for your specific needs .

http://activemq.apache.org/tcp-transport-reference.html

Edited by: rnewcomb on Dec 13, 2010 6:21 PM