I am working on a project where I will probably have two to three ActiveMQ brokers setup in a chain where I will need to be publishing files that can be anywhere from a few MB to over 70 MB in size. I have already been able to setup some topic forwarding across the brokers and that all seems fine. The problem right now is how do I transfer the files efficiently. What I want is to be able to send a set of messages that contain portions of the file and then just have the receiver put them back together. From what I have looked into I think there are two ways of doing this.
It seems that with ActiveMQ you can use ActiveMQ Stream objects where you just write to one stream from the producer side and read from another stream on the subscribers side. I had found an example like this and was able to get it working. The problem is that when I send a message I want it to propagate through the brokers to the client immediately. In my test it looked like the first broker in the chain had to receive at least 64 KB before it passed the message on to the next broker. If I wrote more than 64 KB in one write to the stream it looked like the ActiveMQ broker made it into two messages. While this does seem to work I need more finite control over the size of the messages so unless there is some way to change the 64 KB enforced message size that I saw then I might need to find another way.
The second option I tried was of just sending multiple ByteMessage objects. In this case I was able to control the size of the messages sent and they seemed to propagate immediately across the brokers.
So my question is what is considered the best way for transferring large files over ActiveMQ while having complete control over the size of the messages sent? Is sending multiple ByteMessages the best solution? Is there something about the ActiveMQ Stream messages that I overlooked that could of given me greater control over the message size? Thank you for any help in this.
Edited by: brett on Nov 1, 2010 3:11 PM