-
1. Re: Large Messages disappearing
clebert.suconic Sep 3, 2009 8:54 AM (in response to thammoud)100k is already big for large messages.
Are you playing with rollbacks? We've had an issue with flow control and rollback that is fixed on trunk.
Or maybe you're not ACKing messages? If you have a testcase we would take a look. -
2. Re: Large Messages disappearing
clebert.suconic Sep 3, 2009 10:12 AM (in response to thammoud)"thammoud" wrote:
We increased the size of the large message to 250m
BTW: I assumed you meant 250K (or 250Kib). -
3. Re: Large Messages disappearing
thammoud Sep 3, 2009 10:21 AM (in response to thammoud)OK. I found the issue.
MessageConsumer queueReceiver = outboundSession.createConsumer(tempQueue); defaultPublisher.send(outboundQueue, message); ObjectMessage reply = (ObjectMessage)queueReceiver.receive(); queueReceiver.close(); tempQueue.delete(); return (UserMessageReplyEvent)reply.getObject();
The tempQueue delete is affecting the outcome of the reply.getObject if the size of the buffer is > 100K. i.e If I extract the object from the reply message and then delete the tempQueue, it always works. I assumed that when receive returns, I can safely delete the queue. Will fix my code. There must be size dependent logic taking place place in ObjectMessage.getObject(); -
4. Re: Large Messages disappearing
clebert.suconic Sep 3, 2009 10:43 AM (in response to thammoud)LargeMessage is a streaming of packages on the wire.
When you receive the message on consumer.receive(); you're only receiving what fits on the consumer flow control.
As you take packets out of the stream you will get more packets from the server.
If you move to the next message, the packets are ignored... (there is an explanation about this on the large message doc).
But as you deleted the queue, you broke the flow.
I will look at what could be improved in terms of exceptions or controls when you delete a queue containing large messages. -
5. Re: Large Messages disappearing
clebert.suconic Sep 3, 2009 1:01 PM (in response to thammoud)The problem was you closing the Consumer before reading the large message, and not deleting the queue.
You can't delete the queue if there are active consumers.
I'm adding a check on that. If you close the consumer and read the large message stream after the close you would get an exception"if (streamCancelled) { throw new IllegalAccessError("The consumer associated with this large message was closed before the body was read"); }
BTW: ObjectMessages are slow (on the client). So, as long as you keep all those operations on the client it's fine. If you are reading ObjectMessages on the server I would advice to find some alternative.
- ObjectMessage will need the entire object in memory
- It will perform expensive operations throught Object(input|output)Stream
If you read the LargeMessage as a streaming/file.. you would get much better performance out of that. -
6. Re: Large Messages disappearing
clebert.suconic Sep 3, 2009 1:04 PM (in response to thammoud)If you are reading ObjectMessages on the server I would advice to find some alternative.
BTW: By server here I mean a VM in your server box.. such as Servlet/EJBs/JSPs.. etc.
The HornetQServer doesn't treat an ObjectMessage any different on the server side. Everything are just messages with bodies.
HornetQ core API doesn't treat it any different also. The only differentiation is made on the JMS layer on the client side.
If you're doing a lot of ObjectMessages in your server VM, you will likely require more hardware to process them. -
7. Re: Large Messages disappearing
thammoud Sep 3, 2009 7:02 PM (in response to thammoud)Al messages that process on the server are object messages. All messages will need to be serialized. Ok we are lazy but serialization in the server itself prevents us from writing some nasty streaming code. Most objects are very small. Results can however vary in size.
I guess my question to you is, and without offending anyone from the JMS committee, is there a way to turn off serialization when using the in-vm connector. Here is the thing, all objects tossed back and forth between the various subsystems are guaranteed to be immutable. ActiveMQ provides such an option with the in-vm transport.
Thanks for your prompt answers. Hornet truly looks like a wonderful platform. Keep up the fantastic work. -
8. Re: Large Messages disappearing
clebert.suconic Sep 4, 2009 10:38 AM (in response to thammoud)I guess my question to you is, and without offending anyone from the JMS committee, is there a way to turn off serialization when using the in-vm connector
You could maybe request a feature on JIRA such as "optionally send messages by reference on In-VM transport", and we would discuss about it internally:
https://jira.jboss.org/jira/browse/HORNETQ -
9. Re: Large Messages disappearing
clebert.suconic Sep 4, 2009 12:33 PM (in response to thammoud)Actually, we already have a feature request for this:
https://jira.jboss.org/jira/browse/HORNETQ-72
if you want, you could vote for it. -
10. Re: Large Messages disappearing
thammoud Sep 4, 2009 1:19 PM (in response to thammoud)Will vote for it. Thanks for all your help.
-
11. Re: Large Messages disappearing
timfox Sep 4, 2009 5:39 PM (in response to thammoud)If you are using JMS ObjectMessages they will always get serialized when you call setObject on the message.
This is necessary for JMS compliance.
BTW Ob jectMessage are the slowest kind of message you can use. Also the serialized form is verbose because it used Java serialization.
This is discussed a little in the user manual in the section on performance tuning -
12. Re: Large Messages disappearing
thammoud Sep 11, 2009 7:22 AM (in response to thammoud)Tim,
Per spec you are absolutely right. However, the pattern when using in-vm transport to communicate amongst different subsystems in the same container is very common. It is no different than the local interfaces with EJB calls. In the latter no serialization needs to take place which makes perfect sense. The EJB spec allows us, the developers, to communicate with the proper remote or local interface based on our needs.
Using objects to communicate internally is of great convenience. Doing manual serialization just to abide by a spec is giving up the convenience for no real good reason.
The fact that the JMS folks did not address the serialization issue when communicating locally does not make it an invalid requirement. It should be added as a flag to the in-vm connection factory at a developers discretion. I would say that that will be augmenting the spec and not breaking it given the default behavior. I hope that you implement HORNETQ-72. -
13. Re: Large Messages disappearing
timfox Sep 11, 2009 9:02 AM (in response to thammoud)I don't have a problem with creating a message type which allows an arbitrary object to be passed by reference, but that's not JMS ObjectMessage's role.
A JMS ObjectMessage has well defined semantics, and I don't want to muddy those. -
14. Re: Large Messages disappearing
timfox Sep 11, 2009 9:06 AM (in response to thammoud)Also, on a slightly different node, JMS ObjectMessage is often used as a lazy way to pass any object.
I've seen people use ObjectMessage and they're just setting a String, or an int or a long in the body!
This is an incredibly bad idea, and lazy programming. ObjectMessage always uses java serialization, so the message body will probably be several hundred bytes long, even if you just stored a single int.
Also java serialization is slow.
Don't use Objectmessage unless you really have to! JMS comes with a variety of types - textmessage, mapmessage etc which are optimised for that particular type of data. They are infinitely preferable to using an ObjectMessage.
The only time when an ObjectMessage should be used is when you do not know the type of the object you are passing until run-time.