6 Replies Latest reply on Aug 2, 2012 12:38 PM by zont

Combination of Msg. size and 'min-large-message-size' causes exception

aengineer Jul 20, 2010 2:42 PM

We have a JMS producer that reads a file and publishes the contents of the file as the body of a TextMessage. For a specific file size (i.e. message size), the producer gets an exception:

javax.jms.JMSException
    at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:287)
    at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:285)
    at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:139)
    at org.hornetq.jms.client.HornetQMessageProducer.doSend(HornetQMessageProducer.java:451)
    at org.hornetq.jms.client.HornetQMessageProducer.send(HornetQMessageProducer.java:199)
    at com.putnam.jboss.issues.Send1MMsgFails.run(Send1MMsgFails.java:143)
    at com.putnam.jboss.issues.Send1MMsgFails.main(Send1MMsgFails.java:186)
Caused by: HornetQException[errorCode=0 message=null]
    ... 7 more
An exception occurred

We can publish messages that have a smaller size or a larger size. Now, if I were to comment out the setting for
'min-large-message-size', the producer works just fine. So there seems to be some combination of the actual message size and the value of the 'min-large-message-size' setting that causes the JMS producer to fail. A value of:
<min-large-message-size>10485760</min-large-message-size>
will cause the producer to fail.

We are using 2.1.0.CR1. Before I open a jira, I would like for someone to be able to try the attached client and see if the problem is reproducible.

Attached:
1) Sample java client that can be used to reproduce the problem. Simply run:
java com.putnam.jboss.issues.Send1MMsgFails <file-path>

The argument that you pass in is the complete location of the data file. I am also uploading a zipped up version of the data file.

2) The data file.

Thanks
Aspi Engineer

Putnam Investments

Send1MMsgFails.java.zip 2.1 KB
1M.zip 12.2 KB

1. Re: Combination of Msg. size and 'min-large-message-size' causes exception

clebert.suconic Jul 20, 2010 4:50 PM (in response to aengineer)

When we write persistent messages to the journal, everything goes through a class that we call TimedBuffer, which its default size is 128K. And everything here will be written to the journal.

When writing to the journal, the max size will be the buffer-size:

http://hornetq.sourceforge.net/docs/hornetq-2.1.1.Final/user-manual/en/html/persistence.html#configuring.message.journal.journal-buffer-size

If you increase the buffer-size, the buffer size has to be less than the journal data file size:

http://hornetq.sourceforge.net/docs/hornetq-2.1.1.Final/user-manual/en/html/persistence.html#configuring.message.journal.journal-file-size

So, if you want to write 1MiB records on the journal, you need to increase buffer and journal file sizes accordingly.
Actions
2. Re: Combination of Msg. size and 'min-large-message-size' causes exception

aengineer Jul 20, 2010 5:10 PM (in response to clebert.suconic)
Hi,

I am having a hard time understanding the argument being presented. How can either the journal file size or the journal buffer size cause a message publisher to fail? It just should not happen. How many other combinations are out there that would cause either producers to consumers to fail?

I am using the default value of "journal-buffer-size" which according to the documentation is 490 KiB. And I am using the default value of "journal-file-size" which according to the documentation is 10Mib. Why would any of these settings cause the publisher to fail? And why is it that if I change my 'min-large-message-size' the publisher works just fine.

One would argue that no combination of any of these three settings should cause any issues. If the server start up, then the settings are good and messaging should work as expected.

I am uploading my configuration files just for analysis. But no combination of settings should cause publishers to fail. For my specific test case, one would think that HQ can save a message of size 1Mib to the journal using a journal file size of 10 Mib (the default) and a buffer size of 490 Kib (the default). The min-large-message-size of 10485760 (10 Mib) should not have had any impact because this message should not get identified as a large message.

Thanks
Aspi Engineer

hornetq-jms.xml 3.3 KB

hornetq-configuration.xml 4.2 KB
Actions
3. Re: Combination of Msg. size and 'min-large-message-size' causes exception

clebert.suconic Jul 20, 2010 9:15 PM (in response to aengineer)

You can instantiate a Connection Factory at the client side, and use a large MinMessagesize. On that case the system will throw an exception during the publish.

We could add a warning at the server's side. But those settings needs to be limited like this:

MinlargeMessageSize < BufferSize < JournalSize

Can you try 2.1.1.Final? 2.1.0.CR1 is quite old.
Actions
4. Re: Combination of Msg. size and 'min-large-message-size' causes exception

aengineer Jul 21, 2010 9:37 AM (in response to clebert.suconic)

MinlargeMessageSize < BufferSize < JournalSize

Assuming that I want messages greater than say 1 M or even 10 M classified as large messages, are you suggesting that the journal-buffer be greater than this? A buffer of size > 10 M that holds pending writes just seems too large.

Can you please confirm your suggestions.

Thanks
Aspi
Actions
5. Re: Combination of Msg. size and 'min-large-message-size' causes exception

clebert.suconic Jul 21, 2010 11:29 AM (in response to aengineer)

Think this way:

- We store LargeMessages outside of the journal. That's a caveat.
- We store a pointer to large messages on the journal.
- The journal and the buffer have fixed sizes.
- minLargeMessage size is used to decide what's a big message. If you say a big message is only big when size > 10 M, then you need to provide space at the buffer and journal to receive them.

"A buffer of size > 10 M that holds pending writes just seems too large"

Small messages are not split into packages ever. If you tell us 10 M is small, then.. by definition 10 M will be small to the system.
Actions
6. Re: Combination of Msg. size and 'min-large-message-size' causes exception

zont Aug 2, 2012 12:38 PM (in response to clebert.suconic)
Just to make sure.

The BufferSize determines:
The max. size of non-large-message that can be written on journal
The size of a chunk in case of a large-message
Then:
Is MinlargeMessageSize <= BufferSize <= JournalSize also correct configuration?
non-large-message is never broken into chunks, neither during transport (on hornetq-level of course) nor during writing on the journal

Are all my assumptions right?
Actions

Go to original post