OutOfMemory exception in JMS server

[kgreiner]

I've setup two applications (servers) running in the same JBoss 4.2.2.GA instance. One acts as the JMS client while the other is obviously the server. In my test, the client is simply generating a stream of object messages (about 50-100K per message) to a queue where that should be persisted by the JMS server to MySQL. A second client, that is not part of this test, would consume these messages at a later time.

I'm running Messaging 1.4.0.SP3 and I believe that I've fixed all of the installation problems related to port bindings, remoting, etc. as I am able to send and receive small numbers of messages between my two applications.

The first production-like test involving thousands of messages crashed the JMS server with an OutOfMemoryException after only a few hundred messages where sent to the queue. I increased the max heap size to 160M which, given that the server is simply persisting messages from one client, seems quite sufficient. That only delayed the error.

I then found that, after several aborted tests, I could not restart the JMS server due to an OutOfMemory while opening the messaging queue. Apparently, starting the queue resulted in all of the messages in the database (about 1,500) being loaded into memory. Setting the FullSize attribute to 100 in my queue's destination fixed this problem (I am certain that I could have used a larger value. I picked 100 to test whether this attribute would resolve my problem rather than to optimize future performance).

When I resumed testing, I found that my JMS server again failed with an OutOfMemory exception. I then tried setting the DownCacheSize attribute to 100 but that didn't help.

I've since run my JMS server under JProfiler and found what appears to be a problem. The server presently has 180 JBossObjectMessage instances in memory (I stopped the test when the server was nearly out of memory) which is substantially higher than the 100 that I expected when I lowered DownCacheSize.

Each JBossObjectMessage has two references to it. The first is a weak reference from the SimpleMessageStore . That reference is exactly what I hoped to find. The second reference is a hard reference via the messageRefs field of the org.jboss.messaging.core.impl.MessagingQueue class. That field contains a list of org.jboss.messaging.core.impl.message.SimpleMessageReference that each contain a hard reference, via their message fields, to the JBossObjectMessage instances.

Should SimpleMessageReference directly reference the message or a key into the SimpleMessageStore?

What should I try next to resolve this problem?

Should I be setting FullSize and/or DownCacheSize? Documentation for either is rather lacking right now.

[timfox]

"kgreiner" wrote:
I

Should SimpleMessageReference directly reference the message or a key into the SimpleMessageStore?

Yes, it should reference it directly

Should I be setting FullSize and/or DownCacheSize? Documentation for either is rather lacking right now.

Documentation is here:

http://labs.jboss.com/file-access/default/members/jbossmessaging/freezone/docs/userguide-1.4.0.SP3/html/configuration.html#conf.destination.queue.attributes.paging

[kgreiner]

Thanks.

I changed my FullSize to 100 and both PageSize and DownCacheSize to 20. The OutOfMemory exception did not reoccur in the latest test and the profile logs indicate that memory utilization was stable.

Is it reasonable to conclude that PageSize should always be smaller than FullSize?

[timfox]

FullSize is the maximum number of messages to hold in memory for that queue at any one time

PageSize is the number of messages to to load from disk in one go

So yes, pageSize must be less than downCacheSize

[pschuebl]

I had problems with OutOfMemory as well and the settings for Queues that receive potentially huge messages I use are

<attribute name="FullSize">20</attribute>
<attribute name="PageSize">5</attribute>
<attribute name="DownCacheSize">5</attribute>

This works but you have to be aware of that a QueueBrowser will only list messages that are in memory.


I would be great if there was an option to hold just the message header (the JMS-properties) in memory if the payload exceeds a configurable size.

This way "small" messages would still be in memory and performance would be good. The payload for bigger messages would be loaded from the database when needed keeping memory usage low.

What do you think?