messaging flow control related question

[raghum]

Hi

I am using a JMS topic for communication between a producer and a consumer. The producer generates large quantities of data (many large streaming messages).

I would expect that in case the consumer is slow, JBOSS would throttle the producer. But I am noticing that the the producer keeps producing the messages till the JBoss messaging server runs out of heap space (OutOfMemory Exception).

Is it possible to control this behavior - to have flow control between the producer and consumer? If so, how?

I am using
* JBoss [Zion] 4.0.4.GA (build: CVSTag=JBoss_4_0_4_GA date=200605151000)
* JBoss Messaging 1.0.1.CR2 server
on Linux x86 environment with Java1.5.

I am using this topic - 'testTopic' preconfigured in the system. Its configuration as printed in the output while server starts up -
INFO [Topic] Topic[/topic/testTopic] started, fullSize=75000, pageSize=2000, downCacheSize=2000.

Thanks
Raghu

[timfox]

There is no producer throttling at present.

However JBoss Messaging can cope with very large queues, by paging to disk. See "Destination Paging Parameters" for more details.

[raghum]

Thanks for pointing out the tunables.

Does it mean that if I have messages of fixed size 'm', I should provision for at least (FullSize+downCacheSize+pageSize)*m amount of memory in the JVM to prevent any OutOfMemory exceptions?

Also, even after (FullSize+downCacheSize) number of outstanding messages on the JBoss server, I am not seeing any disk activity. And the producer keeps writing messages and the JVM heap size keeps increasing. (The consumer is forcefully halted to simulate the behavior).

Any ideas on what is happening? I am trying to prevent getting OutOfMemory errors in the runtime and I have set these tunables to small values.

Thanks
Raghu

[timfox]

Some more questions:

How many subscriptions do you have?

How many queues/topics?

Size of messages?

Type of message (persistent/non persistent)?

What values have you set on your queuss/topics? (please post config)

How much memory have you allocated to the VM?

How do you know there is no disk activity?

[timfox]

Also, can you tell me what database you are using?

[raghum]

There is one connection, one topic and one session shared by three threads. Session is created with AUTO_ACKNOWLEDE flag. And there are no queues.

The threads -
1. Sender thread has a producer on the topic.
2. Relayer thread has a producer and a consumer on the topic.
3. Receiver thread has a consumer on the topic.

So, there are two subscribers to the topic.

The communication setup is such that Relayer will receive the message sent by Sender and send it to Receiver thread. The relayer does this by reposting the message on the same topi but with different selctor.

The Sender repeatedly send messages of size 100000 bytes and the message type is StreamMessage. Message contents slightly vary between messages. The Relayer and the Receiver are halted delibrately to create this scenario.

There is a correction since the previous post. The messages *are* flushed to the database and disk io is happening (the file localDB.script keeps increasing in size). I hadn't noticed it earlier as it was on NFS. But the problem remains - the JVM heap size keeps increasing even after database flushes kick in till it runs out of memory.

* The messages are NON_PERSISTANT
* The values on the topic -
fullSize=3000, pageSize=2000, downCacheSize=2000
* VM has 1024MB
* Hypersonic is being used

So, as I asked earlier, how could I set the JBoss parameters to avoid OutofMemory error?

Thanks,
Raghu

[timfox]

Don't use hypersonic, it not intended for any serious use.

This is mentioned in the getting started guide.

The reason you are running out of memory, and see little disk activity is that hypersonic stores it's tables in memory (in the same process).

We should really prevent Jboss messaging from working with HSQL at all since this question comes up again and again.

Also see

http://wiki.jboss.org/wiki/Wiki.jsp?page=ConfigJBossMQDB

[raghum]

I am still having the same problem - the JMS server running out of memory when configured with the mySQL database (4.1.20) too.

The behaviour of the program in brief:

* the program is time limited - terminates after a time period. So, when the program terminates, there are still lots of unconsumed messages in the topic. But the messages are marked as NON_PERSISTENT while sending; Also before terminating the program, all the connections, subscriptions (producer, consumer) and sessions are closed. Messages can be still be found in the database after the program has terminated.

* If the same program is run twice over, the JMS server complains of out-of-memory errors. The JVM heap size is 1GB but it should not matter as whatever be the heap size, the problem is bound to occur later.

Any work arounds?

Thanks
Raghu

[timfox]

It doesn't matter if the messages are non persistent.

The messages will last as long as the topic consumer.

If the topic consumer lasts 100 years, the messages will last 100 years.

Regarding the out of memory error, please casn youi replicate this in a test case and send to me and I will investigate.

[timfox]

Raghu-

Thanks for your test program.

I have been experimenting with it, and here are my findings:

If CONSUMER_SLEEP_TIME and RELAYER_SLEEP_TIME are set to a small value e.g. 1000, then the entire test runs through successfully - this is because the messages are being relayed and consumed at approximately the same rate as they are being sent, so they do not build up in the topic subscriptions.

If CONSUMER_SLEEP_TIME and RELAYER_SLEEP_TIME are set to a large value e.g. 150000, then a great many messages get sent before they start being consumed. Since each message you are sending is about 100K in size, and (on my machine) about 3000 get sent before they start being consumed, this is too much to be stored in RAM at once in my JBoss4.0.4 installation since I am using -Xmx100M.

I therefore configured the paging parameters for the topic so a maximum of 200 messages would be stored in memory at once and the rest paged to storage:

 <mbean code="org.jboss.jms.server.destination.Topic"
 name="jboss.messaging.performance:service=Topic,name=raguTopic"
 xmbean-dd="xmdesc/Topic-xmbean.xml">
 <depends optional-attribute-name="ServerPeer">jboss.messaging:service=ServerPeer</depends>
 <attribute name="FullSize">200</attribute>
 <attribute name="PageSize">10</attribute>
 <attribute name="DownCacheSize">10</attribute>
 </mbean>

With this config the test also completes successfully - albeit more slowly since it has the added overhead of reading and writing from the database. I am using MySQL 5.

Finally I tried increasing the amount of memory for the server to 1.2GB (-XMx1200M), this should allow all 10000 messages to be stored in memory without having to page to the db.

For this setup I also changed fullSize to 10000, so if more than 10000 messages arrive then we won't get an OutOfMemory error.

Again this test ran through fine.

After the tests have completed I am not seeing any messages or message references in the database, which is correct bahviour.

I can repeat the tests multiple times without seeing any significant change in memory on ther server.

I am using the HEAD codebase, which is what will be in RC4 when it is released in a few days. I have made many changes in compared to RC3 but not sure if they are responsible for seeing the difference in behaviour that you are seeing.

Probably your best bet is to upgrade to RC4 when it comes out and see if you still have your problems.

[raghum]

Tim -

Thanks for your reply and the effort you have put in analyzing the problem. But I am sorry if I have failed to communicate the real intent of the test program.

The large values for CONSUMER_SLEEP_TIME and RELAYER_SLEEP_TIME was intentionally put. It is simulating a situation where the system could go down with many messages unconsumed in a topic. So, the test case needs to stop with many messages unconsumed in the topic. And this test case is run multiple times without restarting the JMS server.

To give you an idea of the usage - JMS is used by a communication layer to provide a large scale message passing infrastructure (similar to MPI) between the distributed processes. So, during the application development many applications would terminate abnormally (but we make sure that all the connections, topic producers, topic consumers, topic sessions are closed before the application terminates) leaving messages unconsumed on the topic. But we no longer care for these messages as it does not any meaning outside of the closed connection/session. So, we want to make sure that the unconsumed messages are flushed out (out of memory at least and the database). Multiple runs of the test case essentially tries to test this behavior - if the messages from a previous run is still in JMS server memory.

So, I would rephrase my query - in a scenario where the applications close (connections, sessions, topic producers, topic consumers) arbitrarily and we need the unconsumed messages on the topic to be flushed out automatically. Please note that When the application closes, it terminates the JVM the application was instantiated in, too. So, apart from setting the messages to be NON_PERSISTENT and closing all the JMS objects and the JVM, is there anything that needs to be done to flush all the messages?

Thanks
Raghu

[timfox]

If you read my last reply, you will notice that I said I can successfully repeat your test case with both large and small values for CONSUMER_SLEEP_TIME and RELAYER_SLEEP_TIME multiple times, without seeing memory increase and without seeing any messages left in the database.

So, as far as I am concerned I have not been able to replicate your problem, and everything is working ok.

As I say I am using the latest code, so that may explain the difference in behaviour.

[timfox]

Raghu-

Anyway, I would strongly recommend upgrading to RC4 when it comes out and see if it solves your problems.

Thanks again for your patience.