[henriknyberg]

Hello,
I noticed that you have resolved the JBCACHE-93 issue by submitting two different JGroups configurations. The problem is that I still get a memory problem when I run the test using your JGroups files.

The fc-fast.xml seems to be the original one and does not resolve the problem.

The tcp.xml is a new file and I get the following error when I use it:

TCPPING.setProperties(): the following properties are not recognized:
-- listing properties --
num_ping_requests=5

I do not think this bug is resolved.

[belaban]

Okay, so let me tell you what I did

- I ran the test with fc-fast.xml for some time, and didn't see *any* change in memory after an initial ramp-up period

- Both tests were run under jboss-head (to be: 1.2.1 shortly) and JGroups 2.2.7 (part of jboss-head).

- The tcp.xml file was flawed, the parameter is part of JGroups 2.2.8, so simply remove it.

I'm running the test with fc-fast, and memory consumption for both putters is around 30MB heap

[henriknyberg]

Hello Bela,
thankyou for your prompt reply.

Regarding jar file versions, I am using the ones that are included in the JBossCache 1.2 distribution. If you are using properties from a future version of JGroups that may explain why we get different results. If you can make available your development jars to me I will gladly verify if they solve the problem in my testbench.

As mentioned earlier, I have verified that the "new" fc-fast.xml file does not solve the problem. If I run one stand-alone simple.bat putter it works steadily at around 13-14 MB heap. If I start the other putter after some time, both putters instantly start to consume memory. It takes me only a few minutes to consume 200 MB of JVM memory in each putter and to trigger a full gc.

I will test the corrected tcp file tomorrow to see if it makes any difference.

[belaban]

I'm using JBossCache 1.2.1 (you: 1.2). I *am* using 2.2.7 (like you), I'm not using 2.2.8.
So the diff is JBCache 1.2 versus 1.2.1. I'm starting the tests with -server -Xss16k.
I have slightly modified the test and put in on http://www.jgroups.org/jbcache-93.zip.
With credits of 2MB (in FC), I get a surge within the first minute to 48 MB for each of the 2 members, then they stay there. With 1MB of credits, that's ca 32MB.
I'm running this is 1.4.2_05.
As I'm writing this, the 2 members have been running for 8 minutes, and actually memory went back to 44MB, so I don't think I'll see an increase.

I have a 1-CPU laptop, next I'll try this in our Atlanta lab with 2 or 4-CPU boxes.

[henriknyberg]

First, I ran the test with your new tcp file less the unsupported property and it failed miserably. Not only did a putter take minutes to start, when it later got going it consumed memory in big chunks say 2-3 MB between gc updates.

Next, I looked at your modified testcase and added the Thread.sleep(5) to my own SimpleCacheTest.java after each put as you had done. In all fairness, that's probably a more realistic load than a continuous relentless put. I then ran the test with the fc-fast configuration. This time the memory consumption increases much more slowly and you can also see that the gc manages to free memory. However, the trend is still growing and after about 40 minutes both putters had consumed all JVM memory and a full gc was invoked.

Next, I took your version of SimpleCacheTest.java (with the receiver flag = false, and the System.out.println output commented away) and your NetCache.java and reran the test with the fc-fast configuration. I should now be running virtually the same testbench as you (apart from the console output and your JBosscache 1.2.1 jar). When running two simple.bat putters simultaneously I eat up all 200 MBs of JVM memory in less than 5 minutes in each JVM triggering a full gc.

Of course, after the full garbage collect the memory is cleaned up and in each putter the memory consumption is back to 25 MB after the gc. (And then starts to grow again). The problem is that in some app servers a full gc is something potentially dangerous since the full gc may totally shut down the JVM during the gc. Using e.g. Oracle 9ias the most likely result of a full gc is that the JVM is restarted which is something you don't want to happen in a heavily loaded enterprise application.

What you need to do is to make the gc collect your JGroups garbage objects much more often. This is the problem with Java, in C++ you simply write "delete pObject" when you want to free memory, In Java you can't explicitly handle memory that way. Or maybe you can, in which case would be the solution you need to clean your JGroups objects? Another solution may be to handle memory more carefully in your JGroups stack and avoid using the new operator as often.


Anyway, my conclusions at this point are the following:

* repeated puts on the same key is not a recommended practice when using JBossCache (version 1.2) in a cluster. Apart from this oddity the cache seems to be working ok.
* the fc-fast configuration seems to be most stable one and is preferred over UDP or TCP

Best regards,

Henrik Nyberg

[belaban]

I looked at your modified testcase and added the Thread.sleep(5) to my own SimpleCacheTest.java after each put as you had done.

In my testcase, sleep(5) is commented ! You're not running the same load if you sleep for 5ms between requests !

This time the memory consumption increases much more slowly and you can also see that the gc manages to free memory. However, the trend is still growing and after about 40 minutes both putters had consumed all JVM memory and a full gc was invoked.

I ran 2 senders at full speed on my WindowsXP laptop (1 CPU). I can only reiterate that they didn't run out of memory at all. I used both SUN's 1.4.2_05 and SUN's 1.5.0_01. The second test I ran for ca. 15 minutes, and memory even decreased slightly after a few minutes.

Oh, I re-read your prev statement, you're not saying you run out of memory, but that you're seeing full garbage collections ?
Then I suggest increasing the size of the young generation: JGroups creates a lot of smaller objects that can be garbage collected almost immediately after creation, e.g. Messages, Events etc. So the eden space should be relatively good-sized, so that the max number of bytes generated by Messages should be smaller than the eden space.
From the JGroups side, you can control the number of messages buffered though FC.max_credits/min_threshold and STABLE.max_bytes.

Next, I took your version of SimpleCacheTest.java (with the receiver flag = false, and the System.out.println output commented away) and your NetCache.java and reran the test with the fc-fast configuration. I should now be running virtually the same testbench as you (apart from the console output and your JBosscache 1.2.1 jar). When running two simple.bat putters simultaneously I eat up all 200 MBs of JVM memory in less than 5 minutes in each JVM triggering a full gc.

In my tests, I didn't run with receiver=true because that would have generated less load.