How memory consumption behave?

[jorgemoralespou_2]

Hi,
We have a 2 node clustered application, with JBC 1.4.1-SP9. When we upgrade our app, we see this behaviour:

1- We stop node 1, upgrade our app, and start node 1. (Node 2 keeps processing application traffic)
2- It takes a while to start (~5 minutes) , since we have fetchInMemoryState to true.
3- The server ends starting, and initially it has around 1GB old space memory used.
4- 15 minutes later we do the same thing with node 2. We stop it, update it, and start it. It takes some time (~4 minutes), and finally loads with 1.4Gb.
While this state transfer from node 1 to node 2 is happening, node 1 memory gets up to 1.9Gb.
5- After a while, there is a Full GC, and memory lowers to 1.4 Gb in node1, and 1 Gb in node 2.

My questions are:
1- Why recent loaded apps have such a difference in memory?
2- Why memory fills so much during state transfer?

Another thing we have seen is that we have a region with a default cache loader of 24 hours, but we have in this region data only valid for a day, until 24:00 PM, so we have a scheduler job, that removes such a node, doing a cache.remove(FQN). When we run this job, memory lowers, but after 24 hours from start it lowers much more, so we think eviction policy is happening of internal cache data for the removed nodes. Can this be possible?

We have set log levels to TRACE, but we have to wait another 24 hours to see this behaviour.

Thanks,

[manik]

"jorgemoralespou_2" wrote:
Hi,
My questions are:
1- Why recent loaded apps have such a difference in memory?

Is this running within a JMX environment, either JBoss AS' JmxConsole or Java 5's jconsole? If so, I'd recommend comparing the number of nodes in each cache rather than the memory used, to make sure these match up.

"jorgemoralespou_2" wrote:

2- Why memory fills so much during state transfer?

Short-lived transient objects. State is transferred by creating NodeData objects, and streaming these. These are then unmarshalled on the recipient, data retrieved, real nodes created, and the NodeData objects left for gc to clear up.

"jorgemoralespou_2" wrote:

Another thing we have seen is that we have a region with a default cache loader of 24 hours, but we have in this region data only valid for a day, until 24:00 PM, so we have a scheduler job, that removes such a node, doing a cache.remove(FQN). When we run this job, memory lowers, but after 24 hours from start it lowers much more, so we think eviction policy is happening of internal cache data for the removed nodes. Can this be possible?

Again this could be a gc thing. If you do a remove() on the region root so that everything in that region is removed, the node is removed from it's parent's children map such that the node is no longer accessible. So the node - and all it's children and children's children - are still around until gc clears it all away. I'm guessing this is what you are seeing. Again, if you have jconsole enabled, you can watch gc cycles and I'm guessing these drops in memory usage will correspond with gc kicking in.

[jorgemoralespou_2]

Yes, we are running inside JBossAS.

It seems that nodes/atributes in both servers are very different. Don't understand, as they should be very close to each other, as we have initial state transfer, and a whole day has passed by, so nodes not transferred should have been evicted.

Machine-1:
Number of nodes: 16609
Number of attributes: 408866
Old Memory: 1369195680 MB
Time since reset: 189362 seconds

Machine-2:
Number of nodes: 28446
Number of attributes: 450500
Old memory: 1036104896 MB
Time since reset: 84834 seconds

Machine 1 has less nodes, more memory consummed. I have posted another thing that I have seen in the logs http://www.jboss.com/index.html?module=bb&op=viewtopic&t=138158.

[manik]

yes, stuff in the eviction queue will contribute to memory consumption.

The discrepancy in the number of nodes is interesting though - is this system backed by a cache loader? Could be that stuff is being evicted at different rates on both servers.