3 Replies Latest reply on Jun 26, 2008 6:31 AM by Manik Surtani

    How memory consumption behave?

    Jorge Morales Master

      Hi,
      We have a 2 node clustered application, with JBC 1.4.1-SP9. When we upgrade our app, we see this behaviour:

      1- We stop node 1, upgrade our app, and start node 1. (Node 2 keeps processing application traffic)
      2- It takes a while to start (~5 minutes) , since we have fetchInMemoryState to true.
      3- The server ends starting, and initially it has around 1GB old space memory used.
      4- 15 minutes later we do the same thing with node 2. We stop it, update it, and start it. It takes some time (~4 minutes), and finally loads with 1.4Gb.
      While this state transfer from node 1 to node 2 is happening, node 1 memory gets up to 1.9Gb.
      5- After a while, there is a Full GC, and memory lowers to 1.4 Gb in node1, and 1 Gb in node 2.

      My questions are:
      1- Why recent loaded apps have such a difference in memory?
      2- Why memory fills so much during state transfer?

      Another thing we have seen is that we have a region with a default cache loader of 24 hours, but we have in this region data only valid for a day, until 24:00 PM, so we have a scheduler job, that removes such a node, doing a cache.remove(FQN). When we run this job, memory lowers, but after 24 hours from start it lowers much more, so we think eviction policy is happening of internal cache data for the removed nodes. Can this be possible?

      We have set log levels to TRACE, but we have to wait another 24 hours to see this behaviour.

      Thanks,

        • 1. Re: How memory consumption behave?
          Manik Surtani Master

           

          "jorgemoralespou_2" wrote:
          Hi,
          My questions are:
          1- Why recent loaded apps have such a difference in memory?


          Is this running within a JMX environment, either JBoss AS' JmxConsole or Java 5's jconsole? If so, I'd recommend comparing the number of nodes in each cache rather than the memory used, to make sure these match up.

          "jorgemoralespou_2" wrote:

          2- Why memory fills so much during state transfer?


          Short-lived transient objects. State is transferred by creating NodeData objects, and streaming these. These are then unmarshalled on the recipient, data retrieved, real nodes created, and the NodeData objects left for gc to clear up.

          "jorgemoralespou_2" wrote:

          Another thing we have seen is that we have a region with a default cache loader of 24 hours, but we have in this region data only valid for a day, until 24:00 PM, so we have a scheduler job, that removes such a node, doing a cache.remove(FQN). When we run this job, memory lowers, but after 24 hours from start it lowers much more, so we think eviction policy is happening of internal cache data for the removed nodes. Can this be possible?


          Again this could be a gc thing. If you do a remove() on the region root so that everything in that region is removed, the node is removed from it's parent's children map such that the node is no longer accessible. So the node - and all it's children and children's children - are still around until gc clears it all away. I'm guessing this is what you are seeing. Again, if you have jconsole enabled, you can watch gc cycles and I'm guessing these drops in memory usage will correspond with gc kicking in.


          • 2. Re: How memory consumption behave?
            Jorge Morales Master

            Yes, we are running inside JBossAS.

            It seems that nodes/atributes in both servers are very different. Don't understand, as they should be very close to each other, as we have initial state transfer, and a whole day has passed by, so nodes not transferred should have been evicted.

            Machine-1:
            Number of nodes: 16609
            Number of attributes: 408866
            Old Memory: 1369195680 MB
            Time since reset: 189362 seconds

            Machine-2:
            Number of nodes: 28446
            Number of attributes: 450500
            Old memory: 1036104896 MB
            Time since reset: 84834 seconds

            Machine 1 has less nodes, more memory consummed. I have posted another thing that I have seen in the logs http://www.jboss.com/index.html?module=bb&op=viewtopic&t=138158.

            • 3. Re: How memory consumption behave?
              Manik Surtani Master

              yes, stuff in the eviction queue will contribute to memory consumption.

              The discrepancy in the number of nodes is interesting though - is this system backed by a cache loader? Could be that stuff is being evicted at different rates on both servers.