6 Replies Latest reply on May 1, 2008 3:48 PM by brian.stansberry

    Buddy Replication: Memory usage with Tomcat Session Replicat


      Hi - we have a test cluster with three nodes a1, a2, a3. We have configured Tomcat sessions replication with buddyreplication.NextMemberBuddyLocator and one buddy per node.

      Buddy Replication Question

      I assume that with this setup we would have something like a2 replicates a1's sessions, a3 replicates a2's sessions and a1 replicates a3's sessions.

      However what I see (using memory profiling) is that the the memory usage pattern on app1 is virtually the same as non clustered (i.e. a slow drain util major garbage collection). a2 shows a much more rapid depletion of memory and a3 shows depletion at at least a 3x rate of the depletion as of a2.

      What I was expecting was to see the same memory use on each node and am confused about why I do not. THis seems to imply some basic misunderstanding of how the buddy replication works. Why aren't the memory usage patterns between all three nodes the same?

      Session Replication Question

      I am assuming that all replicas of a session are destroyed when the user logs off the node that the source session was created on. Is that the case?

      The issue we are seeing is that under jmx-console service=TomcatClusteringCache property 'NumberOfAttributes' we see that value always increasing during our load test whereas I was expecting that it would reach a steady state when the number of new logins (session create rate) is matched by the number of users logging out (killing their session). If seems if this number always increases then out of memory is inevitable.








        • 1. Re: Buddy Replication: Memory usage with Tomcat Session Repl
          brian.stansberry

          Re: the memory usage pattern with buddy replication, I too expect the memory usage to be consistent across the 3 nodes if you are applying load evenly to them. When I've tested this, that's what I've seen. So, not sure why you're seeing something different.

          Re: the effect of a logout, if you call session.invalidate() the session is removed cluster-wide. (If you just wait for the session to timeout, each node in the cluster independently does that.) I just looked at the impl of the TreeCache.getNumberOfAttributes() method and don't see any reason why it would overcount. So I'm concerned you've identified some bug here. What AS release are you using?

          • 2. Re: Buddy Replication: Memory usage with Tomcat Session Repl

            Brian - thanks for your input..

            Re memory: I researched further and think this is the reason.. We have 3 app servers. The start sequence is a1, a2, a3. When a2 starts a1 (already started) and a2 form a buddy pair (they appear to both backup each other). Then when a3 starts it looks like a2 also becomes the buddy to backup a3. So a3 backups up two nodes and a3 ends up backing no nodes. I think that is why the memory patterns vary between the nodes (with a2 exhibiting the fastest depletion of memory)

            RE session.invalidate(): I am still seeing unbounded growth of the number of replicated sessions and eventually we get out of memory. This is JBOSS AS 4.0.5. Any further input on this one would be most valuable since its now a show stopper for us at this point..

            Thanks

            • 3. Re: Buddy Replication: Memory usage with Tomcat Session Repl

              BTW - I assume there is no reason (and one should not) configure an eviction policy into the tc5-cluster.sar/jboss-service.xml.

              This understanding is based on your comment

              the effect of a logout, if you call session.invalidate() the session is removed cluster-wide. (If you just wait for the session to timeout, each node in the cluster independently does that.)


              Is this understanding correct?



              • 4. Re: Buddy Replication: Memory usage with Tomcat Session Repl
                brian.stansberry

                Node a2 becoming the buddy of both a1 and a3 sounds broken; it's not meant to be that way.

                There have been a lot of improvements in JBoss Cache related to buddy replication since AS 4.0.5 came out. Can you try replacing the server/all/lib/jboss-cache.jar with the jboss-cache-jdk50.jar that comes with the JBC 1.4.1.SP9 release? For that to work you'll need to replace jgroups.jar, either with the one that comes with the JBC 1.4.1.SP9 download. or even better with the one from the JGroups 2.4.1.SP4 release.

                That may help your session.invalidate() issue as well. I've never heard of a problem like that before; will investigate more.

                Re: setting up eviction, normally I say don't do that, but it may be a valid workaround for now. If you do it, set it up for the _default_ region (i.e. the whole cache) so it also covers the internal buddy replication backup areas. Use LRUPolicy, and configure maxNodes=0 (disabling eviction based on # of nodes). The timeToLiveSeconds value must be greater than the session expiration timeout. I'd say make it a couple minutes greater -- idea is to make sure that the normal JBossWeb session cleanup process gets a chance to flush things out first, with the JBC eviction only cleaning out stuff that gets left behind.

                • 5. Re: Buddy Replication: Memory usage with Tomcat Session Repl

                  Brian - ok, thanks for the input

                  I will try running the new jgroups and jboss-cache still using AS 4.0.5. So you are saying these will still be compatible - right? Is there a matrix somewhere that defines which version of which components can go together. e.g. what versions of jgroups or jboss-cache can we run with AS 4.0.5.

                  Another option maybe just to wait till AS 5.0 is ready. I am assuming that there have been many clustering improvements in that release..

                  After quite a bit of work I also finally figured out the session leak. It was actually being caused by our load test setup. So right now that issue is resolved.

                  We are now seeing a 3x cpu hit using session replication. But I will post that as a new topic in this forum..



                  • 6. Re: Buddy Replication: Memory usage with Tomcat Session Repl
                    brian.stansberry

                    http://www.jboss.org/jbosscache/compatibility/index.html

                    Our support guys keep that fairly up-to-date.

                    AS 5 is going to be a while (at least a couple months). You can look at AS 4.2.2 as well, although again I recommend moving the JBoss Cache to 1.4.1.SP9. That's a small change from the stock 4.2.2; just a later bug fix release.