14 Replies Latest reply on Jan 24, 2006 10:18 AM by Bela Ban

    Management Attributes by Release

    Jerry Gauthier Apprentice

      The following is a non-exhaustive list of cache management attributes and statistics that have been suggested for exposure via the cache management interceptor mbeans. Per Ben's request, can we identify which ones should be targeted for 1.3 and 1.4? Please modify the list as appropriate.

      Attributes
      hits (already implemented for 1.3)
      misses (already implemented for 1.3)
      errors
      stores (already implemented for 1.3)
      lock failures
      transaction prepares, commits, rollbacks
      replication data
      bytes sent and received
      replication units sent and received
      minimum/maximum of messages sent
      initial state transfer bytes sent and received
      average size of send/receive
      summary of JGroups statistics
      number of nodes (already implemented for 1.3)
      number of attributes (already implemented for 1.3)
      evictions
      nodes evicted
      failure rate
      start time
      reset time
      memory consumption
      cache loader
      nodes stored
      nodes loaded
      passivation
      nodes activated
      nodes passivated

      Statistics
      hit/miss ratio = hits / (hits + misses) (already implemented for 1.3)
      read/write ratio = (hits + misses) / stores (already implemented for 1.3)
      elapsed time running
      elapsed time since reset
      average write time
      average read time
      accesses per unit time
      peak accesses per unit time
      % full???
      cache health (time running, cluster membership)

      Operations
      reset statistics (already implemented for 1.3)
      enable/disable statistics (already implemented for 1.3)
      print statistics (to console, file, combined with JGroups?)

      Notifications
      wrap TreeCacheListener into JMX notification events

        • 1. Re: Management Attributes by Release
          Bela Ban Master

          all of them into 1.3, none into 1.4 ? :-)

          • 2. Re: Management Attributes by Release
            Jerry Gauthier Apprentice

            OK. We just need to associate them with the appropriate interceptors and add the corresponding mbeans if they haven't been added yet. Seems like identifying the interceptors that require mbeans and then creating them would be a good place to start.

            • 3. Re: Management Attributes by Release
              Jerry Gauthier Apprentice

              One attribute request was for cache "elapsed time running." This is trivial to provide but the output can be presented in several ways.

              1) Just show elapsed time in seconds (or milliseconds) since the cache was placed in service. The Sun JVM mbean takes this approach as it shows how long the JVM has been in use in milliseconds. Milliseconds does seem like overkill here so maybe seconds would be better.

              2) Just show when the cache was started (e.g., show the corresponding Date object). This is the approach taken by the JBoss Server mbean.

              3) Show the elapsed time in a user friendly String (e.g., 2 days, 3 hours, 5 minutes and 10 seconds would be 2:03:05:10)

              Any recommendations?

              • 4. Re: Management Attributes by Release
                Ben Wang Master

                Jerry,

                If it is not too time consuming, I'd really like to see the following in your list into 1.3. :-)

                Attrs:
                * bytes send and recd
                * replication units send and recd
                * initial state transfer sent and rcvd (from last time since the node can leave and re-join later)

                Stats:
                * avg read/write time

                So for eviction, we will wait for EvictionInterceptor in 1.4 to implement the attrs and stats. I will create this shortly.

                For Cache loader and passivation, we already have the interceptor. Can you work with the corresponding interceptor owner (like Manik and Hany for passivation) to come up with some attrs and stats?

                Great work!

                • 6. Re: Management Attributes by Release
                  Ben Wang Master

                  The other thing about attrs and stats is that they are related wrt to cache stop and start service (or leave and join a group). For example, when a user do:

                  cache.stop() and then cache.start() again, should we re-set the stats?

                  Same question goes for leave and join a group.

                  What do others think?

                  -Ben

                  • 7. Re: Management Attributes by Release
                    Jerry Gauthier Apprentice

                    Ben - what are "bytes sent and received?" What are we measuring here? Same question for "replication units sent and received."

                    Avg read/write time is now implemented.

                    Not sure whether we should reset stats when the cache is stopped and restarted. I suppose it could be handled easily by having the statistics interceptor register a listener with the cache to catch the stop and start events. Since stopping and starting the cache seems to leave the contents intact, I don't think we need to reset the statistics as they're still valid. ElaspedTimeRunning would be somewhat incorrect since the cache was running continuously; this statistic isn't normally reset during a resetStatistics operation. My inclination would be to leave the statistics as is for now.

                    It was useful for me to look at stop and start as I noted the dynamic mbean registration process fails with a stack trace on a restart since the mbeans have already been registered. Will fix asap.


                    • 8. Re: Management Attributes by Release
                      Ben Wang Master

                      1. bytes sent and received. This is more of an attribute from JGroups, actually, to measure the total bytes sent and recevied, either during this call stack (or over a period of time).

                      2. replication units sent and recvd. This measures how many sent/recvd thru JGroups method call in this call stack, e.g., we can have mutiple remove/put calls.

                      3. OK. Somehow we need to have an option to say that stop/restart should flush out the cache contents (and stats). But this is not high priority.

                      THanks,

                      -Ben

                      • 9. Re: Management Attributes by Release
                        Jerry Gauthier Apprentice

                        Some followup questions -

                        1. TreeCache doesn't currently expose its JChannel reference. Consequently in order to obtain bytes sent and received, it seems like I would need to do one of the following.

                        a) Modify TreeCache to expose the JChannel reference so that the statistics interceptor can work with it.

                        b) Modify TreeCache to expose the underlying statistics via its mbean. Then the statistics interceptor can make the information available by querying TreeCache (as it does now to display number of nodes and attributes)

                        2. Regardless of which option we choose for the previous issue, it seems likely that when JGroups channels become sharable, the bytes sent and received by a channel will no longer be associated with a single cluster such as a replicated TreeCache instance. Is this correct?


                        • 10. Re: Management Attributes by Release
                          Brian Stansberry Master

                          Can TreeCacheMarshaller help here? Right now we only use this class if they want to register special classloaders, but it would be trivial to modify things so we always use it and the cache's "useMarshalling" switch gets passed to it to control how serialization/deserialization is done.

                          This way all objectTo/FromByteBuffer calls could update statistics in this class (number of calls and number of bytes). Don't know if this would match what you're looking for.

                          • 11. Re: Management Attributes by Release
                            Bela Ban Master

                            Isn't there a getChannel() method on TreeCache ?
                            I suggest though to expose the channel via the Channel MBean that is aleardy exposed, so let's avoid duplication.

                            • 12. Re: Management Attributes by Release
                              Ben Wang Master

                              Jerry, you question of future channel multiplexing will pollute the bytes sent/recd is right. We need to have a separate counter from each sender. So in a sense we will need to have a way to intercept this.

                              • 13. Re: Management Attributes by Release
                                Jerry Gauthier Apprentice

                                No - there is no getChannel() method on TreeCache.

                                I'm not clear on how to get the channel mbean associated with the TreeCache instance. I know how to get a channel mbean for a standalone JGroups configuration that's defined in an mbean such as is doen with DemoChannel. But how does this relate to a JGroups configuration embedded in a TreeCache mbean?

                                Is is possible to define the standalone JGroups configuration and then refer to this configuration within the TreeCache configuration so that the TreeCache configuration doesn't prescribe its own channel? Is this where we're going with channel multiplexing?

                                • 14. Re: Management Attributes by Release
                                  Bela Ban Master

                                  Yes, I think it might be a better idea to simply not include bytes sent/received, and delegate it to JGroups