1 2 3 Previous Next 33 Replies Latest reply on Dec 28, 2005 10:00 PM by ben.wang

    Exposing JBossCache Management Attributes

      Reference
      http://jira.jboss.com/jira/browse/JBCACHE-56

      Background
      The intent of this issue is to provide common cache characteristics such as hit/miss ratio, read/write ratio, number of cached nodes and attributes, etc. The issue's comments provide an extensive list of cache characteristics that are considered desirable to expose. It's also proposed that this implementation expose cache notifications via JMX in addition to the current TreeCacheListener implementation.

      Design Overview
      The initial design focus is on how to incorporate this capability into JBossCache. Once the design work and initial implementation is complete, it should be possible to review the list of attributes to be monitored and incorporate them over time, expanding or contracting the list as appropriate. Initially, the implementation will focus on simple characteristics such as number of nodes, number of attributes, number of hits, number of misses, and number of puts.

      Design Alternatives
      The simplest approach would be to add code to TreeCache to capture the necessary statistics and then expose them through the TreeCacheMBean. This is already in place for ?number of nodes? and ?number of attributes.? The obvious disadvantages here are that TreeCache.java is already quite large and adding numerous new methods to the TreeCache MBean will further clutter it and make it difficult for users to readily locate relevant statistics.

      An alternative approach here is to provide a new MBean named TreeCacheManagement MBean that will be used to expose the management statistics. This MBean will be similar to the existing TreeCacheView MBean; it will be an optional MBean that's dependent on and associated with a configured TreeCache service. The TreeCacheManagement MBean will have a reference to its associated TreeCache instance so it can easily access statistics that are exposed by the TreeCache such as Number of Nodes and Number of Attributes.

      Providing a new MBean resolves the second disadvantage noted above; the current TreeCache MBean won't be cluttered with new methods. However the first disadvantage remains as the new logic to capture cache statistics will still need to be located in the TreeCache implementation. It will also be necessary to add public methods to the TreeCache to make the statistics available to the new MBean. This could probably be minimized by providing the statistics via a single method (e.g., by returning a collection or array) rather than using individual methods per statistic.

      In order to minimize changes to the TreeCache class, it's possible to introduce a new interceptor named CacheMgmtInterceptor. This interceptor can be used to capture various statistics as it intercepts TreeCache methods. For example, any invocation of TreeCache.getKeyValueMethodLocal() would result in a hit or a miss, depending on whether the method returned a null value. In this case, it's transparent to the TreeCache implementation that hits and misses are being counted. The interceptor accumulates the statistics and provides methods for callers to access them. The new TreeCacheManagement MBean can obtain a reference to the interceptor via the interceptor chain exposed by the TreeCache MBean so the statistics will be accessible to the new MBean.

      There are at least two potential problems with the interceptor approach. First, some statistics may not be accessible via interceptors. Fir example, if a desired statistic isn't the input or output of a method, interception probably won't be able to capture it. In this case, it will probably be necessary to introduce logic into TreeCache to expose the statistic It's not clear whether this applies to any of the statistics currently of interest; this will be determined as they're implemented in turn.

      A second potential problem relates to modifying logic flow in TreeCache. For example, all lookups ultimately invoke the method identified by TreeCache.getKeyValueMethodLocal(). If this situation changes in the future, it may not be obvious to the developer modifying the code that this condition was a prerequisite for logic in the interceptor that ascertained hits and misses. If the logic were in-line in TreeCache, it would be much more obvious to someone modifying the implementation.

      Users should have the ability to disable the statistics gathering implementation if they wish to do so. Since deployment of the TreeCacheManagement MBean is optional, they can easily prevent the MBean from being deployed. However the interceptor still needs to be excluded from the TreeCache interceptor chain. The interceptor chain is established when the TreeCache service is created; at this point the TreeCacheManagement MBean hasn't been created as it's dependent on the TreeCache MBean. Consequently it's not possible for the interceptor factory to detect the presence or absence of the TreeCacheManagement MBean and act accordingly. One possibility here would be for the TreeCacheManagement MBean to dynamically add the interceptor but it appears that the current intention of the TreeCache interceptor stack is to not support dynamic changes. A simpler alternative would be to add a boolean attribute to the TreeCache MBean configuration specifying whether statistics should be maintained. The interceptor factory can then ascertain whether to incorporate the management interceptor or not.

      Design Proposal
      My proposal is to use a new TreeCacheManagment MBean to expose statistics, beginning with simple ones such as hits and misses. I think it makes sense to introduce a new interceptor rather than add the logic directly to TreeCache. I would add a new configuration attribute to TreeCache so that the management interceptor can be enabled or disabled appropriately.

      Thoughts, alternatives?

      I'll follow up with further discussion on individual statistics to ensure that we have agreement on how they should be measured.

      Jerry

        • 1. Re: Exposing JBossCache Management Attributes

          Good proposal! Couple comments here.

          1. Since cache have different kind of aspects like replication, eviction, and cache loader, will it make sense if we have a hierarchical MBean structure? For example, can we define an MBean interface for eviction policy such that different eviction policy provider can implement it (standard one plus their own management attributes extension) and hook into the main TreeCacheMbean automatically?

          Note that we will switch to use EvictionInterceptor pretty soon as well. So all "aspect" will come from the interceptor stack.

          2. I will vote for the boolean flag to enable/disable stats. We can always revisit this issue when we refactor in 2.0.

          3. Yes, probably not all attributes can be captures via interceptors. But I suspect they are all more of static in nautre?

          • 2. Re: Exposing JBossCache Management Attributes
            belaban

            Jerry, can we capture this also in a document in JBossCache/docs/design ?

            • 3. Re: Exposing JBossCache Management Attributes
              belaban

              Okay, if you have to provide methods to expose stats in TreeCache itself, then I don't see the need for a separate management MBean. In 2.0, we will get rid of the TreeCache class anyway, and simply provide a Cache interface.
              My preference would be to simply expose the attrs/ops in TreeCache/TreeCacheMBean, and then think about a better solution once 2.0 comes around. Think about the use case where a user wants to use JBossCache in JDK5, and use jconsole to browse the instance. In this case, he has to create another management MBean, which is unneeded complexity.

              • 4. Re: Exposing JBossCache Management Attributes

                Initial design document located in JBossCache/docs/design/ManagementAttributes.txt

                • 5. Re: Exposing JBossCache Management Attributes

                  Providing a Cache interface will allow for the implementation to be hidden and possibly decomposed into various classes. But if all the management attribute methods are added to the interface, it will remain cluttered. It will also require that all cache implementations expose their management attributes, or at least provide stub methods. Maybe a second interface covering management attributes would be preferable. It would then be up to providers whether to support the interface and how to do it under the covers.

                  Perhaps I'm confusing the 2.0 Cache interface with the corresponding MBean. Is the intent that the 2.0 TreeCache MBean expose the interface or all of the public methods, as it now does?

                  I agree that adding the logic to TreeCache and adding the management attribute methods to TreeCache MBean is the simplest approach and it can be reworked in 2.0. My concern is that piling more methods onto the existing mbean makes it difficult for anyone to focus on the important management attributes when working with the mbean in the console. It's certainly feasible to add the methods to TreeCache (i.e., don't use an interceptor) but only expose them via the new mbean.

                  Your observation about a user having to configure another mbean is correct. So a key design issue here is whether it's preferable to require the use of a new mbean or not. The new mbean would require configuration on the part of the user, while it would offer better segregation of managment attributes.

                  If we don't use interceptors to capture statistics, is it still worthwhile to provide a boolean flag on the TreeCache MBean to enable/disable statistics gathering?



                  "bela@jboss.com" wrote:
                  Okay, if you have to provide methods to expose stats in TreeCache itself, then I don't see the need for a separate management MBean. In 2.0, we will get rid of the TreeCache class anyway, and simply provide a Cache interface.
                  My preference would be to simply expose the attrs/ops in TreeCache/TreeCacheMBean, and then think about a better solution once 2.0 comes around. Think about the use case where a user wants to use JBossCache in JDK5, and use jconsole to browse the instance. In this case, he has to create another management MBean, which is unneeded complexity.



                  • 6. Re: Exposing JBossCache Management Attributes
                    belaban

                    I agree with your point that either way we end up cluttering up the TreeCache class. How about a method call

                    ManagementMBean getStatistics(input ?) ? The management info would be kept in an interceptor, and be exposed via ManagementMBean.

                    • 7. Re: Exposing JBossCache Management Attributes

                      It's not necessary to add a getStatistics method to a new management mbean. Did you mean the existing TreeCacheMBean? If we place the statistics gathering logic in the existing TreeCache code, then yes - I think it would be desirable to expose the statistics via a single method which returned some type of collection. The new management mbean would then expose the statistics via individual methods (e.g., getHits, getMisses).

                      If we use an interceptor, it may not be necessary to add a getStatistics method to TreeCacheMBean. This would be determined by whether the desired statistics could be obtained solely via interceptor(s).

                      So the two key initial design issues are as follows.

                      1) Expose the statistics via the existing TreeCacheMBean or via a new management mbean?

                      2) Add an interceptor to capture the statistics or simply add the logic to TreeCache for now? As noted, an interceptor may not be able to handle everything so some logic might still need to be added to TreeCache.

                      • 8. Re: Exposing JBossCache Management Attributes

                        We need to define various cache management statistics in the context of JBossCache. Here are my current proposals for some basic measurements.

                        Store Operations
                        Stores are used to provide the read/write ratio for the cache. The read/write ratio is (hits + misses) / stores.

                        Putting an attribute is a store operation, reputting an attribute is a store operation, putting a map can be interpreted as one store operation per attribute in the map. What about putting a node? Under this proposal, only attributes are counted as store operations.

                        Hits and Misses
                        Hits and misses are used to provide the hit/miss ratio for the cache. This ratio is calculated as hits / (hits + misses).

                        A hit is a get attribute operation that ultimately results in an object being returned to the client. So a get operation that fails over to a custom cache loader is a hit if the cache loader returns the requested object. A miss is any get attribute operation that fails to return the requested object.

                        What about getting nodes rather than attributes? Under this proposal, only getting attributes are counted as hits or misses. This may be misleading if the user gets the data map for a node and then reads it. Any thoughts on how this should be treated?

                        • 9. Re: Exposing JBossCache Management Attributes

                          I'm still thinking about your question; not sure that I understand the ramifications yet. Would the user see a single mbean incorporating all "aspects" or would the user be presented with multiple mbeans per aspect (e.g., basic management mbean, eviction mbean , replication mbean)?

                          If the former, presumably we would define a standard set of management attributes per aspect (e.g., eviction attributes) and the eviction policy provider could augment this for their own use. We wouldn't be able to display the non-standard attributes via the mbean unless we used dynamic mbeans (I think).

                          A single management mbean will work fine with multiple interceptors as long as it can recognize the interceptor (e.g., the "eviction" interceptor) and it knows what methods to invoke on the interceptor to obtain the desired statistics.



                          "ben.wang@jboss.com" wrote:


                          1. Since cache have different kind of aspects like replication, eviction, and cache loader, will it make sense if we have a hierarchical MBean structure? For example, can we define an MBean interface for eviction policy such that different eviction policy provider can implement it (standard one plus their own management attributes extension) and hook into the main TreeCacheMbean automatically?



                          • 10. Re: Exposing JBossCache Management Attributes

                            To comment on the hit/miss ration first. To collect the stats between get(fqn) and get(fqn, key) is a tricky one. I'd just get(fqn, key), but some may want to get the whole map as well (thru the DataNode). Maybe we should distinguish it down the road but just count get(fqn, key) for now.

                            Thoughts?

                            • 11. Re: Exposing JBossCache Management Attributes

                              Honestly, I am not sure what is the best way either. But I think here are the basic requirements for pluggable components:

                              1. We define a standard MBean attributes across of all components (e.g., eviction has some basic attributes that are shared such as items evicted).

                              2. Individual implementor (provider) should also has a way to expose additional attributes (such as item ages out or items capacity exceeded).

                              To achieve, we will need to call out to an registered MBean. E.g., LRUPolicy will have a LRUPolicyMBean regsitered.

                              • 12. Re: Exposing JBossCache Management Attributes
                                manik

                                The problem with multiple MBeans is that where would you stop . Not all JMX attributes are informative, several are configuration or control features. Once we have interceptors that configure themselves (e.g., ReplicationInterceptor maintaining knowledge of all clustering params or the CacheLoaderInterceptor maintaining cache loader configs) we'd want a separate MBean per interceptor. And then some interceptors may have specific classes attached to them e.g., CacheLoaderInterceptor having one or more CacheLoaders. Each CacheLoader then ought to come with a CacheLoaderMBean so that it can be configured via JMX. Or in Ben's example above, an EvictionPolicy with an EvictionPolicyMBean.

                                Where would we draw the line?

                                I agree that it is still better than clobbing everything together in the TreeCacheMBean, but even fragmenting it like this could soon get out of hand unless we draw certain boundaries.

                                I'd suggest drawing the line at the interceptors - the TreeCacheMBean exists for core functionality/configuration, and each Interceptor has an MBean and is responsible for querying it's dependant classes.

                                • 13. Re: Exposing JBossCache Management Attributes
                                  belaban

                                   

                                  "JerryGauth" wrote:


                                  So the two key initial design issues are as follows.

                                  1) Expose the statistics via the existing TreeCacheMBean or via a new management mbean?

                                  2) Add an interceptor to capture the statistics or simply add the logic to TreeCache for now? As noted, an interceptor may not be able to handle everything so some logic might still need to be added to TreeCache.


                                  #1 I'd say use a new management MBean
                                  #2 Interceptor. If not possible to gather some stats, then (and only then !) implement in TreeCache itself

                                  • 14. Re: Exposing JBossCache Management Attributes
                                    belaban

                                    Regarding each interceptor having an MBean: yes, in 2.0 we will configure each interceptor separately. But we need to expose the stats for each interceptor hierarchically, e.g.

                                    JBossCache
                                     CallInterceptor
                                     PessimisticLockInterceptor
                                     ReplicationInterceptor
                                    

                                    etc. There has to be some function which dynamically registers all of these MBeans under the JBossCache MBean with the MBeanServer.
                                    Since this is similar to how JGroups does it, may I suggest you take a look at the JGroups code ?

                                    1 2 3 Previous Next