3 Replies Latest reply on Jul 9, 2013 3:39 AM by pilhuhn

    RHQ taking advantage of memory/CPU

    genman

      My RHQ host is only using about 8GB out of 100GB of memory, and about 4% of CPU (there are 8 cores).

       

      Is it possible for RHQ to cache things more aggresively for better performance? For example, I know Hibernate/JBoss 7 allows you to cache things in a second-level cache. Has this sort of thing been turned on? An entity cache might make some sense. I'm refering to this document.

        • 1. Re: RHQ taking advantage of memory/CPU
          pilhuhn

          Elias,

           

          you are (from my point of view) totally running into open doors.

           

          Having said that, there are rumors that someone tried the 2nd level cache some loong time ago and failed for whatever reason, so we never really tried again.

           

          When / if we would do more caching we would need to consider a few things (list is for sure incomplete):

          • Synchronyzation of caches in a HA-szenario with multiple servers, where an agent is usually attached to one server, but can switch over. So at least at the switchover time, the caches would need to be (partially) flushed
            • Syncing mechanism: this may need multicast or any other means, which means that admins may have to set up networking and or firewalling for this. One may omit cache to cache communication, but would for sure also loose some benefits. On the other hand flushing some of the data and re-creating from DB is still better than what we have today
          • Depending on where we are caching (Hibernate, app logic):
            • LazyLoad exceptions may arise. I was running into this when developing the REST-api where (at least in the old as4-based container), the rest-code was more less running in the sevlet and objects from the cache did not have all the relations needed. This may be circumvented by just caching a bit more
            • Access rights to objects in the cache. Currently many queries have some sort of "AND where object.subject.role.permissions = <permission>" at the end wich enforces visibility. A caching layer needs to enforce this as well
          • Memory consumption and garbage collections: using a nice 16GB heap for the old generation(1) could mean that a stop-the-world pause for old generation may take up to 16sec (2) ,
            • which has negative impact on the responsiveness of the ui and
            • could (worse) produce riple effect on the agents in the sense that this thinks the server is unreachable and switches over to the other server, which does some cache flushing / regeneration, which causes a full GC, which ...
            • At least on Linux there is the Zing VM, that may help, but it is yet another unknown.
            • Typical garbage collections on the young generation and also partial ones on the old generation are much shorter than our calls to the DB-backend, so that except for the outlier the effect of caching should be positive anyway.

           

          I am sure those concerns can all be addressed and especially with Hibernate caching, the access rights and lazy loading issues may be "just void". Again, I am totally in favor of caching and thus reducing the load on the DB.

           

          1) Cached objects are supposed to be long-lived and will thus end up in the old generation

          2) See e.g. http://es.slideshare.net/jaxconf/understanding-java-garbage-collection-and-what-you-can-do-about-it-gil-tene Slide 10

          1 of 1 people found this helpful
          • 2. Re: RHQ taking advantage of memory/CPU
            genman

            If JBoss does what it is advertised to do, then you shouldn't have to worry about most of these issues. I'm guessing if configured, and with the appropriate classes annotated[*], it should just work out of the box, excepting some of the lazy-load issues which really should be fixed in any event.

             

            Query caching I guess would be pretty much impossible to do well, and the Hibernate folk really don't recommend it too highly anyway. As you say, it's highly variable.

             

            To me it feels like something easy to test. I unfortunately don't have the clustered setup (yet) to do this.

             

            I would like to add, I went down the Hibernate+JBoss Cache road once, but since my application performed fine without any sort of caching, I didn't see it fit to complicate matters and possibly introduce bugs. This was a telecom application for the largest carrier in North America, mind you.

             

            [*] I would probably only cache the stuff that's fairly static: Resouces and their associations, measurement schedules, etc. If you don't see any benefit then I guess no reason to go on...

            • 3. Re: RHQ taking advantage of memory/CPU
              pilhuhn

              Elias,

               

              will answer (some time :-) later in more detail

              [*] I would probably only cache the stuff that's fairly static: Resouces and their associations, measurement schedules, etc. If you don't see any benefit then I guess no reason to go on...

              As I said, you are running into open doors here. And I agree with the static nature. Resource Types also come to mind. Or availability (which gets then written back to DB once it changes) - the latter probably needs some application logic.