6 Replies Latest reply on Aug 25, 2008 2:48 PM by manik

    Eviction node event queue warning

    em161100

      Hello all,

      I'm working JBoss Cache 'Alegrias' 2.1.1.GA under Windows environment.
      Problem : under heavy load, I start receiving the following warning messages :
      WARNING: putNodeEvent(): eviction node event queue size is at 98% threshold value of capacity: 200000 Region: /objects/users You will need to reduce the wakeUpIntervalSeconds parameter.
      Aug 21, 2008 3:54:32 PM org.jboss.cache.RegionImpl putNodeEvent

      I'm working in local configuration and I tried to reduce wakeup time to 5 seconds and it didn't help. This is a snapshot of my configuration :



      5
      org.jboss.cache.eviction.LRUPolicy

      <!--
      the tags max time should be fixed with an eviction method
      -->

      10000
      172800


      <!--
      the profiles will be evicted after 5 hours in the cache
      -->


      30000
      18000







      I'm using sun JVM - V1.6.

      What is the best method for solving this ? From reading in the forums I'm not sure that it is good to have a thread every 1 seconds cleaning the cache and I also feel like it might not solve the problem.

      Any help will do.

      Thanks,
      Elad.

        • 1. Re: Eviction node event queue warning
          skajotde

           

          I'm not sure that it is good to have a thread every 1 seconds cleaning the cache


          This parameter doesn't clean cache - just configure period of scanning cache queue to plan cleaning. On heavy load this parameter sometimes would be set to milliseconds (it is not available feature). Try 1 second.

          This settings helps me in same problem, but maybe my load wasn't so high as yours.

          Maybe try increase size of region or divide objects into subregions (are you using hibernate ?)

          ps
          Post configurations with tags code.

          • 2. Re: Eviction node event queue warning
            em161100

            Hello and thanks for the help,

            The problem came back even after reducing the sleep time to 2 seconds.
            I could see that every 2 seconds the JVM cpu utilization jumps high and I get many warning messages. When the problem happens the JVM takes about 250M, mostly for JBoss cache.
            I will try to reduce the max nodes to 5000 and set the sleep time to 1 second.

            I still don't understand how JBoss should be configured. Should I change the queue size from 200,000 ? Does it have any implication on my situation ?

            Thanks again,
            Elad.

            • 3. Re: Eviction node event queue warning
              manik

              Yes, try increasing the queue size. Implications are memory usage.

              • 4. Re: Eviction node event queue warning
                em161100

                Hello,

                The problem came back again even after setting the queue size to 500,000.
                I can't just ignore this problem because it just takes too much cpu.

                I have a question regarding the way the eviction work. I thought that the problem happens maybe because of the way I evict my objects and for some reason the cache thinks that the object was revisited and therefor adds it again to the cache. This is what I do in my eviction listener :

                public void logNodeEvent(NodeEvent ne) {
                Fqn fqnActionFather = ...

                // handling evicition of nodes according to the object type

                if (ne.getType().name().equals("NODE_EVICTED")) {
                if (ne.getFqn().size() > 0) {
                if (ne.getFqn().isChildOf(fqnActionFather )) { String id= (String) ne.getFqn().getLastElement();
                Node node = ne.getCache().getNode(ne.getFqn());
                if (node != null) {
                Action action = (Action) node.get(id);
                // handle eviction
                }
                .
                .
                .
                .

                Thanks for the help,
                Elad.

                • 5. Re: Eviction node event queue warning
                  em161100

                  Hello,

                  I tried the latest 2.2.0.CR7. I saw on another similar reply that it helped.
                  It didn't help me. My scenario is very basic and when ever I reach more than 10000 nodes it comes back. I tried increasing the queue size up to 500000 and it didn't help either.

                  Did anyone checked this scenario ? I saw several JIRA issues related to this and it is not clear if it was remarked as a bug. From the following issue raised in the forum it looks like someone had this fixed in CR7, meaning that it might was a bug on the latest GA :
                  http://www.jboss.com/index.html?module=bb&op=viewtopic&t=139650

                  Thanks,
                  Elad.

                  • 6. Re: Eviction node event queue warning
                    manik

                    It would certainly re-add the object in the cache - in your eviction listener you are doing a cache.get() again which will trigger a VISIT event. :-)

                    What are you trying to do in your eviction listener anyway? The actual eviction is handled by the eviction policy.