6 Replies Latest reply on Feb 24, 2009 6:52 AM by Chetan Parekh

    Resident Nodes evicted in heavy load conditions

    Mahendra Goyal Newbie


      Hi All,

      We recently implemented Jboss Cache for our application to reduce the memory footprint and we are now caching some common data in cache intead of being copied accross all users sessions

      We have 2 regions below the root node and we have marked the root node as well as the region nodes as resident using the setResident(true) everything was working fine in unit tests and system integration tests but we had a problem in perfromance tests.

      I have a static Hashtable into which I am storing these 2 region nodes maked as resident, for putting/getting in cache client passes the key with which the region node is stored in the Hashtable. The put/get function in turn pulls the region node from Hashtable and adds or gets the node and its data is passed back to the client.

      But in Performance/Load Testing we encontered an issue when the system ran fine for about 1.30 hours but after that suddenly the region nodes were found to be null as if evicted. Although the eviction config was not correct in the sense all the values for LRUPolicy like the maxNodes, timeToLiveSeconds and wakeUpIntervalSeconds for both the regions were very low considering the Load Tests.

      But still we feel that the nodes still should not have been evicted, and at the most the performance shuould have gone down.

      Has any one faced such an issue or is aware of any such limitations ?


        • 1. Re: Resident Nodes evicted in heavy load conditions
          Mircea Markus Master

          this is weird indeed.
          Would it be possible for you to write a UT to try to simulate this? As this issue only hits after 90 mins and the scenario isn't that straight forward, it's really hard to reproduce it otherwise.

          • 2. Re: Resident Nodes evicted in heavy load conditions
            Mahendra Goyal Newbie

            That would be difficult, but after 90 minutes in this case means in more loaded conditions. Since in our load tests the load is gradually increased continously to reach 200% (expected peak user load for current year x 2) but in this case the application failed due to caching even before it reached 100% (about 2000 concurrent users)

            The issue has not come up since we increased all the values of eviction as per load env. But the concern is still there since it has occured once, it shouldn't happen in production

            • 3. Re: Resident Nodes evicted in heavy load conditions
              Mahendra Goyal Newbie

              Hi Markus,

              we have encountered the same issue again, following are the settings used


              Stack Trace

              org.jboss.cache.NodeNotValidException: Node /Products is not valid. Perhaps it has been moved or removed.
               at org.jboss.cache.invocation.NodeInvocationDelegate.assertValid(NodeInvocationDelegate.java:497)
               at org.jboss.cache.invocation.NodeInvocationDelegate.getChild(NodeInvocationDelegate.java:330)
               at com.pearson.commonbiz.app.appservices.caching.JBossCacheAdpater.getFromCache(JBossCacheAdpater.java:316)
               at com.pearson.commonbiz.app.appservices.ResourcesCacheHelper.getFromCache(ResourcesCacheHelper.java:232)
               at com.pearson.commonbiz.app.appservices.ResourcesCacheHelper.getResourceIDTypeListFromCache(ResourcesCacheHelper.java:156)
               at com.pearson.ph.ois.presentation.actions.OISCoreAction.isProductHasService(OISCoreAction.java:734)
               at com.pearson.ph.ois.presentation.actions.GetAllAssignmentsForClassAction.processRequest(GetAllAssignmentsForClassAction.java:319)
               at com.pearson.ph.ois.presentation.actions.OISCoreAction.execute(OISCoreAction.java:87)
               at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
               at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
               at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
               at com.pearson.commonsys.dynaweb.uicontroller.ConcertActionServlet.process(ConcertActionServlet.java:230)
               at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)

              Like I told you earlier we store the "Product" node in a hashtable which is marked as resident in the begining after the creation

              Lately in our performance tests the error was coming up in logs every few hours.

              I am not sure how I can reproduce it since its very inconsistent.

              Could you please help us a bit with this

              Many Thanks,

              • 4. Re: Resident Nodes evicted in heavy load conditions
                Elias Ross Master

                A couple of thoughts: If losing data is undesirable, then disable eviction. If eviction is required but losing data is undesirable, use a cache loader. If eviction is happening sooner than you expect, then don't sent max nodes and set a minimum time to live for nodes.

                • 5. Re: Resident Nodes evicted in heavy load conditions
                  Chetan Parekh Newbie

                  We are facing the same issue. Have anybody resolved this issue?

                  • 6. Re: Resident Nodes evicted in heavy load conditions
                    Chetan Parekh Newbie

                    All our region nodes are resident nodes. Still we are getting this issue.