3 Replies Latest reply on May 5, 2008 5:26 PM by David Murphy

    Large CPU Increase Using Buddy Session Replication

    David Murphy Newbie

      PROBLEM
      We are trying to use session replication with ASS 4.0.5 but running into significant performance problems. Specifically, under load we are seeing a 3-4x cpu increase in cpu on the app servers as a result of them trying to do session replication.

      Unfortunately the application is pretty old and somewhat badly written. As a result there are some pretty large object graphs in the user's session. We are trying to remove those in order to be as session replication friendly as possible. In addition the app repeatedly stores the same attribute value back into the session rather than checking if the value already exists there (and avoiding an unnecessary session.set). We could possibly re-factor this but its going to be a lot of work..

      There is a possibility of upgrading to a later version of the AS, but not in the short term and so we would really like to get this working with our current version - AS 4.0.5.

      BASIC SESSION REPLICATION UNDERSTANDING

      Can I get some info on the basic operation of the session cache replication?
      (Please note we are using AS 4.0.5 so the info I require must be specific to that version, since I imagine many improvements have been done since that version)

      For example, (for the moment without a UseReplQueue queue setup) does replication occur every time the app sets a session attribute? And when it replicates does it then replicate just the value of the attribute that was just set or does it replicate the entire user's session at that point.

      OPTIMAL CONFIGURATION

      here is the best guess settings we have for the session replication cache (org.jboss.cache.aop.TreeCacheAop)

      We use buddy replication with 1 backup node for each node.

      REPEATABLE_READ
      can we use 'NONE'. Will this help performance? What are the semantics of NONE for a session repl cache?

      REPL_ASYNC
      This should be the best choice - right?

      false
      This should be the best choice - right?


      Changes we are considering -

      OPTIMISTIC
      Will this help the cpu overhead?

      true
      10000
      2000
      We are considering to try this since our app updates the same value into the session repeatedly and often. Short of refactoring the app I am hoping maybe the queue mechanism may help?

      OTHER SUGGESTIONS
      Are there any other suggestions that might help us reduce this very significant overheard of using session replication




        • 1. Re: Large CPU Increase Using Buddy Session Replication
          David Murphy Newbie

          Sorry - re-posting with code tags..

          PROBLEM
          We are trying to use session replication with ASS 4.0.5 but running into significant performance problems.
          Specifically, under load We are seeing a 3-4x cpu increase in cpu on the app servers as a result of them
          trying to do session replication.

          Unfortunately the application is pretty old and somewhat badly written. As a result there
          are some pretty large oject graphs in the user's session. We are trying to remove those in order to be
          as session replication friendly as possible. In addition the app repeatedly stores the same
          attribute value back into the session rather than checking if the value already exists there (and avoiding
          an unnecessaryt session.set). We could possibly refactor this but its going to be a lot of work..

          There is a possibility of upgrading to a later version of the AS, but not in the short term and so we would really
          like to get this working with our current verson - AS 4.0.5.


          BASIC SESSION REPLICATION UNDERSTANDING

          Can I get some info on the basic operation of the session cache replication?
          (Please note we are using AS 4.0.5 so the info I require must be specific to that version, since I imagine many improvements have been done since that version)

          For example, (for the moment without a UseReplQueue queue setup) does replication occur every time the app sets a session attribute? And when it replicates does it then replicate just the value of the attribute that was just set
          or does it replicate the entire user's session at that point.

          OPTIMAL CONFIGURATION

          here is the best guess settings we have for the session replication cache (org.jboss.cache.aop.TreeCacheAop)

          We use buddy replication with 1 backup node for each node.Here is our settings-

          <attribute name="IsolationLevel">REPEATABLE_READ</attribute>
          can we use 'NONE'. Will this help performance? What are the semantics of NONE for a session repl cache?
          
          <attribute name="CacheMode">REPL_ASYNC</attribute>
          This should be the best choice - right?
          
          <attribute name="UseRegionBasedMarshalling">false</attribute>
          This should be the best choice - right?
          
          Things we might try ..
          <attribute name="UseReplQueue">true</attribute>
          <attribute name="ReplQueueInterval ">10000</attribute>
          <attribute name="ReplQueueMaxElements">2000</attribute>
          We are considering to ty this since our app updates the same value into the session repeatedly and often.
          Short of refactoring the app I am hoping maybe the queue mechanism may help?
          
          <attribute name="NodeLockingScheme">OPTIMISTIC</attribute>
          Will this help our cpu load?
          
          


          OTHER SUGGESTIONS
          Are there any other suggestions that might help us reduce this very significant overheard of using session replication







          • 2. Re: Large CPU Increase Using Buddy Session Replication
            Brian Stansberry Master

            The session gets replicated at most once per request, so if you call setAttribute multiple times in the course of a single request, that doesn't trigger multiple replications. Replication happens after the request thread exits your application code.

            Performance tweaks:

            1) In jboss-web.xml use replication-trigger SET instead of the default SET_AND_NON_PRIMITIVE_GET. Avoids replication after read-only requests. Only do this if you are sure your code calls setAttribute as part of every request wherein the attribute object is modified.

            2) In jboss-web.xml use replication-granularity ATTRIBUTE instead of the default SESSION. Only the attributes made dirty during the request are replicated. SESSION replicates the whole session object.

            You could also try replication granularity FIELD, but to use that you'd need to bytecode enhance the classes you store in the session.

            Re: other changes:

            NONE instead of REPEATABLE_READ will leave you exposed to providing corrupted state during state transfer to a new node that joins the cluster. You can try READ_COMMITTED and see if that gives you an improvment.

            REPL_ASYNC is the right choice.

            OPTIMISTIC locking will hurt performance of the session repl use case.

            Use of a replication queue isn't going to reduce CPU usage. It will just isolate web request threads from having to block waiting to put replication messages on the wire.

            Using "interval" SnapshotMode with a snapshot interval of 10000 in jbossweb-tomcat55.sar/META-INF/jboss-service.xml may reduce CPU usage if you make multiple requests that trigger session replication within the 10 seconds. The session will get replicated just once. Of course this or a replication queue both reduce the up-to-date-ness of your replicated data.

            • 3. Re: Large CPU Increase Using Buddy Session Replication
              David Murphy Newbie

              Thanks Brian - these settings seem to have made a significant improvement. We have also started trimming down on the amount of session state and that is also a big win for replication.