Large CPU Increase Using Buddy Session Replication

[dmurphy2]

PROBLEM
We are trying to use session replication with ASS 4.0.5 but running into significant performance problems. Specifically, under load we are seeing a 3-4x cpu increase in cpu on the app servers as a result of them trying to do session replication.

Unfortunately the application is pretty old and somewhat badly written. As a result there are some pretty large object graphs in the user's session. We are trying to remove those in order to be as session replication friendly as possible. In addition the app repeatedly stores the same attribute value back into the session rather than checking if the value already exists there (and avoiding an unnecessary session.set). We could possibly re-factor this but its going to be a lot of work..

There is a possibility of upgrading to a later version of the AS, but not in the short term and so we would really like to get this working with our current version - AS 4.0.5.

BASIC SESSION REPLICATION UNDERSTANDING

Can I get some info on the basic operation of the session cache replication?
(Please note we are using AS 4.0.5 so the info I require must be specific to that version, since I imagine many improvements have been done since that version)

For example, (for the moment without a UseReplQueue queue setup) does replication occur every time the app sets a session attribute? And when it replicates does it then replicate just the value of the attribute that was just set or does it replicate the entire user's session at that point.

OPTIMAL CONFIGURATION

here is the best guess settings we have for the session replication cache (org.jboss.cache.aop.TreeCacheAop)

We use buddy replication with 1 backup node for each node.

REPEATABLE_READ
can we use 'NONE'. Will this help performance? What are the semantics of NONE for a session repl cache?

REPL_ASYNC
This should be the best choice - right?

false
This should be the best choice - right?


Changes we are considering -

OPTIMISTIC
Will this help the cpu overhead?

true
10000
2000
We are considering to try this since our app updates the same value into the session repeatedly and often. Short of refactoring the app I am hoping maybe the queue mechanism may help?

OTHER SUGGESTIONS
Are there any other suggestions that might help us reduce this very significant overheard of using session replication

[dmurphy2]

Sorry - re-posting with code tags..

PROBLEM
We are trying to use session replication with ASS 4.0.5 but running into significant performance problems.
Specifically, under load We are seeing a 3-4x cpu increase in cpu on the app servers as a result of them
trying to do session replication.

Unfortunately the application is pretty old and somewhat badly written. As a result there
are some pretty large oject graphs in the user's session. We are trying to remove those in order to be
as session replication friendly as possible. In addition the app repeatedly stores the same
attribute value back into the session rather than checking if the value already exists there (and avoiding
an unnecessaryt session.set). We could possibly refactor this but its going to be a lot of work..

There is a possibility of upgrading to a later version of the AS, but not in the short term and so we would really
like to get this working with our current verson - AS 4.0.5.


BASIC SESSION REPLICATION UNDERSTANDING

Can I get some info on the basic operation of the session cache replication?
(Please note we are using AS 4.0.5 so the info I require must be specific to that version, since I imagine many improvements have been done since that version)

For example, (for the moment without a UseReplQueue queue setup) does replication occur every time the app sets a session attribute? And when it replicates does it then replicate just the value of the attribute that was just set
or does it replicate the entire user's session at that point.

OPTIMAL CONFIGURATION

here is the best guess settings we have for the session replication cache (org.jboss.cache.aop.TreeCacheAop)

We use buddy replication with 1 backup node for each node.Here is our settings-

<attribute name="IsolationLevel">REPEATABLE_READ</attribute>
can we use 'NONE'. Will this help performance? What are the semantics of NONE for a session repl cache?

<attribute name="CacheMode">REPL_ASYNC</attribute>
This should be the best choice - right?

<attribute name="UseRegionBasedMarshalling">false</attribute>
This should be the best choice - right?

Things we might try ..
<attribute name="UseReplQueue">true</attribute>
<attribute name="ReplQueueInterval ">10000</attribute>
<attribute name="ReplQueueMaxElements">2000</attribute>
We are considering to ty this since our app updates the same value into the session repeatedly and often.
Short of refactoring the app I am hoping maybe the queue mechanism may help?

<attribute name="NodeLockingScheme">OPTIMISTIC</attribute>
Will this help our cpu load?

OTHER SUGGESTIONS
Are there any other suggestions that might help us reduce this very significant overheard of using session replication

[brian.stansberry]

The session gets replicated at most once per request, so if you call setAttribute multiple times in the course of a single request, that doesn't trigger multiple replications. Replication happens after the request thread exits your application code.

Performance tweaks:

1) In jboss-web.xml use replication-trigger SET instead of the default SET_AND_NON_PRIMITIVE_GET. Avoids replication after read-only requests. Only do this if you are sure your code calls setAttribute as part of every request wherein the attribute object is modified.

2) In jboss-web.xml use replication-granularity ATTRIBUTE instead of the default SESSION. Only the attributes made dirty during the request are replicated. SESSION replicates the whole session object.

You could also try replication granularity FIELD, but to use that you'd need to bytecode enhance the classes you store in the session.

Re: other changes:

NONE instead of REPEATABLE_READ will leave you exposed to providing corrupted state during state transfer to a new node that joins the cluster. You can try READ_COMMITTED and see if that gives you an improvment.

REPL_ASYNC is the right choice.

OPTIMISTIC locking will hurt performance of the session repl use case.

Use of a replication queue isn't going to reduce CPU usage. It will just isolate web request threads from having to block waiting to put replication messages on the wire.

Using "interval" SnapshotMode with a snapshot interval of 10000 in jbossweb-tomcat55.sar/META-INF/jboss-service.xml may reduce CPU usage if you make multiple requests that trigger session replication within the 10 seconds. The session will get replicated just once. Of course this or a replication queue both reduce the up-to-date-ness of your replicated data.

[dmurphy2]

Thanks Brian - these settings seem to have made a significant improvement. We have also started trimming down on the amount of session state and that is also a big win for replication.