[brian.stansberry]

ASYNC generally isn't valid for an entity, since you can't guarantee cache consistency. It might be OK if all your entities are really "owned" by one node; i.e. it's not expected that another node will access that entity except in case of failover.

I'm somewhat surprised you are seeing a big perf improvement with OL. That would generally imply lots of cache writes, since the benefit of OL is it allows reads to proceed concurrently with writes.

[prabhat.jha]

I will have Julien/Thomas comment on amount of read/write.

What I saw during many iterations of my tests that OL performed and scaled much better {1,2,3} nodes cluster but scalability went down when nodes in cluster went to {4,5}. I have not tried tests with PL with a {4,5} node cluster.

[prabhat.jha]

Brian,

I'm tempted to extract a version of the TreeCacheProvider out of the EAP code, improve it a bit and package it as an independent library. Will help with your issue, and be a small benefit to the community. Won't be as good as the Hibernate 3.3 stuff, but better than what you've got. But I really don't have the cycle to do it, not until AS 5 is done.

JBoss Portal 2.7 is tentatively set to released in August. Is it possible for you to work on this in this time frame?

In mean time I am thinking of making PL as default locking mechanism with a doc on OL and describing potential problems and workaround.

[brian.stansberry]

Depends on when in August. :-) Aug 1st, no. Aug 31, probably. Once AS 5 CR1 goes out in the next week or so I'll put in a half-day to implement the improvement I'm thinking about in Branch_4_2 and to think about what it would take to extract the code into an external project. (Main source is easy, but tests are tied into the EJB3 testsuite fixtures.) I can give you a better answer then.

[brian.stansberry]

Prabhat,

Can you try your tests with your cache configured for READ_COMMITTED instead of REPEATABLE_READ? Once a tx reads an item from the 2nd Level Cache, Hibernate caches it in the Session object and doesn't need to go back to the 2LC for that same object. So there should be no benefit to REPEATABLE_READ.

[prabhat.jha]

Sure. I will give it a shot.

[prabhat.jha]

Brian, a quick question. Unless I looked at the outdated doc, there is a comment in the tree cache configuration file which tells that isolation level attribute is ignored if locking scheme optimistic. Since the problem I am having is with OL, should I need to care about this attribute?

Or do you mean to run the test with repeatable read and PL and see how performance numbers look?

[prabhat.jha]

Sorry, I had a typo which changes the meaning completely.

Do you mean to run the test with read_commited and PL?

[brian.stansberry]

Sorry, meant PL.

[prabhat.jha]

Brian, I reran the tests with PL and READ_COMMITTED isolation level.

<mbean code="org.jboss.cache.TreeCache"
name="portal:service=TreeCache,type=hibernate">
<depends>jboss:service=Naming</depends>
<depends>jboss:service=TransactionManager</depends>
<attribute name="TransactionManagerLookupClass">
org.jboss.cache.JBossTransactionManagerLookup
</attribute>
<attribute name="NodeLockingScheme">PESSIMISTIC</attribute>
<attribute name="IsolationLevel">READ_COMMITTED</attribute>
<attribute name="CacheMode">REPL_SYNC</attribute>

The result is definitely better and at par with what I was getting with OL. However, problem with scalability with {4,5} node cluster still exists. I get almost linear scalability in {1,2,3} node cluster but load handled in {4,5} node cluster is more or less same as that of a 2-node cluster. I have not tested with more than 5 node cluster to see if pattern continues.

Also, I am not sure based on this test if I should relax the isolation level from REPEATABLE_READ to READ_COMMITTED. Currently, RR is default isolation level in portal. Thoughts on this?

[prabhat.jha]

Sorry about code formatting in previous post but I am sure you got what I meant. :-)

 <mbean code="org.jboss.cache.TreeCache"
 name="portal:service=TreeCache,type=hibernate">
 <depends>jboss:service=Naming</depends>
 <depends>jboss:service=TransactionManager</depends>

 <attribute name="TransactionManagerLookupClass">
 org.jboss.cache.JBossTransactionManagerLookup
 </attribute>

 <attribute name="NodeLockingScheme">PESSIMISTIC</attribute>
 <attribute name="IsolationLevel">READ_COMMITTED</attribute>
 <attribute name="CacheMode">REPL_SYNC</attribute>

[brian.stansberry]

Relax to R_C. See http://lists.jboss.org/pipermail/jbosscache-dev/2008-July/002401.html for a full discussion of this.

Re: increasing scalability, the work Galder and I are doing on JBCLUSTER-206 should help there. Basically eliminating cluster-wide messages for the most common cache write operation. Once that is in, there should only be message traffic for new entities that are created and updates to existing entities. And if there are lots of those, the entity type is probably not a good choice for caching anyway.

Do you cache queries? If not, try INVALIDATION_SYNC and see how that improves things.

[brian.stansberry]

Do you cache queries? If not, try INVALIDATION_SYNC and see how that improves things.

Don't bother with that INVALIDATION_SYNC test for now. Am curious whether you cache queries though.

[prabhat.jha]

Thanks for the link Brian. Looking at different hibernate.cfg.xml, query cache is enabled for everything that uses second level cache:

 <property name="cache.use_second_level_cache">true</property>
 <property name="cache.use_query_cache">true</property>

And I believe REPL_SYNC is used because we cache both query and entity.

[brian.stansberry]

Yes, REPL_SYNC is needed if you use query caching.