very slow bdbje loader implementation (CR3)

[srnm]

I have a simple test case that puts 1000 objects into a node.
Using the bdbje loader the test runs in >100s
Using the jdbm loader the test runs in <10s

I looked at the loader implementations and I see that bdbje loader is storing each node as a HashMap, and that each put reads the hashmap from disk, modifies it and then writes it. Put another way, the entire node is being read, and written as it grows.

In contrast, the jdbm implementation uses a "flat" key space and therefore can take advantage of the b-tree structure to find and modify individual items.

I'd go so far as to say that the bdbje loader is currently unusable.
Are there any plans to fix this?
My apologies, I can't volunteer to fix this right now.

I'm new here, so please forgive if I have missed something...

thanks in advance,
Steven Marcus

[genman]

Well, when I wrote the JDBM cache loader I designed it at a finer level of granularity, not understanding that you should really be storing a single object per node.

Locking, replication, eviction all work off of nodes as units of work, not node keys.

[srnm]

Are you saying that the jdbm loader doesn't support the semantics of the cache properly? If so http://jira.jboss.com/jira/browse/JBCACHE-696 would be a negative?

Am I correct in thinking that the FQN is really the "key" and that the fanout of actual keys underneath should be kept very small?

Do the jdbc loaders update a row/single key in a node at a time? If so, how do they support the semantics of the cache?

Are there any hints recorded as to how to design a cache structure for optimal performance?

Lots of questions, apologies...

[manik]

"srnm" wrote:

Am I correct in thinking that the FQN is really the "key" and that the fanout of actual keys underneath should be kept very small?

Correct. This will help you achieve maximum concurrency in the cache.

"srnm" wrote:

Do the jdbc loaders update a row/single key in a node at a time? If so, how do they support the semantics of the cache?

By using a separate shared lock on the Fqn. See, for example, line 181 in JDBCCacheLoader

"srnm" wrote:

Are there any hints recorded as to how to design a cache structure for optimal performance?

It depends. Where are your bottlenecks? Is it in concurrency, replication or cache loading? :-) Either way though, use the tree structure fully, don't put too much in each node.