1 Reply Latest reply on May 17, 2009 1:57 AM by java.mastro

optimal storage parameters for cache node?

sridhar_ratna May 16, 2009 5:10 AM

I am using jboss cache 3.0 stand alone persistence mechanism as BJDB with c3p0 connection pool.
I stored 100,000 key value pairs under a single FQN (node) at the depth of level 3.
There was no data in the intermediate levels except at leaf level.

Key is Long object and value is byte array of size 3k.
So the total data stored in node is around 300 MB

It took 7 seconds to process(read) all the records.
key things are
1) there were no modification to the content
2) Prefetch all the records and no eviction mechanism.
3) assigned 1GB RAM to application (it used only 600 MB) (SWAP space was not used by application)

now i distributed 100,000 key value pairs under 10,000 individual FQNs so that each node has 10 key value
pairs. Now the time taken to process the records is just 1.1 seconds.

But in some other scenario i have a situation where i need to store 10,000 key value pairs where each value
is 0f 1KB. There is no way that i can distribute them to different FQNs

What i would like to know is some of the statistics like
1) what is the optimal no of key value pairs
2) what is the optimal size of each value in key value pair.
3) what is the optimal size for a node.
4) what is the optimal depth.

Is ther any way that i can get good performance with 10,000 key value pairs under a single node.

Please help me.

1. Re: optimal storage parameters for cache node?

java.mastro May 17, 2009 1:57 AM (in response to sridhar_ratna)

Dear Member,

I do faced the same problem some time back.
Right now i can not give the statistics that you requested. But will let you know the work around(s) and
follow if it suits for you.

solution 1
--------------
Instead of keeping 10,000 key value attributes under a single node,
put your all key value pairs in a hash map and keep this hash map in the node.

As effect the jboss cache will have only one node under a node. In this way jboss cache overhead will be
eliminated and will give you the more performance in start up of jboss cache and reading the contents of it.

One side effect of jboss cache is that, as the size of the node increases the performance of cache start up and persisting the updates will be affected.

solution 2
------------
do segmentation using hashing Instead of putting all key value pairs under a single node.

What i mean to say is that, as your key is a long, Apply some kind of hashing so that, for any given number you will get the hash code between 0 to 9.
now distribute your key value pairs under different nodes as below.
if /x/y/z is your node under which you are placing attributes, after segmentation it will be

/x/y/z/0 - some key value pairs
/x/y/z/1 - some key value pairs
...
/x/y/z/9 - some key value pairs

performance wise there wont be any diff between depth of 3 and 4 if your are having values only at the leaf node.

Please note that, using database persistence for a node with MBs of data will be too slow and chances of failures are more. BJDBC is preferred in such cases.

hope your requirement satisfied
Actions