1 Reply Latest reply on May 17, 2009 1:57 AM by java mastro

    optimal storage parameters for cache node?

    sridhar reddy Newbie

      I am using jboss cache 3.0 stand alone persistence mechanism as BJDB with c3p0 connection pool.
      I stored 100,000 key value pairs under a single FQN (node) at the depth of level 3.
      There was no data in the intermediate levels except at leaf level.

      Key is Long object and value is byte array of size 3k.
      So the total data stored in node is around 300 MB

      It took 7 seconds to process(read) all the records.
      key things are
      1) there were no modification to the content
      2) Prefetch all the records and no eviction mechanism.
      3) assigned 1GB RAM to application (it used only 600 MB) (SWAP space was not used by application)

      now i distributed 100,000 key value pairs under 10,000 individual FQNs so that each node has 10 key value
      pairs. Now the time taken to process the records is just 1.1 seconds.

      But in some other scenario i have a situation where i need to store 10,000 key value pairs where each value
      is 0f 1KB. There is no way that i can distribute them to different FQNs

      What i would like to know is some of the statistics like
      1) what is the optimal no of key value pairs
      2) what is the optimal size of each value in key value pair.
      3) what is the optimal size for a node.
      4) what is the optimal depth.

      Is ther any way that i can get good performance with 10,000 key value pairs under a single node.

      Please help me.

        • 1. Re: optimal storage parameters for cache node?
          java mastro Newbie

          Dear Member,

          I do faced the same problem some time back.
          Right now i can not give the statistics that you requested. But will let you know the work around(s) and
          follow if it suits for you.

          solution 1
          Instead of keeping 10,000 key value attributes under a single node,
          put your all key value pairs in a hash map and keep this hash map in the node.

          As effect the jboss cache will have only one node under a node. In this way jboss cache overhead will be
          eliminated and will give you the more performance in start up of jboss cache and reading the contents of it.

          One side effect of jboss cache is that, as the size of the node increases the performance of cache start up and persisting the updates will be affected.

          solution 2
          do segmentation using hashing Instead of putting all key value pairs under a single node.

          What i mean to say is that, as your key is a long, Apply some kind of hashing so that, for any given number you will get the hash code between 0 to 9.
          now distribute your key value pairs under different nodes as below.
          if /x/y/z is your node under which you are placing attributes, after segmentation it will be

          /x/y/z/0 - some key value pairs
          /x/y/z/1 - some key value pairs
          /x/y/z/9 - some key value pairs

          performance wise there wont be any diff between depth of 3 and 4 if your are having values only at the leaf node.

          Please note that, using database persistence for a node with MBs of data will be too slow and chances of failures are more. BJDBC is preferred in such cases.

          hope your requirement satisfied