1 2 Previous Next 16 Replies Latest reply on Aug 23, 2007 6:10 PM by mvlior

    Performance expectations

    mvlior

      Hi,

      We have been evaluating JBC for production use in our system, and have decided to test PojoCache performance on the hardware we intend to use.

      Looking for performance data, we came across this performance test:
      http://wiki.jboss.org/wiki/Wiki.jsp?page=WhatShouldWeExpectOfThePojoCachePerformance

      The reason for this post is that we tried to run our own, differnt, throughput test, but the performance seemed very degraded in comparison to the expected results, implied by the above reference.
      The two tests are not identical, but based on the reference we believe our test should have run significantly faster than it does.
      In practice, the test runs about four time slower than it should take to attach these amount of new, very small POJOs.

      Can anyone please point out what we have been doing wrong, verify our results, cite other benchmarks of JBoss-Cache etc. ?
      Also, if anyone knows about comparison to other caching solutions, we would be interested.


      Below are the test details, our results, the configuration file and the test code itself.


      Thank you in advance,

      Lior Neuman
      R&D Team
      MailVision LTD


      Test environment:
      Single node (Local cache),
      HW: Dual P4 Xeon 3.0, 2Gb, 80Gb
      SW: running Linux 2.6.9-22.ELsmp, JVM: 32bit 1.5.0_06

      Remarks:
      Our HW has 2G RAM whereas the test had 4G, but only 20% of it are being used throughout the test.
      The system was dedicated for the test.
      Logging level is set to INFO, sent to console only

      Test Plan and Load Pattern
      A Local cache, with optimistic-locking and no eviction is created and started.
      A single thread, with no added delay, invokes putObject(key, value) where both key and value are new String instances, which have not been attached before.

      Basically, the code that runs is the following, in an infinite loop: (full code at the end)

      String key = "/root/Key-" + counter;
      Object value = "dummy" + counter;
      
      pojoCache.putObject(key, value);
      
      if (counter % 1000 == 0) {
       // print counter and elapsed time
      }
      



      Test Results

      Col1 = Total Objs
      Col2 = Time Delta[sec]
      Col3 = Throughput [obj/sec]

      Col 1 | Col 2 | Col 3
      ------------------------------
      1000 | 4.84 | 206.75
      2000 | 2.38 | 277.2
      3000 | 2.02 | 324.89
      4000 | 1.95 | 357.55
      5000 | 2.28 | 371.25
      6000 | 2.68 | 371.56
      7000 | 3.14 | 363
      8000 | 3.52 | 350.82
      9000 | 3.95 | 336.34
      10000 | 4.31 | 321.84
      11000 | 4.71 | 307.41
      12000 | 5.14 | 293.26
      13000 | 5.70 | 278.84
      14000 | 6.13 | 265.38
      15000 | 6.52 | 253.05
      16000 | 6.89 | 241.82

      Graphically, the results show that the throughput starts at approx. 200 Objects/Sec,
      climbs to 370 (for 6000 objects), and then linearily
      decreases at approx. 12 Objects/Sec for every 1000 objects
      added.


      Cache Configuration File

      <?xml version="1.0" encoding="UTF-8"?>
      
      <!-- ===================================================================== -->
      <!-- -->
      <!-- Sample PojoCache Service Configuration -->
      <!-- -->
      <!-- ===================================================================== -->
      
      <server>
      
       <!-- Used inside JBoss AS -->
       <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar" />
      
      
       <!-- ==================================================================== -->
       <!-- Defines configuration -->
       <!-- ==================================================================== -->
      
       <mbean code="org.jboss.cache.aop.PojoCache"
       name="jboss.cache:service=testPojoCache">
      
       <!-- Used inside JBoss AS -->
       <depends>jboss:service=Naming</depends>
       <depends>jboss:service=TransactionManager</depends>
      
       <!--
       Configure the TransactionManager
       -->
       <attribute name="TransactionManagerLookupClass">org.jboss.cache.DummyTransactionManagerLookup</attribute>
      
       <!--
       Node locking scheme:
       OPTIMISTIC
       PESSIMISTIC (default)
       -->
       <attribute name="NodeLockingScheme">OPTIMISTIC</attribute>
      
       <!--
       Note that this attribute is IGNORED if your NodeLockingScheme above is OPTIMISTIC.
      
       Isolation level : SERIALIZABLE
       REPEATABLE_READ (default)
       READ_COMMITTED
       READ_UNCOMMITTED
       NONE
       -->
       <attribute name="IsolationLevel">NONE</attribute>
      
       <!--
       Valid modes are LOCAL
       REPL_ASYNC
       REPL_SYNC
       INVALIDATION_ASYNC
       INVALIDATION_SYNC
       -->
       <attribute name="CacheMode">LOCAL</attribute>
      
       <!-- Name of cluster. Needs to be the same for all clusters, in order
       to find each other
       -->
       <attribute name="ClusterName">MVCache-Cluster</attribute>
      
       <!-- JGroups protocol stack properties. Can also be a URL,
       e.g. file:/home/bela/default.xml
       <attribute name="ClusterProperties"></attribute>
       -->
      
       <attribute name="ClusterConfig">
       <config>
       <!-- UDP: if you have a multihomed machine,
       set the bind_addr attribute to the appropriate NIC IP address, e.g bind_addr="192.168.0.2"
       -->
       <!-- UDP: On Windows machines, because of the media sense feature
       being broken with multicast (even after disabling media sense)
       set the loopback attribute to true -->
       <UDP mcast_addr="228.1.2.3" mcast_port="48866" ip_ttl="64" ip_mcast="true" mcast_send_buf_size="150000"
      mcast_recv_buf_size="80000" ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
      loopback="true" />
       <PING timeout="2000" num_initial_members="3" up_thread="false" down_thread="false" />
       <MERGE2 min_interval="10000" max_interval="20000" />
       <!-- <FD shun="true" up_thread="true" down_thread="true" />-->
       <FD_SOCK />
       <VERIFY_SUSPECT timeout="3000" up_thread="false" down_thread="false" />
       <pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800" max_xmit_size="8192"
      up_thread="false" down_thread="false" />
       <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10" down_thread="false" />
       <pbcast.STABLE desired_avg_gossip="20000" up_thread="false" down_thread="false" />
       <FRAG frag_size="8192" down_thread="false" up_thread="false" />
       <pbcast.GMS join_timeout="5000" join_retry_timeout="2000" shun="true" print_local_addr="true" />
       <pbcast.STATE_TRANSFER up_thread="true" down_thread="true" />
       </config>
       </attribute>
      
      
       <!--
       Whether or not to fetch state on joining a cluster
       NOTE this used to be called FetchStateOnStartup and has been renamed to be more descriptive.
       -->
       <attribute name="FetchInMemoryState">true</attribute>
      
       <!--
       The max amount of time (in milliseconds) we wait until the
       initial state (ie. the contents of the cache) are retrieved from
       existing members in a clustered environment
       -->
       <attribute name="InitialStateRetrievalTimeout">20000</attribute>
      
       <!--
       Number of milliseconds to wait until all responses for a
       synchronous call have been received.
       -->
       <attribute name="SyncReplTimeout">15000</attribute>
      
       <!-- Max number of milliseconds to wait for a lock acquisition -->
       <attribute name="LockAcquisitionTimeout">10000</attribute>
      
       <!-- Name of the eviction policy class. -->
       <attribute name="EvictionPolicyClass"></attribute>
      
       <!--
       Indicate whether to use region based marshalling or not. Set this to true if you are running under a scoped
       class loader, e.g., inside an application server. Default is "false".
       -->
       <attribute name="UseRegionBasedMarshalling">false</attribute>
       </mbean>
      
      
       <!-- Uncomment to get a graphical view of the TreeCache MBean above -->
       <!-- <mbean code="org.jboss.cache.TreeCacheView" name="jboss.cache:service=TreeCacheView">-->
       <!-- <depends>jboss.cache:service=TreeCache</depends>-->
       <!-- <attribute name="CacheService">jboss.cache:service=TreeCache</attribute>-->
       <!-- </mbean>-->
      
      
      </server>


      Test Code
      package test.stress;
      
      import java.util.Properties;
      
      import javax.naming.Context;
      
      import junit.framework.TestCase;
      
      import org.apache.log4j.Logger;
      import org.jboss.cache.PropertyConfigurator;
      import org.jboss.cache.aop.PojoCache;
      
      /**
       *
       * Tests performance with JBC 1.4.0 SP1. Adding new primitive objects only.
       *
       */
      public class StressTestAdd extends TestCase {
      
       static {
       org.apache.log4j.PropertyConfigurator.configure("resources/test.log4j");
       }
      
       private static final long TEST_TIME_SECONDS = 70;
       private static final String CONFIG_FILENAME = "resources/session-cache/local_cache.xml";
       private static final Logger log = Logger.getLogger(StressTestAdd.class);
      
       public void testOneCache() throws Exception {
      
       // configure cache, don't start it yet
       Properties prop = new Properties();
       prop.put(Context.INITIAL_CONTEXT_FACTORY, "org.jboss.cache.transaction.DummyContextFactory");
       final PojoCache pojoCache = new PojoCache();
       org.jboss.cache.PropertyConfigurator config = new PropertyConfigurator();
       config.configure(pojoCache, CONFIG_FILENAME);
      
      
       // declare runner thread
       Thread runner = new Thread(new Runnable() {
      
       private int counter = 0;
       boolean running = true;
       private final long start = System.currentTimeMillis();
      
       public void run() {
       try {
       while (running) {
       counter++;
      
       // key and value for attach(key, value);
       String key = "/root/Key-" + counter;
       Object value = "dummy" + counter;
      
       // put in cache
       Object previous = pojoCache.putObject(key, value);
      
       // verify no object was already attached with that key
       if (previous != null) {
       log.error("A previous key existed");
       running = false;
       fail("Attempt to update an existing key " + key);
       }
      
       // display counter every 1000 invocations
       if (counter % 1000 == 0) {
       long elapsed = System.currentTimeMillis() - start; // approx.
       log.info("Number = " + counter + ", after approx. "
       + elapsed / 1000.0 + " seconds.");
       }
       }
       } catch (Exception e) {
       log.error("Exception caught: ", e);
       fail(e.toString());
       }
       }
       });
      
      
       // start stress
       log.info("Stress test about to begin...");
      
       pojoCache.start();
       runner.start();
       Thread.sleep(TEST_TIME_SECONDS * 1000);
      
       log.info("...Stress test ended");
      
       }
      
      }



        • 1. Re: Performance expectations
          genman


          There are many differences in what you tested and the wiki page.

          Mainly, they were testing field updates on attached objects across a cluster. And you are just testing the speed of attaching objects.

          If your objects are simple strings, just use the plain TreeCache.

          • 2. Re: Performance expectations
            mvlior

            Hi,
            Thanks for replying; please see inline.

            There are many differences in what you tested and the wiki page.

            Mainly, they were testing field updates on attached objects across a cluster. And you are just testing the speed of attaching objects.


            Yes, you are right, they are different. In fact, the expectations from our test are based on the wiki's results for the cases named "TreeCache" and "100-0 PojoCache". Unless they were totally misinterpreted, then even a very cautious estimate for the throughput would be, in my opinion, about 1500 "different POJO attachment" per second.
            At no point in the stress does the throughput go above a quarter of that figure.
            It should also be noted that no network traffic / local replication efforts are involved in our case since there is only one local cache.

            If your objects are simple strings, just use the plain TreeCache.


            Actually, the system's needs are for more complex POJOs. Attaching small String objects was chosen for the stress test because:
            1. They are small objects, so throughput is expected to be high, and
            2. Strings are not aspectized, so this test can be run by other people without the need to run aopc on our classes.

            I guess that at the end of the day, the question is how fast this code is supposed to run, given that environment.
            Our hope is that perhaps something can be done to increase the throughput to a level acceptable for our system.


            Thanks for reading,

            Lior Neuman
            R&D Team
            MailVision LTD


            • 3. Re: Performance expectations
              mvlior

              We were hoping for an official response for such a question.
              Perhaps a light version of the above would help :)


              How fast can putObject("Key...", "Value...") be called on a local cache ?
              The keys/values are different Strings every time.


              Thanks in advance,

              Lior Neuman
              R&D Team
              MailVision LTD

              • 4. Re: Performance expectations
                genman

                PojoCache isn't optimized for attaching strings using putObject(). I'm not an official developer, but I know PojoCache is designed for lightweight replication and is not optimized for attachment speed.

                I know PojoCache isn't trying to be the fastest for this sort of use case. There is the TreeCache class for this use case instead. PojoCache's advantages are allowing you to store complex object trees and fine-grained replication.

                • 5. Re: Performance expectations

                  I have noticed that you used optimistic locking, if I am reading it correctly. Have you tried the pessimistic one where the wiki results were based?

                  • 6. Re: Performance expectations
                    mvlior

                    Hi,

                    No, we did not try the pessimistic locking in the test.
                    Do the above test results seem normal to you, or is there something wrong ?

                    BTW, at first it was planned to run the wiki test, but it seems to have been removed from the trunk (last time I checked).

                    Thank you.

                    • 7. Re: Performance expectations

                      You need to check out Branch_JBossCache_1_4_0 and look under tests/scripts. It is there since I just check. :-)

                      I'd recommend to try it with pessimistic locking. Not sure how much of the performance impact it will have but I think it does without further tuning.

                      • 8. Re: Performance expectations
                        mvlior

                        Oh... I must have checked in the wrong place then.

                        I understand that with optimistic locking, these are the results to expect,
                        please correct me if I'm wrong.

                        Thank you.

                        • 9. Re: Performance expectations
                          mvlior

                           

                          I'd recommend to try it with pessimistic locking. Not sure how much of the performance impact it will have but I think it does without further tuning.


                          After changing to pessimistic locking, our test results have dramatically improved. It turns out to be a known issue in 1.2.4 SP1.

                          In case anyone stumbles upon this issue, below is a very useful link referring to the phenomenon.

                          http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossCacheOptimisticLockingPerformance

                          Also referred to from:
                          http://wiki.jboss.org/wiki/Wiki.jsp?page=OptimisticNodeLocking

                          It would be nice to be able to reach (one of) the above pages from a page titled "Performance we should expect from PojoCache"
                          (http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossCacheOptimisticLockingPerformance),
                          but that's my opinion.
                          Maybe the results would have been expected - 170 (!) times faster.


                          Thanks for all the help,

                          Lior Neuman
                          R&D Team
                          MailVision LTD

                          • 10. Re: Performance expectations
                            manik

                             


                            After changing to pessimistic locking, our test results have dramatically improved. It turns out to be a known issue in 1.2.4 SP1.

                            What is a known issue? Optimistic locking did not officially exist till 1.3.0. :-)



                            • 11. Re: Performance expectations
                              brucespringfield

                              Out of curiosity, is Optimistic locking still an issue with POJOCache in 2.0 Habanero?

                              • 12. Re: Performance expectations
                                brucespringfield

                                Which version of JBOSS Cache was Lior using?

                                • 13. Re: Performance expectations
                                  mvlior

                                   

                                  "manik.surtani@jboss.com" wrote:

                                  After changing to pessimistic locking, our test results have dramatically improved. It turns out to be a known issue in 1.2.4 SP1.

                                  What is a known issue? Optimistic locking did not officially exist till 1.3.0. :-)



                                  :)
                                  The Wiki referred to it nonetheless
                                  http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossCacheOptimisticLockingPerformance

                                  • 14. Re: Performance expectations
                                    jason.greene

                                     

                                    "BruceSpringfield" wrote:
                                    Out of curiosity, is Optimistic locking still an issue with POJOCache in 2.0 Habanero?


                                    No this is actually a good combination.

                                    1 2 Previous Next