13 Replies Latest reply on Aug 5, 2008 5:13 AM by manik

    Stress Tests

    lovelyliatroim

      Ok I have been doing a bit of stress testing on my side. I have a scenario where my client has asked me to see if it is possible. Scenario is as follows, they current have at peak time 1500 pushes/writes a second going on in there DB. This 1500 writes a occurs on a data size of about 300,000 records. Due to re-design in their architecture they want to move this feed out of their DB.

      Ive been asked to see if it is feasible to push it into there cache.

      So following is used in the simulation Tomcat 6.0.13,JDK 1.6.0_06 and Jboss Cache 2.1.0

      I created a local cache with 300,000 records. Code looks like so


      public void handleRequest(HashMap params, OutputStream out) {
       // TODO Auto-generated method stub
       //Create the cache on the first call and load it with records
       if(cache == null){
       cache = CacheManager.createCache("RealTime", "realtime-cache.xml");
       for(int i = 0; i < noOfRecords;i++){
       Map record = createQuoteMap();
       String path = "/"+ i+ "/" + i +"/" +i ;
       Fqn nodePath = Fqn.fromString(path);
       cache.put(nodePath, "item", record);
       if(i%500 == 0){
       log.debug("Currently at Index"+i);
       }
       }
       log.debug("Set up cache with "+noOfRecords +" records in the cache");
      
       }
      
       requestCounter++;
      
       String action= RequestUtil.getString("action", params);
       int index = generator.nextInt(noOfRecords);
       String path = "/"+ index+ "/" + index +"/" + index ;
       Fqn nodePath = Fqn.fromString(path);
      
       try {
       if(action == null || action.compareTo("add") == 0){
       Map record = createQuoteMap();
       cache.put(path, "item", record);
       writeCounter++;
       }else{
       //read operation
       Map record = (Map)cache.get(nodePath, "item");
       path = path +"\n";
       out.write(path.getBytes());
       DebugUtil.dumpMap(record, new PrintStream(out));
       out.close();
       readCounter++;
       }
      
       if(requestCounter%1000 == 0){
       log.debug("Read = "+readCounter + " Write = "+writeCounter);
       }
       } catch (IOException e) {
       // TODO Auto-generated catch block
       e.printStackTrace();
       }
       }
      


      JMeter was set up to generate requests. Tests carried out on my local machine, Xp professional,2g ram, VM given 1G. VM is warmed up before recording begins.

      Results are like so

      Test Case Duration No of Read Clients No of Write/Update Threads No of Reads No of Updates/Writes Throughput Reads (sec) Throughput Updates/Writes(sec) Isolation Level Comments

      Reading and Writing 1 Hour 10 10 1,889,946 1,754,769 525/sec 487.4/sec NONE 39% error recorded in JMeter for updates and 36% error recorded as well for reads.Heap size between 100-175MB

      Reading and Writing 1 Hour 10 10 5,741,899 5,974,218 1,594/sec 1,659/sec READ_UNCOMMITTED Heap size in between 300MB and 400MB range during processing. 0% error on both reading and writing.

      Reading and Writing 30 Mins 10 10 2,940,466 3,046,956 1633.56/sec 1692.70/sec READ_UNCOMMITTED Heap size in between 300MB and 450MB range during processing. 0% error on both reading and writing.

      Reading and Writing 30 Mins 10 10 728,026 743353 404.45/sec 412.95/sec NONE Heap size in between 100MB and 175MB range during processing. 44% error on reads and 43% on the writes.

      -
      Writing 30 mins - 10 - 1,151,919 - 640/sec NONE 56% error reported during run

      Writing 30 mins - 20 - 1,047,963 - 582.2/sec NONE 63% error reported during run

      Writing 30 mins - 10 - 1,928,110 - 1071/sec READ_UNCOMMITTED 21% error reported during run. Heap size between 300-400MB during run.

      -
      Reading 30 mins 10 - 5,670,148 - 3150.1/sec - NONE 0% error reported during run

      Reading 30 mins 10 - 5,443,498 - 3018/sec/sec - READ_UNCOMMITTED 0% error reported during run. Heap size ranges from 280-400MB

      Reading 30 mins 20 - 5,313,351 - 2951.8/sec - NONE 0% error reported during run



      Ok figures are taken from JMeters Summary report
      http://jakarta.apache.org/jmeter/usermanual/component_reference.html#Summary_Report.

      Things that struck me are this
      1. heap size difference, just by having a different isolation level from NONE to READ_UNCOMMITTED. I have seen this https://jira.jboss.org/jira/browse/JBCACHE-1383 and maybe that is playing a factor in this case. Will download latest CR version and see if it makes a difference. Heap Size when cache loaded with records in NONE mode was 100MB, when loaded in "READ_UNCOMIITED" 225MB. I possibly would have expected abit more in memory but not that much,its practically bigger than whats put in the cache.
      2. Error % which is logged by JMeter, i tried to find their definition of what is classed as an error, unfortunately no such luck, best i could find was this definition " "Error % - Percent of requests with errors". So i will assume that this is possibly timeouts, maybe cache taking to long to write/read.
      3. I would have thought that having an ISOLATION level of NONE would turn out to be quicker but apparently not. What goes on behind the scenes when its set to "NONE"?.
      4. What I also found interesting was when in "READ_UNCOMMITTED" mode for just doing writes, I got 21% error back. Possibly contention to write to the same node?

      Just thought Id post it, any feedback is welcome, any tips to improve throughput??

      Thanks,
      LL



        • 1. Re: Stress Tests
          lovelyliatroim

          VM Sizes

          * Create cache with 100,000 records, heap size = 42MB with ISOLATION set to "NONE"
          * Create cache with 100,000 records, heap size = 84MB with ISOLATION set to "READ_UNCOMMITTED"
          * Create cache with 200,000 records, heap size = 62MB with ISOLATION set to "NONE"
          * Create cache with 200,000 records, heap size = 150MB with ISOLATION set to "READ_UNCOMMITTED"
          * Create cache with 300,000 records, heap size = 90MB with ISOLATION set to "NONE"
          * Create cache with 300,000 records, heap size = 225MB with ISOLATION set to "READ_UNCOMMITTED"

          Will try it with the latest CR candidate and see if it produces the same results.

          • 2. Re: Stress Tests
            lovelyliatroim

            Ok this is is tried with the latest 2.2 CR6, I just replaced the core lib the rest of the libraries belong to the 2.1 release,dont think it should make a difference.

            Updated to the latest JBoss Cache 2.2 CR6 VM Sizes

            * Create cache with 100,000 records, heap size = 42MB with ISOLATION set to "NONE"
            * Create cache with 100,000 records, heap size = 82MB with ISOLATION set to "READ_UNCOMMITTED"
            * Create cache with 200,000 records, heap size = 68MB with ISOLATION set to "NONE"
            * Create cache with 200,000 records, heap size = 150MB with ISOLATION set to "READ_UNCOMMITTED"
            * Create cache with 300,000 records, heap size = 98MB with ISOLATION set to "NONE"
            * Create cache with 300,000 records, heap size = 225MB with ISOLATION set to "READ_UNCOMMITTED"

            Still interesting that the heap size doubles just because i have a different ISOLATION level.

            Will get the profiler set up and see what it says.........jmap aint working for me on bindows.

            • 3. Re: Stress Tests
              mircea.markus

               

              Still interesting that the heap size doubles just because i have a different ISOLATION level.

              If you are using 'NONE' then no lock object is being created for each node. I think there won't be significant differences between other isolation levels (except serialiable, which I should be quite memory-efficient due to same reasons as NONE)

              • 4. Re: Stress Tests
                jason.greene

                The major memory increase is because of JBCACHE-1383, which creates 16 CHM segments and 17 locks per node. We have not yet done a 2.2 release that includes the fix, which is reducing it to 4. You could build the latest 2.2.x branch to try it.

                The 3.0.0 Alpha release completely eliminates the problem (when using the MVCC locking mode). Perhaps you want to give that a try?

                • 5. Re: Stress Tests
                  lovelyliatroim

                   


                  The 3.0.0 Alpha release completely eliminates the problem (when using the MVCC locking mode). Perhaps you want to give that a try?

                  Unfortunately my time line wont allow for that.

                  Seeing what I would say is a strange result.

                  When I run the tests with 10 reading threads and 10 writing threads in parallel it is quicker when i just run the test with with 10 writing threads,alot quicker!!


                  Test Case Duration No of Read Clients No of Write/Update Threads No of Reads No of Updates/Writes Throughput Reads (sec) Throughput Updates/Writes(sec) Isolation Level Comments
                  Reading and Writing 30 Mins 10 10 2,940,466 3,046,956 1633.56/sec 1692.70/sec READ_UNCOMMITTED Heap size in between 300MB and 450MB range during processing. 0% error on both reading and writing.
                  Writing 30 mins - 10 - 1,200,578 - 666.98/sec READ_UNCOMMITTED Reran this test just to see if the first one was just bad luck.38% error reported during run. Heap size between 300-400MB during run. Interesting to see that reading and writing performs better than just writing alone.


                  Any idea why this would be??

                  Thanks,
                  LL

                  • 6. Re: Stress Tests
                    lovelyliatroim

                    Just back to this question


                    I would have thought that having an ISOLATION level of NONE would turn out to be quicker but apparently not. What goes on behind the scenes when its set to "NONE"?.


                    Any ideas as to why it would perform better with an isolation level set compared to NONE??

                    Thanks,
                    LL

                    • 7. Re: Stress Tests
                      genman

                      The best use of JBossCache is primarily for data read frequently but written rarely...according to the docs.

                      Consider testing using an actual application scenario.

                      • 8. Re: Stress Tests
                        jason.greene

                         

                        "lovelyliatroim" wrote:


                        When I run the tests with 10 reading threads and 10 writing threads in parallel it is quicker when i just run the test with with 10 writing threads,alot quicker!!


                        That has to be an anomaly, probably associated with the errors you are getting.

                        • 9. Re: Stress Tests
                          lovelyliatroim

                           


                          That has to be an anomaly, probably associated with the errors you are getting.


                          Yeh its hard to judge on windows, test servers arrive in 2 weeks so then i can re-run the fun. Just wanted to get a rough feeling for what to expect.



                          • 10. Re: Stress Tests
                            manik

                             

                            "lovelyliatroim" wrote:
                            Just back to this question


                            I would have thought that having an ISOLATION level of NONE would turn out to be quicker but apparently not. What goes on behind the scenes when its set to "NONE"?.


                            Any ideas as to why it would perform better with an isolation level set compared to NONE??

                            Thanks,
                            LL


                            I suspect this has to do with the error levels you are seeing. With NONE, all access to the cache is unguarded. Which is good for memory (no lock objects, CHM segments, etc.) and *possibly* good for performance (if there is no contention). The moment there is contention you will start to see concurrent modification exceptions and this is possibly what JMeter is seeing as an error. You also need to look at how JMeter calculates this throughput - e.g., if it doesn't count erroneous connections as successful (i.e., remove from numerator), doesn't count them at all (remove from numerator and denominator) or adds the time taken to generate and throw an exception. :-)


                            • 11. Re: Stress Tests
                              manik

                               

                              "jason.greene@jboss.com" wrote:
                              The major memory increase is because of JBCACHE-1383, which creates 16 CHM segments and 17 locks per node. We have not yet done a 2.2 release that includes the fix, which is reducing it to 4. You could build the latest 2.2.x branch to try it.


                              2.2.0.CR7 is now available which does have JBCACHE-1383. You should see your heap size drop.

                              "jason.greene@jboss.com" wrote:
                              The 3.0.0 Alpha release completely eliminates the problem (when using the MVCC locking mode). Perhaps you want to give that a try?


                              While you may not have the time to play with this, I would advise to at least keep it on your radar. It is _significantly_ faster than 2.X, and has a smaller memory footprint as well. I expect to release Alpha2 this evening/tomorrow morning, and my goal is to have this in GA by the end of September.

                              • 12. Re: Stress Tests
                                lovelyliatroim

                                 


                                While you may not have the time to play with this, I would advise to at least keep it on your radar. It is _significantly_ faster than 2.X, and has a smaller memory footprint as well. I expect to release Alpha2 this evening/tomorrow morning, and my goal is to have this in GA by the end of September.


                                So soon, will keep it on the radar so, how hard/easy would a migration step up be from 2.x to 3.0?? Roughly?


                                2.2.0.CR7 is now available which does have JBCACHE-1383. You should see your heap size drop.

                                Downloaded it yesterday and has improved the heap size.


                                • 13. Re: Stress Tests
                                  manik

                                   

                                  "lovelyliatroim" wrote:

                                  While you may not have the time to play with this, I would advise to at least keep it on your radar. It is _significantly_ faster than 2.X, and has a smaller memory footprint as well. I expect to release Alpha2 this evening/tomorrow morning, and my goal is to have this in GA by the end of September.


                                  So soon, will keep it on the radar so, how hard/easy would a migration step up be from 2.x to 3.0?? Roughly?


                                  Easier than the 1.x -> 2.x move, I expect. The only real API differences that may cause problems:

                                  1. De-generification of Fqn
                                  2. Removed deprecated public constructors on Fqn - only use static factory methods like Fqn.fromString() from now on.
                                  3. Removed deprecated DefaultCacheFactory.getInstance() method - now create one using "new".

                                  Apart from that, the only other changes are internal so while they will affect you (better performance, lower memory usage, different tuning and config options) they won't affect any interfacing code.