7 Replies Latest reply on Dec 17, 2010 11:33 AM by mreasy

    JBoss 6 performance evaluation

    mreasy

      Hi all,

       

      First of all, thanks for all your effort so far and what is still to come, for releasing JBoss 6.

       

      We are building an Business-Process-application, which curently runs on JBoss 4.0.5 with JBossMQ, which we plan to migrate to JBoss6. This application heavily uses JMS messaging, which we hoped to benefit from the HornetQ integration.

       

      Unfortunately the tests we ran so far were disappointing and we reached a thruput which only reaches 30% using JBoss6 CR1 of the same test with JBoss4. Maybe you can share your obervations if you have comparisions between JBoss6 and 5/4, or raise points where we could improve set-up / configuration / etc. I'll describe the test in the following. It is clear that JBoss6 is not yet GA, but afaik there is no known issue in this area that should get addressed until GA.

       

      Setup

      CPU, memory, filesystem and DB should not be limitating factors (see spec attached at end), DB-load decreases by nearly 80% which is partly expected since JMS traffic goes from DB to filesystem.

       

      HornetQ is using default configuration aside from the No. and size of the data files:

      <!-- Default journal file size is set to 1MB for fast booting, we use 10MB here  -->
      <journal-file-size>${hornetq.journal.file.size:10485760}</journal-file-size>
      <!-- Default journal min file is 2, increase for higher average msg rates, we use 5 here -->
      <journal-min-files>${hornetq.journal.min.files:5}</journal-min-files>

       

      Tested HornetQ with NIO and AIO, which did not make a measureable difference, since the filesystem has a battery-backed buffer anyways.

      It is clear, that the thruput is limited by the No. of possible fsyncs, but that should not be the limiting factor here (also see Thread#19114 discussing this).

       

      Further tests included:

      • set-up on other machines, which showed the same relation before/after.
      • Putting HornetQ files (journal, data, ...) on a RAM-disk - no real difference
      • Putting Arjuna transaction files on a RAM-disk - no real difference

      Observations

      Thruput reaches 30% as mentioned. The load on the JBoss machine goes into sky, see screenshot below. But also using a profiler, it is not easy to tell what uses that much CPU (I/O waits which also count into the load are not seen that much). There is not 'that one hotspot' nor locking.

      htop.png

       

      Open Questions

      • Is this expected /known?
      • Is there a chance that the poor performance derives from using beans using standard EJB 2.1 and maybe EJB 3 beans would fit-in better?
      • Is HornetQ the bottleneck? From the benchmarks, that are known I would not say this, but maybe there are known limitations with the integration?

       

       

       

      Besides the performance topic, I also experience transaction problems, if the system is running for some time (12 - 24 hours). But this is another topic...

       

      For information machine and VM details:

      VM Arguments:
      jvm_args: -XX:NewRatio=1 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:PermSize=100M -XX:MaxPermSize=300M -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -XX:+HeapDumpOnOutOfMemoryError -Xloggc:gc.log -Xverify:none -Xms4G -Xmx8G -Djava.awt.headless=true -Djava.endorsed.dirs=/home/username/MyApp/runtime/jboss/lib/endorsed -Djava.io.tmpdir=/home/username/MyApp/temp -Djboss.home.dir=/home/username/MyApp/runtime/jboss -Djboss.server.base.dir=/home/username/MyApp/jboss -Djboss.server.base.url=/home/username/MyApp/jboss/ -Djboss.server.log.dir=/home/username/MyApp/log/jboss -Djboss.server.temp.dir=/home/username/MyApp/temp/jboss -Djboss.server.data.dir=/home/username/MyApp/data/jboss -Dcatalina.base=/home/username/MyApp/temp/tomcat -Dcatalina.home=/home/username/MyApp/temp/tomcat -Dlogging.configuration=file:/home/username/MyApp/jboss/SeeBisAs/conf/logging.properties -Djava.library.path=/home/username/MyApp/runtime/jboss/bin/native/lib64 -Djava.endorsed.dirs=/home/username/MyApp/runtime/jboss/lib/endorsed
      java_command: org.jboss.Main -c MyApp -b 0.0.0.0


      ---------------  S Y S T E M  ---------------
      OS:Red Hat Enterprise Linux Server release 5.5 (Tikanga)

      uname:Linux 2.6.18-194.26.1.el5 #1 SMP Fri Oct 29 14:21:16 EDT 2010 x86_64
      libc:glibc 2.5 NPTL 2.5
      rlimit: STACK 10240k, CORE 0k, NPROC 278528, NOFILE 65536, AS infinity

      CPU:total 8 (4 cores per cpu, 1 threads per core) family 6 model 15 stepping 11, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3
      Memory: 4k page, physical 20544364k(3235836k free), swap 8388600k(8388584k free)

      vm_info: Java HotSpot(TM) 64-Bit Server VM (17.1-b03) for linux-amd64 JRE (1.6.0_22-b04), built on Sep 15 2010 01:07:59 by "java_re" with gcc 3.2.2 (SuSE Linux)

        • 1. Re: JBoss 6 performance evaluation
          jaikiran

          This mostly looks like a HornetQ/messaging question. From your question, I'm guessing that this application is using EJB2.x MDBs. Is that right?

          By the way, have you looked at http://hornetq.sourceforge.net/docs/hornetq-2.1.2.Final/user-manual/en/html/perf-tuning.html and made sure that you follow the recommended configurations?

           

          If none of this applies in your case or doesn't work, then feel free to report back with details.

          • 2. Re: JBoss 6 performance evaluation
            jaikiran

            By the way, if this involves MDBs, then post the relevant configs on the MDB.

            • 3. Re: JBoss 6 performance evaluation
              mreasy

              Hi Jaykiran, thanks for your reply.

              I would also bet on messaging, but that is not yet a fact I would say.

               

              Yes, we are using EJBs 2.1 - this might change after the migration, but is the current state.

              I know the HornetQ settings and tried them out as well, but 1. with no significant change and 2. imho also the default should work at least as well as JBoss4

              Here is the MDB configuration (which is the default except of MaxMessages and MaxTimesRedelivered):

               

              <invoker-proxy-binding>
               <name>message-driven-bean</name>
               <invoker-mbean>default</invoker-mbean>
               <proxy-factory>org.jboss.ejb.plugins.jms.JMSContainerInvoker</proxy-factory>
               <proxy-factory-config>
                 <JMSProviderAdapterJNDI>DefaultJMSProvider</JMSProviderAdapterJNDI>
                 <ServerSessionPoolFactoryJNDI>StdJMSPool</ServerSessionPoolFactoryJNDI>
                 <CreateJBossMQDestination>false</CreateJBossMQDestination>
                 <MinimumSize>1</MinimumSize>
                 <MaximumSize>30</MaximumSize>
                 <KeepAliveMillis>30000</KeepAliveMillis>
                 <MaxMessages>1</MaxMessages>
                 <MDBConfig>
                   <ReconnectIntervalSec>10</ReconnectIntervalSec>
                   <DLQConfig>
                     <DestinationQueue>queue/DLQ</DestinationQueue>
                     <MaxTimesRedelivered>2</MaxTimesRedelivered>
                     <TimeToLive>0</TimeToLive>
                   </DLQConfig>
                 </MDBConfig>
               </proxy-factory-config>
              </invoker-proxy-binding>
              
              • 4. Re: JBoss 6 performance evaluation
                mreasy

                The root of the problem is found. JBoss6 defines the "Standard CMP 2.x EntityBean" configuration in standardjboss.xml by referencing the "instance per transaction" configuration, which has no transaction-independant entity bean cache:

                <container-configuration extends="Instance Per Transaction CMP 2.x EntityBean">
                 <container-name>Standard CMP 2.x EntityBean</container-name>
                </container-configuration>
                

                 

                This was a killer for our application, since there are huge imutable entities, which get loaded in the cache once and should not be reloaded afterwards.

                By replacing "Standard CMP 2.x EntityBean" to use a configuration like "Standard Pessimistic CMP 2.x EntityBean" this problem was gone and performance is good - about 50% more than with JBoss4. But some transaction issues remain as mentioned.

                 

                • 5. Re: JBoss 6 performance evaluation
                  jaikiran

                  Rico Neubauer wrote:

                   

                  But some transaction issues remain as mentioned.

                   

                   

                   

                  Besides the performance topic, I also experience transaction problems, if the system is running for some time (12 - 24 hours). But this is another topic...

                   

                  Got a link to that thread?

                  • 6. Re: JBoss 6 performance evaluation
                    mreasy

                    jaikiran pai schrieb:

                     

                    Got a link to that thread?

                     

                    One issue is https://issues.jboss.org/browse/HORNETQ-583 which is HornetQ-internal.

                    Another issue I see seems to derive from its JBoss integration - don't have logs currently, will open-up a new thread when I have access to them.

                    • 7. Re: JBoss 6 performance evaluation
                      mreasy

                      @jaikiran pai

                      see http://community.jboss.org/message/576663 for another issue