JBoss 6 performance evaluation
mreasy Dec 6, 2010 8:36 AMHi all,
First of all, thanks for all your effort so far and what is still to come, for releasing JBoss 6.
We are building an Business-Process-application, which curently runs on JBoss 4.0.5 with JBossMQ, which we plan to migrate to JBoss6. This application heavily uses JMS messaging, which we hoped to benefit from the HornetQ integration.
Unfortunately the tests we ran so far were disappointing and we reached a thruput which only reaches 30% using JBoss6 CR1 of the same test with JBoss4. Maybe you can share your obervations if you have comparisions between JBoss6 and 5/4, or raise points where we could improve set-up / configuration / etc. I'll describe the test in the following. It is clear that JBoss6 is not yet GA, but afaik there is no known issue in this area that should get addressed until GA.
Setup
CPU, memory, filesystem and DB should not be limitating factors (see spec attached at end), DB-load decreases by nearly 80% which is partly expected since JMS traffic goes from DB to filesystem.
HornetQ is using default configuration aside from the No. and size of the data files:
<!-- Default journal file size is set to 1MB for fast booting, we use 10MB here --> <journal-file-size>${hornetq.journal.file.size:10485760}</journal-file-size> <!-- Default journal min file is 2, increase for higher average msg rates, we use 5 here --> <journal-min-files>${hornetq.journal.min.files:5}</journal-min-files>
Tested HornetQ with NIO and AIO, which did not make a measureable difference, since the filesystem has a battery-backed buffer anyways.
It is clear, that the thruput is limited by the No. of possible fsyncs, but that should not be the limiting factor here (also see Thread#19114 discussing this).
Further tests included:
- set-up on other machines, which showed the same relation before/after.
- Putting HornetQ files (journal, data, ...) on a RAM-disk - no real difference
- Putting Arjuna transaction files on a RAM-disk - no real difference
Observations
Thruput reaches 30% as mentioned. The load on the JBoss machine goes into sky, see screenshot below. But also using a profiler, it is not easy to tell what uses that much CPU (I/O waits which also count into the load are not seen that much). There is not 'that one hotspot' nor locking.
Open Questions
- Is this expected /known?
- Is there a chance that the poor performance derives from using beans using standard EJB 2.1 and maybe EJB 3 beans would fit-in better?
- Is HornetQ the bottleneck? From the benchmarks, that are known I would not say this, but maybe there are known limitations with the integration?
Besides the performance topic, I also experience transaction problems, if the system is running for some time (12 - 24 hours). But this is another topic...
For information machine and VM details:
VM Arguments:
jvm_args: -XX:NewRatio=1 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:PermSize=100M -XX:MaxPermSize=300M -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -XX:+HeapDumpOnOutOfMemoryError -Xloggc:gc.log -Xverify:none -Xms4G -Xmx8G -Djava.awt.headless=true -Djava.endorsed.dirs=/home/username/MyApp/runtime/jboss/lib/endorsed -Djava.io.tmpdir=/home/username/MyApp/temp -Djboss.home.dir=/home/username/MyApp/runtime/jboss -Djboss.server.base.dir=/home/username/MyApp/jboss -Djboss.server.base.url=/home/username/MyApp/jboss/ -Djboss.server.log.dir=/home/username/MyApp/log/jboss -Djboss.server.temp.dir=/home/username/MyApp/temp/jboss -Djboss.server.data.dir=/home/username/MyApp/data/jboss -Dcatalina.base=/home/username/MyApp/temp/tomcat -Dcatalina.home=/home/username/MyApp/temp/tomcat -Dlogging.configuration=file:/home/username/MyApp/jboss/SeeBisAs/conf/logging.properties -Djava.library.path=/home/username/MyApp/runtime/jboss/bin/native/lib64 -Djava.endorsed.dirs=/home/username/MyApp/runtime/jboss/lib/endorsed
java_command: org.jboss.Main -c MyApp -b 0.0.0.0
--------------- S Y S T E M ---------------
OS:Red Hat Enterprise Linux Server release 5.5 (Tikanga)
uname:Linux 2.6.18-194.26.1.el5 #1 SMP Fri Oct 29 14:21:16 EDT 2010 x86_64
libc:glibc 2.5 NPTL 2.5
rlimit: STACK 10240k, CORE 0k, NPROC 278528, NOFILE 65536, AS infinity
CPU:total 8 (4 cores per cpu, 1 threads per core) family 6 model 15 stepping 11, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3
Memory: 4k page, physical 20544364k(3235836k free), swap 8388600k(8388584k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (17.1-b03) for linux-amd64 JRE (1.6.0_22-b04), built on Sep 15 2010 01:07:59 by "java_re" with gcc 3.2.2 (SuSE Linux)