We are trying to benchmark JBoss 7.1.1 on AIX 126.96.36.199. (6 Power6 CPU's, 24Gb of memory)
We have also tried with JBoss 7.1.0 and Redhat EAP 6.0, results are the same
java version "1.6.0"
Java(TM) SE Runtime Environment (build pap6460sr7-20091215_02(SR7))
IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 AIX ppc64-64 jvmap6460sr7-20091214_49398 (JIT enabled, AOT enabled)
J9VM - 20091214_049398
JIT - r9_20091123_13891
GC - 20091111_AA)
JCL - 20091202_01
Database: Oracle 10g under Hibernate using ojdbc14_g.jar or ojdbc6_g.jar (we tried both, normally ojdbc6_g.jar is for use with 11g but we tried anyway).
Injector: Apache Jmeter 2.7 injecting 100 streams of 4 transactions each, delivering 50 TPS (each stream injects the same transactions with a different branch number, user number and account number).
We have done the same benchmark on CentOS 6 Linux boxes (HP Proliant and PC), Windows XP and Windows 7
The results of these tests are good, there is nothing out of the ordinary.
Under AIX we see the following strange behaviour (list is not exhaustive).
- Update transaction trigger frequent optimistic locking errors (rollbacks). By comparison there is no way that I can trigger optimistic locking errors on the HP Proliant, no matter how fast I inject transactions. The HP is a single processor box, the AIX box has 6 CPU's.
- When we inject non update transactions everything goes well until we start a second Jmeter session from a different client. Errors (see below) start to appear instantly after starting the second session.
- The AIX box at times shows 100% wait on all processors (we use nmon to view performance).
Errors include the following, all are extremely intermittant.
- Invalid date formats: 2012-04-06 appears as 0002-04-06 and variations
- Oracle cannot insert a numeric value because the the value exceeds the maximum allowed for a column.
- Index out of range errors in Hibernate.
- Complaints about values not being a number (non numeric data in numeric data item).
- Null pointer errors.
We have tried many things incl:
- Using different JBoss version
- Moving the database to another box
- Run Jmeter on the AIX box using vnc (to eliminate network issues)
- Move the exact database export, standalone-full.xml, standalone.conf etc to a Windows box (only to find that all was ok)
It looks like we are missing something here.
All comments will be much appreciated.