JBoss 7.1.1 on AIX 6.1.3.0 gives strange results

[johanvermeij]

We are trying to benchmark JBoss 7.1.1 on AIX 6.1.3.0. (6 Power6 CPU's, 24Gb of memory)

We have also tried with JBoss 7.1.0 and Redhat EAP 6.0, results are the same

Details:

JBoss Standalone

Java version:

java version "1.6.0"

Java(TM) SE Runtime Environment (build pap6460sr7-20091215_02(SR7))

IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 AIX ppc64-64 jvmap6460sr7-20091214_49398 (JIT enabled, AOT enabled)

J9VM - 20091214_049398

JIT  - r9_20091123_13891

GC   - 20091111_AA)

JCL  - 20091202_01

Database: Oracle 10g under Hibernate using ojdbc14_g.jar or ojdbc6_g.jar (we tried both, normally ojdbc6_g.jar is for use with 11g but we tried anyway).

Injector: Apache Jmeter 2.7 injecting 100 streams of 4 transactions each, delivering 50 TPS (each stream injects the same transactions with a different branch number, user number and account number).

We have done the same benchmark on CentOS 6 Linux boxes (HP Proliant and PC), Windows XP and Windows 7

The results of these tests are good, there is nothing out of the ordinary.

Under AIX we see the following strange behaviour (list is not exhaustive).

1. Update transaction trigger frequent optimistic locking errors (rollbacks). By comparison there is no way that I can trigger optimistic locking errors on the HP Proliant, no matter how fast I inject transactions. The HP is a single processor box, the AIX box has 6 CPU's.
2. When we inject non update transactions everything goes well until we start a second Jmeter session from a different client. Errors (see below) start to appear instantly after starting the second session.
3. The AIX box at times shows 100% wait on all processors (we use nmon to view performance).

Errors include the following, all are extremely intermittant.

1. Invalid date formats: 2012-04-06 appears as 0002-04-06 and variations
2. Oracle cannot insert a numeric value because the the value exceeds the maximum allowed for a column.
3. Index out of range errors in Hibernate.
4. Complaints about values not being a number (non numeric data in numeric data item).
5. Null pointer errors.
6. etc.

We have tried many things incl:

1. Using different JBoss version
2. Moving the database to another box
3. Run Jmeter on the AIX box using vnc (to eliminate network issues)
4. Move the exact database export, standalone-full.xml, standalone.conf etc to a Windows box (only to find that all was ok)

It looks like we are missing something here.

All comments will be much appreciated.

[jaikiran]

Johan Vermeij wrote:

Errors include the following, all are extremely intermittant.

1. Invalid date formats: 2012-04-06 appears as 0002-04-06 and variations
2. Oracle cannot insert a numeric value because the the value exceeds the maximum allowed for a column.
3. Index out of range errors in Hibernate.
4. Complaints about values not being a number (non numeric data in numeric data item).
5. Null pointer errors.
6. etc.

If all this works fine on other systems, then the first thing I would check is the JDK/JRE. Are you sure you are using the latest (bug fix) version of that JDK/JRE? You might want to check if there are any bug reports around this.

[jaikiran]

By the way, althought not related to this, we have had at least a couple of threads in this forum where users ran into issues with a specific JDK/JRE version on AIX when using JBoss AS. Like here https://community.jboss.org/message/758708#758708