Benchmark between different versions of Narayana JTA

[taknev]

Hi,

We have an application running using narayana 5.5.32.Final. We recently upgraded it to narayana 5.9.0.Final.

We are seeing a 10% performance degradation.

Is there any benchmark available for between versions?

[mmusgrov]

We do do performance comparisons between different versions of Narayana and we have set off a run to get figures for 5.5.32.Final versus 5.9.0.Final on our own CI infrastructure.

We use JMH to do microbenchmark testing and we tune that using an env variable called JMHARGS.

So first clone our performance repo:

git clone https://github.com/jbosstm/performance

cd performance

export JMHARGS="-t 240 -r 30 -f 3 -wi 5 -i 5"

and then do the two runs:

export version=5.5.32.Final

./build.sh -f narayana/ArjunaJTA/jta/pom.xml clean install -DskipTests -Dnarayana.version=$version

java -classpath narayana/ArjunaJTA/jta/target/classes/ -Dcom.arjuna.ats.arjuna.common.propertiesFile=jbossts-properties.xml -jar narayana/ArjunaJTA/jta/target/benchmarks.jar io.narayana.perf.product.NarayanaComparison.* $JMHARGS -rf csv -rff io.narayana.perf.product.NarayanaComparison-$version.csv

Then do the same for the second version

export version=5.9.0.Final

./build.sh -f narayana/ArjunaJTA/jta/pom.xml clean install -DskipTests -Dnarayana.version=$version

java -classpath narayana/ArjunaJTA/jta/target/classes/ -Dcom.arjuna.ats.arjuna.common.propertiesFile=jbossts-properties.xml -jar narayana/ArjunaJTA/jta/target/benchmarks.jar io.narayana.perf.product.NarayanaComparison.* $JMHARGS -rf csv -rff io.narayana.perf.product.NarayanaComparison-$version.csv

We also have a variety of transaction log stores (the location where we put transaction logs) to get the ACID properties expected from transactions. We normally test against a journaling based store which seems to perform quite well compared with the filesystem based store. You can see which Transaction Manager properties we tune by looking at the jbossts properties file (cat narayana/ArjunaJTA/jta/etc/jbossts-properties.xml)

When we have some figures to share with you we will update this thread.

[mmusgrov]

We performed a micro benchmark runs to get the comparative figures for the two versions you are using. We did not see any significant difference so it would be great if you could give us more information about how we can duplicate the performance degradation on our systems. In particular the bottleneck for transactions is the logging that occurs after the prepare phase of an XA transaction so what kind of storage medium are you using (typically users will use JDBC, an artemis journal or a filesystem) and what are the characteristics of your storage are of particular interest to us.

Our test output was as follows. The *Score* column represents TPS and in this test the difference was 217253 versus 216552 (so a 0.3% degradation which is within tolerance):

"Version","Benchmark","Mode","Threads","Samples","Score","Score Error (99.9%)","Unit"

5.5.32.Final,"io.narayana.perf.product.NarayanaComparison.test","thrpt",240,15,217253.976024,8720.877211,"ops/s"

5.9.0.Final,"io.narayana.perf.product.NarayanaComparison.test","thrpt",240,15,216552.983213,6586.482104,"ops/s"

NB our JMH benchmark method is quite simple: start transaction, enlist two dummy XA resources, commit the transaction. So it would also be useful to know what kind of work your XA resources need to do during the prepare and commit phases of the transaction.

[taknev]

Thanks for the quick response.

Ours is a spring boot application with Postgresql DB (JDBC driver version 42.2.5).

"NB our JMH benchmark method is quite simple: start transaction, enlist two dummy XA resources, commit the transaction. So it would also be useful to know what kind of work your XA resources need to do during the prepare and commit phases of the transaction."

- Our application has extensive DB queries. To simplify I run a job which does all CRUD operations (on ~500 GB dataset).

The job which ran at 8 mins before takes about 10 mins now.

I just saw the below Hibernate issue which I need to investigate more to narrow down.

Issue Navigator - Hibernate JIRA

I need some more time for debugging from my side, before I can give more info. Thanks again for your valuable time.

[tomjenkinson]

Thanks for the response. If I understood your response correctly it would be totally unexpected for a Narayana upgrade to have increased the length of a single transaction by 2 minutes.

Are you doing this standalone or in WildFly or some other container? If you upgraded Hibernate and Narayana from a standalone perspective I wonder if they can be upgraded independently to help isolate where the regression might come from?