4 Replies Latest reply on Dec 5, 2019 11:12 AM by Thomas Jenkinson

    Benchmark between different versions of Narayana JTA

    Venkatramanan Swaminathan Newbie

      Hi,

      We have an application running using narayana 5.5.32.Final. We recently upgraded it to narayana 5.9.0.Final.

      We are seeing a 10% performance degradation.

      Is there any benchmark available for between versions?

        • 1. Re: Benchmark between different versions of Narayana JTA
          Michael Musgrove Master

          We do do performance comparisons between different versions of Narayana and we have set off a run to get figures for 5.5.32.Final versus 5.9.0.Final on our own CI infrastructure.

           

          We use JMH to do microbenchmark testing and we tune that using an env variable called JMHARGS.

           

          So first clone our performance repo:

           

          git clone https://github.com/jbosstm/performance

          cd performance

          export JMHARGS="-t 240 -r 30 -f 3 -wi 5 -i 5"

           

          and then do the two runs:

           

          export version=5.5.32.Final

          ./build.sh -f narayana/ArjunaJTA/jta/pom.xml clean install -DskipTests -Dnarayana.version=$version

          java -classpath narayana/ArjunaJTA/jta/target/classes/ -Dcom.arjuna.ats.arjuna.common.propertiesFile=jbossts-properties.xml -jar narayana/ArjunaJTA/jta/target/benchmarks.jar io.narayana.perf.product.NarayanaComparison.* $JMHARGS -rf csv -rff io.narayana.perf.product.NarayanaComparison-$version.csv

           

          Then do the same for the second version

           

          export version=5.9.0.Final

          ./build.sh -f narayana/ArjunaJTA/jta/pom.xml clean install -DskipTests -Dnarayana.version=$version

          java -classpath narayana/ArjunaJTA/jta/target/classes/ -Dcom.arjuna.ats.arjuna.common.propertiesFile=jbossts-properties.xml -jar narayana/ArjunaJTA/jta/target/benchmarks.jar io.narayana.perf.product.NarayanaComparison.* $JMHARGS -rf csv -rff io.narayana.perf.product.NarayanaComparison-$version.csv

           

          We also have a variety of transaction log stores (the location where we put transaction logs) to get the ACID properties expected from transactions. We normally test against a journaling based store which seems to perform quite well compared with the filesystem based store. You can see which Transaction Manager properties we tune by looking at the jbossts properties file (cat narayana/ArjunaJTA/jta/etc/jbossts-properties.xml)

           

          When we have some figures to share with you we will update this thread.

          • 2. Re: Benchmark between different versions of Narayana JTA
            Michael Musgrove Master

            We performed a micro benchmark runs to get the comparative figures for the two versions you are using. We did not see any significant difference so it would be great if you could give us more information about how we can duplicate the performance degradation on our systems. In particular the bottleneck for transactions is the logging that occurs after the prepare phase of an XA transaction so what kind of storage medium are you using (typically users will use JDBC, an artemis journal or a filesystem) and what are the characteristics of your storage are of particular interest to us.

             

            Our test output was as follows. The *Score* column represents TPS and in this test the difference was 217253 versus 216552 (so a 0.3% degradation which is within tolerance):

             

            "Version","Benchmark","Mode","Threads","Samples","Score","Score Error (99.9%)","Unit"

            5.5.32.Final,"io.narayana.perf.product.NarayanaComparison.test","thrpt",240,15,217253.976024,8720.877211,"ops/s"

            5.9.0.Final,"io.narayana.perf.product.NarayanaComparison.test","thrpt",240,15,216552.983213,6586.482104,"ops/s"

             

            NB our JMH benchmark method is quite simple: start transaction, enlist two dummy XA resources, commit the transaction. So it would also be useful to know what kind of work your XA resources need to do during the prepare and commit phases of the transaction.

            • 3. Re: Benchmark between different versions of Narayana JTA
              Venkatramanan Swaminathan Newbie

              Thanks for the quick response.

               

              Ours is a spring boot application with Postgresql DB (JDBC driver version 42.2.5).

               

              "NB our JMH benchmark method is quite simple: start transaction, enlist two dummy XA resources, commit the transaction. So it would also be useful to know what kind of work your XA resources need to do during the prepare and commit phases of the transaction."

              - Our application has extensive DB queries. To simplify I run a job which does all CRUD operations (on ~500 GB dataset).

              The job which ran at 8 mins before takes about 10 mins now.

               

              I just saw the below Hibernate issue which I need to investigate more to narrow down.

              Issue Navigator - Hibernate JIRA

               

              I need some more time for debugging from my side, before I can give more info. Thanks again for your valuable time.

              • 4. Re: Benchmark between different versions of Narayana JTA
                Thomas Jenkinson Master

                Thanks for the response. If I understood your response correctly it would be totally unexpected for a Narayana upgrade to have increased the length of a single transaction by 2 minutes.

                 

                Are you doing this standalone or in WildFly or some other container? If you upgraded Hibernate and Narayana from a standalone perspective I wonder if they can be upgraded independently to help isolate where the regression might come from?