14 Replies Latest reply on Oct 6, 2014 3:57 AM by pmm

    Java/OJDBC performance issue with JBoss 7.1.1

      Suggestions Please

       

       

      Guys,  i am actively working on migrating an application running on JBoss 5 to JBoss 7.

       

       

      After migration, I have noticed that the database calls took a huge performance hit.

       

       

      I was using ojdbc14 with a pool(min - 5 max - 100) on JBoss 5 and it was working pretty good for me.

       

      With JBoss 7, I have installed the driver as a module.

       

      All my queries are now taking 10 times longer.

      For ex, if a query took 30ms while I was on JBoss 5 , it is taking 400-600ms on JBoss 7.

       

      I have tried with both the drivers shown in the config below (ojdbc6, ojdbc14)

       

      An Observation is that , the perfomance decrease is more noticable on a linux box(i.e. terribly slower) than a OS X box.

       

      FYI. I have tried running JBoss 7.1.1Final on both Java 1.6 and 1.7

       

      My application itself contains:

      - struts 2 front end (used for request processing, no web ui involved)

      - Session Beans in the backend

      - QuartzPlugin for batch jobs.

      - a custom MBean

       

      I have tried

      1. lowering and increasing the pool size

      2. prefill = true and false

      3. switching between ojdbc6 and ojdbc14

       

       

       

       

      {code:xml}

              <subsystem xmlns="urn:jboss:domain:datasources:1.0">

                  <datasources>

                      <datasource jta="true" jndi-name="java:jboss/datasources/MyDS" pool-name="hive-datasource" enabled="true" use-java-context="true" use-ccm="true">

                          <connection-url>jdbc:oracle:thin:@host:port:service</connection-url>

                          <driver-class>oracle.jdbc.driver.OracleDriver</driver-class>

                          <connection-property name="defaultRowPrefetch">

                              50

                          </connection-property>

                          <driver>oracle14</driver>

                          <pool>

                              <min-pool-size>20</min-pool-size>

                              <max-pool-size>100</max-pool-size>

                              <prefill>true</prefill>

                              <use-strict-min>false</use-strict-min>

                              <flush-strategy>FailingConnectionOnly</flush-strategy>

                          </pool>

                          <security>

                              <user-name>theusername</user-name>

                              <password>thepassword</password>

                          </security>

                          <validation>

                              <check-valid-connection-sql>select * from dual</check-valid-connection-sql>

                              <validate-on-match>false</validate-on-match>

                              <background-validation>false</background-validation>

                              <use-fast-fail>false</use-fast-fail>

                              <exception-sorter class-name="org.jboss.resource.adapter.jdbc.vendor.OracleExceptionSorter"/>

                          </validation>

                          <timeout>

                              <set-tx-query-timeout>true</set-tx-query-timeout>

                              <blocking-timeout-millis>300000</blocking-timeout-millis>

                              <idle-timeout-minutes>30</idle-timeout-minutes>

                          </timeout>

                          <statement>

                              <track-statements>false</track-statements>

                              <prepared-statement-cache-size>0</prepared-statement-cache-size>

                          </statement>

                      </datasource>

                      <drivers>

                          <driver name="oracle6" module="com.oracle.ojdbc6">

            <xa-datasource-class>oracle.jdbc.xa.client.OracleXADataSource</xa-datasource-class>

                          </driver>

                          <driver name="oracle14" module="com.oracle.ojdbc14">

                              <driver-class>oracle.jdbc.driver.OracleDriver</driver-class>

                          </driver>

                      </drivers>

                  </datasources>

              </subsystem>

      {code}

       

       

        • 1. Re: Oracle/ OJDBC performance issue with JBoss 7.1.1
          nickarls

          Can you determine if it's the actual query that takes a long time or the getting of the connection?

          • 2. Re: Oracle/ OJDBC performance issue with JBoss 7.1.1

            I have already timed the queries on both Jboss 5 and Jboss 7, the queries are just fine on Jboss 5.

             

            for example one of the query is taking 30ms on JBoss5 while it takes 490ms on JBoss 7.  I am giving you an average number after repeated iterations.

             

            Also, On Jboss 5 when i get the connection by looking a jndi lookup on datasource, it executes in 1-2ms, whereas the same takes 40ms on JBoss 7

            • 3. Re: Oracle/ OJDBC performance issue with JBoss 7.1.1
              nickarls

              Have you tried enabling trace logging in order to pin down the problem?

              • 4. Re: Oracle/ OJDBC performance issue with JBoss 7.1.1

                Not yet, I am going to do that next. One thing that i notice after further timing these is that it is not just the DB operations that are running slow. But just about everything(i.e. the regular java methods) are running slow and i tend to notice the performance degradion more on the database as I am timing it in my logs.

                 

                 

                Also, i am noticing this performance reduction on Linux in particular. the box that I am running on has a pretty good Xeon processor with 12G ram.

                 

                On OSX i.e. my local laptop (connecting to same database running on a remote machine) , the performance of JBoss 5 and JBoss 7 are similar.

                 

                Also, noticed that the cpu and memory usage of the 'java' process is normal as well.

                • 5. Re: Oracle/ OJDBC performance issue with JBoss 7.1.1
                  nickarls

                  Does it slow down over time? How is the performance on a fresh-booted instance?

                  • 6. Re: Oracle/ OJDBC performance issue with JBoss 7.1.1
                    wdfink

                    BTW what you use for persistence? EJB2 CMP/BMP JPA or SQL?

                     

                    Do you see that the performance is slow over all or only if you have IO.

                    I run a Fedora17@Xeon and did not see such differences

                    • 7. Re: Oracle/ OJDBC performance issue with JBoss 7.1.1
                      jesper.pedersen

                      Retest with WildFly 8.0.0.Alpha1

                      • 8. Re: Oracle/ OJDBC performance issue with JBoss 7.1.1

                        The JBoss with the datasource configured and without any deployments starts up in about 6-7seconds.

                         

                        The deployment/app start up takes about 10-15 secs which is not bad.

                         

                        But the performance of the application itself it bad right from the beginning.

                         

                        Also, if it might help, i have noticed that the deployment scanner does not recognize the *.dodeploy instantaneously on the Linux box, it takes anywhere from 5sec - 2 min to start the deployment. I have left the scan interval to the default 5sec and turned off the auto deploy

                         

                        the deployment scanner is instantaneous on my OS X laptop though.

                        :

                        • 9. Re: Oracle/ OJDBC performance issue with JBoss 7.1.1

                          We use plain JDBC with theh datasource configured as a pool in JBoss. I tried setting prefill = true but that didnt make a difference.

                          • 10. Re: Oracle/ OJDBC performance issue with JBoss 7.1.1

                            Thanks for the suggestion. I have tried the application today on 7.2.0 Release by getting it from github

                             

                            The application is way faster now. The only thing that i think is slow is the web server performance which probably can by tuned.

                             

                            i have use the abolute same setting, i.e. i copied my standalone.conf and standalone.xml file to the newer build.

                             

                            But since this is not a true RELEASE, it would be difficult/impossible for it to be accepted in my production environment.

                             

                             

                             

                            Also, I have observed that the performance degradation on JBoss 7.1.1 was not just specific to JDBC. I see the degradation in entire application.

                             

                            For example, a piece of code that creates a JSON response out of data retrieved from DB executes in 2ms on JBoss 5 running on the same box. The same code takes 4-5 sec on JBoss 7.

                            • 11. Re: Oracle/ OJDBC performance issue with JBoss 7.1.1
                              jaikiran

                              Kalyan Kantam wrote:

                               

                               

                              But since this is not a true RELEASE, it would be difficult/impossible for it to be accepted in my production environment.

                               

                              An actual release is here http://wildfly.org/download/

                              • 12. Re: Oracle/ OJDBC performance issue with JBoss 7.1.1

                                Since, Wildfly is in ALPHA as well, i would face the same problem that I would face with getting 7.2 accepted for Production.

                                 

                                But now that, I have seen the app run good in 7.2.0 with the same standalone.conf/xml i think that isolating the problem in 7.1.1.Final would be an optimal solution for me. I would continue to research.

                                 

                                Please post any suggestions.

                                 

                                FYI... the only configuration i have done in the standalone.xml is for logging, datasource and http executors.

                                • 13. Re: Java/OJDBC performance issue with JBoss 7.1.1
                                  dnovo

                                  Hi Kalyan,

                                   

                                  I'm also trying to migrate my app from Jboss AS6.1 to Wildfly 8.1 and i'm facing the same problem.

                                   

                                  Do you find anything?

                                   

                                  The server boot is fast, but de overall application performance is slower.

                                   

                                  I'm also moving from Java 6 to 7.

                                   

                                  Thanks

                                  • 14. Re: Re: Java/OJDBC performance issue with JBoss 7.1.1
                                    pmm

                                    The following configuration works fine for us

                                     

                                    <datasources>
                                      <datasource jndi-name="java:/global/datasources/OracleAcmeDS" pool-name="AcmeDS">
                                      <connection-url>${com.acme.db.url}</connection-url>
                                      <driver>oracle</driver>
                                      <connection-property name="v$session.program">ACME App</connection-property>
                                      <timeout>
                                      <set-tx-query-timeout>true</set-tx-query-timeout>
                                      <query-timeout>300</query-timeout>
                                      </timeout>
                                      <pool>
                                      <min-pool-size>5</min-pool-size>
                                      <max-pool-size>60</max-pool-size>
                                      <prefill>true</prefill>
                                      </pool>
                                      <security>
                                      <user-name>scott</user-name>
                                      <password>tiger</password>
                                      </security>
                                      <validation>
                                      <valid-connection-checker class-name="org.jboss.jca.adapters.jdbc.extensions.oracle.OracleValidConnectionChecker"/>
                                      <stale-connection-checker class-name="org.jboss.jca.adapters.jdbc.extensions.oracle.OracleStaleConnectionChecker"/>
                                      <exception-sorter class-name="org.jboss.jca.adapters.jdbc.extensions.oracle.OracleExceptionSorter"/>
                                      </validation>
                                      </datasource>
                                      <drivers>
                                      <driver name="oracle" module="com.oracle.jdbc">
                                      <driver-class>oracle.jdbc.OracleDriver</driver-class>
                                      </driver>
                                      </drivers>
                                      </datasources>
                                    
                                    

                                     

                                    We deploy the ojdbc7-12.1.0.2 as a module:

                                    • ojdbc7 because we're on Java 8, we will not get AbstractMethodErrors "only" Java 8 Date and Time classes will not work
                                    • version 12 because buffer management is massively better than in version 11, if you check the Interoperability Matrix version 12 of the driver works with versions 12 to 10.2 of the database
                                    • 12.1.0.2 because 12.1.0.1 has a horrible bug when it comes to statement metadata that causes IronJacamar to jank the connection from the pool (which is correct I believe)

                                     

                                    Not a lot of though has gone into our configuration (except the pool size). We more or less stick with the defaults from the IronJacamar documentation.