13 Replies Latest reply on Apr 15, 2009 1:39 PM by frank.feinbube

    JBoss Performance Problems with SPECjAppServer2004

    frank.feinbube

      I am benchmarking JBoss with the SPECjAppServer2004-Benchmark for my master thesis.

      When I use an Injectionrate of 30 (which is really low) or higher, I get lots of connection problems on the driver side. These lead to a very poor Benchmark result.

      I am working on my master thesis on comparison of J2EE app servers using the SPECjAppServer2004 benchmark. I am using the JBoss SPECj kit and the current version of the benchmark (1.08).

      I finally made it run, but there are still lots of error message which result in a very bad performance.


      Here is the problem:

      On the server-side:

      10:18:19,952 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator_4] TwoPhaseCoordinator.afterCompletion - returned failure for com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple@29917

      07:24:09,895 WARN [loggerI18N] [com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa] [com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa]
      Could not find new XAResource to use for recovering non-serializable XAResource
      < 131075, 31, 29, 1--53efe8f4:bf4:4990ac87:24f9b5-53efe8f4:bf4:4990ac87:24f9cd
      >

      On the client-side:

      Error: Connection: IOException: java.net.ConnectException: Connection refused: connect

      Error: Connection: BindException: java.net.BindException: Address already in use: connect


      I hope you can help me make the JBoss perform well in the Benchmark.

      Best regards,
      Frank Feinbube


        • 1. Re: JBoss Performance Problems with SPECjAppServer2004
          frank.feinbube

          I tried a lot in solving the problem.

          My current guess is, that it has to do with the MySQL 5.1 Database Management System.


          M1:0: Error occured in scheduleWorkOrderjava.rmi.ServerException: EJBException:; nested exception is:
          javax.ejb.EJBException: javax.ejb.TransactionRolledbackLocalException: javax.ejb.FinderException: Could not create connection; - nested throwable: (java.sql.SQLException: null, message from server: "Can't create a new thread (errno 12); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug"); - nested throwable: (org.jboss.resource.JBossResourceException: Could not create connection; - nested throwable: (java.sql.SQLException: null, message from server: "Can't create a new thread (errno 12); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug"))

          • 2. Re: JBoss Performance Problems with SPECjAppServer2004
            peterj

            We have run specjappserver with injection rates in the high hundreds, using MySQL as the database, so it is possibly.

            How many connections are you allowing in MySQL? Both the database and the *-ds.xml file must agree on the max number of connections, and I recommend setting both min and max connections to the same value. Also, I think that you need to set close you the IR*8 because each request will need a connection.

            Also, have you seen:
            http://www.jboss.org/community/docs/DOC-11980
            http://www.jboss.org/community/wiki/SPECjAppServer2002

            • 3. Re: JBoss Performance Problems with SPECjAppServer2004
              frank.feinbube

              Thanks for the reply. I used the kit from this page:
              http://www.jboss.org/community/docs/DOC-11980

              This way I was able to make the benchmark run. I spend a lot time in playing with the connections pools.

              My latest configuration was 2000 connections on the mysql-server site and about 1000 connections on the jboss site. But I always had the problems I described.

              Why do I need a connection for every request?

              Perhaps this problem only occurs when combining JBoss, MySQL and Windows ;)

              • 4. Re: JBoss Performance Problems with SPECjAppServer2004
                peterj

                 

                Frank wrote:
                Why do I need a connection for every request?


                Because every request that is made will cause some interaction with the database. Therefore, each request needs a connection.

                Also, you have to realize that every connection in the database creates a new thread in the database. You can monitor the number of connections that the database has - perhaps you have reached the limit on the number of threads it can run. What OS are you using?

                A better approach might be to have only a few connection, say setting max connections to 100, and see how far you can push the IR. Keep track of the queue depth (that will tell you how many app server threads are waiting for an available database connection), and then increase the database connections by a small number, say 20, at a time.

                • 5. Re: JBoss Performance Problems with SPECjAppServer2004
                  frank.feinbube

                  I tried the following configuration:

                  mysql-ds.xml

                  <?xml version="1.0" encoding="UTF-8"?>
                  
                  <datasources>
                   <xa-datasource>
                   <jndi-name>DefaultDS</jndi-name>
                  
                   <xa-datasource-class>com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</xa-datasource-class>
                   <xa-datasource-property name="URL">jdbc:mysql://DBHost/specj</xa-datasource-property>
                   <xa-datasource-property name="User">specj2004</xa-datasource-property>
                   <xa-datasource-property name="Password">*****</xa-datasource-property>
                  
                   <track-connection-by-tx>true</track-connection-by-tx>
                   <no-tx-separate-pools>true</no-tx-separate-pools>
                  
                  
                   <min-pool-size>20</min-pool-size>
                   <max-pool-size>20</max-pool-size>
                  
                   <blocking-timeout-millis>15001</blocking-timeout-millis>
                   <idle-timeout-minutes>15</idle-timeout-minutes>
                  
                   <transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation>
                  
                   <metadata>
                   <type-mapping>mySQL</type-mapping>
                   </metadata>
                   </xa-datasource>
                  
                   <xa-datasource>
                   <jndi-name>SpecjDS_RC</jndi-name>
                  
                   <xa-datasource-class>com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</xa-datasource-class>
                   <xa-datasource-property name="URL">jdbc:mysql://DBHost/specj</xa-datasource-property>
                   <xa-datasource-property name="User">specj2004</xa-datasource-property>
                   <xa-datasource-property name="Password">*****</xa-datasource-property>
                  
                   <track-connection-by-tx>true</track-connection-by-tx>
                   <no-tx-separate-pools>true</no-tx-separate-pools>
                  
                   <min-pool-size>320</min-pool-size>
                   <max-pool-size>320</max-pool-size>
                  
                   <prepared-statement-cache-size>512</prepared-statement-cache-size>
                  
                   <blocking-timeout-millis>60002</blocking-timeout-millis>
                   <idle-timeout-minutes>90</idle-timeout-minutes>
                  
                   <transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation>
                  
                   <metadata>
                   <type-mapping>mySQL</type-mapping>
                   </metadata>
                   </xa-datasource>
                  
                   <no-tx-datasource>
                   <jndi-name>SpecjJMSDS</jndi-name>
                   <connection-url>jdbc:mysql://DBHost/jms</connection-url>
                   <driver-class>com.mysql.jdbc.Driver</driver-class>
                   <user-name>jbossjms</user-name>
                   <password>*****</password>
                  
                   <min-pool-size>20</min-pool-size>
                   <max-pool-size>20</max-pool-size>
                  
                   <prepared-statement-cache-size>512</prepared-statement-cache-size>
                  
                   <blocking-timeout-millis>60003</blocking-timeout-millis>
                   <idle-timeout-minutes>15</idle-timeout-minutes>
                  
                   <metadata>
                   <type-mapping>mySQL</type-mapping>
                   </metadata>
                  
                   </no-tx-datasource>
                  
                  </datasources>


                  my.ini
                  [client]
                  port=3306
                  
                  [mysql]
                  default-character-set=latin1
                  
                  [mysqld]
                  max_connections=800
                  thread_cache_size=38
                  
                  port=3306
                  lower_case_table_names = 1
                  sql-mode = IGNORE_SPACE
                  basedir="C:/Program Files/MySQL/MySQL Server 5.1/"
                  datadir="C:/Documents and Settings/All Users/Application Data/MySQL/MySQL Server 5.1/Data/"
                  default-character-set=latin1
                  default-storage-engine=INNODB
                  query_cache_size=84M
                  table_cache=1520
                  tmp_table_size=30M
                  
                  myisam_max_sort_file_size=100G
                  myisam_sort_buffer_size=30M
                  key_buffer_size=129M
                  read_buffer_size=64K
                  read_rnd_buffer_size=256K
                  sort_buffer_size=256K
                  
                  
                  #*** INNODB Specific options ***
                  innodb_additional_mem_pool_size=6M
                  innodb_flush_log_at_trx_commit=1
                  innodb_log_buffer_size=3M
                  innodb_buffer_pool_size=250M
                  innodb_log_file_size=50M
                  innodb_thread_concurrency=8
                  


                  This should work for an Injectionrate of 40. But I got lots of errors.

                  • 6. Re: JBoss Performance Problems with SPECjAppServer2004
                    frank.feinbube

                    The OS I am using is Windows XP with SP3.

                    (I am sorry for that ;) )

                    • 7. Re: JBoss Performance Problems with SPECjAppServer2004
                      frank.feinbube

                      I still have all the errors mentioned before. And I have no ideas anymore.

                      I tried small thread_cache_size (8) and big ones (2000). I tried to increase connections (up to 2000) on mysql-site. I tried lots of configurations for the mysql-ds.xml.

                      But I get the same errors every time.

                      Perhaps its really a problem of the Windows XP + JBoss 4.2.3 + MySQL 5.1 combination.

                      Any suggestions?

                      I will try JBoss 5.0.0. Perhaps he will be nicer to me.

                      • 8. Re: JBoss Performance Problems with SPECjAppServer2004
                        peterj

                        With the settings you posted, what is the highest IR you get without any errors?

                        What is your hardware setup? Everything running on one box? Or multiple boxes? What is the size (RAM, CPU speed, number of cores) of each box?

                        • 9. Re: JBoss Performance Problems with SPECjAppServer2004
                          frank.feinbube

                          I have one real Node per Role:

                          Emulator: Intel E6600 @ 2.4 Ghz, 3.00 GB
                          Database: Intel E6600 @ 2.4 Ghz, 3.00 GB
                          Driver: Intel E8400 @ 3.0 Ghz, 3.25 GB
                          Appserver: Intel E8400 @ 3.0 Ghz, 3.25 GB
                          (every CPU has 2 cores)

                          The highest IR I was able to achieve without errors was 20. (The next I tried was 30.)

                          The GlassFish application server was able to perform an IR of 100 and more without any problems.

                          • 10. Re: JBoss Performance Problems with SPECjAppServer2004
                            peterj

                            According to this post: http://lists.mysql.com/mysql/212112, try decreasing the max-connections in my.ini to 400 (or even 360 - that it the most connections you will make based on the *-ds.xml file setting).

                            Also, monitor the thread count on the mysqld.exe process - this will give a rough idea of how many connections you have. Or you can use the MySQL Administrator to monitor the number of connections. I'm sure there is also a query you can run in the mysql command-line client to get this info, but I don't know the syntax offhand.

                            Another thing to consider is the memory usage on the database node. Are you, by any chance, booting that system with the /3GB boot option?

                            • 11. Re: JBoss Performance Problems with SPECjAppServer2004
                              frank.feinbube

                              Actually I tried to set the max_connections count to 400 in the first place. But this led to errors on the JBoss stating that the pool could not be filled. So I increased it to 800 which worked fine. (At least for the startup of JBoss. And the preparation phase of the benchmark.)

                              The overall thread count of the database node is about 560 when running the benchmark. Which is 60 threads more than in the idle state. This is the case even when I set the thread_cache_size to 2000.

                              I think that I am not using the /3GB option. I don't even know what you mean by it ;)
                              But actually I monitored for memory paging errors and RAM utilization and there is no problem. In addition as I mentioned before GlassFish had no problems with this configuration. So I think that the interaction of JBoss, MySQL and Windows are causing the problem.

                              • 12. Re: JBoss Performance Problems with SPECjAppServer2004
                                frank.feinbube

                                Unfortunately JBoss 5.0.1.GA seem to be no solution either cause it doesn't work with the kit. :(

                                • 13. Re: JBoss Performance Problems with SPECjAppServer2004
                                  frank.feinbube

                                  Has anyone any further suggestions? I am running out of time and it seems that no one has a solution to this problem.