-
1. Re: JBoss Performance Problems with SPECjAppServer2004
frank.feinbube Apr 7, 2009 9:54 AM (in response to frank.feinbube)I tried a lot in solving the problem.
My current guess is, that it has to do with the MySQL 5.1 Database Management System.
M1:0: Error occured in scheduleWorkOrderjava.rmi.ServerException: EJBException:; nested exception is:
javax.ejb.EJBException: javax.ejb.TransactionRolledbackLocalException: javax.ejb.FinderException: Could not create connection; - nested throwable: (java.sql.SQLException: null, message from server: "Can't create a new thread (errno 12); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug"); - nested throwable: (org.jboss.resource.JBossResourceException: Could not create connection; - nested throwable: (java.sql.SQLException: null, message from server: "Can't create a new thread (errno 12); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug")) -
2. Re: JBoss Performance Problems with SPECjAppServer2004
peterj Apr 7, 2009 12:02 PM (in response to frank.feinbube)We have run specjappserver with injection rates in the high hundreds, using MySQL as the database, so it is possibly.
How many connections are you allowing in MySQL? Both the database and the *-ds.xml file must agree on the max number of connections, and I recommend setting both min and max connections to the same value. Also, I think that you need to set close you the IR*8 because each request will need a connection.
Also, have you seen:
http://www.jboss.org/community/docs/DOC-11980
http://www.jboss.org/community/wiki/SPECjAppServer2002 -
3. Re: JBoss Performance Problems with SPECjAppServer2004
frank.feinbube Apr 7, 2009 4:35 PM (in response to frank.feinbube)Thanks for the reply. I used the kit from this page:
http://www.jboss.org/community/docs/DOC-11980
This way I was able to make the benchmark run. I spend a lot time in playing with the connections pools.
My latest configuration was 2000 connections on the mysql-server site and about 1000 connections on the jboss site. But I always had the problems I described.
Why do I need a connection for every request?
Perhaps this problem only occurs when combining JBoss, MySQL and Windows ;) -
4. Re: JBoss Performance Problems with SPECjAppServer2004
peterj Apr 7, 2009 6:10 PM (in response to frank.feinbube)Frank wrote:
Why do I need a connection for every request?
Because every request that is made will cause some interaction with the database. Therefore, each request needs a connection.
Also, you have to realize that every connection in the database creates a new thread in the database. You can monitor the number of connections that the database has - perhaps you have reached the limit on the number of threads it can run. What OS are you using?
A better approach might be to have only a few connection, say setting max connections to 100, and see how far you can push the IR. Keep track of the queue depth (that will tell you how many app server threads are waiting for an available database connection), and then increase the database connections by a small number, say 20, at a time. -
5. Re: JBoss Performance Problems with SPECjAppServer2004
frank.feinbube Apr 8, 2009 2:36 AM (in response to frank.feinbube)I tried the following configuration:
mysql-ds.xml<?xml version="1.0" encoding="UTF-8"?> <datasources> <xa-datasource> <jndi-name>DefaultDS</jndi-name> <xa-datasource-class>com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</xa-datasource-class> <xa-datasource-property name="URL">jdbc:mysql://DBHost/specj</xa-datasource-property> <xa-datasource-property name="User">specj2004</xa-datasource-property> <xa-datasource-property name="Password">*****</xa-datasource-property> <track-connection-by-tx>true</track-connection-by-tx> <no-tx-separate-pools>true</no-tx-separate-pools> <min-pool-size>20</min-pool-size> <max-pool-size>20</max-pool-size> <blocking-timeout-millis>15001</blocking-timeout-millis> <idle-timeout-minutes>15</idle-timeout-minutes> <transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation> <metadata> <type-mapping>mySQL</type-mapping> </metadata> </xa-datasource> <xa-datasource> <jndi-name>SpecjDS_RC</jndi-name> <xa-datasource-class>com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</xa-datasource-class> <xa-datasource-property name="URL">jdbc:mysql://DBHost/specj</xa-datasource-property> <xa-datasource-property name="User">specj2004</xa-datasource-property> <xa-datasource-property name="Password">*****</xa-datasource-property> <track-connection-by-tx>true</track-connection-by-tx> <no-tx-separate-pools>true</no-tx-separate-pools> <min-pool-size>320</min-pool-size> <max-pool-size>320</max-pool-size> <prepared-statement-cache-size>512</prepared-statement-cache-size> <blocking-timeout-millis>60002</blocking-timeout-millis> <idle-timeout-minutes>90</idle-timeout-minutes> <transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation> <metadata> <type-mapping>mySQL</type-mapping> </metadata> </xa-datasource> <no-tx-datasource> <jndi-name>SpecjJMSDS</jndi-name> <connection-url>jdbc:mysql://DBHost/jms</connection-url> <driver-class>com.mysql.jdbc.Driver</driver-class> <user-name>jbossjms</user-name> <password>*****</password> <min-pool-size>20</min-pool-size> <max-pool-size>20</max-pool-size> <prepared-statement-cache-size>512</prepared-statement-cache-size> <blocking-timeout-millis>60003</blocking-timeout-millis> <idle-timeout-minutes>15</idle-timeout-minutes> <metadata> <type-mapping>mySQL</type-mapping> </metadata> </no-tx-datasource> </datasources>
my.ini[client] port=3306 [mysql] default-character-set=latin1 [mysqld] max_connections=800 thread_cache_size=38 port=3306 lower_case_table_names = 1 sql-mode = IGNORE_SPACE basedir="C:/Program Files/MySQL/MySQL Server 5.1/" datadir="C:/Documents and Settings/All Users/Application Data/MySQL/MySQL Server 5.1/Data/" default-character-set=latin1 default-storage-engine=INNODB query_cache_size=84M table_cache=1520 tmp_table_size=30M myisam_max_sort_file_size=100G myisam_sort_buffer_size=30M key_buffer_size=129M read_buffer_size=64K read_rnd_buffer_size=256K sort_buffer_size=256K #*** INNODB Specific options *** innodb_additional_mem_pool_size=6M innodb_flush_log_at_trx_commit=1 innodb_log_buffer_size=3M innodb_buffer_pool_size=250M innodb_log_file_size=50M innodb_thread_concurrency=8
This should work for an Injectionrate of 40. But I got lots of errors. -
6. Re: JBoss Performance Problems with SPECjAppServer2004
frank.feinbube Apr 8, 2009 3:47 AM (in response to frank.feinbube)The OS I am using is Windows XP with SP3.
(I am sorry for that ;) ) -
7. Re: JBoss Performance Problems with SPECjAppServer2004
frank.feinbube Apr 8, 2009 4:53 AM (in response to frank.feinbube)I still have all the errors mentioned before. And I have no ideas anymore.
I tried small thread_cache_size (8) and big ones (2000). I tried to increase connections (up to 2000) on mysql-site. I tried lots of configurations for the mysql-ds.xml.
But I get the same errors every time.
Perhaps its really a problem of the Windows XP + JBoss 4.2.3 + MySQL 5.1 combination.
Any suggestions?
I will try JBoss 5.0.0. Perhaps he will be nicer to me. -
8. Re: JBoss Performance Problems with SPECjAppServer2004
peterj Apr 8, 2009 11:23 AM (in response to frank.feinbube)With the settings you posted, what is the highest IR you get without any errors?
What is your hardware setup? Everything running on one box? Or multiple boxes? What is the size (RAM, CPU speed, number of cores) of each box? -
9. Re: JBoss Performance Problems with SPECjAppServer2004
frank.feinbube Apr 8, 2009 11:30 AM (in response to frank.feinbube)I have one real Node per Role:
Emulator: Intel E6600 @ 2.4 Ghz, 3.00 GB
Database: Intel E6600 @ 2.4 Ghz, 3.00 GB
Driver: Intel E8400 @ 3.0 Ghz, 3.25 GB
Appserver: Intel E8400 @ 3.0 Ghz, 3.25 GB
(every CPU has 2 cores)
The highest IR I was able to achieve without errors was 20. (The next I tried was 30.)
The GlassFish application server was able to perform an IR of 100 and more without any problems. -
10. Re: JBoss Performance Problems with SPECjAppServer2004
peterj Apr 8, 2009 3:33 PM (in response to frank.feinbube)According to this post: http://lists.mysql.com/mysql/212112, try decreasing the max-connections in my.ini to 400 (or even 360 - that it the most connections you will make based on the *-ds.xml file setting).
Also, monitor the thread count on the mysqld.exe process - this will give a rough idea of how many connections you have. Or you can use the MySQL Administrator to monitor the number of connections. I'm sure there is also a query you can run in the mysql command-line client to get this info, but I don't know the syntax offhand.
Another thing to consider is the memory usage on the database node. Are you, by any chance, booting that system with the /3GB boot option? -
11. Re: JBoss Performance Problems with SPECjAppServer2004
frank.feinbube Apr 8, 2009 4:33 PM (in response to frank.feinbube)Actually I tried to set the max_connections count to 400 in the first place. But this led to errors on the JBoss stating that the pool could not be filled. So I increased it to 800 which worked fine. (At least for the startup of JBoss. And the preparation phase of the benchmark.)
The overall thread count of the database node is about 560 when running the benchmark. Which is 60 threads more than in the idle state. This is the case even when I set the thread_cache_size to 2000.
I think that I am not using the /3GB option. I don't even know what you mean by it ;)
But actually I monitored for memory paging errors and RAM utilization and there is no problem. In addition as I mentioned before GlassFish had no problems with this configuration. So I think that the interaction of JBoss, MySQL and Windows are causing the problem. -
12. Re: JBoss Performance Problems with SPECjAppServer2004
frank.feinbube Apr 8, 2009 6:02 PM (in response to frank.feinbube)Unfortunately JBoss 5.0.1.GA seem to be no solution either cause it doesn't work with the kit. :(
-
13. Re: JBoss Performance Problems with SPECjAppServer2004
frank.feinbube Apr 15, 2009 1:39 PM (in response to frank.feinbube)Has anyone any further suggestions? I am running out of time and it seems that no one has a solution to this problem.