Low performance of jboss in windows server 2003

[lquenta]

We are using a web service developed in java hosted in a jboss 4.0.2, that is running on windows 2003 server and is using a MySql 1.1.20 as DBMS.
This application receives an xml string from the client, deserializes it and insert it on a table on the mySql DB, and finally sends it to another server that is outside our network.
The client application sends requests to this webservice and waits for an answer. This client application is a aspx web page, that is, multiple requests can occur at one time from multiple client machines.
Our problem: the more requests JBoss receives from client machines through the web service, the lower the performance gets. But even so, when only one client machine sends many requests (say 25+) the performance get reduce as well. We are seeing mainly three things:
1.Memory usage gets higher and full garbage collections become more usual, reaching a 4000 total CG and 10 Full GC. The process starts by inserting a record per second on MySQL, but ends up doing up to one record every 30-40 seconds.
2.Number of Threads goes up, from an initial 4-5 to pass 50. We assume this is linked to the first issue.
3.A lot of memory swap (all the time the process is running), which reduces disk space until eventually there is “0 bytes� of free space in drive C:
What can we do to improve the performance of JBoss and make it restore the disk space when it finishes processing all the requests? We are running JBoss on a developing environment, thus our server is virtual and has 512MB of RAM. Could this too have an effect on the performance of JBoss? Are there any known issues in using MySQL with JBoss?
We are manually executing JBoss . The JVM heap configuration is as follows:
-Xms350m -Xmx350m -XX:NewSize=100m -XX:MaxNewSize=100m -XX:SurvivorRatio=30
This configuration has been the best we’ve tried so far.
Any help will be cheerfully received. Thanks a lot.

[peterj]

If you bring up your virtual server, but do not bring up JBossAS, how much free physical memory do you have? That is the maximum you should set the Java heap.

On my Server 2003 system, which has 1024MB RAM, I have 550MB free, which suggest to me that you have very little free RAM. Also, is MySQL running on the same virtual system? If so, it also eats up quite a bit of RAM I imagine that you are thrashing your memory.

If you plan to run both MySQL and JBossAS on the same system I would not go below 2GB of RAM. If you run them on separate systems, 1GB for each system.

Your full GC to total GC ratio is very low, which is good, and makes me assume that your heap settings are good for your application. A detailed analysis of the gc data might offer some suggestions for improvement. See my presentation at http://www.cecmg.de/doc/tagung_2007/agenda07/24-mai/2b3-peter-johnson/index.html

The number of threads has more to do with the number of connections made. If you mean Java threads, then this represents the number of simultaneous HTTP connections. If you mean MySQL threads, then this depends on the number of database connections made, and the settings for the database connection pool.

We have gotten very good performance using JBossAS 4.0.2 and MySQL 4.1.

[lquenta]

Peter: thanks for you reply.
The physical memory available on the server is 351,728 K. Thats why we set the heap to that size.
We saw your presentation and we have followed your suggestions for improvement, and we have a graph of performance of our server when a single client sent 25 requests. This requests have 16 retries per request for tests purpouses.
We would like you to see this result, is there a chance that we can send you this graph?.
Yes, we mean Java threads, we only have one mySql connection configured in the connection pool.
Thanks again for you quick answer, regards.

[lquenta]

One more thing, in the performance graph we see that total heap size is bigger than the heap used during all the process. Does this mean that JVM has free heap space all the time?.

[peterj]

Only one connection in the connection pool? That will work for a single client, but if there are multiple clients you will need more connections or database access will be an issue. (Based on your description I cannot tell if you are running a single or multiple clients, but the number of threads leads me to believe you have multiple clients.)

Also, is MySQL on the same system with JBossAS?

How does the app run if you put it on a physical box? I would tune the app on a physical box first and once I had acceptable performance, then I would move it to a virtual machine.

And yes, there will always be unused heap, typically equal to the size of the young generation. So in your case there will usually be no more than 250MB of heap used, with 100MB always unused. This has to do with the algorithm the Sun JVM uses to determine if it should do a full gc.

[lquenta]

We have single server that is sending all the request to Jboss from multiple web clients. This server is a MBI(Microsoft Business Integrator) and is the only connected to Jboss.

I wonder if JBoss is creating a Thread for each Java connection it receives. Is it queuing all the requests made by the clients?

How many connections can we set in the MySql connection pool?, does it depend on the number of total requests that we are sending?,
Or is there a number of recommended connections that we can use for tests?

Yes, MySql and JbossAS are running in the same virtual machine, and sadly we can't use a physical box because thats beyond our control.

[peterj]

Yes, JBossAS creates a thread for each HTTP connection. You can adjust the maximum number of threads allowed. See http://tomcat.apache.org/tomcat-5.5-doc/config/http.html, look for maxThreads amd acceptCount.

You can set up hundreds of connections in the MySQL connection pool, though in your setup I would recommend perhaps 20-50 connections. JBossAS will establish as many connections as there are simultaneous requests to the database. One way to tell how many connections you need is to allow hundreds of connections, run your test, and use the JMX Console to monitor the number of connections in use. Of course, the more connections you have the more memory MySQL will use. Once you decide on the number of connections, set max-pool-size in your *-ds.xml file, and max_connections in your my.ini file, to the same value.

Because JBossAS and MySQL are on the same virtual machine, you will need to assign more memory to the virtual machine. You should also adjust the MySQL memory settings.

You don't have a desktop system, or even a laptop, that you could try it on? It doesn't have to be running Server 2003, XP would be fine. Even Vista would do in a pinch.

[lquenta]

We are going over the HTTP connections and the MYSQL configurations to see if we can find something wrong.

We would like to know if upgrading to the lates JBoss version could help al all. Would you recomend it?

Thank for all the help Peter. Best of luck.

[peterj]

I would recommend the lastest AS version for the functionality and features. But I would not expect a big performance improvement.