trying to increase thoroughput to server, what am I missing?

[jamicide]

I am trying to load test a newly installed JBoss server.

I am using JMeter to send continuous HTTP POSTs to a servlet running on the server but I can't seem to send more than ~50 requests a second, and this doesn't max out the server (it handles these requests with no problems). I have tried sending the requests from multiple machines simultaneously with the same result. I have also modified JBoss's server.xml config file to up the HTTP connector threadpool size, but again this seems to have no affect - I am stuck around 50 requests a second.

Are there other configurations I should be modifying or could I be missing something with my current setup?

Any help is appreciated!

[peterj]

Is JMeter running on the same machine as JBossAS? Perhaps JMeter, or the network connection between the JMeter machine and the JBossAS machine, is where the problem is.

What processor utilization are you seeing?

[jamicide]

Thanks for the response.

I have ran JMeter from 3 seperate machines (never the same machine as JBoss) both seperately and simulataneously, all with the same cumulative thoroughput of ~50 per second. This led me to believe that the the problem was on the JBoss box.

The processor load is around 85% at its peak.

[peterj]

OK, if the JBossAS box is running at 85% CPU load, you now have to find the bottleneck there and fix it.

I assume when you said earlier that "this doesn't max out the server" that you consider 85% as not being maxed out. For an app server, actually, it is. Realistically, you don't want to push an app server much beyond this.

[jamicide]

Thanks for your input.

For this particular test I was trying to overload the server with requests to see if anything extremely bad happened. Is it reasonable to suppose that the server isn't accepting/processing the HTTP requests because it is already so loaded with work? Like you said, this functionality is hindered by another bottleneck?

[peterj]

How many threads are you running in JMeter? Have you tried starting out with only a few threads, and slowly increasing the number of threads, noting how the system behaves for each run?

How many processors (or cores) are on the JBossAS box?

What Windows performance statistics are you monitoring, if any? Are you monitoring the gc data (using -verbose:gc or one of the other options)?

What are the JVM arguments to JBossAS? I am especially looking for heap settings.

I have some JVM performance tuning info at http://www.cecmg.de/doc/tagung_2007/agenda07/24-mai/2b3-peter-johnson/index.html.

If I have some time I will set up a JMeter run on my box against one of my servlets and see how that goes.

[peterj]

Which listener are you using?

Why do I ask? Well, I used the Summary Report listener. It said Throughput = 2.0/sec (running a single thread for 200 requests) and = 44.0/sec (running 10 threads for 200 requests). The problem is, my 1 thread run lasted less than 1 second, and the 10 thread run about 3 seconds.

I then used the Graph Results listener and it says 41,039/min (=683/sec), which is closer to the truth.

My CPU usage does hit 99%.

I decided to do a longer run - 10000 requests. This lasted 3 min 25 secs (100,000/205=487.8/sec). This time the Summary Report was more correct, Throughput = 482.9/sec.

By the way, I have a Core 2 Quad processor, and am running JDK 1.5.0_15 (64-bit version) on Vista 64. Unfortunately, I noticed after doing this run that my servlet was writing to the console log, which probably slowed things down tremendously. I noticed that 3 times the CPU was pegged at 100% for several seconds, but was mostly ran around 60-70%.

[peterj]

Looks like if I want to take this any further I will have to put JMeter on another machine. It is the one doing all of the processing and JBossAS is hardly doing anything.

I am also seeing a lot of kernel time, which usually indicates GC processing. At this time, I let everything run with defaults. Based on past experience, if I tune the heap sizes properly I should see at least another 50% increase in throughput from than. And moving JMeter off the box should get me even more throughput (assuming I can find enough boxes on which to run JMeter, my laptop might not be up to it. Hmm, maybe I can convince my daughter to bring her quad core system over...)

(I also need a new copy of listdlls that works with Vista64. Guess that that kind of thing happens when you are breaking in a new OS...)

[jamicide]

Hi Peter - I'll try and answer all your Qs here :)

Post #1:

I have tried numerous thread group settings thus far - low numbers of threads w/ many repetitions, high threads w/ many repetitions, various ramp-up times - I still get a max of around 50 /s, be it when running JMeter on my windows machine or on a linux box.

For instance, I just tried 50 repetitions of 60 threads, and the test ran in almost exactly a minute (50/s).

On the JBossAS box (which runs on linux), there are 2 dual-core processors. I have been monitoring the CPU and MEM load on the machine, but not the GC stats. The JVM arguments I use are "-Xms4096m -Xmx4096m -XX:NewSize=1024m -XX:MaxNewSize=1024m -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000".

Post #2:

I use both the Graph Listener and the Summary Report listener, and they seem to more or less be in agreement on the throughput. You say your CPU usage running Jmeter hits 99% - mine never goes about 15-20%? I have a Core 2 Quad processor using Java 1.6.0 on Win XP.

Thanks again!

[peterj]

Are you running JBossAS on both Linux and WinXP? And getting the same results?

I assume your Linux is 64-bit (based on the heap settings used - 32-bit is limited to around 2.7GB heaps on Linux and 1.5 on WinXP), am I correct? Which distro?

Are you are running 64-bit WinXP? If only 32-bit, what are the heap settings there?

I suspect that if I reboot into 32-bit XP and use JDK 6 that I will get a higher throughput. Ditto if I reboot into Ubuntu. And I am not changing the default hep settings specified in the run script (heap resizing possibly accounts for much of the kernel time I was seeing.)

I guess I need to see the code for your servlet. My servlet is a simple "hello world" servlet that does no other processing.

[jamicide]

I am running JBoss solely on the Linux box. Any yes the linux is 64-bit CentOS.

Inspired by your "Hello world" test, I tried POSTing to a dummy servlet running on a Tomcat server (on another machine) that simply prints the request to file and returns. I can get the requests up to ~60/second, but still nowhere near your results :(

Perhaps I'll copy this thread over to the JMeter forums after we're done :)

[peterj]

Hmm, I was using GET. Though I doubt that using POST would be different (doPost() calls doGet()).

What are all of the JMeter components in your Thread Group? All I have are:

* HTTP Request Defaults
* HTTP Request (doe the single GET)
* Summary Report

The only other thing I can think of is that I ran JMeter on the same box as JBossAS. I'll try running JMeter on another box. Maybe there is some network issue.

[peterj]

Ran JMeter on my laptop (dual core AMD) and JBossAS on my desktop (quad core Intel). Laptop showed about 70-80% CPU usage. Desktop barely registered 10% usage. I was getting 333 requests/second. Hmm, that was using wireless on my laptop. Turned off wireless and used a wired connection. Now my laptop shows 100% CPU (desktop 10-15% CPU) and I got 534 requests/second.

Have you tried running JMeter on the CentOS box? That would at least eliminate the network as a possibility.

In JMeter, when you specify the host, do you use hostname or IP address? How many routers/switches are you going through between the JMeter and JBossAS boxes?

Do other network operations run acceptably on the CentOS box? Things like ftping a large file.