HTTP threads waiting on JIOEndpoint$Worker.await

[jsmith085]

I am doing some load testing using a http playback tool and simulating 150 users, with 40 second think times between transactions.  I'm on Red Hat Enterprise 5.2 on Wintel, on VMWare !

My load test response times have become worse and worse as I've added the last 50 or so users and I'm curious about the following.

A thread dump of one of our running JBoss AS 5.1 app server shows large numbers of our HTTP connector threads WAITING as follows:

Thread: http-0.0.0.0-8180-1 : priority:5, demon:true, threadId:129, threadState:WAITING

- waiting on <0x1d30afa> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
java.lang.Thread.run(Thread.java:619)

Is this normal/expected? 

In a different post elsewhere I saw what appeared to be the code for the await method of the JIOEndpoint$Worker class and it appeared to be trying to obtain a socket.  This led to other post talking about file descriptors (lack thereof) potentially being an issue....how can I definitely confirm if they are an issue?

If threads waiting in this method is not normal then what other causes could account for it?

[peterj]

The waiting threads are expected. These are threads that are in the pool waiting to be assigned to process a user request.

Given the think time and number of users, and assuming that the requests are evenly distributed across the think time (in other words, all 150 users are NOT starting thinking at the same time and after 40 seconds making their requests all at the same time),  and if a user request takes about 500 milliseconds to process, you will have about 2 requests in the system at any one time: (150 / 40) * 0.5

So how long does a request take? Are you monitoing the number of active HTTP threads? You might have to do some digging to uncover the bottleneck, You might also consider running on physical hardware instead of in a VM just for comparison.

BTW, please don't bold your entire post.

[amarhcl]

How to control this increase this threads , do we need to do any tunings

"http-/0.0.0.0:9443-99": awaiting notification on [0x00000007aab780a0]

  at java.lang.Object.wait(Native Method)

  at java.lang.Object.wait(Object.java:502)

  at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:875)

  at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:901)

  at java.lang.Thread.run(Thread.java:701)