JBoss http thread block reading in SPECjEnterprise2010

[hypheng]

I got a problem when benching SPECjEnterprise2010.

The http threads in the thread pool all waiting for http input.

tack trace:

java.net.SocketInputStream.socketRead0(Native Method)

java.net.SocketInputStream.read(SocketInputStream.java:146)

org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:731)

org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:368)

org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:838)

org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:679)

org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:519)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

java.lang.Thread.run(Thread.java:679)

org.jboss.threads.JBossThread.run(JBossThread.java:122)

The executor was idle waiting while thousands of requests were not served, queued and never served in the end.

I don't know which configuration caused this problem.

My configuration about web is:

        <subsystem xmlns="urn:jboss:domain:threads:1.1">

            <unbounded-queue-thread-pool name="JBossWeb">

                <max-threads count="200"/>

                <keepalive-time time="10" unit="second"/>

            </unbounded-queue-thread-pool>

        </subsystem>

        <subsystem xmlns="urn:jboss:domain:web:1.2" default-virtual-server="default-host" native="true">

            <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http" executor="JBossWeb" max-connections="3000"/>

            <virtual-server name="default-host" enable-welcome-root="true">

                <alias name="localhost"/>

                <alias name="example.com"/>

            </virtual-server>

        </subsystem>

If I use the defaut executor, there will be thousands of threads created.

Most of the threads does nothing but waiting i/o.

Could anyone tell me what can lead to this problem?

[hypheng]

Anyone can help with this issue?

I can reach injection rate 500 now, but I used 9000 threads.

I don't think it's reasonable to use so much threads on a 8 core machine.

[asoldano]

Hi Haifenf,

I suggest moving this thread to a different section of the forum, this does not actually seem a directly WS related issue, but something related to the lower web layer.

[hypheng]

Thank you for reminding

[jfclere]

make sure that native is used otherwise you need a lot of threads (which seems to be the problem).

Adittionally it would be nice to know the platform/os as well as the version of AS7 you are testing.

[whitingjr]

Jean-Frederic, it seems Haifenf is already using the native connector and is having problems with it.

Hi Haifenf,

This behaviour has been seen before when scaling up with the native connector.

My advice is to try using the Java NIO connector recently integrated in as7.2. Change your configuration with native="false" to see if the problem continues.

<subsystem xmlns="urn:jboss:domain:web:1.2" default-virtual-server="default-host" native="false">

Regards,

Jeremy

[whitingjr]

Not fogetting the following line in the configuration.

<connector name="http" protocol="org.apache.coyote.http11.Http11NioProtocol" scheme="http" socket-binding="http" executor="JBossWeb" max-connections="xxxx"/>

Jeremy

[hypheng]

I'm running on rhel 6.3 and using JBoss EAP 6.0.0.GA (AS 7.1.2.Final-redhat-1)

Thank you, Jeremy.

I can't use the nio http.

The class cannot be found.

Do you mean I need to use AS 7.2?

I can't find where to download the binary.

Should I build from the git source base?

[whitingjr]

Do you mean I need to use AS 7.2?

Yes

Should I build from the git source base?

Yes.

"master" branch is the one to build.

[hypheng]

Thank Jeremy.

I've not built AS 7.2.

I used ARP connector and it can reduce the threads to hundreds.

Just curious, why ARP connector shipped with JBoss binary.