AJP threads waiting on socket reads in Jboss 7.1.1.final

[jacobmarcus20]

Hi all,

I see many threads in a waiting state with the following trace. Is this a problem?

State: RUNNABLE

Total blocked: 3    Total waited: 45

Strack trace:

    java.net.SocketInputStream.socketRead0(Native Method)

    java.net.SocketInputStream.read(SocketInputStream.java:129)

    org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:1131)

    org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:1213)

    org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:451)

    org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445)

    org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:518)

    org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33)

    org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:801)

    org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45)

    org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:842)

    java.lang.Thread.run(Thread.java:662)

    org.jboss.threads.JBossThread.run(JBossThread.java:122)

I see some in the follwoing state. This is what I woud expect as I have configured some core threads.

State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@38380f4b

Total blocked: 46    Total waited: 66

Strack trace:

    sun.misc.Unsafe.park(Native Method)

    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

    org.jboss.threads.QueueExecutor.takeTask(QueueExecutor.java:746)

    org.jboss.threads.QueueExecutor.access$400(QueueExecutor.java:45)

    org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:835)

    java.lang.Thread.run(Thread.java:662)

    org.jboss.threads.JBossThread.run(JBossThread.java:122)

My thread pool configuration looks like this.

<bounded-queue-thread-pool name="test-ajp" allow-core-timeout="false">

                <core-threads count="50"/>

                <queue-length count="1"/>

                <max-threads count="650"/>

                <keepalive-time time="60" unit="seconds"/>

</bounded-queue-thread-pool>

Please let me know why those some of these threads seem to be in a waiting mode. I suspect I am misisng a timeout configuration.

[jacobmarcus20]

Any ideas anyone? Is the stacktrace something that I need to be concerned about or is this some kind of optimization? I see more and more threads in that state over time. I am using mod_jk and do see some errors in the mod_jk logs.

[Mon Jul 16 20:21:33.226 2012] [4985:1246734656] [error] ajp_connect_to_endpoint::jk_ajp_common.c (957): (app1) cping/cpong after connecting to the backend server failed (errno=110)

[Mon Jul 16 20:21:33.226 2012] [4985:1246734656] [error] ajp_send_request::jk_ajp_common.c (1507): (app1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=110)

Here is my standard worker template.

worker.template1.port=8009

worker.template1.type=ajp13

worker.template1.lbfactor=1

worker.template1.socket_connect_timeout=2000

worker.template1.prepost_timeout=6000

worker.template1.connect_timeout=6000

I am seeing in some messages suggesting to use jbosssnative. I plan to give that a shot.

[mrapcooper]

Hi Jacob,

Did you ever get any progress on this issue?  On our cluster each node has 400 ajp threads waiting too!

[pandes3]

Hi Jacob, Andrew,

Did you guys find any solution for this issue ?

If so please let me know

Thanks and Regards

[alexh_97]

We're having the same issue.  Our symptom actually ends up being a "Too many open files" error. It almost looks like JBoss is not closing down AJP connections that are not in use.

[jprio]

You can try to use a specific thread pool for your connector :

<connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp" executor="ajp-executor"/>

and then specify a keepalive-time to stop idle threads :

              <bounded-queue-thread-pool name="ajp-executor">  

                    <core-threads count="10"/>

                    <queue-length count="50"/>

                    <max-threads count="200"/>

                    <keepalive-time time="30" unit="seconds"/>

                </bounded-queue-thread-pool>

[joghurt]

I'm getting

JBWEB003011: Error allocating socket processor: java.lang.OutOfMemoryError: unable to create new native thread