6 Replies Latest reply on Jan 17, 2014 6:03 AM by joghurt

AJP threads waiting on socket reads in Jboss 7.1.1.final

jacobmarcus20 Jul 12, 2012 7:36 AM

Hi all,

I see many threads in a waiting state with the following trace. Is this a problem?

State: RUNNABLE

Total blocked: 3 Total waited: 45

Strack trace:

java.net.SocketInputStream.socketRead0(Native Method)

java.net.SocketInputStream.read(SocketInputStream.java:129)

org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:1131)

org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:1213)

org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:451)

org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445)

org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:518)

org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33)

org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:801)

org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45)

org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:842)

java.lang.Thread.run(Thread.java:662)

org.jboss.threads.JBossThread.run(JBossThread.java:122)

I see some in the follwoing state. This is what I woud expect as I have configured some core threads.

State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@38380f4b

Total blocked: 46 Total waited: 66

Strack trace:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

org.jboss.threads.QueueExecutor.takeTask(QueueExecutor.java:746)

org.jboss.threads.QueueExecutor.access$400(QueueExecutor.java:45)

org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:835)

java.lang.Thread.run(Thread.java:662)

org.jboss.threads.JBossThread.run(JBossThread.java:122)

My thread pool configuration looks like this.

<bounded-queue-thread-pool name="test-ajp" allow-core-timeout="false">

<core-threads count="50"/>

<queue-length count="1"/>

<max-threads count="650"/>

<keepalive-time time="60" unit="seconds"/>

</bounded-queue-thread-pool>

Please let me know why those some of these threads seem to be in a waiting mode. I suspect I am misisng a timeout configuration.

1. Re: AJP threads waiting on socket reads in Jboss 7.1.1.final

jacobmarcus20 Jul 16, 2012 10:49 PM (in response to jacobmarcus20)

Any ideas anyone? Is the stacktrace something that I need to be concerned about or is this some kind of optimization? I see more and more threads in that state over time. I am using mod_jk and do see some errors in the mod_jk logs.

[Mon Jul 16 20:21:33.226 2012] [4985:1246734656] [error] ajp_connect_to_endpoint::jk_ajp_common.c (957): (app1) cping/cpong after connecting to the backend server failed (errno=110)
[Mon Jul 16 20:21:33.226 2012] [4985:1246734656] [error] ajp_send_request::jk_ajp_common.c (1507): (app1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=110)

Here is my standard worker template.

worker.template1.port=8009
worker.template1.type=ajp13
worker.template1.lbfactor=1
worker.template1.socket_connect_timeout=2000
worker.template1.prepost_timeout=6000
worker.template1.connect_timeout=6000

I am seeing in some messages suggesting to use jbosssnative. I plan to give that a shot.
Actions
2. Re: AJP threads waiting on socket reads in Jboss 7.1.1.final

mrapcooper Mar 11, 2013 8:11 PM (in response to jacobmarcus20)

Hi Jacob,

Did you ever get any progress on this issue? On our cluster each node has 400 ajp threads waiting too!
Actions
3. Re: AJP threads waiting on socket reads in Jboss 7.1.1.final

pandes3 Sep 10, 2013 11:33 PM (in response to mrapcooper)

Hi Jacob, Andrew,

Did you guys find any solution for this issue ?

If so please let me know

Thanks and Regards
Actions
4. Re: AJP threads waiting on socket reads in Jboss 7.1.1.final

alexh_97 Sep 13, 2013 1:57 PM (in response to pandes3)

We're having the same issue. Our symptom actually ends up being a "Too many open files" error. It almost looks like JBoss is not closing down AJP connections that are not in use.
Actions
5. Re: Re: AJP threads waiting on socket reads in Jboss 7.1.1.final

jprio Nov 7, 2013 5:24 AM (in response to alexh_97)

You can try to use a specific thread pool for your connector :

<connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp" executor="ajp-executor"/>

and then specify a keepalive-time to stop idle threads :

              <bounded-queue-thread-pool name="ajp-executor">
                    <core-threads count="10"/>
                    <queue-length count="50"/>
                    <max-threads count="200"/>
                    <keepalive-time time="30" unit="seconds"/>
                </bounded-queue-thread-pool>
Actions
6. Re: AJP threads waiting on socket reads in Jboss 7.1.1.final

joghurt Jan 17, 2014 6:03 AM (in response to alexh_97)

I'm getting
JBWEB003011: Error allocating socket processor: java.lang.OutOfMemoryError: unable to create new native thread
Actions

Go to original post