[edlam_2000]

Thank you, besides the thread dump file, threads.html, attached in post#14, I have just taken another thread dump file, threads2010020501.html as attached from another machine experiencing the same kind of Recv-Q thread hung problem.

The JMX http://localhost:8080/jboss?status=full is showing

     Max threads: 1000 Current thread count: 42 Current thread busy: 4
     Max processing time: 46824 ms Processing time: 11450.632 s Request count: 1499921 Error count: 2854 Bytes received: 0.00 MB Bytes sent: 2933.76 MB

The netstat shows that most Recv-Q are hung, except 3 are still ok as below:-

Proto Recv-Q Send-Q Local Address               Foreign Address             State

tcp      179      0 127.0.0.1:8009              127.0.0.1:49943             ESTABLISHED
tcp      516      0 127.0.0.1:8009              127.0.0.1:49686             ESTABLISHED
tcp      500      0 127.0.0.1:8009              127.0.0.1:49678             ESTABLISHED
tcp      179      0 127.0.0.1:8009              127.0.0.1:49932             ESTABLISHED
tcp      500      0 127.0.0.1:8009              127.0.0.1:49410             ESTABLISHED
tcp      297      0 127.0.0.1:8009              127.0.0.1:49921             ESTABLISHED
tcp      223      0 127.0.0.1:8009              127.0.0.1:49920             ESTABLISHED
tcp      300      0 127.0.0.1:8009              127.0.0.1:49214             ESTABLISHED
tcp      499      0 127.0.0.1:8009              127.0.0.1:49213             ESTABLISHED
tcp      223      0 127.0.0.1:8009              127.0.0.1:49978             ESTABLISHED
tcp      219      0 127.0.0.1:8009              127.0.0.1:49977             ESTABLISHED
tcp      223      0 127.0.0.1:8009              127.0.0.1:49976             ESTABLISHED
tcp      219      0 127.0.0.1:8009              127.0.0.1:49975             ESTABLISHED
tcp      179      0 127.0.0.1:8009              127.0.0.1:49974             ESTABLISHED
tcp        0      0 127.0.0.1:8009              127.0.0.1:50254             ESTABLISHED
tcp      179      0 127.0.0.1:8009              127.0.0.1:49996             ESTABLISHED
tcp      179      0 127.0.0.1:8009              127.0.0.1:49484             ESTABLISHED
tcp      374      0 127.0.0.1:8009              127.0.0.1:49482             ESTABLISHED
tcp      179      0 127.0.0.1:8009              127.0.0.1:49481             ESTABLISHED
tcp      368      0 127.0.0.1:8009              127.0.0.1:49223             ESTABLISHED
tcp      298      0 127.0.0.1:8009              127.0.0.1:49477             ESTABLISHED
tcp      366      0 127.0.0.1:8009              127.0.0.1:49513             ESTABLISHED
tcp      219      0 127.0.0.1:8009              127.0.0.1:49512             ESTABLISHED
tcp      223      0 127.0.0.1:8009              127.0.0.1:50023             ESTABLISHED
tcp        0      0 127.0.0.1:8009              127.0.0.1:50322             ESTABLISHED
tcp      300      0 127.0.0.1:8009              127.0.0.1:50057             ESTABLISHED
tcp        0      0 127.0.0.1:8009              127.0.0.1:50355             ESTABLISHED
tcp      219      0 127.0.0.1:8009              127.0.0.1:50098             ESTABLISHED
tcp      219      0 127.0.0.1:8009              127.0.0.1:50097             ESTABLISHED
tcp      223      0 127.0.0.1:8009              127.0.0.1:50086             ESTABLISHED
tcp        0      0 127.0.0.1:8009              127.0.0.1:50139             ESTABLISHED
tcp        0      0 127.0.0.1:8009              127.0.0.1:50138             ESTABLISHED
tcp      499      0 127.0.0.1:8009              127.0.0.1:49406             ESTABLISHED
tcp        0      0 127.0.0.1:8009              127.0.0.1:50168             ESTABLISHED
tcp      500      0 127.0.0.1:8009              127.0.0.1:49651             ESTABLISHED
tcp      219      0 127.0.0.1:8009              127.0.0.1:49650             ESTABLISHED
tcp      219      0 127.0.0.1:8009              127.0.0.1:49649             ESTABLISHED

[edlam_2000]

I have greatly improved the situation by setting the connection_pool_timeout to the same number of seconds as the connectionTimeout of the AJP 8009 port connection in the server.xml, however, occasionally, there're still some data stuck in the Recv-Q for reason unknown.

It appears that if these 2 values are different, then the JBOSS might have closed the connection without acknowledging the tomcat workers on the Apache side, and thus whent the tomcat worker sent a AJP13 request to the JBOSS end, the Recv-Q is stuck with data without it being serviced by the JBOSS thread.  This is just my logical guess to explain why the data got hung in the Recv-Q for a long time.