Pardon my verbosity but I want to include as much info as possible...
We're a mid-size software company and we've recently launched a web forum using JForum and JBoss -- we hadn't known about JBoss's own forum that's part of JBoss Portal(?) or we might have considered it first. Anyhow, after issuing a press release yesterday, our fora were swamped with traffic and at 3am went offline. Here's what I'm seeing in the logs earlier in the day but nothing of the sort when it finally went offline in the middle of the night...
12:48:16,806 ERROR [ThreadPool] All threads (250) are currently busy, waiting. Increase maxThreads (250) or check the servlet status
12:49:24,829 WARN [MsgContext] Error sending end packet
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
12:49:24,830 WARN [ChannelSocket] processCallbacks status 2
12:50:49,812 WARN [MsgContext] Error sending end packet
We're using Apache 2.2.2 with mod_proxy and the following settings:
I've found the likely place to change the thread count in jboss.web in the jk-8009 ThreadPool but I want to know if my Apache settings are causing problems as well. I started out with a 60 second time out and lowering has helped considerably but it only manages to postpone our crashes. Until we upgrade in a few weeks, we're running all pieces on RedHat Enterprise Linux 2.1 with a 2.4.9-e.59custom kernel.
I've been monitoring the web server and noticed that after dropping the ProxyTimeout value, the number of connections in a wait state have dropped considerably so it seems to be helping (the following output is snipped):
$ netstat -a|grep 8009
tcp 0 0 forums.example.com:41174 app2.extensis.com:8009 ESTABLISHED
tcp 0 0 forums.example.com:41180 app2.example.com:8009 ESTABLISHED
tcp 0 0 forums.example.com:41163 app2.example.com:8009 ESTABLISHED
tcp 0 0 forums.example.com:41205 app2.example.com:8009 TIME_WAIT
tcp 0 0 forums.example.com:41204 app2.example.com:8009 ESTABLISHED
tcp 0 0 forums.example.com:41207 app2.example.com:8009 TIME_WAIT
tcp 0 0 forums.example.com:41206 app2.example.com:8009 ESTABLISHED
Any help would be most appreciated.