0 Replies Latest reply on Aug 24, 2006 3:22 PM by gdriggs

    MaxThread issue with Apache mod_proxy

      Hi folks,

      Pardon my verbosity but I want to include as much info as possible...

      We're a mid-size software company and we've recently launched a web forum using JForum and JBoss -- we hadn't known about JBoss's own forum that's part of JBoss Portal(?) or we might have considered it first. Anyhow, after issuing a press release yesterday, our fora were swamped with traffic and at 3am went offline. Here's what I'm seeing in the logs earlier in the day but nothing of the sort when it finally went offline in the middle of the night...

      12:48:16,806 ERROR [ThreadPool] All threads (250) are currently busy, waiting. Increase maxThreads (250) or check the servlet status
      12:49:24,829 WARN [MsgContext] Error sending end packet
      java.net.SocketException: Broken pipe
      at java.net.SocketOutputStream.socketWrite0(Native Method)
      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
      at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
      at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:518)
      at org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112)
      at org.apache.jk.core.MsgContext.action(MsgContext.java:293)
      at org.apache.coyote.Response.action(Response.java:182)
      at org.apache.coyote.Response.finish(Response.java:304)
      at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:204)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684)
      at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)
      12:49:24,830 WARN [ChannelSocket] processCallbacks status 2
      12:50:49,812 WARN [MsgContext] Error sending end packet

      We're using Apache 2.2.2 with mod_proxy and the following settings:

      ProxyTimeout 10
      <Proxy balancer://cluster>
      BalancerMember ajp://app2.example.com:8009

      <Location /jforum>
      ProxyPass balancer://cluster/jforum


      I've found the likely place to change the thread count in jboss.web in the jk-8009 ThreadPool but I want to know if my Apache settings are causing problems as well. I started out with a 60 second time out and lowering has helped considerably but it only manages to postpone our crashes. Until we upgrade in a few weeks, we're running all pieces on RedHat Enterprise Linux 2.1 with a 2.4.9-e.59custom kernel.

      I've been monitoring the web server and noticed that after dropping the ProxyTimeout value, the number of connections in a wait state have dropped considerably so it seems to be helping (the following output is snipped):

      $ netstat -a|grep 8009
      tcp 0 0 forums.example.com:41174 app2.extensis.com:8009 ESTABLISHED
      tcp 0 0 forums.example.com:41180 app2.example.com:8009 ESTABLISHED
      tcp 0 0 forums.example.com:41163 app2.example.com:8009 ESTABLISHED
      tcp 0 0 forums.example.com:41205 app2.example.com:8009 TIME_WAIT
      tcp 0 0 forums.example.com:41204 app2.example.com:8009 ESTABLISHED
      tcp 0 0 forums.example.com:41207 app2.example.com:8009 TIME_WAIT
      tcp 0 0 forums.example.com:41206 app2.example.com:8009 ESTABLISHED

      Any help would be most appreciated.

      -Gary