0 Replies Latest reply on Dec 6, 2012 3:33 AM by litalh12

    JBoss 5.0.1 AJP worker thread hangs?

    litalh12

      Recently we're experiencing a problem with our JBoss AS and we're not able to identify the root cause.

      we have an apache load balancer sitting infront of our jboss servers which delivers the requests to the jboss using AJP protocol on port 8009.

      recently we're experiencing an issue where the worker threads are stuck in the system (we see it through the jmx-console on jboss.web, and also when issuing thread dumps).

      We're currently set to have 700 maxThreads for AJP connections and it takes up to 3 days for it to completely max. at that point we restart the jboss service to solve it.

      These threads looks like this in the thread dump:

       

      **Thread: ajp-0.0.0.0-8009-4 : priority:5, demon:true, threadId:90, threadState:RUNNABLE**
      java
      .net.SocketInputStream.socketRead0(Native Method)
      java
      .net.SocketInputStream.read(SocketInputStream.java:129)
      org
      .apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:1036)
      org
      .apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:1115)
      org
      .apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:383)
      org
      .apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:384)
      org
      .apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
      java
      .lang.Thread.run(Thread.java:662)

      **Thread: ajp-0.0.0.0-8009-5 : priority:5, demon:true, threadId:91, threadState:RUNNABLE**
      java
      .net.SocketInputStream.socketRead0(Native Method)
      java
      .net.SocketInputStream.read(SocketInputStream.java:129)
      org
      .apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:1036)
      org
      .apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:1115)
      org
      .apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:383)
      org
      .apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:384)
      org
      .apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
      java
      .lang.Thread.run(Thread.java:662)

      **Thread: ajp-0.0.0.0-8009-6 : priority:5, demon:true, threadId:92, threadState:RUNNABLE**
      java
      .net.SocketInputStream.socketRead0(Native Method)
      java
      .net.SocketInputStream.read(SocketInputStream.java:129)
      org
      .apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:1036)
      org
      .apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:1115)
      org
      .apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:383)
      org
      .apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:384)
      org
      .apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
      java
      .lang.Thread.run(Thread.java:662)

      **Thread: ajp-0.0.0.0-8009-7 : priority:5, demon:true, threadId:93, threadState:WAITING**
      - waiting on <0x42ee5ce8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
      java
      .lang.Object.wait(Native Method)
      java
      .lang.Object.wait(Object.java:485)
      org
      .apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
      org
      .apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
      java
      .lang.Thread.run(Thread.java:662)

       

      The web app running on the jboss uses Jersey implementation to either return the client a list of available files or the files itself using application/octet-stream (Adding the InputStream as an entity to the Jersey resposne).

      Thanks in advance, Lital