10 Replies Latest reply on Jan 27, 2006 1:35 PM by atait

    Dead locked threads encountered when upgrading from 3.2.3 to

      We have upgraded our Jboss instance and the JDK to 4.0.3 and 1.5.0_06 from 3.2.3 and 1.4.2_03 respectively. Our OS platform is Linux Enterprise 2.6. On the upgraded platform our web application runs for about 12 hours before becoming unresponsive. I'm attaching the JVM stack dump as it indicates many locked threads. I have been looking at this for a while and could use some assistance from others with more experience.

      Notice that TP-Processor137 has locked the java object that most other threads are waiting for. This thread is waiting on a condition but it's not clear on what condition and why it is not completing. It appears that this is the root cause. Other thread dumps indicate the same scenario.

      Any light that can be shed is appreciated.

      Allen



      Full thread dump Java HotSpot(TM) Client VM (1.5.0_06-b05 mixed mode, sharing):

      "http-0.0.0.0-8080-2" daemon prio=1 tid=0x09039ae8 nid=0x802 in Object.wait() [0x75cff000..0x75cff74c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x4f9ccd68> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread)
      at java.lang.Object.wait(Object.java:474)
      at org.apache.tomcat.util.net.MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
      - locked <0x4f9ccd68> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread)
      at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor144" daemon prio=1 tid=0x495b27d8 nid=0x7f7 runnable [0x75c7f000..0x75c7f7cc]
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
      at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
      - locked <0x4fb84c90> (a java.io.BufferedInputStream)
      at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:601)
      at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:538)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:666)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor143" daemon prio=1 tid=0x71c10ba8 nid=0x7f6 runnable [0x75bff000..0x75bff84c]
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
      at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
      - locked <0x4fb8d5a8> (a java.io.BufferedInputStream)
      at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:601)
      at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:538)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:666)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor142" daemon prio=1 tid=0x71b0e680 nid=0x7f5 waiting on condition [0x75b7e000..0x75b7f4cc]
      at java.lang.ThreadLocal$ThreadLocalMap.cleanSomeSlots(ThreadLocal.java:601)
      at java.lang.ThreadLocal$ThreadLocalMap.getAfterMiss(ThreadLocal.java:377)
      at java.lang.ThreadLocal$ThreadLocalMap.get(ThreadLocal.java:347)
      at java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:225)
      at java.lang.ThreadLocal.get(ThreadLocal.java:127)
      at java.lang.StringCoding.deref(StringCoding.java:49)
      at java.lang.StringCoding.encode(StringCoding.java:361)
      at java.lang.String.getBytes(String.java:812)
      at java.io.UnixFileSystem.getLastModifiedTime(Native Method)
      at java.io.File.lastModified(File.java:795)
      at com.opensymphony.module.sitemesh.factory.DefaultFactory.refresh(DefaultFactory.java:282)
      at com.opensymphony.module.sitemesh.factory.DefaultFactory.isPathExcluded(DefaultFactory.java:84)
      at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:37)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      at com.idexx.publicweb.filter.IdexxI18NFilter.doFilter(IdexxI18NFilter.java:97)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:81)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
      at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:39)
      at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:159)
      at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:59)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
      at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:307)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:385)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor141" daemon prio=1 tid=0x71b13808 nid=0x7f4 waiting for monitor entry [0x75aff000..0x75aff54c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor140" daemon prio=1 tid=0x71d6b010 nid=0x7ee waiting for monitor entry [0x75a7f000..0x75a7f5cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor139" daemon prio=1 tid=0x71bc4050 nid=0x7ed waiting for monitor entry [0x759ff000..0x759ff64c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor138" daemon prio=1 tid=0x71b17478 nid=0x7ec waiting for monitor entry [0x7597f000..0x7597f6cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor137" daemon prio=1 tid=0x71d20158 nid=0x7eb waiting on condition [0x758ff000..0x758ff74c]
      at java.lang.String.intern(Native Method)
      at javax.management.ObjectName.setCanonicalName(ObjectName.java:733)
      at javax.management.ObjectName.construct(ObjectName.java:565)
      at javax.management.ObjectName.(ObjectName.java:1304)
      at org.apache.jk.common.ChannelSocket.registerRequest(ChannelSocket.java:434)
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:459)
      - locked <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor136" daemon prio=1 tid=0x71d44230 nid=0x7e7 waiting for monitor entry [0x7587f000..0x7587f7cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor135" daemon prio=1 tid=0x0928b378 nid=0x7e6 waiting for monitor entry [0x74b7f000..0x74b7f84c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor134" daemon prio=1 tid=0x71c15df8 nid=0x7e5 waiting for monitor entry [0x757ff000..0x757ff4cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor133" daemon prio=1 tid=0x71c4cf70 nid=0x7e4 waiting for monitor entry [0x7577f000..0x7577f54c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor132" daemon prio=1 tid=0x71c39a30 nid=0x7cf waiting for monitor entry [0x756ff000..0x756ff5cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor131" daemon prio=1 tid=0x71cd1730 nid=0x7ce waiting for monitor entry [0x7567f000..0x7567f64c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor130" daemon prio=1 tid=0x71c71668 nid=0x7cd waiting for monitor entry [0x755ff000..0x755ff6cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor129" daemon prio=1 tid=0x71c709b8 nid=0x7cc waiting for monitor entry [0x7557f000..0x7557f74c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor128" daemon prio=1 tid=0x495d9008 nid=0x7c9 waiting for monitor entry [0x754ff000..0x754ff7cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor127" daemon prio=1 tid=0x71c08b68 nid=0x7c8 waiting for monitor entry [0x7547f000..0x7547f84c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor126" daemon prio=1 tid=0x71c07eb8 nid=0x7c7 waiting for monitor entry [0x753ff000..0x753ff4cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor125" daemon prio=1 tid=0x49585d68 nid=0x7c6 waiting for monitor entry [0x7537f000..0x7537f54c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor124" daemon prio=1 tid=0x714d2cb0 nid=0x7be waiting for monitor entry [0x752ff000..0x752ff5cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor123" daemon prio=1 tid=0x71550390 nid=0x7bd waiting for monitor entry [0x7527f000..0x7527f64c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor122" daemon prio=1 tid=0x71acdd68 nid=0x7bc waiting for monitor entry [0x751ff000..0x751ff6cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor121" daemon prio=1 tid=0x49571d98 nid=0x7ba waiting for monitor entry [0x72b7f000..0x72b7f74c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor120" daemon prio=1 tid=0x47e03650 nid=0x7ad waiting for monitor entry [0x7517f000..0x7517f7cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor119" daemon prio=1 tid=0x495a2800 nid=0x7ac waiting for monitor entry [0x750ff000..0x750ff84c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor118" daemon prio=1 tid=0x494e83a0 nid=0x7ab waiting for monitor entry [0x7507f000..0x7507f4cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor117" daemon prio=1 tid=0x7144df00 nid=0x7aa waiting for monitor entry [0x74fff000..0x74fff54c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor116" daemon prio=1 tid=0x49587c98 nid=0x7a5 waiting for monitor entry [0x74f7f000..0x74f7f5cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor115" daemon prio=1 tid=0x49509c48 nid=0x7a4 waiting for monitor entry [0x74eff000..0x74eff64c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor114" daemon prio=1 tid=0x495ea1f8 nid=0x7a3 waiting for monitor entry [0x74e7f000..0x74e7f6cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor113" daemon prio=1 tid=0x49506c60 nid=0x7a2 waiting for monitor entry [0x74dff000..0x74dff74c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor112" daemon prio=1 tid=0x71d10980 nid=0x79c waiting for monitor entry [0x74d7f000..0x74d7f7cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor111" daemon prio=1 tid=0x71d0c4b0 nid=0x79b waiting for monitor entry [0x74cff000..0x74cff84c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor110" daemon prio=1 tid=0x71bfcbd8 nid=0x79a waiting for monitor entry [0x74c7f000..0x74c7f4cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor109" daemon prio=1 tid=0x71b34708 nid=0x799 waiting for monitor entry [0x74bff000..0x74bff54c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor108" daemon prio=1 tid=0x71bc55f8 nid=0x794 waiting for monitor entry [0x74aff000..0x74aff64c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor107" daemon prio=1 tid=0x71d482e0 nid=0x793 waiting for monitor entry [0x74a7f000..0x74a7f6cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor106" daemon prio=1 tid=0x71d11e78 nid=0x792 waiting for monitor entry [0x749ff000..0x749ff74c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor105" daemon prio=1 tid=0x71b02ed8 nid=0x791 waiting for monitor entry [0x7477f000..0x7477f7cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor104" daemon prio=1 tid=0x71d4a488 nid=0x779 waiting for monitor entry [0x7497f000..0x7497f4cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor103" daemon prio=1 tid=0x71bd91f0 nid=0x778 waiting for monitor entry [0x748ff000..0x748ff54c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor102" daemon prio=1 tid=0x71bfe178 nid=0x777 waiting for monitor entry [0x7487f000..0x7487f5cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor101" daemon prio=1 tid=0x71bfe3f0 nid=0x776 waiting for monitor entry [0x747ff000..0x747ff64c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor100" daemon prio=1 tid=0x71d61ba0 nid=0x771 waiting for monitor entry [0x746ff000..0x746ff74c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor99" daemon prio=1 tid=0x71d06230 nid=0x770 waiting for monitor entry [0x7467f000..0x7467f7cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor98" daemon prio=1 tid=0x71d17760 nid=0x76f waiting for monitor entry [0x745ff000..0x745ff84c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor97" daemon prio=1 tid=0x71b4b618 nid=0x76e waiting for monitor entry [0x7457f000..0x7457f4cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor96" daemon prio=1 tid=0x71d19888 nid=0x766 waiting for monitor entry [0x744ff000..0x744ff54c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor95" daemon prio=1 tid=0x71b1bac8 nid=0x765 waiting for monitor entry [0x7447f000..0x7447f5cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor94" daemon prio=1 tid=0x71d1ad30 nid=0x764 waiting for monitor entry [0x743ff000..0x743ff64c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor93" daemon prio=1 tid=0x71b49920 nid=0x763 waiting for monitor entry [0x7437f000..0x7437f6cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor92" daemon prio=1 tid=0x71bc5038 nid=0x74e waiting for monitor entry [0x742ff000..0x742ff74c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor91" daemon prio=1 tid=0x71b46578 nid=0x74d waiting for monitor entry [0x7427f000..0x7427f7cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor90" daemon prio=1 tid=0x71bcdbc8 nid=0x74c waiting for monitor entry [0x741ff000..0x741ff84c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor89" daemon prio=1 tid=0x71b337e8 nid=0x74b waiting for monitor entry [0x7417f000..0x7417f4cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor88" daemon prio=1 tid=0x71b60f28 nid=0x74a waiting for monitor entry [0x740ff000..0x740ff54c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor87" daemon prio=1 tid=0x71b07e90 nid=0x749 waiting for monitor entry [0x7407f000..0x7407f5cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor86" daemon prio=1 tid=0x71d32208 nid=0x746 waiting for monitor entry [0x73fff000..0x73fff64c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor85" daemon prio=1 tid=0x09aceca8 nid=0x745 waiting for monitor entry [0x73f7f000..0x73f7f6cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor84" daemon prio=1 tid=0x09655c78 nid=0x743 waiting for monitor entry [0x73eff000..0x73eff74c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor83" daemon prio=1 tid=0x09654db8 nid=0x742 waiting for monitor entry [0x73e7f000..0x73e7f7cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor82" daemon prio=1 tid=0x4958a878 nid=0x741 waiting for monitor entry [0x73dff000..0x73dff84c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor81" daemon prio=1 tid=0x715b6c80 nid=0x740 waiting for monitor entry [0x73d7f000..0x73d7f4cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor80" daemon prio=1 tid=0x495db640 nid=0x735 waiting for monitor entry [0x73cff000..0x73cff54c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor79" daemon prio=1 tid=0x714f8e68 nid=0x734 waiting for monitor entry [0x73c7f000..0x73c7f5cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor78" daemon prio=1 tid=0x714c82b0 nid=0x733 waiting for monitor entry [0x73bff000..0x73bff64c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor77" daemon prio=1 tid=0x714c2670 nid=0x732 waiting for monitor entry [0x73b7f000..0x73b7f6cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor76" daemon prio=1 tid=0x71b93278 nid=0x703 waiting for monitor entry [0x73aff000..0x73aff74c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor75" daemon prio=1 tid=0x71b9daf8 nid=0x702 waiting for monitor entry [0x73a7f000..0x73a7f7cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor74" daemon prio=1 tid=0x71b2cde8 nid=0x701 waiting for monitor entry [0x739ff000..0x739ff84c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor73" daemon prio=1 tid=0x71d48130 nid=0x700 waiting for monitor entry [0x7397f000..0x7397f4cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor72" daemon prio=1 tid=0x71b17650 nid=0x6f2 waiting for monitor entry [0x738ff000..0x738ff54c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor71" daemon prio=1 tid=0x71b82d10 nid=0x6f1 waiting for monitor entry [0x7387f000..0x7387f5cc]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:363)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
      at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)

      "TP-Processor70" daemon prio=1 tid=0x71d69fb8 nid=0x6ef waiting for monitor entry [0x737ff000..0x737ff64c]
      at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:458)
      - waiting to lock <0x4dd937c8> (a java.lang.Object)
      at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:470)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRe

        • 1. Re: Dead locked threads encountered when upgrading from 3.2.

          After looking at the javax.management code, it appears that the thread TP-Processor137 is constructing an Mbean. Is there any other reason these methods would be called? Trying to get some traction on this ....

          • 2. Re: Dead locked threads encountered when upgrading from 3.2.
            starksm64

            By moving to jdk5 you are using the jdk implementation of the jmx components and it appears that the jdk ObjectName is causing contention.

            You can disable this registration by setting registerRequests="false" on the connector.

            • 3. Re: Dead locked threads encountered when upgrading from 3.2.

              I can see this property in JMX, although when I submit the change I get the following error:

              Cannot find setter method setModelerType

              I see a JIRA entry for this issue with no resolution at this point. It appears this will be corrected in 4.0.4.

              I grepped through config for this property with no success. Is there some place I can set this manually?

              Thanks

              • 4. Re: Dead locked threads encountered when upgrading from 3.2.
                starksm64

                What is the jira issue?

                • 5. Re: Dead locked threads encountered when upgrading from 3.2.

                  Here's the link to Jira issue:

                  http://jira.jboss.com/jira/browse/JBAS-1169

                  Is there a jk.properties file deployed with jboss4.0.3? I believe this is the right place to make the suggested configuration. I'm not finding this file in our deployment.

                  Allen

                  • 6. Re: Dead locked threads encountered when upgrading from 3.2.

                    Does Jboss provide a way to set this attribute <registerRequests=false> in 4.0.3?

                    JMX throws the error described above when attempting to apply changes to this Mbean. This boolean value does not appear to be managed in a config file. The java class "HandlerRequest" sets this attribute to true, right in the code.




                    • 7. Re: Dead locked threads encountered when upgrading from 3.2.
                      starksm64

                      Edit jbossweb-tomcat55.sar/server.xml and add request.registerRequests="false"

                       <!-- A AJP 1.3 Connector on port 8009 -->
                       <Connector port="8009" address="${jboss.bind.address}"
                       emptySessionPath="true" enableLookups="false" redirectPort="8443"
                       protocol="AJP/1.3"
                       request.registerRequests="false"
                       />
                      



                      • 8. Re: Dead locked threads encountered when upgrading from 3.2.

                        Scott,

                        The config change suggested did not solve this issue. Those waiting threads are likely a side affect and not cause. Note that this issue (accumulation of waiting threads, leading to full degradation of performance) can be duplicated by running our link checker against all 12 web sites. This activity, it seems, would be similar to a crawler indexing our sites. The assumption is that this activity is responsible for initiating the downward spiral in performance ending in no response from the server. From testing it seems this is not necessarily related to the level of concurrent activity. If we link check one site at a time, after a number of sites, we end up in this state. Interesting to note that if we link check the same site repeatedly, the server does not progress to this bad state. Instead some extra threads are created on the first test, subsequent tests do not increase the thread count. Of course when we link check all at once, threads pile up, performance degrades

                        When the server gets in this bad state CPU utilization is tapped. Our environment is running on a 2 CPU box. While in this bad state, one CPU will me maxed (90+ %) the other near (0%). At pretty short intervals (few seconds) the load is swapped from CPU1 to CPU2 and then back again, continuing to swap in this fashion. I assume the JVM is utilizing one CPU at a time.

                        In worst state, thread count (in JVM stack trace) is high, usually over 200. No dead locks are indicated in the JVM stack trace. When the system is healthy, we typically see 80 threads. I took 3 traces a few minutes apart as system performance went from bad to worse. The majority of threads are ?waiting on monitor entry?. The thread traces appear to be all executing, or waiting at a low level in the code (below our application). There are only a few threads in the runnable state. These runnable threads appear to be in normal processing based on thread stack traces, nothing stands out.

                        I?d like to attach the 3 stack traces but these are large. Can I send these via email to get more input?

                        Allen

                        • 9. Re: Dead locked threads encountered when upgrading from 3.2.
                          starksm64
                          • 10. Re: Dead locked threads encountered when upgrading from 3.2.

                            Since JIRA entry has been closed I'm posting back here with update. I have profiled this issue and see the following with respect to garbage collection:

                            As the application is exercised heavily, heap utilization increases and decreases due to GC activity. The heap space used is generally around 100m (according to profiling tool), max heap space is set to 512m. At some point heap utilization drops from a typical peak (result of GC) and flattens out, and GC begins to execute repeatedly (excessively). While this is going on, no additional heap space is recovered (heap utilization appears to be well below the max throughout). It appears that no other processes get a chance to run while the GC continues in this manner for a period of a few minutes. While in this state, CPU is pegged at 100%. After a few minutes GC goes back to one minute intervals and CPU utilization returns to 1% or 2%. Heap usage remains unchanged throughout excessive GC activity and beyond. The application continues to be unresponsive and requires a restart to clean up.

                            I continue to drill in to this. Any additional direction is appreciated.

                            Allen