4 Replies Latest reply on Oct 21, 2015 8:27 AM by nsoman

    apache-tomcat - APR does not understand this error code

    nsoman
    nsoman Oct 20, 2015 11:28 AM

      We are on production with Apache/2.2.27 and Tomcat Version 7.0.55 on linux(2.6.32-431.37.1.el6.x86_64) with 6 core. what we're seeing is below error on webserver which has no pattern as to when it occurs.

      (104)Connection reset by peer: ajp_ilink_receive() can't receive header ajp_read_header: ajp_ilink_receive failed (120006)APR does not understand this error code: proxy: read response failed from <server ip>:<port> (<server ip>)

      it happens quite frequently when we do load test and on normal traffic it happens but rarely. As load ramps up we see cpu utilization going up(not hinting that this is reason) but we start seeing this error usually happens once we cross about 300 user load.

      we also have firewall in play between webserver and appserver.

      After we see above error webserver jumps from one app server to another app server. probably thinks original server is marked down. This has become very comman with load going up hence I took thread dump to see what's happening and I did not find any blocked threads. I can post whole thread dump if required here is unique thread stack trace where most of threads are under load.

      1) "ajp-bio-8009-exec-3396" daemon prio=10 tid=0x00007fa165b99800 nid=0x538b runnable [0x00007fa069ad9000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:312) at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:367) at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:118) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314) - locked <0x00000007d2998118> (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) 
      

       
      

      2) "pool-8-thread-1281" prio=10 tid=0x00007fa0de140800 nid=0x5368 waiting on condition [0x00007fa0721d9000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000050e5f11f8> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 
      

       
      

      3) "PooledThread[50]" prio=10 tid=0x00007fa0c4125000 nid=0x734 in Object.wait() [0x00007fa0f064e000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000006915ccf60> (a de.hybris.platform.util.threadpool.PoolableThread) at java.lang.Object.wait(Object.java:503) at de.hybris.platform.util.threadpool.PoolableThread.resetAndReturnToPool(PoolableThread.java:246) - locked <0x00000006915ccf60> (a de.hybris.platform.util.threadpool.PoolableThread) at de.hybris.platform.util.threadpool.PoolableThread.run(PoolableThread.java:219) 
      

       
      

      4) "PooledThread[49]" prio=10 tid=0x00007fa0b801f800 nid=0x733 in Object.wait() [0x00007fa066cab000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x0000000653594388> (a de.hybris.platform.util.threadpool.PoolableThread) at java.lang.Object.wait(Object.java:503) at de.hybris.platform.util.threadpool.PoolableThread.resetAndReturnToPool(PoolableThread.java:246) - locked <0x0000000653594388> (a de.hybris.platform.util.threadpool.PoolableThread) at de.hybris.platform.util.threadpool.PoolableThread.run(PoolableThread.java:219) 
      

       
      

      5) "FD_SOCK client connection handler,hybris-broadcast,hybrisnode-2" daemon prio=10 tid=0x00007fa0bc09d800 nid=0x67b7 runnable [0x00007fa06b9f8000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.net.SocketInputStream.read(SocketInputStream.java:210) at org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1120) at java.lang.Thread.run(Thread.java:745)

      Here's our AJP connector config on tomcat

      <Connector protocol="AJP/1.3" port="8009" redirectPort="443" useIPVHosts="true" URIEncoding="UTF-8" maxThreads="850" minSpareThreads="100" connectionTimeout="30000" backlog="1000" keepAliveTimeout="60000" maxConnections="700" acceptorThreadCount="4"/>

      And below is apache conf

      prefork.c StartServers 20 MaxSpareServers 40 MinSpareServers 40 ServerLimit 700 MaxClients 700 IfModule

      and lb.conf

      BalancerMember ajp://<host_ip>:8009 route=app5 retry=60 ProxySet stickysession=JSESSIONID ProxySet lbmethod=byrequests failonstatus=503,504 ProxySet disablereuse=On