7 Replies Latest reply on Feb 25, 2008 8:16 AM by Jean-Frederic Clere

    CPU (threads) at 100%, Tomcat sockets in CLOSE_WAIT

    gggriffith Newbie

      Hello,
      I was happily testing our Jboss-hosted app today which is hosted on a Quad-Core IBM x3550 running Fedora Core 6 (2.6.18-1.2798.fc6) with a fairly recent Sun JVM for Linux "Java HotSpot(TM) Server VM (build 1.5.0_10-b03, mixed mode)". We started JBoss 4.2.1GA and I was load testing our site over Tomcat interface http//server:8080 with a browser test tool. After the test completed, I noticed 2 of my 4 cores were stuck at 100% utilization each.

      [root@devjb12 log]# top
      top - 11:47:31 up 14 days, 19:31, 2 users, load average: 2.10, 2.03, 1.49
      Tasks: 90 total, 1 running, 89 sleeping, 0 stopped, 0 zombie
      Cpu0 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
      Cpu1 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
      Cpu2 : 0.0%us, 0.0%sy, 0.0%ni, 99.3%id, 0.7%wa, 0.0%hi, 0.0%si, 0.0%st
      Cpu3 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
      Mem: 3107224k total, 2783312k used, 323912k free, 220592k buffers
      Swap: 2031608k total, 0k used, 2031608k free, 1224688k cached


      Correlating "ps" info with Linux thread ID's and the Java thread dump shows me that the threads spinning out of control are:

      "http-0.0.0.0-8080-14" daemon prio=1 tid=0x09a33b40 nid=0x5903 runnable [0x58ffd000..0x58ffef30]
      "http-0.0.0.0-8080-3" daemon prio=1 tid=0x092958c0 nid=0x46d1 runnable [0x5c6e6000..0x5c6e7db0]


      The dump file shows these two (-3 and -14) look very unlike the other http-0.0.0.0-8080-## threads:

      "http-0.0.0.0-8080-14" daemon prio=1 tid=0x09a33b40 nid=0x5903 runnable [0x58ffd000..0x58ffef30]
      at java.util.HashMap.put(HashMap.java:420)
      at com.tracker.web.servlet.SessionCounter.sessionCreated(SessionCounter.java:19)
      at org.apache.catalina.session.StandardSession.tellNew(StandardSession.java:397)
      at org.apache.catalina.session.StandardSession.setId(StandardSession.java:369)
      at org.jboss.web.tomcat.service.session.ClusteredSession.setId(ClusteredSession.java:1040)
      at org.jboss.web.tomcat.service.session.JBossCacheManager.createSession(JBossCacheManager.java:609)
      at org.apache.catalina.connector.Request.doGetSession(Request.java:2293)
      at org.apache.catalina.connector.Request.getSession(Request.java:2066)
      at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:833)
      at org.apache.tapestry.web.ServletWebRequest.getSession(ServletWebRequest.java:82)
      at $WebRequest_11784a0bc35.getSession($WebRequest_11784a0bc35.java)
      at $WebRequest_11784a0bbe9.getSession($WebRequest_11784a0bbe9.java)
      at org.apache.tapestry.engine.state.SessionScopeManager.getSession(SessionScopeManager.java:48)
      at org.apache.tapestry.engine.state.SessionScopeManager.get(SessionScopeManager.java:64)
      at $StateObjectPersistenceManager_11784a0bd0f.get($StateObjectPersistenceManager_11784a0bd0f.java)
      at org.apache.tapestry.engine.state.StateObjectManagerImpl.get(StateObjectManagerImpl.java:50)
      at org.apache.tapestry.engine.state.ApplicationStateManagerImpl.get(ApplicationStateManagerImpl.java:60)
      at $ApplicationStateManager_11784a0bbbf.get($ApplicationStateManager_11784a0bbbf.java)
      at $ApplicationStateManager_11784a0bbc0.get($ApplicationStateManager_11784a0bbc0.java)
      at $PatientInformation_14.getUserSession($PatientInformation_14.java)
      at com.tracker.web.action.AuthenticationRequiredPage.pageValidate(AuthenticationRequiredPage.java:34)
      at org.apache.tapestry.AbstractPage.firePageValidate(AbstractPage.java:585)
      at org.apache.tapestry.AbstractPage.validate(AbstractPage.java:307)
      at org.apache.tapestry.engine.RequestCycle.activate(RequestCycle.java:625)
      at org.apache.tapestry.engine.DirectService.service(DirectService.java:99)
      at $IEngineService_11784a0bc48.service($IEngineService_11784a0bc48.java)
      at org.apache.tapestry.services.impl.EngineServiceOuterProxy.service(EngineServiceOuterProxy.java:66)
      at org.apache.tapestry.engine.AbstractEngine.service(AbstractEngine.java:248)
      at org.apache.tapestry.services.impl.InvokeEngineTerminator.service(InvokeEngineTerminator.java:60)
      at $WebRequestServicer_11784a0bc24.service($WebRequestServicer_11784a0bc24.java)
      at $WebRequestServicer_11784a0bc20.service($WebRequestServicer_11784a0bc20.java)
      at org.apache.tapestry.services.impl.WebRequestServicerPipelineBridge.service(WebRequestServicerPipelineBridge.java:56)
      at $ServletRequestServicer_11784a0bc03.service($ServletRequestServicer_11784a0bc03.java)
      at org.apache.tapestry.request.DecodedRequestInjector.service(DecodedRequestInjector.java:55)
      at $ServletRequestServicerFilter_11784a0bbff.service($ServletRequestServicerFilter_11784a0bbff.java)
      at $ServletRequestServicer_11784a0bc05.service($ServletRequestServicer_11784a0bc05.java)
      at com.carmanconsulting.tapernate.filter.TapernateFilter.serviceOnce(TapernateFilter.java:110)
      at com.carmanconsulting.tapernate.filter.OncePerServletRequestServicerFilter.service(OncePerServletRequestServicerFilter.java:23)
      at $ServletRequestServicerFilter_11784a0bbfd.service($ServletRequestServicerFilter_11784a0bbfd.java)
      at $ServletRequestServicer_11784a0bc05.service($ServletRequestServicer_11784a0bc05.java)
      at org.apache.tapestry.multipart.MultipartDecoderFilter.service(MultipartDecoderFilter.java:52)
      at $ServletRequestServicerFilter_11784a0bbfb.service($ServletRequestServicerFilter_11784a0bbfb.java)
      at $ServletRequestServicer_11784a0bc05.service($ServletRequestServicer_11784a0bc05.java)
      at org.apache.tapestry.services.impl.SetupRequestEncoding.service(SetupRequestEncoding.java:53)
      at $ServletRequestServicerFilter_11784a0bc01.service($ServletRequestServicerFilter_11784a0bc01.java)
      at $ServletRequestServicer_11784a0bc05.service($ServletRequestServicer_11784a0bc05.java)
      at $ServletRequestServicer_11784a0bbf5.service($ServletRequestServicer_11784a0bbf5.java)
      at org.apache.tapestry.ApplicationServlet.doService(ApplicationServlet.java:123)
      at org.apache.tapestry.ApplicationServlet.doPost(ApplicationServlet.java:168)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at com.tracker.service.impl.TimeServiceImpl$ResetTimeServiceFilter.doFilterInternal(TimeServiceImpl.java:133)
      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.apache.tapestry.RedirectFilter.doFilter(RedirectFilter.java:103)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
      at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
      at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:87)
      at org.jboss.web.tomcat.service.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
      at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
      at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
      at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)
      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
      at java.lang.Thread.run(Thread.java:595)

      "http-0.0.0.0-8080-13" daemon prio=1 tid=0x0948cf80 nid=0x5902 in Object.wait() [0x593fa000..0x593faeb0]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x77cec2b0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
      at java.lang.Object.wait(Object.java:474)
      at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
      - locked <0x77cec2b0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
      at java.lang.Thread.run(Thread.java:595)

      "http-0.0.0.0-8080-12" daemon prio=1 tid=0x09691fe0 nid=0x5901 in Object.wait() [0x599fe000..0x599fee30]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x77cec338> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
      at java.lang.Object.wait(Object.java:474)
      at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
      - locked <0x77cec338> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
      at java.lang.Thread.run(Thread.java:595)


      etc.... One unhappy thread and two "normal" idle threads among others....


      Now, I do have two sockets lingering on CLOSE_WAIT (dell420 is the system I launched the browser test client from).

      [root@devjb12 ipv4]# lsof|grep webcache
      java 8799 javauser 62u IPv4 1416460 TCP *:webcache (LISTEN)
      java 8799 javauser 116u IPv4 1490728 TCP devjb12:webcache->dell420:alarm-clock-s (CLOSE_WAIT)
      java 8799 javauser 145u IPv4 1490721 TCP devjb12:webcache->dell420:gcmonitor (CLOSE_WAIT)

      [root@devjb12 ipv4]# netstat -an | grep :8080
      tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN
      tcp 1 0 10.20.0.28:8080 10.20.10.66:2667 CLOSE_WAIT
      tcp 1 0 10.20.0.28:8080 10.20.10.66:2660 CLOSE_WAIT


      My suspicions are that the CLOSE_WAITs are related the CPU to run off. In an hour or so, the /proc/sys/net/ipv4/tcp_keepalive_time of 7200 seconds will pass and the system will forget about the sockets and we'll see if Tomcat and the system as a whole calms down too.

      Any ideas? I'll followup as things progress, as I said, I should see calm in another 30 minutes...

      Thanks,
      -Gary

        • 1. Re: CPU (threads) at 100%, Tomcat sockets in CLOSE_WAIT
          Brian Stansberry Master

          If you take a couple more thread dumps, are the -3 and -14 threads progressing?

          Also, what does com.tracker.web.servlet.SessionCounter.sessionCreated do? Obviously it's updating a java.util.HashMap. Is it doing anything to synchronize access to that non-thread-safe object? Perhaps a ConcurrentHashMap is a better choice, since you can concurrently get the sessionCreated callback from multiple threads.

          • 2. Re: CPU (threads) at 100%, Tomcat sockets in CLOSE_WAIT
            Georges Goebel Newbie

            Hi,

            I use JBoss 4.2 on Redhat eneterprise 4 and 5 and have the same problem. JBoss runs without any problems for 4 to 5 days when suddenly the CPU usage goes to 100% without any reason (no hot redeploy, ...).
            I use the ajp port with mod_proxy to load balance between 2 nodes.
            There are several applications (+- 10 applications) on the JBoss nodes. Mainly EJB3/Seam/JSF(Icefaces), EJB3/JSF(Icefaces) or only webapps (struts,spring)

            Can somebody give me a hint ?

            Thanks,

            Georges Goebel

            • 3. Re: CPU (threads) at 100%, Tomcat sockets in CLOSE_WAIT
              Peter Johnson Master

              Did you follow Brian's suggestion and take some thread dumps? Take 2 or 3 a few seconds apart. That should pinpoint what code is using the processor. Then look at that code. (The last two times someone presented this same issue to me I found that the user's code had gone into an infinite loop.)

              • 4. Re: CPU (threads) at 100%, Tomcat sockets in CLOSE_WAIT
                gggriffith Newbie

                We've taken out the functions that were using java.util.HashMap as they were unneccessary for operation and haven't seen the issue since. Our developer related that there may be a chance for things to spin due to the thread-safe issues.

                For Georges, do you know how to dump thread CPU utilization using ps and correlating to the thread dump TID's from JBoss?

                • 5. Re: CPU (threads) at 100%, Tomcat sockets in CLOSE_WAIT
                  Georges Goebel Newbie

                  Hi,

                  I upgraded to JBoss 4.2.2 but I notice that there are also connections in "CLOSE_WAIT" status.
                  Can you show me how to "dump thread CPU utilization using ps and correlating to the thread dump TID's from JBoss"

                  Furthermore I noticed that starting the same JBoss config (same version with the same applications deployed) on RedHat 5 needs more than twice the time than on RedHat 4. (with the same JAVA_OPTS). (On VmWare ESX 3.0.X)

                  Georges Goebel

                  • 6. Re: CPU (threads) at 100%, Tomcat sockets in CLOSE_WAIT
                    Georges Goebel Newbie

                    Hi,

                    Today I had again the same problem (plenty of CLOSE_WAIT sessions and the CPU is on 100%).
                    Is somebody out there who can solve this problem. I use apache 2.2 with mod_proxy_balancer to the ajp port.

                    Changing from ajp to http does not solve my problem because the absolute url is changed. http://bla.pch.etat.lu/ap1 -> http://node1.pch.etat.lu:8080/ap1

                    Neither using the native JBoss library because the SAP native library does not like the binaries of JBoss navtive :-( (JBoss does not starte whene the SAP JCO libraries are in LD_LIBRARY_PATH)

                    Thanks for any help

                    Georges

                    <Proxy balancer://myCluster>
                     BalancerMember ajp://jbosssrv01.pch.etat.lu:8009 route=worker1 loadfactor=10
                     BalancerMember ajp://jbosssrv02.pch.etat.lu:8009 route=worker2 loadfactor=10
                    </Proxy>
                    


                    [java@jbosssrv01 ~]$ netstat -na | grep 8009
                    tcp 0 0 0.0.0.0:8009 0.0.0.0:* LISTEN
                    tcp 0 0 148.110.55.140:8009 148.110.55.136:33657 SYN_RECV
                    tcp 0 0 148.110.55.140:8009 148.110.55.136:33663 SYN_RECV
                    tcp 0 0 148.110.55.140:8009 148.110.55.136:33655 SYN_RECV
                    tcp 0 0 148.110.55.140:8009 148.110.55.136:33660 SYN_RECV
                    tcp 0 0 148.110.55.140:8009 148.110.55.136:33651 SYN_RECV
                    tcp 0 0 148.110.55.140:8009 148.110.55.136:33652 SYN_RECV
                    tcp 387 0 148.110.55.140:8009 148.110.55.136:34073 CLOSE_WAIT
                    tcp 394 0 148.110.55.140:8009 148.110.55.136:34072 CLOSE_WAIT
                    tcp 394 0 148.110.55.140:8009 148.110.55.136:34079 CLOSE_WAIT
                    tcp 375 0 148.110.55.140:8009 148.110.55.136:34078 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:34067 CLOSE_WAIT
                    tcp 440 0 148.110.55.140:8009 148.110.55.136:44816 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:37131 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:37130 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:39177 CLOSE_WAIT
                    tcp 394 0 148.110.55.140:8009 148.110.55.136:34063 CLOSE_WAIT
                    tcp 375 0 148.110.55.140:8009 148.110.55.136:34062 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:44814 CLOSE_WAIT
                    tcp 607 0 148.110.55.140:8009 148.110.55.136:44812 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:37123 CLOSE_WAIT
                    tcp 440 0 148.110.55.140:8009 148.110.55.136:44801 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:44807 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:37126 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:47929 CLOSE_WAIT
                    tcp 667 0 148.110.55.140:8009 148.110.55.136:53053 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:53042 CLOSE_WAIT
                    tcp 0 0 148.110.55.140:8009 148.110.55.136:55346 ESTABLISHED
                    tcp 272 0 148.110.55.140:8009 148.110.55.136:53046 CLOSE_WAIT
                    tcp 272 0 148.110.55.140:8009 148.110.55.136:53033 CLOSE_WAIT
                    tcp 387 0 148.110.55.140:8009 148.110.55.136:34088 CLOSE_WAIT
                    tcp 742 0 148.110.55.140:8009 148.110.55.136:53039 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:44588 CLOSE_WAIT
                    tcp 532 0 148.110.55.140:8009 148.110.55.136:53027 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:34083 CLOSE_WAIT
                    tcp 537 0 148.110.55.140:8009 148.110.55.136:53026 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:53025 CLOSE_WAIT
                    tcp 394 0 148.110.55.140:8009 148.110.55.136:34087 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:33575 CLOSE_WAIT
                    tcp 649 0 148.110.55.140:8009 148.110.55.136:53029 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:53082 CLOSE_WAIT
                    tcp 0 0 148.110.55.140:8009 148.110.55.136:55647 ESTABLISHED
                    tcp 313 0 148.110.55.140:8009 148.110.55.136:53843 CLOSE_WAIT
                    tcp 721 0 148.110.55.140:8009 148.110.55.136:53842 CLOSE_WAIT
                    tcp 532 0 148.110.55.140:8009 148.110.55.136:53840 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:42071 CLOSE_WAIT
                    tcp 537 0 148.110.55.140:8009 148.110.55.136:53839 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:36173 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:47939 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:47938 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:53314 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:44613 CLOSE_WAIT
                    tcp 313 0 148.110.55.140:8009 148.110.55.136:53828 CLOSE_WAIT
                    tcp 527 0 148.110.55.140:8009 148.110.55.136:42107 CLOSE_WAIT
                    tcp 887 0 148.110.55.140:8009 148.110.55.136:35707 CLOSE_WAIT
                    tcp 533 0 148.110.55.140:8009 148.110.55.136:42106 CLOSE_WAIT
                    tcp 492 0 148.110.55.140:8009 148.110.55.136:42105 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:35704 CLOSE_WAIT
                    tcp 533 0 148.110.55.140:8009 148.110.55.136:42110 CLOSE_WAIT
                    tcp 527 0 148.110.55.140:8009 148.110.55.136:42109 CLOSE_WAIT
                    tcp 417 0 148.110.55.140:8009 148.110.55.136:35709 CLOSE_WAIT
                    tcp 0 0 148.110.55.140:8009 148.110.55.136:33139 ESTABLISHED
                    tcp 492 0 148.110.55.140:8009 148.110.55.136:42098 CLOSE_WAIT
                    tcp 492 0 148.110.55.140:8009 148.110.55.136:42092 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:48739 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:41826 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:41830 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:41828 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:44388 CLOSE_WAIT
                    tcp 769 0 148.110.55.140:8009 148.110.55.136:35730 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:57232 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:35734 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:37782 CLOSE_WAIT
                    tcp 566 0 148.110.55.140:8009 148.110.55.136:43158 CLOSE_WAIT
                    tcp 416 0 148.110.55.140:8009 148.110.55.136:57236 CLOSE_WAIT
                    tcp 489 0 148.110.55.140:8009 148.110.55.136:35732 CLOSE_WAIT
                    tcp 468 0 148.110.55.140:8009 148.110.55.136:43146 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:41097 CLOSE_WAIT
                    tcp 533 0 148.110.55.140:8009 148.110.55.136:42121 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:35720 CLOSE_WAIT
                    tcp 0 0 148.110.55.140:8009 148.110.55.136:59279 ESTABLISHED
                    tcp 814 0 148.110.55.140:8009 148.110.55.136:35724 CLOSE_WAIT
                    tcp 527 0 148.110.55.140:8009 148.110.55.136:43139 CLOSE_WAIT
                    tcp 416 0 148.110.55.140:8009 148.110.55.136:57223 CLOSE_WAIT
                    tcp 489 0 148.110.55.140:8009 148.110.55.136:35719 CLOSE_WAIT
                    tcp 533 0 148.110.55.140:8009 148.110.55.136:43143 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:41094 CLOSE_WAIT
                    tcp 527 0 148.110.55.140:8009 148.110.55.136:43142 CLOSE_WAIT
                    tcp 589 0 148.110.55.140:8009 148.110.55.136:35717 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:43141 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:57220 CLOSE_WAIT
                    tcp 769 0 148.110.55.140:8009 148.110.55.136:35716 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:57528 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:57535 CLOSE_WAIT
                    tcp 570 0 148.110.55.140:8009 148.110.55.136:57533 CLOSE_WAIT
                    tcp 516 0 148.110.55.140:8009 148.110.55.136:57525 CLOSE_WAIT
                    tcp 570 0 148.110.55.140:8009 148.110.55.136:57524 CLOSE_WAIT
                    tcp 318 0 148.110.55.140:8009 148.110.55.136:44714 CLOSE_WAIT
                    tcp 308 0 148.110.55.140:8009 148.110.55.136:44713 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:57512 CLOSE_WAIT
                    tcp 570 0 148.110.55.140:8009 148.110.55.136:57518 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:44718 CLOSE_WAIT
                    tcp 566 0 148.110.55.140:8009 148.110.55.136:43170 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:57511 CLOSE_WAIT
                    tcp 308 0 148.110.55.140:8009 148.110.55.136:44711 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:44709 CLOSE_WAIT
                    tcp 0 0 148.110.55.140:8009 148.110.55.136:41691 ESTABLISHED
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:42459 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:45787 CLOSE_WAIT
                    tcp 643 0 148.110.55.140:8009 148.110.55.136:55000 CLOSE_WAIT
                    tcp 560 0 148.110.55.140:8009 148.110.55.136:42463 CLOSE_WAIT
                    tcp 513 0 148.110.55.140:8009 148.110.55.136:45790 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:41693 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:55004 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:41682 CLOSE_WAIT
                    tcp 560 0 148.110.55.140:8009 148.110.55.136:42450 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:50386 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:41680 CLOSE_WAIT
                    tcp 416 0 148.110.55.140:8009 148.110.55.136:42454 CLOSE_WAIT
                    tcp 651 0 148.110.55.140:8009 148.110.55.136:50389 CLOSE_WAIT
                    tcp 651 0 148.110.55.140:8009 148.110.55.136:54997 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:54984 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:42446 CLOSE_WAIT
                    tcp 596 0 148.110.55.140:8009 148.110.55.136:54989 CLOSE_WAIT
                    tcp 570 0 148.110.55.140:8009 148.110.55.136:57539 CLOSE_WAIT
                    tcp 518 0 148.110.55.140:8009 148.110.55.136:57542 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:33221 CLOSE_WAIT
                    tcp 516 0 148.110.55.140:8009 148.110.55.136:57541 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:33220 CLOSE_WAIT
                    tcp 422 0 148.110.55.140:8009 148.110.55.136:45818 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:44793 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:38905 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:36600 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:37112 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:44799 CLOSE_WAIT
                    tcp 607 0 148.110.55.140:8009 148.110.55.136:44797 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:37116 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:45810 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:44790 CLOSE_WAIT
                    tcp 589 0 148.110.55.140:8009 148.110.55.136:42472 CLOSE_WAIT
                    tcp 422 0 148.110.55.140:8009 148.110.55.136:45805 CLOSE_WAIT
                    tcp 306 0 148.110.55.140:8009 148.110.55.136:38893 CLOSE_WAIT
                    tcp 513 0 148.110.55.140:8009 148.110.55.136:45804 CLOSE_WAIT
                    tcp 518 0 148.110.55.140:8009 148.110.55.136:45794 CLOSE_WAIT
                    tcp 521 0 148.110.55.140:8009 148.110.55.136:45793 CLOSE_WAIT
                    tcp 417 0 148.110.55.140:8009 148.110.55.136:42465 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:37095 CLOSE_WAIT
                    tcp 1 0 148.110.55.140:8009 148.110.55.136:37094 CLOSE_WAIT
                    tcp 416 0 148.110.55.140:8009 148.110.55.136:42468 CLOSE_WAIT
                    


                    • 7. Re: CPU (threads) at 100%, Tomcat sockets in CLOSE_WAIT
                      Jean-Frederic Clere Master

                      For mod_proxy_http configuration see:
                      http://wiki.jboss.org/wiki/Wiki.jsp?page=UsingMod_proxyWithJBoss
                      You just need 2 ProxyPassReverse directive.

                      The CLOSE_WAIT is the known problem it has been fixed.
                      http://jira.jboss.org/jira/browse/JBPAPP-366