4 Replies Latest reply on May 5, 2014 8:50 AM by Tomaz Cerar

    High CPU usage after redeployment

    Stanislav Belyakov Newbie

      Hello everyone, after migrating from Jboss 7.1.1 to Wildfly I faced a problem. Our environment uses maven for compiling and deploying ear servlet package. First deployment is always OK (Wildfly CPU usage is normal), any other redeployments are OK too if there wasn't any attempts to open servlet pages. Otherwise after redeployment Wildfly uses 100 of CPU. Profiler shows permanent creating and terminating tremendous count of threads (see attached pics below). This issue has been reproduced on Windows 7 x64 (JVM - JDK7u55 x86), used Wildfly 8.0.0.Final (Undertow 1.0.0 and 1.0.6), Wildfly 8.1.0.CR1 (Undertow 1.0.6) and Wildfly nightly build №1085 (Undertow 1.0.7). Note: everything works fine on OS X.

       

      Threads usage:

      ThreadsUsage.png

      CPU usage:

      CPUUsage.png

        • 1. Re: High CPU usage after redeployment
          jaikiran pai Master

          "XNIO-1 I/O-2" prio=6 tid=0x396dfc00 nid=0x350c runnable [0x3b91f000]

             java.lang.Thread.State: RUNNABLE

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

              at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

              - locked <0x0fa8f460> (a sun.nio.ch.Util$2)

              - locked <0x0fa8f450> (a java.util.Collections$UnmodifiableSet)

              - locked <0x0fa8f308> (a sun.nio.ch.WindowsSelectorImpl)

              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

              at org.xnio.nio.WorkerThread.run(WorkerThread.java:502)

           

             Locked ownable synchronizers:

              - None

           

          "XNIO-1 I/O-1" prio=6 tid=0x397d0c00 nid=0x5f90 runnable [0x3b83f000]

             java.lang.Thread.State: RUNNABLE

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

              at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

              - locked <0x0fa8bc38> (a sun.nio.ch.Util$2)

              - locked <0x0fa8bc28> (a java.util.Collections$UnmodifiableSet)

              - locked <0x0fa8bae0> (a sun.nio.ch.WindowsSelectorImpl)

              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

              at org.xnio.nio.WorkerThread.run(WorkerThread.java:502)

           

             Locked ownable synchronizers:

              - None

           

          "Remoting "alexander-win7:MANAGEMENT" I/O-1" prio=6 tid=0x39597800 nid=0x2488 runnable [0x3a94f000]

             java.lang.Thread.State: RUNNABLE

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

              at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

              - locked <0x0f786290> (a sun.nio.ch.Util$2)

              - locked <0x0f786280> (a java.util.Collections$UnmodifiableSet)

              - locked <0x0f786138> (a sun.nio.ch.WindowsSelectorImpl)

              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

              at org.xnio.nio.WorkerThread.run(WorkerThread.java:502)

           

             Locked ownable synchronizers:

              - None

           

          "default I/O-8" prio=6 tid=0x393ba800 nid=0x46e0 runnable [0x3a7af000]

             java.lang.Thread.State: RUNNABLE

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

              at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

              - locked <0x0f7a4688> (a sun.nio.ch.Util$2)

              - locked <0x0f7a4678> (a java.util.Collections$UnmodifiableSet)

              - locked <0x0f7a4530> (a sun.nio.ch.WindowsSelectorImpl)

              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

              at org.xnio.nio.WorkerThread.run(WorkerThread.java:502)

           

             Locked ownable synchronizers:

              - None

           

          "default I/O-7" prio=6 tid=0x393ba000 nid=0x4790 runnable [0x3ab4f000]

             java.lang.Thread.State: RUNNABLE

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

              at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

              - locked <0x0f7a0e58> (a sun.nio.ch.Util$2)

              - locked <0x0f7a0e48> (a java.util.Collections$UnmodifiableSet)

              - locked <0x0f7a0d00> (a sun.nio.ch.WindowsSelectorImpl)

              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

              at org.xnio.nio.WorkerThread.run(WorkerThread.java:502)

           

             Locked ownable synchronizers:

              - None

           

          "default I/O-6" prio=6 tid=0x393b9c00 nid=0x28bc runnable [0x3a8be000]

             java.lang.Thread.State: RUNNABLE

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

              at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

              - locked <0x0f79d628> (a sun.nio.ch.Util$2)

              - locked <0x0f79d618> (a java.util.Collections$UnmodifiableSet)

              - locked <0x0f79d4d0> (a sun.nio.ch.WindowsSelectorImpl)

              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

              at org.xnio.nio.WorkerThread.run(WorkerThread.java:502)

           

             Locked ownable synchronizers:

              - None

           

          "default I/O-5" prio=6 tid=0x393b9400 nid=0x2204 runnable [0x3a44e000]

             java.lang.Thread.State: RUNNABLE

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

              at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

              - locked <0x0f799df8> (a sun.nio.ch.Util$2)

              - locked <0x0f799de8> (a java.util.Collections$UnmodifiableSet)

              - locked <0x0f799ca0> (a sun.nio.ch.WindowsSelectorImpl)

              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

              at org.xnio.nio.WorkerThread.run(WorkerThread.java:502)

           

             Locked ownable synchronizers:

              - None

           

          "default I/O-4" prio=6 tid=0x393b9000 nid=0x5244 runnable [0x3a75f000]

             java.lang.Thread.State: RUNNABLE

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

              at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

              - locked <0x0f7965c8> (a sun.nio.ch.Util$2)

              - locked <0x0f7965b8> (a java.util.Collections$UnmodifiableSet)

              - locked <0x0f796470> (a sun.nio.ch.WindowsSelectorImpl)

              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

              at org.xnio.nio.WorkerThread.run(WorkerThread.java:502)

           

             Locked ownable synchronizers:

              - None

           

          "default I/O-3" prio=6 tid=0x393b8800 nid=0x44ec runnable [0x3a86f000]

             java.lang.Thread.State: RUNNABLE

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

              at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

              - locked <0x0f78eb88> (a sun.nio.ch.Util$2)

              - locked <0x0f78eb78> (a java.util.Collections$UnmodifiableSet)

              - locked <0x0f78ea30> (a sun.nio.ch.WindowsSelectorImpl)

              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

              at org.xnio.nio.WorkerThread.run(WorkerThread.java:502)

           

             Locked ownable synchronizers:

              - None

           

          "default I/O-2" prio=6 tid=0x393b8400 nid=0x4818 runnable [0x3a61e000]

             java.lang.Thread.State: RUNNABLE

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

              at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

              - locked <0x0f78b358> (a sun.nio.ch.Util$2)

              - locked <0x0f78b348> (a java.util.Collections$UnmodifiableSet)

              - locked <0x0f78b200> (a sun.nio.ch.WindowsSelectorImpl)

              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

              at org.xnio.nio.WorkerThread.run(WorkerThread.java:502)

           

             Locked ownable synchronizers:

              - None

           

          "default I/O-1" prio=6 tid=0x393b7c00 nid=0x61a0 runnable [0x3a5ae000]

             java.lang.Thread.State: RUNNABLE

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

              at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

              at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

              - locked <0x0f7879e8> (a sun.nio.ch.Util$2)

              - locked <0x0f7879d8> (a java.util.Collections$UnmodifiableSet)

              - locked <0x0f787890> (a sun.nio.ch.WindowsSelectorImpl)

              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

              at org.xnio.nio.WorkerThread.run(WorkerThread.java:506)

           

             Locked ownable synchronizers:

              - None

          This looks related to these:

           

          https://community.jboss.org/message/870675

          [8.1.0] Add sun.jdk dependancy to xnio nio module so provider detection works properly by ctomc · Pull Request #6201 · w…

           

          You mention you tried one of the nightly builds. I haven't checked if that build number 1085 had this fix. Can you check against the latest nightly build WildFly nightly builds available and see if it works there?

          • 2. Re: High CPU usage after redeployment
            Stanislav Belyakov Newbie

            Still able to reproduce on latest Wildfly build https://ci.jboss.org/hudson/job/WildFly-latest-master/1092/ . Notice that redeployment works fine until first http request to deployed server content.

            • 3. Re: High CPU usage after redeployment
              jaikiran pai Master

              Please file a JIRA here WildFly - JBoss Issue Tracker with relevant details and if possible attach a sample application which reproduces this.

              • 4. Re: High CPU usage after redeployment
                Tomaz Cerar Master

                jaikiran pai wrote:

                 

                Please file a JIRA here WildFly - JBoss Issue Tracker with relevant details and if possible attach a sample application which reproduces this.

                Yes! I cannot stress enough that we cannot help with problems like this, without reproducible case.