High CPU usage after redeployment

[stanislav.belyakov]

Hello everyone, after migrating from Jboss 7.1.1 to Wildfly I faced a problem. Our environment uses maven for compiling and deploying ear servlet package. First deployment is always OK (Wildfly CPU usage is normal), any other redeployments are OK too if there wasn't any attempts to open servlet pages. Otherwise after redeployment Wildfly uses 100 of CPU. Profiler shows permanent creating and terminating tremendous count of threads (see attached pics below). This issue has been reproduced on Windows 7 x64 (JVM - JDK7u55 x86), used Wildfly 8.0.0.Final (Undertow 1.0.0 and 1.0.6), Wildfly 8.1.0.CR1 (Undertow 1.0.6) and Wildfly nightly build №1085 (Undertow 1.0.7). Note: everything works fine on OS X.

Threads usage:

CPU usage:

[jaikiran]

"XNIO-1 I/O-2" prio=6 tid=0x396dfc00 nid=0x350c runnable [0x3b91f000]

   java.lang.Thread.State: RUNNABLE

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

    at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

    - locked <0x0fa8f460> (a sun.nio.ch.Util$2)

    - locked <0x0fa8f450> (a java.util.Collections$UnmodifiableSet)

    - locked <0x0fa8f308> (a sun.nio.ch.WindowsSelectorImpl)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

    at org.xnio.nio.WorkerThread.run(WorkerThread.java:502)

   Locked ownable synchronizers:

    - None

"XNIO-1 I/O-1" prio=6 tid=0x397d0c00 nid=0x5f90 runnable [0x3b83f000]

   java.lang.Thread.State: RUNNABLE

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

    at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

    - locked <0x0fa8bc38> (a sun.nio.ch.Util$2)

    - locked <0x0fa8bc28> (a java.util.Collections$UnmodifiableSet)

    - locked <0x0fa8bae0> (a sun.nio.ch.WindowsSelectorImpl)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

    at org.xnio.nio.WorkerThread.run(WorkerThread.java:502)

   Locked ownable synchronizers:

    - None

"Remoting "alexander-win7:MANAGEMENT" I/O-1" prio=6 tid=0x39597800 nid=0x2488 runnable [0x3a94f000]

   java.lang.Thread.State: RUNNABLE

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

    at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

    - locked <0x0f786290> (a sun.nio.ch.Util$2)

    - locked <0x0f786280> (a java.util.Collections$UnmodifiableSet)

    - locked <0x0f786138> (a sun.nio.ch.WindowsSelectorImpl)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

    at org.xnio.nio.WorkerThread.run(WorkerThread.java:502)

   Locked ownable synchronizers:

    - None

"default I/O-8" prio=6 tid=0x393ba800 nid=0x46e0 runnable [0x3a7af000]

   java.lang.Thread.State: RUNNABLE

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

    at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

    - locked <0x0f7a4688> (a sun.nio.ch.Util$2)

    - locked <0x0f7a4678> (a java.util.Collections$UnmodifiableSet)

    - locked <0x0f7a4530> (a sun.nio.ch.WindowsSelectorImpl)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

    at org.xnio.nio.WorkerThread.run(WorkerThread.java:502)

   Locked ownable synchronizers:

    - None

"default I/O-7" prio=6 tid=0x393ba000 nid=0x4790 runnable [0x3ab4f000]

   java.lang.Thread.State: RUNNABLE

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

    at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

    - locked <0x0f7a0e58> (a sun.nio.ch.Util$2)

    - locked <0x0f7a0e48> (a java.util.Collections$UnmodifiableSet)

    - locked <0x0f7a0d00> (a sun.nio.ch.WindowsSelectorImpl)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

    at org.xnio.nio.WorkerThread.run(WorkerThread.java:502)

   Locked ownable synchronizers:

    - None

"default I/O-6" prio=6 tid=0x393b9c00 nid=0x28bc runnable [0x3a8be000]

   java.lang.Thread.State: RUNNABLE

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

    at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

    - locked <0x0f79d628> (a sun.nio.ch.Util$2)

    - locked <0x0f79d618> (a java.util.Collections$UnmodifiableSet)

    - locked <0x0f79d4d0> (a sun.nio.ch.WindowsSelectorImpl)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

    at org.xnio.nio.WorkerThread.run(WorkerThread.java:502)

   Locked ownable synchronizers:

    - None

"default I/O-5" prio=6 tid=0x393b9400 nid=0x2204 runnable [0x3a44e000]

   java.lang.Thread.State: RUNNABLE

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

    at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

    - locked <0x0f799df8> (a sun.nio.ch.Util$2)

    - locked <0x0f799de8> (a java.util.Collections$UnmodifiableSet)

    - locked <0x0f799ca0> (a sun.nio.ch.WindowsSelectorImpl)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

    at org.xnio.nio.WorkerThread.run(WorkerThread.java:502)

   Locked ownable synchronizers:

    - None

"default I/O-4" prio=6 tid=0x393b9000 nid=0x5244 runnable [0x3a75f000]

   java.lang.Thread.State: RUNNABLE

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

    at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

    - locked <0x0f7965c8> (a sun.nio.ch.Util$2)

    - locked <0x0f7965b8> (a java.util.Collections$UnmodifiableSet)

    - locked <0x0f796470> (a sun.nio.ch.WindowsSelectorImpl)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

    at org.xnio.nio.WorkerThread.run(WorkerThread.java:502)

   Locked ownable synchronizers:

    - None

"default I/O-3" prio=6 tid=0x393b8800 nid=0x44ec runnable [0x3a86f000]

   java.lang.Thread.State: RUNNABLE

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

    at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

    - locked <0x0f78eb88> (a sun.nio.ch.Util$2)

    - locked <0x0f78eb78> (a java.util.Collections$UnmodifiableSet)

    - locked <0x0f78ea30> (a sun.nio.ch.WindowsSelectorImpl)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

    at org.xnio.nio.WorkerThread.run(WorkerThread.java:502)

   Locked ownable synchronizers:

    - None

"default I/O-2" prio=6 tid=0x393b8400 nid=0x4818 runnable [0x3a61e000]

   java.lang.Thread.State: RUNNABLE

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

    at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

    - locked <0x0f78b358> (a sun.nio.ch.Util$2)

    - locked <0x0f78b348> (a java.util.Collections$UnmodifiableSet)

    - locked <0x0f78b200> (a sun.nio.ch.WindowsSelectorImpl)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

    at org.xnio.nio.WorkerThread.run(WorkerThread.java:502)

   Locked ownable synchronizers:

    - None

"default I/O-1" prio=6 tid=0x393b7c00 nid=0x61a0 runnable [0x3a5ae000]

   java.lang.Thread.State: RUNNABLE

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

    at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

    at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

    - locked <0x0f7879e8> (a sun.nio.ch.Util$2)

    - locked <0x0f7879d8> (a java.util.Collections$UnmodifiableSet)

    - locked <0x0f787890> (a sun.nio.ch.WindowsSelectorImpl)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

    at org.xnio.nio.WorkerThread.run(WorkerThread.java:506)

   Locked ownable synchronizers:

    - None

This looks related to these:

https://community.jboss.org/message/870675

[8.1.0] Add sun.jdk dependancy to xnio nio module so provider detection works properly by ctomc · Pull Request #6201 · w…

You mention you tried one of the nightly builds. I haven't checked if that build number 1085 had this fix. Can you check against the latest nightly build WildFly nightly builds available and see if it works there?

[stanislav.belyakov]

Still able to reproduce on latest Wildfly build https://ci.jboss.org/hudson/job/WildFly-latest-master/1092/ . Notice that redeployment works fine until first http request to deployed server content.

[jaikiran]

Please file a JIRA here WildFly - JBoss Issue Tracker with relevant details and if possible attach a sample application which reproduces this.

[ctomc]

jaikiran pai wrote:

Please file a JIRA here WildFly - JBoss Issue Tracker with relevant details and if possible attach a sample application which reproduces this.

Yes! I cannot stress enough that we cannot help with problems like this, without reproducible case.