High CPU Load: Default I/O threads continuously active around poll0()

[paolo.andriolo]

Hi,

I've a server (wildfly-8.0.0.Final) that use ~200% cpu.

with jconsole I observed some threads (Default I/O) that are continuously running:

with this stack:

sun.nio.ch.PollArrayWrapper.poll0(Native Method)

sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:117)

sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:73)

sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

org.xnio.nio.WorkerThread.run(WorkerThread.java:497)

System enviroment:

OS Debian 3.11-2-amd64 #1 SMP Debian 3.11.10-1

java version "1.7.0_21"

OpenJDK Runtime Environment (IcedTea 2.3.9) (7u21-2.3.9-5)

OpenJDK Server VM (build 23.7-b01, mixed mode)

Do you think is a jvm bug, or xnio bug?

Thanks

[ctomc]

two things

1) you should absolutely update your JVM, anything u45 and up should do it

2) also try with newer WildFly 8.1.0.CR1 if you still see problems.

[paolo.andriolo]

Hi,

i've updated jvm:

java version "1.7.0_51"

OpenJDK Runtime Environment (IcedTea 2.4.6) (7u51-2.4.6-1)

OpenJDK Server VM (build 24.51-b03, mixed mode)

and the problem occurred again.

this is the stacks:

sun.nio.ch.PollArrayWrapper.poll0(Native Method)

sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:117)

sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:73)

sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

org.xnio.nio.WorkerThread.run(WorkerThread.java:497)

sun.nio.ch.AbstractPollSelectorImpl.updateSelectedKeys(AbstractPollSelectorImpl.java:119)

sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:78)

sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

org.xnio.nio.WorkerThread.run(WorkerThread.java:497)

sun.nio.ch.PollArrayWrapper.poll0(Native Method)

sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:117)

sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:73)

sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

org.xnio.nio.WorkerThread.run(WorkerThread.java:501)

[swd847]

Can you try with Wildfly 8.1.0.CR1 ?

[ctomc]

Also, do i read that correctly, you are using debian 3.11? or is that 3.11 kernel on debian?

can you paste output of

uname -a

It is strange that jvm is not using epoll but pool implementation, it is like OS would not support epoll.

[ctomc]

Also can you try forcing epoll selector by adding system property:

-Dxnio.nio.selector.provider=sun.nio.ch.EPollSelectorProvider

this should force usage of epoll selector.

[paolo.andriolo]

Hi,

this is uname -a output:

Linux xxxxx 3.11-2-amd64 #1 SMP Debian 3.11.10-1 (2013-12-04) x86_64 GNU/Linux

I try to put the system property but I can't get EPollSelectorProvider. I added some printStackTrace on NioXnio.java and I get:

java.lang.ClassNotFoundException: sun.nio.ch.EPollSelectorProvider from [Module "org.jboss.xnio.nio:main" from local module loader @1fd01f7 (finder: local module finder @18c2d48 (roots: /opt/wildfly-8.0.0.Final/modules,/opt/wildfly-8.0.0.Final/modules/system/layers/base,/home/bolapp/data/modules))]

        at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:197)

        at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:443)

        at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:431)

        at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:373)

        at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:118)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:270)

In order to debug I try to get some ohter java classes (java.lang.String, java.awt.Book) and I get no problem, if I try to load some other sun.nio.ch classes (sun.nio.ch.AbstractPollArrayWrapper) I get ClassNotFoundException.

[dmlloyd]

There is definitely something very strange (and classloading-related) going on.  But just out of curiosity, can you add this line to modules/system/layers/base/org/jboss/xnio/nio/main/module.xml:

    <module name="sun.jdk"/>

in the obvious place?

[paolo.andriolo]

Hi,

adding sun.jdk module I get EPollSelector:

2014-04-25 00:15:58,766 DEBUG [org.xnio.nio] (default I/O-3) Started channel thread 'default I/O-3', selector sun.nio.ch.EPollSelectorImpl@530a7a

[ctomc]

This is now with system property set?

but without it, it still doesn't work?

also with EPoll selector do you still see high cpu load?

[paolo.andriolo]

with sun.jdk module dependency I get EPollSelector with or without system property.

I'll let you know if I stiil get high cpu load.