10 Replies Latest reply on Apr 24, 2014 7:08 PM by paolo.andriolo

    High CPU Load: Default I/O threads continuously active around poll0()

    paolo.andriolo

      Hi,

      I've a server (wildfly-8.0.0.Final) that use ~200% cpu.

      with jconsole I observed some threads (Default I/O) that are continuously running:

       

      bugXNIO.png

      with this stack:

      sun.nio.ch.PollArrayWrapper.poll0(Native Method)

      sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:117)

      sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:73)

      sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

      sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

      sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

      org.xnio.nio.WorkerThread.run(WorkerThread.java:497)

       

      System enviroment:

      OS Debian 3.11-2-amd64 #1 SMP Debian 3.11.10-1

      java version "1.7.0_21"

      OpenJDK Runtime Environment (IcedTea 2.3.9) (7u21-2.3.9-5)

      OpenJDK Server VM (build 23.7-b01, mixed mode)

       

      Do you think is a jvm bug, or xnio bug?

      Thanks


       

        • 1. Re: High CPU Load: Default I/O threads continuously active around poll0()
          ctomc

          two things

          1) you should absolutely update your JVM, anything u45 and up should do it

          2) also try with newer WildFly 8.1.0.CR1 if you still see problems.

          • 2. Re: High CPU Load: Default I/O threads continuously active around poll0()
            paolo.andriolo

            Hi,

            i've updated jvm:

            java version "1.7.0_51"

            OpenJDK Runtime Environment (IcedTea 2.4.6) (7u51-2.4.6-1)

            OpenJDK Server VM (build 24.51-b03, mixed mode)

             

            and the problem occurred again.

            this is the stacks:

             

            sun.nio.ch.PollArrayWrapper.poll0(Native Method)

            sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:117)

            sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:73)

            sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

            sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

            sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

            org.xnio.nio.WorkerThread.run(WorkerThread.java:497)

             

            sun.nio.ch.AbstractPollSelectorImpl.updateSelectedKeys(AbstractPollSelectorImpl.java:119)

             

            sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:78)

            sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

            sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

            sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

            org.xnio.nio.WorkerThread.run(WorkerThread.java:497)

             

            sun.nio.ch.PollArrayWrapper.poll0(Native Method)

             

            sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:117)

            sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:73)

            sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

            sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

            org.xnio.nio.WorkerThread.run(WorkerThread.java:501)

            • 3. Re: High CPU Load: Default I/O threads continuously active around poll0()
              swd847

              Can you try with Wildfly 8.1.0.CR1 ?

              • 4. Re: High CPU Load: Default I/O threads continuously active around poll0()
                ctomc

                Also, do i read that correctly, you are using debian 3.11? or is that 3.11 kernel on debian?

                 

                can you paste output of

                uname -a

                 

                It is strange that jvm is not using epoll but pool implementation, it is like OS would not support epoll.

                • 5. Re: High CPU Load: Default I/O threads continuously active around poll0()
                  ctomc

                  Also can you try forcing epoll selector by adding system property:

                  -Dxnio.nio.selector.provider=sun.nio.ch.EPollSelectorProvider

                  this should force usage of epoll selector.

                  • 6. Re: High CPU Load: Default I/O threads continuously active around poll0()
                    paolo.andriolo

                    Hi,

                    this is uname -a output:

                     

                    Linux xxxxx 3.11-2-amd64 #1 SMP Debian 3.11.10-1 (2013-12-04) x86_64 GNU/Linux

                     

                    I try to put the system property but I can't get EPollSelectorProvider. I added some printStackTrace on NioXnio.java and I get:

                     

                    java.lang.ClassNotFoundException: sun.nio.ch.EPollSelectorProvider from [Module "org.jboss.xnio.nio:main" from local module loader @1fd01f7 (finder: local module finder @18c2d48 (roots: /opt/wildfly-8.0.0.Final/modules,/opt/wildfly-8.0.0.Final/modules/system/layers/base,/home/bolapp/data/modules))]

                            at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:197)

                            at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:443)

                            at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:431)

                            at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:373)

                            at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:118)

                            at java.lang.Class.forName0(Native Method)

                            at java.lang.Class.forName(Class.java:270)

                     

                    In order to debug I try to get some ohter java classes (java.lang.String, java.awt.Book) and I get no problem, if I try to load some other sun.nio.ch classes (sun.nio.ch.AbstractPollArrayWrapper) I get ClassNotFoundException.

                    • 7. Re: Re: High CPU Load: Default I/O threads continuously active around poll0()
                      dmlloyd

                      There is definitely something very strange (and classloading-related) going on.  But just out of curiosity, can you add this line to modules/system/layers/base/org/jboss/xnio/nio/main/module.xml:

                       

                          <module name="sun.jdk"/>
                      

                       

                      in the obvious place?

                      • 8. Re: High CPU Load: Default I/O threads continuously active around poll0()
                        paolo.andriolo

                        Hi,

                         

                        adding sun.jdk module I get EPollSelector:

                         

                        2014-04-25 00:15:58,766 DEBUG [org.xnio.nio] (default I/O-3) Started channel thread 'default I/O-3', selector sun.nio.ch.EPollSelectorImpl@530a7a

                        • 9. Re: High CPU Load: Default I/O threads continuously active around poll0()
                          ctomc

                          This is now with system property set?

                          but without it, it still doesn't work?

                           

                          also with EPoll selector do you still see high cpu load?

                          • 10. Re: High CPU Load: Default I/O threads continuously active around poll0()
                            paolo.andriolo

                            with sun.jdk module dependency I get EPollSelector with or without system property.

                             

                            I'll let you know if I stiil get high cpu load.