1 2 Previous Next 17 Replies Latest reply on Apr 15, 2014 1:29 PM by Yaroslav Mudryy

    Wildfly8-Idle Remover Thread taking 100% CPU Usage

    karna spartacus Newbie

      hi, i am using wildfly-8.0.0.Final with embedded HornetQ. i have an web application deployed on it. After few minutes of server startup, CPU usage of machine going to 100%. i ran Jconsole and found that a thread named idleRemover is taking full CPU usage.

      In my  standalone.xml, in Datasource pool configuration, i have <flush-strategy>IdleConnections</flush-strategy> tag. i thought that may be causing the problem. so i removed that tag and tried, still same problem.

       

      Stack trace of the idle Remover thread is,

       

      "IdleRemover" daemon prio=10 tid=0x0000000001774000 nid=0x36bd runnable [0x00007f7440541000]

         java.lang.Thread.State: RUNNABLE

        at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.removeIdleConnections(SemaphoreArrayListManagedConnectionPool.java:880)

        at org.jboss.jca.core.connectionmanager.pool.idle.IdleRemover$IdleRemoverRunner.run(IdleRemover.java:275)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:744)

       

      Attached screen shots of jconsole and standalone.xml file. Can anybody tell me how can i get rid of this problem.?

        • 1. Re: Wildfly8-Idle Remover Thread taking 100% CPU Usage
          Jesper Pedersen Master

          Try with a nightly build of WildFly, and provide TRACE log from org.jboss.jca

          • 2. Re: Wildfly8-Idle Remover Thread taking 100% CPU Usage
            karna spartacus Newbie

            Thank you for your response Jesper. i'll try and get back to you with result.

            but can you please tell me what is that idleRemover thread and why it is causing the problem?

            if you have any idea about it,please share it...

             

            thank you.

            • 3. Re: Wildfly8-Idle Remover Thread taking 100% CPU Usage
              Luis Garcia Newbie

              Wildfly pegs one core to 99% usage. Test environment is Ubuntu 12.0.3 64 & 32 bits with JDK 1.7.0_u51

              Good evening - I'm testing the Wildfly 8.0.1 Snapshots and the issue is still present.

              • 4. Re: Wildfly8-Idle Remover Thread taking 100% CPU Usage
                Jesper Pedersen Master

                A "Me too" doesn't provide me with any information.

                 

                Please, take the latest WildFly 8.0.1 nightly build, and apply the latest IronJacamar 1.1.5-SNAPSHOT (http://www.ironjacamar.org/doc/userguide/1.1/en-US/html/ch03.html#installas).

                 

                Submit your configuration, steps to reproduce, and TRACE log of org.jboss.jca and org.jboss.as.connector.

                 

                You can use https://bitbucket.org/pjtr/topthreads/overview to get easy access to a thread dump.

                • 5. Re: Wildfly8-Idle Remover Thread taking 100% CPU Usage
                  karna spartacus Newbie

                  hi, Jesper.. i downloaded wildfly-8.0.1.Final-SNAPSHOT. i added my datasources to the standalone.xml by removing default ones

                  i have a java web application,when i try to deploy my application it is searching for exampleDS (which came default in standalone.xml)

                   

                  ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014613: Operation ("deploy") failed - address: ([("deployment" => "YBA.war")]) - failure description: {"JBAS014771: Services with missing/unavailable dependencies" => ["jboss.naming.context.java.module.YBA.YBA.DefaultDataSource is missing [jboss.naming.context.java.jboss.datasources.ExampleDS]"]}

                   

                  then i added back that datasource configuration then my application is working. so, why it is searching for ExampleDS?

                  • 6. Re: Re: Wildfly8-Idle Remover Thread taking 100% CPU Usage
                    karna spartacus Newbie

                    i solved that ExampleDS problem. In EE subsystem, you can find default bindings, in that  remove r change datasource to your datasource name, problem will be solved.


                    i deployed my application in wildfly-8.0.1.Final-SNAPSHOT. after 5 minutes CPU usage of my machine is going 100%. here also same "idleRemover" thread is the culprit.But the interesting thing is, in WINDOWS, i didn't face any problem. But when i try in Linux (CentOS 6.5) operating system, CPU usage is hitting 100%.

                    My datasource configuration is,

                     

                    <subsystem xmlns="urn:jboss:domain:datasources:2.0">

                                <datasources>

                                    <datasource jta="false" jndi-name="java:jboss/datasources/yba" pool-name="yba" enabled="true" use-java-context="true" use-ccm="false">

                                        <connection-url>jdbc:mysql://localhost:3306/yba</connection-url>

                                        <driver>m2mmysql</driver>

                                        <transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation>

                                        <pool>

                                            <min-pool-size>5</min-pool-size>

                                            <max-pool-size>20</max-pool-size>

                                            <prefill>true</prefill>

                                            <use-strict-min>true</use-strict-min>

                                            <flush-strategy>IdleConnections</flush-strategy>

                                        </pool>

                                        <security>

                                            <user-name>root</user-name>

                                            <password>root</password>

                                        </security>

                                        <timeout>

                                            <idle-timeout-minutes>5</idle-timeout-minutes>

                                        </timeout>

                                    </datasource>

                                    <datasource jta="false" jndi-name="java:jboss/datasources/m2m" pool-name="m2m" enabled="true" use-java-context="true" use-ccm="false">

                                        <connection-url>jdbc:mysql:localhost:3306/m2m</connection-url>

                                        <driver>mymysql</driver>

                                        <transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation>

                                        <pool>

                                            <min-pool-size>5</min-pool-size>

                                            <max-pool-size>20</max-pool-size>

                                            <prefill>true</prefill>

                                            <use-strict-min>true</use-strict-min>

                                            <flush-strategy>IdleConnections</flush-strategy>

                                        </pool>

                                        <security>

                                            <user-name>root</user-name>

                                            <password>root</password>

                                        </security>

                                        <timeout>

                                            <idle-timeout-minutes>5</idle-timeout-minutes>

                                        </timeout>

                                    </datasource>

                                    <drivers>

                                        <driver name="h2" module="com.h2database.h2">

                                            <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>

                                        </driver>

                                        <driver name="mymysql" module="my.mysql">

                                            <xa-datasource-class>com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</xa-datasource-class>

                                        </driver>

                                    </drivers>

                                </datasources>

                            </subsystem>

                     

                     

                     

                    Then, i removed below tag in both datasource configurations.

                                        <timeout>

                                            <idle-timeout-minutes>5</idle-timeout-minutes>

                                        </timeout>

                    this time CPU didn't go 100% usage, even if i acces my application left and right. Just to confirm, i again put this tag and changed idle-timeout-minutes to 2, then obviously after 2 mins CPU usage hit 100% and stayed there only.

                    so, Can you please tell me,

                    1)Why is that tag causing the problem?

                    2)why specifically in Linux operating system?

                    seems to be problem is in idle connections removal logic...

                     

                    Attached org.jboss.jca log (which has taken when CPU usage @ 100%...

                    • 7. Re: Re: Wildfly8-Idle Remover Thread taking 100% CPU Usage
                      Tomaz Cerar Master

                      Can you try with newer nightly build, one that has IronJacamar 1.1.5 in it which was merged few days back

                       

                      in your log you can see that your build is still using 1.1.4

                       

                      23:28:57,183 INFO  [org.jboss.as.connector.logging] (MSC service thread 1-1) JBAS010408: Starting JCA Subsystem (IronJacamar 1.1.4.Final)

                      • 8. Re: Wildfly8-Idle Remover Thread taking 100% CPU Usage
                        pkolaric Newbie

                        I've tested the latest nightly build and it's still the same.

                        Capture.PNG

                        • 9. Re: Wildfly8-Idle Remover Thread taking 100% CPU Usage
                          Tomaz Cerar Master

                          Guys, do you all use mysql when you see this issue?

                           

                          Also can you verify that you are using latest JDBC driver & have proper mysql datasource configuration, example can be found here https://access.redhat.com/site/documentation/en-US/JBoss_Enterprise_Application_Platform/6/html/Administration_and_Configuration_Guide/Example_MySQL_XA_Datasource1.html

                           

                           

                          --

                          tomaz

                          • 10. Re: Wildfly8-Idle Remover Thread taking 100% CPU Usage
                            pkolaric Newbie

                            I use postgresql. (with pgjdbc-ng JDBC driver)

                            • 11. Re: Wildfly8-Idle Remover Thread taking 100% CPU Usage
                              ajcmartins Newbie

                              Hello, i'am experiencing the exact same issue with Postgres and its postgresql-9.2-1000.jdbc4 driver.

                               

                              Cheers,

                              • 13. Re: Wildfly8-Idle Remover Thread taking 100% CPU Usage
                                Jesper Pedersen Master

                                Run with <use-strict-min>false</use-strict-min> -- which is the default

                                • 14. Re: Wildfly8-Idle Remover Thread taking 100% CPU Usage
                                  Luis Garcia Newbie

                                  Jesper,

                                  I can confirm that this work-around brings the CPU/Core usage back to normal levels on WildFly 8.1.0.CR1

                                  What is the root cause of the issue?

                                   

                                  Regards

                                  1 2 Previous Next