1 Reply Latest reply on Mar 28, 2018 6:07 AM by rhusar

    Huge number of Jgroup's Timer runner threads (Wildfly 11)

    jason513
    jason513 Mar 27, 2018 11:49 PM

      We are using Jolokia for exposing JMX over http. For this purpose we deployed jolokia.war file in Wildfly 11 server and using telegraph agent. What we noticed is querying jolokia endpoint at regular intervals causing creation of huge number of threads, especially JGorups related "Timer runner" threads. When we did the thread dump we noticed following stack trace.

       

      "Timer runner-10909,shared=<CLUSTER>" #11747 daemon prio=5 os_prio=0 tid=0x0000000015c17000 nid=0x4b60 waiting on condition [0x00007fdc9dae2000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007b9bf33e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
    at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
    at org.jgroups.util.TimeScheduler3.run(TimeScheduler3.java:166)
    at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
    at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory$$Lambda$932/964967519.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:748)

       

       

      I tried following settings in standlaone-ha.xml file to limit the JGroup threads but it is not limiting threads created by "Timer runner" thead pool and it looks like these threads are stuck.

       

      <transport type="TCP" socket-binding="jgroups-tcp" site="${env.DATA_CENTER}">  

          <property name="external_addr"> ${jgroups.tcp.external_addr:}</property>

          <default-thread-pool min-threads="10" max-threads="150" queue-length="100" keepalive-time="60000"/>

          <internal-thread-pool min-threads="2" max-threads="4" queue-length="100" keepalive-time="60000"/>

          <oob-thread-pool min-threads="10" max-threads="150" queue-length="0" keepalive-time="60000"/>

          <timer-thread-pool min-threads="2" max-threads="4" queue-length="500" keepalive-time="5000"/>

      </transport>

       

      Any insight into why we are having lot of "Timer runner" threads when jolokia endpoint is queried and why above settings are not limiting number of JGroup threads is appreciated?