0 Replies Latest reply on Aug 29, 2018 11:02 PM by chihpenglin

    [Remoting "endpoint" task] threads keep growing

    chihpenglin
    chihpenglin Aug 29, 2018 11:02 PM

      We are using Wildfly 10.1.0 Final.

      We encountered an OutOfMemoryError caused by threads kept growing.

      After examining the thread dump.

      We found that there are thousands of Remoting "endpoint" task-N threads.

       

      What are Remoting "endpoint" task-N threads for?

      Are they created by jobss-remoting?

       

      After restarting the server, we found that in the begining, there were only 16 threads of them:

      Remoting "endpoint" task-1 ... Remoting "endpoint" task-16.

       

      After the server run for serveral days or months, there may be hundred or thousands of Remoting threads:

       

      A snippet of thread dump is listed below.

      In this thread dump, there are several "Remoting "endpoint" task-11" with different number.

      So are other tasks such as task-1 to task-16.

      All these threads were doing nothing but waiting.

       

      "Remoting "endpoint" task-11" #55415 daemon prio=5 os_prio=0 tid=0x00007f2b8c0a8000 nid=0x276e waiting on condition [0x00007f280a36c000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabee2b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-11" #55417 daemon prio=5 os_prio=0 tid=0x00007f2ba003f800 nid=0x276d waiting on condition [0x00007f2794818000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabecf40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-11" #55414 daemon prio=5 os_prio=0 tid=0x00007f2b98023800 nid=0x276b waiting on condition [0x00007f2792bfc000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabeda50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-10" #55411 daemon prio=5 os_prio=0 tid=0x00007f2ba003e000 nid=0x276a waiting on condition [0x00007f27926f7000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabecf40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-10" #55413 daemon prio=5 os_prio=0 tid=0x00007f2b8c0a7000 nid=0x2769 waiting on condition [0x00007f27927f8000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabee2b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-10" #55412 daemon prio=5 os_prio=0 tid=0x00007f2b98022800 nid=0x2768 waiting on condition [0x00007f27c4815000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabeda50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-9" #55372 daemon prio=5 os_prio=0 tid=0x00007f2c7408f000 nid=0x41df waiting on condition [0x00007f27907d8000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabece88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-8" #55369 daemon prio=5 os_prio=0 tid=0x00007f2c7408d000 nid=0x41dd waiting on condition [0x00007f27909da000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabece88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-7" #55368 daemon prio=5 os_prio=0 tid=0x00007f2c7408b000 nid=0x41dc waiting on condition [0x00007f2790adb000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabece88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-6" #55367 daemon prio=5 os_prio=0 tid=0x00007f2c74089000 nid=0x41db waiting on condition [0x00007f2790bdc000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabece88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-5" #55366 daemon prio=5 os_prio=0 tid=0x00007f2c74087000 nid=0x41da waiting on condition [0x00007f2790cdd000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabece88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-4" #55365 daemon prio=5 os_prio=0 tid=0x00007f2c74085000 nid=0x41d9 waiting on condition [0x00007f2790dde000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabece88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-9" #55364 daemon prio=5 os_prio=0 tid=0x00007f2bd813c000 nid=0x41d8 waiting on condition [0x00007f2790edf000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabed500> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-9" #55363 daemon prio=5 os_prio=0 tid=0x00007f2bf4044000 nid=0x41d7 waiting on condition [0x00007f2790fe0000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabee3c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

      ....

      • 195 Views
      • Tags: