0 Replies Latest reply on Aug 29, 2018 11:02 PM by Chihpeng Lin

    [Remoting "endpoint" task] threads keep growing

    Chihpeng Lin Newbie

      We are using Wildfly 10.1.0 Final.

      We encountered an OutOfMemoryError caused by threads kept growing.

      After examining the thread dump.

      We found that there are thousands of Remoting "endpoint" task-N threads.

       

      What are Remoting "endpoint" task-N threads for?

      Are they created by jobss-remoting?

       

      After restarting the server, we found that in the begining, there were only 16 threads of them:

      Remoting "endpoint" task-1 ... Remoting "endpoint" task-16.

       

      After the server run for serveral days or months, there may be hundred or thousands of Remoting threads:

       

      A snippet of thread dump is listed below.

      In this thread dump, there are several "Remoting "endpoint" task-11" with different number.

      So are other tasks such as task-1 to task-16.

      All these threads were doing nothing but waiting.

       

      "Remoting "endpoint" task-11" #55415 daemon prio=5 os_prio=0 tid=0x00007f2b8c0a8000 nid=0x276e waiting on condition [0x00007f280a36c000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabee2b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-11" #55417 daemon prio=5 os_prio=0 tid=0x00007f2ba003f800 nid=0x276d waiting on condition [0x00007f2794818000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabecf40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-11" #55414 daemon prio=5 os_prio=0 tid=0x00007f2b98023800 nid=0x276b waiting on condition [0x00007f2792bfc000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabeda50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-10" #55411 daemon prio=5 os_prio=0 tid=0x00007f2ba003e000 nid=0x276a waiting on condition [0x00007f27926f7000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabecf40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-10" #55413 daemon prio=5 os_prio=0 tid=0x00007f2b8c0a7000 nid=0x2769 waiting on condition [0x00007f27927f8000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabee2b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-10" #55412 daemon prio=5 os_prio=0 tid=0x00007f2b98022800 nid=0x2768 waiting on condition [0x00007f27c4815000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabeda50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-9" #55372 daemon prio=5 os_prio=0 tid=0x00007f2c7408f000 nid=0x41df waiting on condition [0x00007f27907d8000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabece88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-8" #55369 daemon prio=5 os_prio=0 tid=0x00007f2c7408d000 nid=0x41dd waiting on condition [0x00007f27909da000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabece88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-7" #55368 daemon prio=5 os_prio=0 tid=0x00007f2c7408b000 nid=0x41dc waiting on condition [0x00007f2790adb000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabece88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-6" #55367 daemon prio=5 os_prio=0 tid=0x00007f2c74089000 nid=0x41db waiting on condition [0x00007f2790bdc000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabece88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-5" #55366 daemon prio=5 os_prio=0 tid=0x00007f2c74087000 nid=0x41da waiting on condition [0x00007f2790cdd000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabece88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-4" #55365 daemon prio=5 os_prio=0 tid=0x00007f2c74085000 nid=0x41d9 waiting on condition [0x00007f2790dde000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabece88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-9" #55364 daemon prio=5 os_prio=0 tid=0x00007f2bd813c000 nid=0x41d8 waiting on condition [0x00007f2790edf000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabed500> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

       

      "Remoting "endpoint" task-9" #55363 daemon prio=5 os_prio=0 tid=0x00007f2bf4044000 nid=0x41d7 waiting on condition [0x00007f2790fe0000]

         java.lang.Thread.State: WAITING (parking)

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for  <0x00000006eabee3c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

      ....