0 Replies Latest reply on Feb 14, 2017 4:56 PM by brittlevy

Leaking XNIO WorkerThreads when JMX RemotingConnector.connect fails

brittlevy Feb 14, 2017 4:56 PM

We have an application that uses JMX to monitor JBoss application servers. If the user misconfigures the JMX service URL or the JBoss server is down then we see two XNIO WorkThreads get created and left running after each failed connection attempt. Since this is a monitory application that is intended to run continuously, we eventually run out of thread resources.

We have seen the issue from wildfly-10.0.0.Final/bin/client/jboss-client.jar and wildfly-10.1.0.Final/bin/client/jboss-client.jar. Have not tried other versions.

This can be easily duplicated using JConsole, so we don't believe it is our code.

To duplicate, launch wildfly-10.0.0.Final/bin/jconsole.sh. Use VisaulVM to monitor jconsole. Try to connect in JConsole using a JMX connection URL (service:jmx:remote+http://server:port) for a server that is not running. Each time you hit Connect in JConsole you will see two additional threads get created.

I also created a simple test application that just tries to connect to the JMX server.

When I start the test application and the XNIO library initializes it prints out:

Feb 13, 2017 4:32:31 PM org.xnio.Xnio <clinit>

INFO: XNIO version 3.3.4.Final

Feb 13, 2017 4:32:31 PM org.xnio.nio.NioXnio <clinit>

INFO: XNIO NIO Implementation Version 3.3.4.Final

Feb 13, 2017 4:32:31 PM org.jboss.remoting3.EndpointImpl <clinit>

INFO: JBoss Remoting version 4.0.18.Final

Below is the stack trace that is generated when the connect request fails.

java.net.ConnectException: Connection refused

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)

at org.xnio.nio.WorkerThread$ConnectHandle.handleReady(WorkerThread.java:319)

at org.xnio.nio.WorkerThread.run(WorkerThread.java:559)

at ...asynchronous invocation...(Unknown Source)

at org.jboss.remoting3.EndpointImpl.doConnect(EndpointImpl.java:294)

at org.jboss.remoting3.EndpointImpl.doConnect(EndpointImpl.java:275)

at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:379)

at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:363)

at org.jboss.remotingjmx.RemotingConnector.internalRemotingConnect(RemotingConnector.java:239)

at org.jboss.remotingjmx.RemotingConnector.internalConnect(RemotingConnector.java:158)

at org.jboss.remotingjmx.RemotingConnector.connect(RemotingConnector.java:105)

at org.jboss.remotingjmx.RemotingConnector.connect(RemotingConnector.java:100)

at hbl.jboss.jmx.ThreadLeakTest.runJBossNativeTest2(ThreadLeakTest.java:162)

at hbl.jboss.jmx.ThreadLeakTest.main(ThreadLeakTest.java:42)

We see this in the thread dump from VisualVM:

"Remoting "endpoint" Accept" #91 daemon prio=6 os_prio=31 tid=0x00007f8e84c7e800 nid=0x18a03 runnable [0x000070000cd65000]

java.lang.Thread.State: RUNNABLE

at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)

at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)

at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

- locked <0x00000006c16b54c8> (a sun.nio.ch.Util$2)

- locked <0x00000006c16b54d8> (a java.util.Collections$UnmodifiableSet)

- locked <0x00000006c16b5478> (a sun.nio.ch.KQueueSelectorImpl)

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)

at org.xnio.nio.WorkerThread.run(WorkerThread.java:509)

Locked ownable synchronizers:

- None

"Remoting "endpoint" I/O-1" #90 daemon prio=6 os_prio=31 tid=0x00007f8e84888000 nid=0x18803 runnable [0x000070000cc62000]