Unable to terminate Daemon XNIO threads created by Teiid admin client

[mtawk]

We have a web application deployed over Tomcat and it is using AdminAPI to administrate Teiid Standalone. When connecting to Teiid through AdminAPI, Daemon XNIO threads are created.

if we stop/start the web application using Tomcat manager, the daemon XNIO threads are not removed, and accordingly, the web application is not starting properly.

We are forced to restart the Tomcat server to terminate those threads. So the threads are not destroyed until the JVM is terminated.

Is there a way to terminate those threads on web application restart? we tried to close teiid admin client but the threads are still hanging.

[jaikiran]

I have zero knowledge of Teiid, so can't really help much. But, since you say it's a web application, can you show how you construct the client and in which component (is it a servlet, is it some kind of listener)? Also can you post the thread dump which shows the threads which aren't killed after undeploying your application?

[mtawk]

Teiid is hosted on Wildfly.

Teiid provides us AdminAPI to administrate datasources, deployment, resource adapaters....

here is the code we are using to create the admin client, (the user is a wildfly management user):

Admin admin = AdminFactory.getInstance().createAdmin("127.0.0.1", 9990, "wildflyuser", "wildflypass".toCharArray());

Whenever we stop the web application, we are not able to terminate the threads created by client API.

We are getting the following error:

WARNING: The web application [mywebapp] appears to have started a thread named [XNIO-1 I/O-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:

sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)

sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)

sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)

sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)

sun.nio.ch.SelectorImpl.select(Unknown Source)

sun.nio.ch.SelectorImpl.select(Unknown Source)

we tried admin.close() but did not work.

You find below the hanging threads list:

Daemon Thread [XNIO-1 I/O-1] (Running)          

Daemon Thread [XNIO-1 Accept] (Running)      

Daemon Thread [XNIO-1 task-1] (Running)        

Daemon Thread [XNIO-1 task-2] (Running)        

Daemon Thread [XNIO-1 task-3] (Running)        

Daemon Thread [XNIO-1 task-4] (Running)        

Daemon Thread [XNIO-1 task-5] (Running)        

Daemon Thread [XNIO-1 task-6] (Running)        

Daemon Thread [XNIO-1 task-7] (Running)        

Daemon Thread [XNIO-1 task-8] (Running)        

Daemon Thread [XNIO-1 task-9] (Running)        

Daemon Thread [XNIO-1 task-10] (Running)      

Daemon Thread [XNIO-1 task-11] (Running)      

Daemon Thread [XNIO-1 task-12] (Running)      

Daemon Thread [XNIO-1 task-13] (Running)      

Daemon Thread [XNIO-1 task-14] (Running)      

Daemon Thread [XNIO-1 task-15] (Running)      

  Daemon Thread [XNIO-1 task-16] (Running)  

Is there any mechanism to control how XNIO manages its threads for the admin client?

[jaikiran]

Which version of WildFly and Java runtime is this? We would also need to know which version of WildFly client libraries are being used on the the client webapp (on Tomcat).

[jaikiran]

Now that I (briefly) checked the Teiid AdminFactory code, I think this is expected because of the way XNIO works/is implemented/designed. The threads that you are seeing are a global JVM level threads which XNIO uses. There seems to be a way in XNIO to allow clients (like Teiid) to specify a application level (via the application's classloader for example) XNIO executor service, but I don't know how easy/complicated it's going to be to manage it, more so because I have zero knowledge of Teiid.

[shawkins]

> There seems to be a way in XNIO to allow clients (like Teiid) to specify a application level (via the application's classloader for example) XNIO executor service, but I don't know how easy/complicated it's going to be to manage it, more so because I have zero knowledge of Teiid.

There are 4 layers in play here:  Teiid admin client, wildfly controller client, remoting, and xnio.  There's nothing in the controller client api/docs covering this case, and I'm unable to determine what could influence this behavior at a lower level.

In the remoting library when a new EndpointImpl is created it passes false for ourWorker - jboss-remoting/EndpointImpl.java at b06e67aad653ce3b48d7e5896c3c15edda56bcea · jboss-remoting/jboss-remoting · GitHub  When that flag is false, the EndpointImpl.finishPhase1 method will not shut the worker down as it's assumed that it's managed statically by XNIO.  So it appears that there's nothing that can be done in the remoting layer.

In XNIO the thread pooling / executor mechanism is baked in: xnio/WorkerThread.java at 63cfe60bdf7270c5db842be682272a116df50d64 · xnio/xnio · GitHub

About the only thing that seems remotely (no pun intended) possible is to add:

XnioWorker.getContextManager().get().shutdown();

to the Teiid admin client somewhere as optional behavior.

[shawkins]

A related issue would be that there are still org.wildfly.common.ref.ReaperThread instances that are similarly created and are not shutdown.

[jaikiran]

I don't have enough knowledge of the XNIO library to suggest something for this. Maybe dmlloyd might have some suggestions?

[dmlloyd]

Basically the XNIO worker is static and lives for the life of the application in this case.  It's not an error or a leak; I'm guessing Tomcat is just not designed to cope with this situation.  A workaround would be for Remoting to be configured to construct its own XNIO worker instead of using the static one.  But I'm not sure how or why Teiid is using Remoting to begin with (JSR 160 maybe?) so I can't say for sure how to go about doing that.