Teiid standalone stop client threads

[mtawk]

We have a web application deployed over Tomcat and it is using AdminAPI to administrate Teiid Standalone.

Whenever we stop the web application, we are not able to terminate the threads created by Teiid client.

We are getting the following error:

WARNING: The web application [mywebapp] appears to have started a thread named [XNIO-1 I/O-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:

sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)

sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)

sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)

sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)

sun.nio.ch.SelectorImpl.select(Unknown Source)

sun.nio.ch.SelectorImpl.select(Unknown Source)

we tried admin.close() but did not work.

Is there a way to terminate the below threads for a clean shutdown?

Daemon Thread [XNIO-1 I/O-1] (Running)           

Daemon Thread [XNIO-1 Accept] (Running)       

Daemon Thread [XNIO-1 task-1] (Running)         

Daemon Thread [XNIO-1 task-2] (Running)         

Daemon Thread [XNIO-1 task-3] (Running)         

Daemon Thread [XNIO-1 task-4] (Running)         

Daemon Thread [XNIO-1 task-5] (Running)         

Daemon Thread [XNIO-1 task-6] (Running)         

Daemon Thread [XNIO-1 task-7] (Running)         

Daemon Thread [XNIO-1 task-8] (Running)         

Daemon Thread [XNIO-1 task-9] (Running)         

Daemon Thread [XNIO-1 task-10] (Running)       

Daemon Thread [XNIO-1 task-11] (Running)       

Daemon Thread [XNIO-1 task-12] (Running)       

Daemon Thread [XNIO-1 task-13] (Running)       

Daemon Thread [XNIO-1 task-14] (Running)       

Daemon Thread [XNIO-1 task-15] (Running)       

  Daemon Thread [XNIO-1 task-16] (Running)       

[shawkins]

Any xnio threads from the admin client would be created by the jboss dmr client libraries.  Calling admin.close should clean up any resources created - it's the only handle to close given to use by the library.  The library also appears to have reference/finalizer logic to ensure cleanup, so it should cleanup even if you don't call close as long as the clients are garbage collected.  Is it possible that some admin clients are not being closed, and not being garbage collected?

[mtawk]

The admin.close is not terminating the threads. They remain hanging even after 10 minutes

we tried with a simple example like the below, and all the threads remained hanging after admin.close() as long as the jvm is not terminated

public static void main(String[] args) throws Exception {

Admin admin = AdminFactory.getInstance().createAdmin("127.0.0.1", 9990, "teiiduser", "teiidpass".toCharArray());

admin.close();

boolean stopProcess = false;

while (!stopProcess) {

Thread.sleep(2000);

}

}

[shawkins]

The example won't complete because the main thread never completes.

All the XNIO threads you show above are daemon threads, so they will go away once the main thread completes.

Retry your example as:

public static void main(String[] args) throws Exception {

Admin admin = AdminFactory.getInstance().createAdmin("127.0.0.1", 9990, "teiiduser", "teiidpass".toCharArray());

admin.close();

// if there are non-daemon threads, this process won't die at this point

}

[mtawk]

The issue is that those threads are remaining as long as the JVM is not terminated.

In the case of a web application hosted over Tomcat, if you stop/start the web application using Tomcat manager (Tomcat server is still using the same JVM), those threads are not removed, and accordingly, the web application is not starting properly.

We are forced to restart Tomcat server for those threads to be garbage collected.

Note that the above issue does not occur when Teiid is embedded in the web application.

[shawkins]

So the issue is not that you have some non-daemon threads preventing the vm from exiting, but that you are in a container environment and running the admin client spawns threads, which Tomcat does not like.  As far as I can tell this is by design with XNIO.   Those threads are effectively managed as a thread pool and as accessed via a static context, so they are assumed to be shared for the entire vm.

This is probably something to ask on the WildFly forum - if there is any mechanism to control how XNIO manages its threads for the admin client.

[mtawk]

I have asked the question in the wildfly forum: Unable to terminate Daemon XNIO threads created by Teiid admin client

They say there might be a way to allow clients to specify an application level for XNIO executor service.

Could this help in resolving the issue?

[shawkins]

We're essentially saying the same thing.  This is by design for the Remoting/XNIO library, which is used underneath the DMR library, which is used by the Teiid admin client.  The DMR library offers no methods nor documentation on controlling the XNIO behavior.  I can only see a hack of a solution at the present time, which I added to the other thread.

[shawkins]

To understand things a little better, what does your classloading situation look like?  Are the jboss dependencies in the web app, or are they in a common classloader?  It seems like a simple solution is to move at least the xnio library to a higher classloader such that the static instance would be associated with the whole tomcat instance and not a particular web context.