1 Reply Latest reply on Dec 13, 2016 6:43 AM by dariusz.mrowka

    SEVERE errors during stopping of Tomcat after ejb calls (using jboss ejb client)- tomcat failed to stop threads

    dariusz.mrowka

      My environment: Windows, java 1.8.0.112, Tomcat 7.0.73, WildFly 10.1.0.Final

      My web applications (JSF/RichFaces) calls EJB services (on WildFly) using jboss ejb client library (taken from WildFly distribution catalog- so the proper one). Simplifying: ejb's are called from servlets.

       

      I noticed that during stopping of Tomcat there are messages in its log like:

       

          gru 09, 2016 1:32:37 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads

          SEVERE: The web application [/xxx] appears to have started a thread named [Remoting "client-endpoint" xxx] but has failed to stop it. This is very likely to create a memory leak.

       

      for about 18-21 (ie. 16 + few) threads, separately for each my web applications (each application has it's own classloader).

      This happens only after web application calls ejb.

      I think ejb-client creates some threads to async ejb calls (using connection pool) and Tomcat has problem to stop them (but why?). This is probably a bug in ejb-client/remonting lib.

      Sometimes I have to stop Tomcat using force (end process), because Tomcat hangs during stopping (probably is waiting for some thread to stop).

      Fortunatelly this errors do not affect my web applications (i did some stress tests, but maybe not full enough)- I didn't noticed any memory leaks, ports not being closed.

       

      But this kind of errors in tomcat log looks awful. Does anybody smart expert know what can be done with this?

      Any help would be appreciated!

        • 1. Re: SEVERE errors in during stopping of Tomcat after ejb calls (using jboss ejb client)- tomcat failed to stop threads
          dariusz.mrowka

          I looked into jboss-ejb-client source code (widlfly 10.1 has jboss-ejb-client 2.1.4) and finally I've found the solution

          There's a class EJBClientContext with nice close() method (public) which does some clean-up (to prevent memory leak- I assume during closing JVM).

          Because each web app has its own classloader (jboss-ejb-client was creating separate threads for each web app) I used it when each web app is stopped:

          in ServletContextListener.contextDestroyed of web application:

              EJBClientContext.getCurrent().close();

          (in try/catch)

          and errors with thread named [Remoting "client-endpoint" xxx]  (16+2 threads) disappeard in Tomcat (and there are no memory leaks for sure). Moreover, sice then Tomcat didn't hang during stoping any more.

          So this EJBClientContext.close() method is useful (especially in Tomcat environment). Why there's no info about that in https://docs.jboss.org/author/display/WFLY10/EJB+invocations+from+a+remote+client+using+JNDI???

          Maybe jboss-ejb-client even without calling thant method by ejb-client cleans for itself, but threads are left, and Tomcat prints awful errors about that, and "no class found exception" sometimes, due to these previous errors).

           

          So after calling this close method errors about 18 threads (that cant' be stopped by Tomcat) disappeard. But 3 other severe errors was left, about 3 other threads named [ejb-client-context-tasks-1-thread-xxx].

          I dug into the source code of jboss-ejb-client and find out where these 3 thread are created (in EJBClientContext class). I look into source code of version 2.x and 1.x and noticed, that in 1.x close method was stopping these threads (but no cleaning was done to prevent memory leaks). In 2.x cleaning was added, but stopping of these threads was removed. Maybe because in standalone client it's not needed (after JVM exits). But Tomcat tryes to stopp then and prints errors.

          So .. I changed close method (import source code ver 2.1.4 into my eclipse, the same as WildFly 10.1 has got), i.e. I've added one line at the end (the same which was in 1.x version):

               this.ejbClientContextTasksExecutorService.shutdownNow();

          compiled whole project, replace class in jboss-ejb-client.jar ("jar uf" command), copied it to my environment and ... Tomcat prints no errors any more!

          I've tested my web apps, did start-stop Tomcat meny times, everything is ok, no errors in log, no Tomcat hangs during stopping ... perfect