2 Replies Latest reply on Dec 7, 2016 10:11 AM by kim_zeevaarders

    JMX connection terminates every minute on Wildfly 8.2.0

    mariomann
    mariomann Oct 22, 2015 4:12 AM

      Hi guys,

       

      I have a wildfly 8.2.0.Final server, some applications are deployed. Now I want to connect with Jconsole to this server. This works very well. But after every minute the connection to the server terminates.

       

      In the server.log are the following entries:

       

      09:38:00,892 ERROR [org.jboss.remotingjmx.protocol.v2.ServerCommon] (pool-2-thread-6) org.jboss.remoting3.NotOpenException: Writes closed

      09:39:07,164 ERROR [org.jboss.remotingjmx.protocol.v2.ServerCommon] (pool-2-thread-2) org.jboss.remoting3.NotOpenException: Writes closed

      09:40:19,218 ERROR [org.jboss.remotingjmx.protocol.v2.ServerCommon] (pool-2-thread-3) org.jboss.remoting3.NotOpenException: Writes closed

      09:41:38,035 ERROR [org.jboss.remotingjmx.protocol.v2.ServerCommon] (pool-2-thread-3) org.jboss.remoting3.NotOpenException: Writes closed

      09:42:44,144 ERROR [org.jboss.remotingjmx.protocol.v2.ServerCommon] (pool-2-thread-3) org.jboss.remoting3.NotOpenException: Writes closed

      09:43:46,700 ERROR [org.jboss.remotingjmx.protocol.v2.ServerCommon] (pool-2-thread-7) org.jboss.remoting3.NotOpenException: Writes closed

      09:45:25,908 ERROR [org.jboss.remotingjmx.protocol.v2.ServerCommon] (pool-2-thread-2) org.jboss.remoting3.NotOpenException: Writes closed

      09:46:31,227 ERROR [org.jboss.remotingjmx.protocol.v2.ServerCommon] (pool-2-thread-3) org.jboss.remoting3.NotOpenException: Writes closed

       

      And in Jconsole I get the fowolling Java StackTrace:

       

      Exception in thread "Remoting "endpoint" task-3" java.util.concurrent.RejectedExecutionException: Task org.jboss.remotingjmx.protocol.v2.ClientCommon$MessageReceiver$1@313c1901 rejected from java.util.concurrent.ThreadPoolExecutor@608da9e7[Shutting down, pool size = 2, active threads = 0, queued tasks = 0, completed tasks = 269]

          at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)

          at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)

          at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)

          at org.jboss.remotingjmx.protocol.v2.ClientExecutorManager.execute(ClientExecutorManager.java:63)

          at org.jboss.remotingjmx.protocol.v2.ClientCommon$MessageReceiver.handleMessage(ClientCommon.java:118)

          at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:463)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

          at java.lang.Thread.run(Thread.java:722)

       

      After this StackTrace appears it is not possible to reconnect via JMX to the Wildfly server. But the applications, which are deployed on this server are not affected and work still stable.

       

      Reconnect.png

       

      Any ideas why this is happen?

       

      Thanks in advance!

        • 1. Re: JMX connection terminates every minute on Wildfly 8.2.0
          mchoma
          mchoma Oct 22, 2015 7:26 AM (in response to mariomann)

          It sounds to me as one similar issue with jmx I resolved with seting -Dorg.jboss.remoting-jmx.timeout=120. You can give a try.

            • Actions
          • 2. Re: JMX connection terminates every minute on Wildfly 8.2.0
            kim_zeevaarders
            kim_zeevaarders Dec 7, 2016 10:11 AM (in response to mariomann)

            Hi,

             

            I had a similar problem today and found the cause!

             

            In our case, from one day to the other, the mbean tab of the jconsole renderd empty and would not show any mbeans anymore . After a minute or so the connection was terminated, with the only visible failure in the wildfly logging being:

             

            09:38:00,892 ERROR [org.jboss.remotingjmx.protocol.v2.ServerCommon] (pool-2-thread-6) org.jboss.remoting3.NotOpenException: Writes closed

             

            After a day long analysis,  we noticed that the issue was caused by too many and big log files from several of our ears/wars. This caused the jmx-bean tab to timeout since the logging subsystem of wildfly tries to load these files over jmx (jboss.as.expr:subsystem=logging).

             

            In our case we needed to delete the log dirs and and change the logging implementation from periodic-size-rotating-file-handler to size-rotating-file-handler to reduce the number of logfiles generated.

             

            After that the problem was solved...

             

            Hope this helps anyone!

             

            Regards,

             

            Kim

              • Actions