JMX connection terminates every minute on Wildfly 8.2.0

[mariomann]

Hi guys,

I have a wildfly 8.2.0.Final server, some applications are deployed. Now I want to connect with Jconsole to this server. This works very well. But after every minute the connection to the server terminates.

In the server.log are the following entries:

09:38:00,892 ERROR [org.jboss.remotingjmx.protocol.v2.ServerCommon] (pool-2-thread-6) org.jboss.remoting3.NotOpenException: Writes closed

09:39:07,164 ERROR [org.jboss.remotingjmx.protocol.v2.ServerCommon] (pool-2-thread-2) org.jboss.remoting3.NotOpenException: Writes closed

09:40:19,218 ERROR [org.jboss.remotingjmx.protocol.v2.ServerCommon] (pool-2-thread-3) org.jboss.remoting3.NotOpenException: Writes closed

09:41:38,035 ERROR [org.jboss.remotingjmx.protocol.v2.ServerCommon] (pool-2-thread-3) org.jboss.remoting3.NotOpenException: Writes closed

09:42:44,144 ERROR [org.jboss.remotingjmx.protocol.v2.ServerCommon] (pool-2-thread-3) org.jboss.remoting3.NotOpenException: Writes closed

09:43:46,700 ERROR [org.jboss.remotingjmx.protocol.v2.ServerCommon] (pool-2-thread-7) org.jboss.remoting3.NotOpenException: Writes closed

09:45:25,908 ERROR [org.jboss.remotingjmx.protocol.v2.ServerCommon] (pool-2-thread-2) org.jboss.remoting3.NotOpenException: Writes closed

09:46:31,227 ERROR [org.jboss.remotingjmx.protocol.v2.ServerCommon] (pool-2-thread-3) org.jboss.remoting3.NotOpenException: Writes closed

And in Jconsole I get the fowolling Java StackTrace:

Exception in thread "Remoting "endpoint" task-3" java.util.concurrent.RejectedExecutionException: Task org.jboss.remotingjmx.protocol.v2.ClientCommon$MessageReceiver$1@313c1901 rejected from java.util.concurrent.ThreadPoolExecutor@608da9e7[Shutting down, pool size = 2, active threads = 0, queued tasks = 0, completed tasks = 269]

    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)

    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)

    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)

    at org.jboss.remotingjmx.protocol.v2.ClientExecutorManager.execute(ClientExecutorManager.java:63)

    at org.jboss.remotingjmx.protocol.v2.ClientCommon$MessageReceiver.handleMessage(ClientCommon.java:118)

    at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:463)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

    at java.lang.Thread.run(Thread.java:722)

After this StackTrace appears it is not possible to reconnect via JMX to the Wildfly server. But the applications, which are deployed on this server are not affected and work still stable.

Any ideas why this is happen?

Thanks in advance!

[mchoma]

It sounds to me as one similar issue with jmx I resolved with seting -Dorg.jboss.remoting-jmx.timeout=120. You can give a try.

[kim_zeevaarders]

Hi,

I had a similar problem today and found the cause!

In our case, from one day to the other, the mbean tab of the jconsole renderd empty and would not show any mbeans anymore . After a minute or so the connection was terminated, with the only visible failure in the wildfly logging being:

09:38:00,892 ERROR [org.jboss.remotingjmx.protocol.v2.ServerCommon] (pool-2-thread-6) org.jboss.remoting3.NotOpenException: Writes closed

After a day long analysis,  we noticed that the issue was caused by too many and big log files from several of our ears/wars. This caused the jmx-bean tab to timeout since the logging subsystem of wildfly tries to load these files over jmx (jboss.as.expr:subsystem=logging).

In our case we needed to delete the log dirs and and change the logging implementation from periodic-size-rotating-file-handler to size-rotating-file-handler to reduce the number of logfiles generated.

After that the problem was solved...

Hope this helps anyone!

Regards,

Kim