Shutting down WildFly with configured remote ejb connection seems to cause java.util.concurrent.RejectedExecutionException being thrown at server shutdown

[ochaloup]

Hi,

I have a test application where remote connection to second WildFly is configured. The app works fine just at the time of container is going to be shutdown there is logged error exception

ERROR [org.jboss.remoting.handler-errors] (default I/O-7) Close handler threw an exception: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@83fbfbf rejected from java.util.concurrent.ThreadPoolExecutor@114598d4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6]
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
    at org.jboss.ejb.client.EJBClientContext.unregisterEJBReceiver(EJBClientContext.java:440)
    at org.jboss.ejb.client.EJBReceiverContext.close(EJBReceiverContext.java:57)
    at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver$1$1.handleClose(RemotingConnectionEJBReceiver.java:185)
    at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver$1$1.handleClose(RemotingConnectionEJBReceiver.java:182)
    at org.jboss.remoting3.spi.SpiUtils.safeHandleClose(SpiUtils.java:54)
    at org.jboss.remoting3.spi.AbstractHandleableCloseable$CloseHandlerTask.run(AbstractHandleableCloseable.java:501)
    at org.jboss.remoting3.spi.AbstractHandleableCloseable.runCloseTask(AbstractHandleableCloseable.java:406)
    at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeComplete(AbstractHandleableCloseable.java:277)
    at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:531)
    at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:359)
    at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAllChannels(RemoteConnectionHandler.java:423)
    at org.jboss.remoting3.remote.RemoteConnectionHandler.receiveCloseRequest(RemoteConnectionHandler.java:213)
    at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:110)
    at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:45)
    at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
    at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:199)
    at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:113)
    at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
    at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092)
    at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
    at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
    at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:88)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:539)

Do you consider it as a bug - as I expect that shutdown should be clean from error messages in log - or there could be some problem in configuration of my remote connection?

<subsystem xmlns="urn:jboss:domain:remoting:3.0">
  <endpoint worker="default"/>
  <http-connector name="http-remoting-connector" connector-ref="default" security-realm="ApplicationRealm"/>
  <outbound-connections>
    <remote-outbound-connection name="remote-ejb-connection" outbound-socket-binding-ref="binding-remote-ejb-connection" username="user1" security-realm="PasswordRealm" protocol="http-remoting">
      <properties>
        <property name="SASL_POLICY_NOANONYMOUS" value="false"/>
        <property name="SSL_ENABLED" value="false"/>
      </properties>
    </remote-outbound-connection>
  </outbound-connections>
</subsystem>

Thanks

Ondra

[dmlloyd]

This looks like the recently-fixed REM3-183.  The problem should be harmless but you should be able to work around it by compiling Remoting from the 4.0 branch and using that version.

[ochaloup]

Ok, thanks for pointing me at that issue. I've tried to run it with current 4.0.7 snapshot but unfortunately I still get the same exception. Agree that it's harmless from functionality point of view. But it's logged as an error.

I found this thread with the same exception (in different occasion):

https://developer.jboss.org/message/904088

And there is mentioned citation from ThreadPoolExecutor javadoc which says

If the task cannot be submitted for execution, either because this executor has been shutdown or because its capacity has been reached, the task is handled by the current RejectedExecutionHandler.

This seems to be a logic explanation for the existence of the exception but it's not so nice having logged an error when it's not an error.

Would you think creating a jira for it is ok?

Btw. I had a trouble to compile the remoting 4.0 branch as maven central does not contain byteman in version 2.0.0

[jaikiran]

I remember bmaxwell had fixed this in the EJB client library. However, it looks like the 8.2 WildFly release missed out on upgrading to a newer version of EJB client, although it was requested in the wildfly-dev mailing list http://lists.jboss.org/pipermail/wildfly-dev/2014-October/003202.html

[ochaloup]

Hi Jaikiran,

It seems that the fix for [EJBCLIENT-98] java.util.concurrent.RejectedExecutionException if a remote-naming InitialContext should be closed - JBo… solves the problem (I've taken the latest ejb client library in "version" jboss-ejb-client-2.0.3.Final-SNAPSHOT.jar from github).

The solution is to wait for the new release of ejb client and its propagation to WildFly.

Thanks

Ondra

[bmaxwell]

Yes, looks like it is in 2.0, 2.x & master branches as well, just needs to be updated in wildfly.