WildFly 11: stopping the ManagedExecutorService during server stop

[hostalp]

Hello,

we're testing one application in an earlier development state at WildFly 11 and have hit the following issue:

The application makes use of Spring's DefaultMessageListenerContainer for receiving some messages. The related message listener threads utilize the ManagedExecutorService. This works as expected under normal circumstances.

When we try to stop the server in domain mode (e.g. server-group:stop or server-config:stop-server commands) the server will attempt to enter the suspended state (serverState=running, suspendState=SUSPENDING) and will be stuck in this state unable to proceed further and fully stop.

We identified that this is due to those message listeners which keep running even in the suspended state with no problem.

Based on the JSR-236 spec - The Java Community Process(SM) Program - communityprocess - final

3.1.6.1 Java EE Product Provider Requirements

This subsection describes additional requirements for ManagedExecutorService providers.

1. All tasks, when executed from the ManagedExecutorService, will run with the Java EE component

identity of the component that submitted the task.

2. The lifecycle of a ManagedExecutorService is managed by an application server. All lifecycle operations

on the ManagedExecutorService interface will throw a java.lang.IllegalStateException exception.

This includes the following methods that are defined in the java.util.concurrent.ExecutorService

interface: awaitTermination(), isShutdown(), isTerminated(), shutdown(), and shutdownNow().

Final Release 3-11

3. No task submitted to an executor can run if task’s component is not started.

When a ManagedExecutorService instance is being shutdown by the Java EE Product Provider:

1. All attempts to submit new tasks are rejected.

2. All submitted tasks are cancelled if not running.

3. All running task threads are interrupted.

4. All registered ManagedTaskListeners are invoked.

E.g. the container should reject new tasks, cancel those already submitted, interrupt the running ones etc.

Sort of this can be seen with the ManagedScheduledExecutorService where in this state we get:

ERROR [org.jboss.as.ee] (EE-ManagedScheduledExecutorService-default-Thread-4) WFLYEE0110: Failed to run scheduled task: java.lang.IllegalStateException: WFLYEE0111: Cannot run scheduled task javax.enterprise.concurrent.ManagedExecutors$ManagedRunnable@11f8633a as container is suspended

        at org.jboss.as.ee.concurrent.ControlPointUtils$ControlledScheduledRunnable.run(ControlPointUtils.java:164)

        at org.jboss.as.ee.concurrent.ControlPointUtils$ControlledManagedRunnable.run(ControlPointUtils.java:246)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at org.glassfish.enterprise.concurrent.internal.ManagedFutureTask.run(ManagedFutureTask.java:141)

        at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$101(ManagedScheduledThreadPoolExecutor.java:383)

        at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:532)

        at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedTriggerSingleFutureTask.run(ManagedScheduledThreadPoolExecutor.java:587)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

        at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250)

        at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78)

However tasks that are submitted to the ManagedExecutorService are still seen to be successfully accepted and processed, none are interrupted etc. - e.g. they just keep running, preventing the suspend operation from completing.

Is this really correct behavior? Then how is one supposed to detect this state in order to stop the related mesage listeners?

Note shutdown or kill commands are able to stop the server, but not the stop or server-stop.

[dmlloyd]

During graceful shutdown, services are not stopped until all outstanding requests are complete.  If you want to terminate immediately, you need a non-graceful shutdown which should have the effect of rejecting subsequent task submissions (though we don't normally initiate the shut down the executor until all dependent services (like EJBs) are shut down in any event).

[hostalp]

I see, but then what about that graceful shutdown scenario which is the most common one in the domain environments? Here the server would be stuck in the suspending state probably forever. Is there any elegant way how to handle this? We tried several approaches but to no avail.

One more idea we're considering is making use of the jboss.as:management-root=server MBean attribute suspendState to detect that the server is suspending and stop the message listeners, but is there anything more suitable perhaps?

[dmlloyd]

I'm not sure about how requests would be detected from third-party request sources, and I'm also not familiar with how the DefaultMessageListenerContainer works in Spring (or what it even is TBH), so I can't really be more specific than that at the moment.

[hostalp]

What does Spring there is that it just repeadedly submits a task to the ManagedExecutorService.

The task attempts to receive a message. Once the task completes, Spring resubmits it again.

If the application is undeployed, the Spring web application context gets destroyed which includes Spring beans responsible for the above operations and as a result submitting of any such tasks is stopped.

Submitting of those tasks can be also stopped programatically if some particular situation occurs.

However when the server is being gracefully stopped it tries to enter the suspended state first. At this point the web application context destruction doesn't start yet so Spring doesn't have any knowledge that it should stop submitting those tasks. They can be still successfully submitted and processed - there's not even any sort of failure at all. It thus keeps running and never really stops because there's no way how to figure out ihat it should stop.

Simply look at it as if you've got some task for some kind of asynchronous processing that's supposed to be running during the whole application lifetime, thus it gets repeatedly resubmitted all the time. And since during the graceful stop this task is still allowed to be submitted and run and there's no regular way how to figure out that the server is about to stop/suspend and since the task running state blocks the server from stopping at the same time, the server will never stop.

If that's going to be the standard WildFly behavior then it seems we'll need to come up with some additional code that's able to detect when the server is trying to enter the suspended state and stop scheduling of those tasks ourselves.

The question thus was about whether it is really necessary to do so (it seems a little weird that it would be necessary since there doesn't seem to be any regular way how to handle such situation cleanly) and if so, what would be the most elegant approach.

[hostalp]

Just a note for anyone potentially interested:

We "solved" this via implementing a JMX notification listener that subscribes to the jboss.root:type=state MBean notifications (the above mentioned jboss.as:management-root=server MBean doesn't produce any notifications so we had to look for another one) and detects changes to the RunningState attribute.

If the RunningState changes from normal to suspending we stop all the Spring message listeners ourselves - and we can also use this to stop anything else we like (like scheduling anything via ManagedScheduledExecutorService so that WFLYEE0111 don't keep occurring until the server is stopped)

Similarly we can also detect completion of the server startup/resume by detecting the change of RunningState from suspended to normal and start all those tasks afterwards.

Not sure if there isn't a better approach to solve this.

[cyron]

HI,  maybe a later reply.

I think u can inject or set the managed scheduled executor service of wildfly into DefaultMessegeListenerContainer via setTaskExecutor method.

coz managed scheduled executor service is supported by graceful shutdown.