[jaikiran]

Is there anyway you can provide some reproducible application for this and create a JIRA? I do think there's a bug in here somewhere but given the nature of the issue (and timers in general), it's hard to say what's going on.

[cboehme]

I am a colleague of Guido and also involved in solving the timer issue.

I finally managed to reproduce the problem. A simple application that shows it can be found on https://github.com/cboehme/wildfly-timer-issue/. The stalled timers appear to be caused by a bug in Wildfly's RequestController that only shows when a maximum request limit is set. I have created a Jira for it (https://issues.jboss.org/browse/WFCORE-3565) and made a pull request (https://github.com/wildfly/wildfly-core/pull/3073) that should fix the problem.

Thank you for your help!

[gjaekel]

Indeed, the maximum number of concurrent request was limited to 64 (from the unlimited default) as a DOS protection. As a workaround against the "stucking queue" issue, I've increased the maximum number of concurrent requests to 256.

/subsystem=request-controller:write-attribute(name=max-requests,value=256)

The Model Reference states:

    max-requests The maximum number of all types of requests that can be running in a server at a time

[jaikiran]

To be honest, I'm not really sure whether the request controller subsystem's rate control should be applicable to timer runs. Based on the conclusion that you and your colleague have arrived at, it does look like this is currently playing a role in timer execution and is preventing the timers that are eligible for execution, from being run. Maybe swd847 might know whether this is expected.

[gjaekel]

Yesterday, the queue length touched the new length of 256 and the whole mechanism blocked, again. The suggested workaround to suspend (and resume) the Wildlfy don't work; the suspend itself run into a timeout. Also, "touching" the queue by increasing the maximum queue length don't unblock it entirely: There were a bunch of "missing EJB3-timer" log messages, but the queue size don't fall. The only "solution" was to restart the Wildfly.