Hi All,

We have installed Wildfly 10.0.0 Final on a Linux OS Cluster Rhel 7.4

Within wildfly installation we embed our own application to be run simultaneously as a combined process with wildfly on the server.

There are 2 nodes (High Availability mode) for fail-over on which our software is installed.

When we try to stop the server after a long period of time ( say 5 days) the 'services' process which is wildfly takes about 15 minutes to stop.

This issue does not appear if we stop the server soon after starting.

I wanted to know if there are any compatibility issues within Wildfly 10.0.0 and RHEL 7.4

The issue was not observed when we had the Wildfly with RHEL 6.x version.

Preliminary investigation points to some threads which are in waiting state and can see them in this state throughout different thread dumps ( We took 9 dumps in total).

It will be helpful to identify if the issue is in wildfly or our own application files.

Below is the log trace from server.log We see it takes 936564ms ( about 15 minutes) to stop the server.

The exception below are noted during short shutdown as well which works fine. So we are sure it is not the root/possible cause of the issue.

Also attached are the thread dump files in case they are helpful. attaching only 3 files as they show all the threads which are commonly waiting across.

Log trace -

2018-02-13 08:12:57,928 INFO  [org.jboss.as.server] (Thread-2) WFLYSRV0220: Server shutdown has been requested.

2018-02-13 08:12:57,964 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = activemq-rar.rar_org.apache.derby.jdbc.AutoloadedDriver_10_4

2018-02-13 08:12:57,970 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-7) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/BppmThresholdTopic]

2018-02-13 08:12:57,971 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-7) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/queue/NGPRateGDQueue]

2018-02-13 08:12:57,971 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-7) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/CAThresholdTopic]

2018-02-13 08:12:57,971 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-7) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/CAServerSideOperationResponseTopic]

2018-02-13 08:12:57,971 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-5) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/ps_notifications]

2018-02-13 08:12:57,971 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-7) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/SLAChangedTopic]

2018-02-13 08:12:57,971 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-5) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/queue/NGPJSQueue]

2018-02-13 08:12:57,971 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-7) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/AgentControllerDataTopic]

2018-02-13 08:12:57,971 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-7) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]

2018-02-13 08:12:57,971 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-5) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/DDTopic]

2018-02-13 08:12:57,970 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-8) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/queue/NGPTSGDQueue]

2018-02-13 08:12:57,971 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-5) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/queue/NGPCAGDQueue]

2018-02-13 08:12:57,971 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-5) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/queue/NGPACQueue]

2018-02-13 08:12:57,971 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-5) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/CAStatusTopic]

2018-02-13 08:12:57,971 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-8) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/HighAvailabilityStatusChangeTopic]

2018-02-13 08:12:57,971 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-8) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/eis/ao/ActiveMQQueue]

2018-02-13 08:12:57,972 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-8) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/ProcessStatusChangeTopic]

2018-02-13 08:12:57,972 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-8) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/MOChangedTopic]

2018-02-13 08:12:57,972 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-8) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/BlSeasonalityTopic]

2018-02-13 08:12:57,972 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-8) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/ReloadServicesTopic]

2018-02-13 08:12:57,972 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-8) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/EventTopic]

2018-02-13 08:12:57,972 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-8) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/queue/ps_requests]

2018-02-13 08:12:57,972 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-8) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/SystemConstTopic]

2018-02-13 08:12:57,972 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-8) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/CAServerSideOperationRequestTopic]

2018-02-13 08:12:57,973 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-8) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/AlarmChangedTopic]

2018-02-13 08:12:57,974 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-8) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/queue/NGPACGDQueue]

2018-02-13 08:12:57,974 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-6) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/AlarmRuleTopic]

2018-02-13 08:12:57,974 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-8) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/queue/NGPRateQueue]

2018-02-13 08:12:57,974 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-6) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/DependencyTopic]

2018-02-13 08:12:57,974 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-8) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/PartitionTopic]

2018-02-13 08:12:57,975 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-8) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/MOTypeChangedTopic]

2018-02-13 08:12:57,975 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-1) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/queue/ps_jserver_replies]

2018-02-13 08:12:57,975 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-1) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/CloseAlarmTopic]

2018-02-13 08:12:57,975 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-1) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/CentralRPMCrudTopic]

2018-02-13 08:12:57,975 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-1) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/AnalysisInfoTopic]

2018-02-13 08:12:57,975 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-1) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/AttributeGroupTopic]

2018-02-13 08:12:57,975 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-3) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/eis/ao/ActiveMQTopic]

2018-02-13 08:12:57,971 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-7) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/queue/NGPTSQueue]

2018-02-13 08:12:57,975 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-8) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/BlResetTopic]

2018-02-13 08:12:57,971 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-5) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/queue/NGPJSGDQueue]

2018-02-13 08:12:57,993 INFO  [com.proactivenet.util.RMIRegistryService] (ServerService Thread Pool -- 68) stopping rmi registry instance

2018-02-13 08:12:58,006 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-7) WFLYUT0019: Host default-host stopping

2018-02-13 08:12:58,006 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-3) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/BppmTenantTopic]

2018-02-13 08:12:58,007 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-7) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/GroupTopic]

2018-02-13 08:12:58,007 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-8) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/topic/BlFreezeTopic]

2018-02-13 08:12:58,008 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-2) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/queue/ps_psservice_replies]

2018-02-13 08:12:58,007 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-7) WFLYJCA0011: Unbound JCA AdminObject [java:jboss/exported/queue/NGPCAQueue]

2018-02-13 08:12:58,020 WARN  [com.bmc.ActiveMQNonJCAConnectionFactory] (ServerService Thread Pool -- 72) WFLYSYSJMX0007: Stopping failed activemq:service=amqbootstrap: javax.naming.NamingException: WFLYNAM0020: Could not resolve service service jboss.naming.context.java.jboss.exported.ConnectionFactory

at org.jboss.as.naming.WritableServiceBasedNamingStore.unbind(WritableServiceBasedNamingStore.java:135)

at org.jboss.as.naming.NamingContext.unbind(NamingContext.java:318)

at org.jboss.as.naming.InitialContext$DefaultInitialContext.unbind(InitialContext.java:282)

at org.jboss.as.naming.NamingContext.unbind(NamingContext.java:326)

at javax.naming.InitialContext.unbind(InitialContext.java:441)

at javax.naming.InitialContext.unbind(InitialContext.java:441)

at com.bmc.ActiveMQNonJCAConnectionFactory.stopService(ActiveMQNonJCAConnectionFactory.java:117)

at org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:291)

at org.jboss.system.ServiceMBeanSupport.stop(ServiceMBeanSupport.java:167)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.jboss.as.service.AbstractService.invokeLifecycleMethod(AbstractService.java:76)

at org.jboss.as.service.StartStopService$2.run(StartStopService.java:86)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:748)

at org.jboss.threads.JBossThread.run(JBossThread.java:320)

2018-02-13 08:12:58,022 WARN  [org.jboss.msc.inject] (ServerService Thread Pool -- 67) MSC000100: Unexpected failure to uninject public void com.proactivenet.util.RMIRegistryService.setPort(int): java.lang.IllegalArgumentException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.jboss.msc.inject.MethodInjector.uninject(MethodInjector.java:119)

at org.jboss.msc.service.ServiceControllerImpl$StopContextImpl.complete(ServiceControllerImpl.java:2469)

at org.jboss.as.service.CreateDestroyService$2.run(CreateDestroyService.java:104)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:748)

at org.jboss.threads.JBossThread.run(JBossThread.java:320)

2018-02-13 08:12:58,034 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0019: Stopped Driver service with driver-name = h2

2018-02-13 08:12:58,047 INFO  [org.apache.activemq.broker.BrokerService] (ServerService Thread Pool -- 74) Apache ActiveMQ 5.11.3 (bppm-jms-broker, ID:eng-bmims02-40686-1518036483763-0:1) is shutting down

2018-02-13 08:12:58,085 INFO  [org.hibernate.validator.internal.util.Version] (MSC service thread 1-4) HV000001: Hibernate Validator 5.2.3.Final

2018-02-13 08:12:58,122 INFO  [org.apache.activemq.broker.TransportConnector] (ServerService Thread Pool -- 74) Connector bppm-jms-broker-tcp-conn stopped

2018-02-13 08:12:58,134 INFO  [org.apache.activemq.broker.BrokerService] (ServerService Thread Pool -- 74) Apache ActiveMQ 5.11.3 (bppm-jms-broker, ID:eng-bmims02-40686-1518036483763-0:1) uptime 5 days 18 hours

2018-02-13 08:12:58,134 INFO  [org.apache.activemq.broker.BrokerService] (ServerService Thread Pool -- 74) Apache ActiveMQ 5.11.3 (bppm-jms-broker, ID:eng-bmims02-40686-1518036483763-0:1) is shutdown

2018-02-13 08:12:58,183 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-7) WFLYUT0008: Undertow HTTP listener default suspending

2018-02-13 08:12:58,183 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-7) WFLYUT0007: Undertow HTTP listener default stopped, was bound to [0:0:0:0:0:0:0:0]:1100

2018-02-13 08:12:58,184 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0004: Undertow 1.3.25.Final stopping

2018-02-13 08:12:58,186 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-5) WFLYSRV0028: Stopped deployment pw_j2ee_logging.sar (runtime-name: pw_j2ee_logging.sar) in 234ms

2018-02-13 08:12:58,189 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0028: Stopped deployment amqbootstrap.sar (runtime-name: amqbootstrap.sar) in 238ms

2018-02-13 08:12:58,189 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0028: Stopped deployment rmiregistry.sar (runtime-name: rmiregistry.sar) in 238ms

2018-02-13 08:12:58,219 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0028: Stopped deployment activemq-rar.rar (runtime-name: activemq-rar.rar) in 267ms

2018-02-13 08:28:34,515 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0050: WildFly Full 10.0.0.Final (WildFly Core 2.0.10.Final) stopped in 936564ms

Any help/pointers are greatly appreciated as I am very new to wildfly.

jaikiran ctomc

Thanks,

Tariq