Wildfly-10.1.Final cluster deployment fails: "WFLYCTL0344: Operation timed out awaiting service container stability"

[kpreeta12]

I am migrating our application from wildfly-8.2 to wildfly-10.1. The standalone setup on wildfly10.1 is deployed successfully. However the cluster has issue. I tried to deploy to a one node cluster with 2 server instances. The domain.xml and host.xml is attached. The deployment of a war to server-instance-1 was successful. However during the deployment of another war to server-instance-2  get the below error.

I have attached the domain.xml and host.xml for your reference.

Below is the error:-

2017-06-16 05:43:23,204 INFO  [javax.enterprise.resource.webcontainer.jsf.config] (ServerService Thread Pool -- 239) Initializing Mojarra 2.2.13.SP1 20160303-1204 for context '/RequestCenter'

2017-06-16 05:44:49,188 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 76) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'deploy' at address '[("deployment" => "RequestCenter.war")]'

2017-06-16 05:44:49,204 ERROR [org.jboss.as.server] (ServerService Thread Pool -- 76) WFLYSRV0021: Deploy of deployment "RequestCenter.war" was rolled back with the following failure message: "WFLYCTL0344: Operation timed out awaiting service container stability"

2017-06-16 05:44:54,219 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 76) WFLYCTL0190: Step handler org.jboss.as.server.deployment.DeploymentHandlerUtil$1@474278ae for operation {"operation" => "deploy","address" => [("deployment" => "RequestCenter.war")],"operation-headers" => {"access-mechanism" => "HTTP","domain-uuid" => "961514ce-8b8f-4667-b724-67043a2a4ea5"}} at address [("deployment" => "RequestCenter.war")] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException

    at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:498)

    at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1369)

    at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1328)

    at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1301)

    at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1185)

    at org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:767)

    at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:644)

    at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)

    at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1329)

    at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:400)

    at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:222)

    at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler.internalExecute(TransactionalProtocolOperationHandler.java:247)

    at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler.doExecute(TransactionalProtocolOperationHandler.java:185)

    at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$1.run(TransactionalProtocolOperationHandler.java:138)

    at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$1.run(TransactionalProtocolOperationHandler.java:134)

    at java.security.AccessController.doPrivileged(Native Method)

    at javax.security.auth.Subject.doAs(Subject.java:360)

    at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:120)

    at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2$1.run(TransactionalProtocolOperationHandler.java:157)

    at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2$1.run(TransactionalProtocolOperationHandler.java:153)

    at java.security.AccessController.doPrivileged(Native Method)

    at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2.execute(TransactionalProtocolOperationHandler.java:153)

    at org.jboss.as.protocol.mgmt.AbstractMessageHandler$ManagementRequestContextImpl$1.doExecute(AbstractMessageHandler.java:363)

    at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:472)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

    at java.lang.Thread.run(Thread.java:748)

    at org.jboss.threads.JBossThread.run(JBossThread.java:320)

[kpreeta12]

The timeout issue is not seen in the logs now after I set the system property as

<property name="jboss.as.management.blocking.timeout" value="600"/>

However still the war is not getting deployed with the below in the server.log

017-06-16 06:22:31,722 INFO  [javax.enterprise.resource.webcontainer.jsf.config] (ServerService Thread Pool -- 149) Initializing Mojarra 2.2.13.SP1 20160303-1204 for context '/RequestCenter'

2017-06-16 06:23:54,817 ERROR [org.jboss.as.server] (ServerService Thread Pool -- 70) WFLYSRV0022:Deploy of deployment "RequestCenter.war" was rolled back with no failure message

2017-06-16 06:23:54,817 WARN  [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 70) WFLYCTL0027: Operation was interrupted before stability could be reached

2017-06-16 06:23:54,848 INFO  [org.jboss.as.protocol] (ServerService Thread Pool -- 248) WFLYPRT0057: cancelled task by interrupting thread Thread[ServerService Thread Pool -- 70,5,ServerService ThreadGroup]

2017-06-16 06:24:15,131 INFO  [org.apache.cxf.transport.servlet.AbstractCXFServlet] (ServerService Thread Pool -- 149) Load the bus with application context

[subhashc37]

"Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'deploy' at address '[("deployment" => "RequestCenter.war")]'"

This indiacate that there is an issue with your RequestCenter.war, you should take multiple threadumps during the deployment and see any issue.