0 Replies Latest reply on May 13, 2019 5:05 AM by shengche

Wildfly 10 domain mode startup issue

shengche May 13, 2019 5:05 AM

Dear all

I had one front end apache httpd server as load balancing service (aka. Server A) , and 3 back-end wildfly 10 application server with domain mode cluster. Recently, I faced some strang issue , when I restart Server A and restart rest 3 wildfly servers , there were some error log:

2019-05-10 10:06:49,376 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[("interface" => "unsecure")]' 
2019-05-10 10:06:54,380 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler@501a652f for operation {"operation" => "add-deployer-chains","address" => []} at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException
        at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:511)
        at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1369)
        at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1328)
        at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1301)
        at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1185)
        at org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:767)
        at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:644)
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
        at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1344)
        at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:485)
        at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:387)
        at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:349)
        at org.jboss.as.server.ServerService.boot(ServerService.java:392)
        at org.jboss.as.server.ServerService.boot(ServerService.java:365)
        at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:299)
        at java.lang.Thread.run(Thread.java:748)
 
2019-05-10 10:06:54,381 ERROR [org.jboss.as.controller.client] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler@501a652f for operation {"operation" => "add-deployer-chains","address" => []} at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException
2019-05-10 10:06:59,382 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.domain.management.security.SecurityRealmAddHandler$ServiceInstallStepHandler@468f86eb for operation {"operation" => "add","address" => [("core-service" => "management"),("security-realm" => "ApplicationRealm")],"map-groups-to-roles" => undefined} at address [
    ("core-service" => "management"),
    ("security-realm" => "ApplicationRealm")
] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException

        at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:511)
        at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1369)
        at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1328)
        at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1311)
        at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1185)
        at org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:767)
        at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:644)
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
        at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1344)
        at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:485)
        at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:387)
        at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:349)
        at org.jboss.as.server.ServerService.boot(ServerService.java:392)
        at org.jboss.as.server.ServerService.boot(ServerService.java:365)
        at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:299)
        at java.lang.Thread.run(Thread.java:748)

2019-05-10 10:06:59,383 ERROR [org.jboss.as.controller.client] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.domain.management.security.SecurityRealmAddHandler$ServiceInstallStepHandler@468f86eb for operation {"operation" => "add","address" => [("core-service" => "management"),("security-realm" => "ApplicationRealm")],"map-groups-to-roles" => undefined} at address [
    ("core-service" => "management"),
    ("security-realm" => "ApplicationRealm")
] failed handling operation rollback -- java.util.concurrent.TimeoutException
2019-05-10 10:07:04,383 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.domain.management.security.SecurityRealmAddHandler$ServiceInstallStepHandler@468f86eb for operation {"operation" => "add","address" => [("core-service" => "management"),("security-realm" => "ManagementRealm")],"map-groups-to-roles" => undefined} at address [
    ("core-service" => "management"),
    ("security-realm" => "ManagementRealm")
] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException
        at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:511)
        at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1369)
        at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1328)
        at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1311)
        at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1185)
        at org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:767)
        at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:644)
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
        at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1344)
        at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:485)
        at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:387)
        at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:349)
        at org.jboss.as.server.ServerService.boot(ServerService.java:392)
        at org.jboss.as.server.ServerService.boot(ServerService.java:365)
        at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:299)
        at java.lang.Thread.run(Thread.java:748)

2019-05-10 10:07:04,384 ERROR [org.jboss.as.controller.client] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.domain.management.security.SecurityRealmAddHandler$ServiceInstallStepHandler@468f86eb for operation {"operation" => "add","address" => [("core-service" => "management"),("security-realm" => "ManagementRealm")],"map-groups-to-roles" => undefined} at address [
    ("core-service" => "management"),
    ("security-realm" => "ManagementRealm")
] failed handling operation rollback -- java.util.concurrent.TimeoutException
2019-05-10 10:07:09,384 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.controller.AbstractControllerService$ModelControllerServiceInitializationBootStepHandler$1@1682cab1 for operation {"operation" => "boottime-controller-initializer-step","address" => []} at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException
        at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:511)
        at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1369)
        at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1328)
        at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1311)
        at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1185)
        at org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:767)
        at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:644)
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
        at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1344)
        at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:485)
        at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:387)
        at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:349)
        at org.jboss.as.server.ServerService.boot(ServerService.java:392)
        at org.jboss.as.server.ServerService.boot(ServerService.java:365)
        at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:299)
        at java.lang.Thread.run(Thread.java:748)

2019-05-10 10:07:09,384 ERROR [org.jboss.as.controller.client] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.controller.AbstractControllerService$ModelControllerServiceInitializationBootStepHandler$1@1682cab1 for operation {"operation" => "boottime-controller-initializer-step","address" => []} at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException
2019-05-10 10:07:09,385 ERROR [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0022: Deploy of deployment "miniserver-0.03.war" was rolled back with no failure message
2019-05-10 10:07:14,386 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.deployment.DeploymentHandlerUtil$1@4781bf57 for operation {"operation" => "add","address" => [("deployment" => "miniserver-0.03.war")],"runtime-name" => "miniserver-0.03.war","content" => [{"hash" => bytes { 0x71, 0x18, 0x85, 0xa1, 0x2c, 0x92, 0xf0, 0x35, 0xe6, 0x76, 0x16, 0x89, 0xda, 0x1a, 0xa0, 0xdd, 0x0e, 0x4c, 0x2c, 0x70 }}],"enabled" => true,"persistent" => true,"owner" => undefined} at address [("deployment" => "miniserver-0.03.war")] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException
        at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:511)
        at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1369)
        at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1328)
        at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1311)
        at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1185)
        at org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:767)
        at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:644)
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
        at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1344)
        at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:485)
        at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:387)
        at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:349)
        at org.jboss.as.server.ServerService.boot(ServerService.java:392)
        at org.jboss.as.server.ServerService.boot(ServerService.java:365)
        at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:299)
        at java.lang.Thread.run(Thread.java:748)

2019-05-10 10:07:14,386 ERROR [org.jboss.as.controller.client] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.deployment.DeploymentHandlerUtil$1@4781bf57 for operation {"operation" => "add","address" => [("deployment" => "miniserver-0.03.war")],"runtime-name" => "miniserver-0.03.war","content" => [{"hash" => bytes { 0x71, 0x18, 0x85, 0xa1, 0x2c, 0x92, 0xf0, 0x35, 0xe6, 0x76, 0x16, 0x89, 0xda, 0x1a, 0xa0, 0xdd, 0x0e, 0x4c, 0x2c, 0x70 }}],"enabled" => true,"persistent" => true,"owner" => undefined} at address [("deployment" => "miniserver-0.03.war")] failed handling operation rollback -- java.util.concurrent.TimeoutException
2019-05-10 10:07:14,386 ERROR [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0022: Deploy of deployment "CncResource-0.0.1.war" was rolled back with no failure message
2019-05-10 10:07:19,387 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.deployment.DeploymentHandlerUtil$1@3ae66b24 for operation {"operation" => "add","address" => [("deployment" => "CncResource-0.0.1.war")],"runtime-name" => "CncResource-0.0.1.war","content" => [{"hash" => bytes { 0x55, 0x28, 0xfa, 0xdf, 0x38, 0x35, 0x6c, 0x09, 0xde, 0x16, 0x2e, 0x1a, 0x2c, 0x0a, 0x67, 0xfa, 0x3f, 0xc6, 0x99, 0xad }}],"enabled" => true,"persistent" => true,"owner" => undefined} at address [("deployment" => "CncResource-0.0.1.war")] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException
        at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:511)
        at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1369)
        at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1328)

It seems like concurrency issue? Any suggestions?