0 Replies Latest reply on May 13, 2019 5:05 AM by Shengche Hsiao

    Wildfly 10 domain mode startup issue

    Shengche Hsiao Newbie

      Dear all

       

      I had one front end apache httpd server as load balancing service (aka. Server A) , and 3 back-end wildfly 10 application server with domain mode cluster. Recently, I faced some strang issue , when I restart Server A and restart rest 3 wildfly servers , there were some error log:

       

      2019-05-10 10:06:49,376 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[("interface" => "unsecure")]'


      2019-05-10 10:06:54,380 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler@501a652f for operation {"operation" => "add-deployer-chains","address" => []} at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException

              at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:511)

              at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1369)

              at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1328)

              at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1301)

              at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1185)

              at org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:767)

              at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:644)

              at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)

              at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1344)

              at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:485)

              at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:387)

              at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:349)

              at org.jboss.as.server.ServerService.boot(ServerService.java:392)

              at org.jboss.as.server.ServerService.boot(ServerService.java:365)

              at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:299)

              at java.lang.Thread.run(Thread.java:748)

      
 

      2019-05-10 10:06:54,381 ERROR [org.jboss.as.controller.client] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler@501a652f for operation {"operation" => "add-deployer-chains","address" => []} at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException

      2019-05-10 10:06:59,382 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.domain.management.security.SecurityRealmAddHandler$ServiceInstallStepHandler@468f86eb for operation {"operation" => "add","address" => [("core-service" => "management"),("security-realm" => "ApplicationRealm")],"map-groups-to-roles" => undefined} at address [

          ("core-service" => "management"),

          ("security-realm" => "ApplicationRealm")

      ] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException

       

              at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:511)

              at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1369)

              at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1328)

              at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1311)

              at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1185)

              at org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:767)

              at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:644)

              at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)

              at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1344)

              at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:485)

              at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:387)

              at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:349)

              at org.jboss.as.server.ServerService.boot(ServerService.java:392)

              at org.jboss.as.server.ServerService.boot(ServerService.java:365)

              at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:299)

              at java.lang.Thread.run(Thread.java:748)

       

      2019-05-10 10:06:59,383 ERROR [org.jboss.as.controller.client] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.domain.management.security.SecurityRealmAddHandler$ServiceInstallStepHandler@468f86eb for operation {"operation" => "add","address" => [("core-service" => "management"),("security-realm" => "ApplicationRealm")],"map-groups-to-roles" => undefined} at address [

          ("core-service" => "management"),

          ("security-realm" => "ApplicationRealm")

      ] failed handling operation rollback -- java.util.concurrent.TimeoutException

      2019-05-10 10:07:04,383 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.domain.management.security.SecurityRealmAddHandler$ServiceInstallStepHandler@468f86eb for operation {"operation" => "add","address" => [("core-service" => "management"),("security-realm" => "ManagementRealm")],"map-groups-to-roles" => undefined} at address [

          ("core-service" => "management"),

          ("security-realm" => "ManagementRealm")

      ] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException

              at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:511)

              at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1369)

              at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1328)

              at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1311)

              at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1185)

              at org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:767)

              at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:644)

              at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)

              at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1344)

              at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:485)

              at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:387)

              at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:349)

              at org.jboss.as.server.ServerService.boot(ServerService.java:392)

              at org.jboss.as.server.ServerService.boot(ServerService.java:365)

              at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:299)

              at java.lang.Thread.run(Thread.java:748)

       

      2019-05-10 10:07:04,384 ERROR [org.jboss.as.controller.client] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.domain.management.security.SecurityRealmAddHandler$ServiceInstallStepHandler@468f86eb for operation {"operation" => "add","address" => [("core-service" => "management"),("security-realm" => "ManagementRealm")],"map-groups-to-roles" => undefined} at address [

          ("core-service" => "management"),

          ("security-realm" => "ManagementRealm")

      ] failed handling operation rollback -- java.util.concurrent.TimeoutException

      2019-05-10 10:07:09,384 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.controller.AbstractControllerService$ModelControllerServiceInitializationBootStepHandler$1@1682cab1 for operation {"operation" => "boottime-controller-initializer-step","address" => []} at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException

              at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:511)

              at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1369)

              at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1328)

              at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1311)

              at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1185)

              at org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:767)

              at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:644)

              at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)

              at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1344)

              at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:485)

              at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:387)

              at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:349)

              at org.jboss.as.server.ServerService.boot(ServerService.java:392)

              at org.jboss.as.server.ServerService.boot(ServerService.java:365)

              at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:299)

              at java.lang.Thread.run(Thread.java:748)

       

      2019-05-10 10:07:09,384 ERROR [org.jboss.as.controller.client] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.controller.AbstractControllerService$ModelControllerServiceInitializationBootStepHandler$1@1682cab1 for operation {"operation" => "boottime-controller-initializer-step","address" => []} at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException

      2019-05-10 10:07:09,385 ERROR [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0022: Deploy of deployment "miniserver-0.03.war" was rolled back with no failure message

      2019-05-10 10:07:14,386 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.deployment.DeploymentHandlerUtil$1@4781bf57 for operation {"operation" => "add","address" => [("deployment" => "miniserver-0.03.war")],"runtime-name" => "miniserver-0.03.war","content" => [{"hash" => bytes { 0x71, 0x18, 0x85, 0xa1, 0x2c, 0x92, 0xf0, 0x35, 0xe6, 0x76, 0x16, 0x89, 0xda, 0x1a, 0xa0, 0xdd, 0x0e, 0x4c, 0x2c, 0x70 }}],"enabled" => true,"persistent" => true,"owner" => undefined} at address [("deployment" => "miniserver-0.03.war")] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException

              at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:511)

              at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1369)

              at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1328)

              at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1311)

              at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1185)

              at org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:767)

              at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:644)

              at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)

              at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1344)

              at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:485)

              at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:387)

              at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:349)

              at org.jboss.as.server.ServerService.boot(ServerService.java:392)

              at org.jboss.as.server.ServerService.boot(ServerService.java:365)

              at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:299)

              at java.lang.Thread.run(Thread.java:748)

       

      2019-05-10 10:07:14,386 ERROR [org.jboss.as.controller.client] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.deployment.DeploymentHandlerUtil$1@4781bf57 for operation {"operation" => "add","address" => [("deployment" => "miniserver-0.03.war")],"runtime-name" => "miniserver-0.03.war","content" => [{"hash" => bytes { 0x71, 0x18, 0x85, 0xa1, 0x2c, 0x92, 0xf0, 0x35, 0xe6, 0x76, 0x16, 0x89, 0xda, 0x1a, 0xa0, 0xdd, 0x0e, 0x4c, 0x2c, 0x70 }}],"enabled" => true,"persistent" => true,"owner" => undefined} at address [("deployment" => "miniserver-0.03.war")] failed handling operation rollback -- java.util.concurrent.TimeoutException

      2019-05-10 10:07:14,386 ERROR [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0022: Deploy of deployment "CncResource-0.0.1.war" was rolled back with no failure message

      2019-05-10 10:07:19,387 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.deployment.DeploymentHandlerUtil$1@3ae66b24 for operation {"operation" => "add","address" => [("deployment" => "CncResource-0.0.1.war")],"runtime-name" => "CncResource-0.0.1.war","content" => [{"hash" => bytes { 0x55, 0x28, 0xfa, 0xdf, 0x38, 0x35, 0x6c, 0x09, 0xde, 0x16, 0x2e, 0x1a, 0x2c, 0x0a, 0x67, 0xfa, 0x3f, 0xc6, 0x99, 0xad }}],"enabled" => true,"persistent" => true,"owner" => undefined} at address [("deployment" => "CncResource-0.0.1.war")] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException

              at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:511)

              at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1369)

              at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1328)

       

      It seems like concurrency issue? Any suggestions?