4 Replies Latest reply on Jun 27, 2016 8:56 AM by ggmoura

    Wildfly 9.0.0  startup issues with domain:infinispan:3.0 subsystem, works fine with domain:infinispan:2.0

    thomash

      The infinspan subsystem of wildfly 9.0.0 has an updated namespace (3.0) compared with 2.0 in wildfly 8

      When using the 3.0 config, the server fails to start. It seems to hang somewhere, and after a fuw minutes it starts to spit errors:

       

      nov:03,11:07:13,003 ERROR (:Controller Boot Thread:) [org.jboss.as.controller.management-operation] WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[
          ("core-service" => "management"),
          ("management-interface" => "http-interface")
      ]'
      (...)
      nov:03,11:07:18,009 ERROR (:Controller Boot Thread:) [org.jboss.as.controller.management-operation] WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler@2f266b09 for operation {"operation" => "add-deployer-chains","address" => []} at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException
        at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:396) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
        at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1384) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
        at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1332) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
        at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1307) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
        at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1180) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
        at org.jboss.as.controller.AbstractOperationContext.handleContainerStabilityFailure(AbstractOperationContext.java:964) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
        at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:590) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
        at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:354) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:330) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
        at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1183) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
        at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:453) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
        at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:327) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
        at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:313) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
        at org.jboss.as.server.ServerService.boot(ServerService.java:384) [wildfly-server-1.0.0.Final.jar:1.0.0.Final]
        at org.jboss.as.server.ServerService.boot(ServerService.java:359) [wildfly-server-1.0.0.Final.jar:1.0.0.Final]
        at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:271) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_45]
      

       

      This is with the following infinspan config: (if I change the xmlns to 2.0, instead of 3.0, everything works fine)

       

      <subsystem xmlns="urn:jboss:domain:infinispan:3.0">
                  <cache-container name="interserver" default-cache="default" jndi-name="java:jboss/infinispan/interserver" module="org.wildfly.clustering.server">
                      <transport stack="tcp-interserver" lock-timeout="60000" />
        <replicated-cache name="default" mode="ASYNC">
                          <transaction mode="BATCH"/>
                      </replicated-cache>
                      <replicated-cache name="com.my.cache1" mode="ASYNC">
                      </replicated-cache>
                      <replicated-cache name="com.my.cache2" mode="ASYNC">
                          <transaction mode="BATCH"/>
                      </replicated-cache>
                  </cache-container>
                  <cache-container name="singleton" default-cache="default" jndi-name="java:jboss/infinispan/cluster" aliases="cluster ha-partition" module="org.wildfly.clustering.server" >
                   <transport lock-timeout="60000"/>
                      <local-cache name="default" />
                      <local-cache name="com.my.cache3">
                  <transaction mode="BATCH"/>
                      </local-cache>
                  </cache-container>
                  <cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">
                      <transport lock-timeout="60000"/>
                      <distributed-cache name="dist" mode="ASYNC" owners="2" l1-lifespan="0">
                          <locking isolation="REPEATABLE_READ"/>
                          <transaction mode="BATCH"/>
                          <file-store/>
                      </distributed-cache>
                  </cache-container>
                  <cache-container name="ejb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan" aliases="sfsb">
                      <transport lock-timeout="60000"/>
                      <distributed-cache name="dist" mode="ASYNC" owners="2" l1-lifespan="0">
                          <locking isolation="REPEATABLE_READ"/>
                          <transaction mode="BATCH"/>
                          <file-store/>
                      </distributed-cache>
                  </cache-container>
                  <cache-container name="hibernate" default-cache="local-query" module="org.hibernate.infinispan">
                      <transport lock-timeout="60000"/>
                      <local-cache name="local-query">
                          <eviction strategy="LRU" max-entries="10000"/>
                          <expiration max-idle="100000"/>
                      </local-cache>
                      <invalidation-cache name="entity" mode="SYNC">
                          <transaction mode="NON_XA"/>
                          <eviction strategy="LRU" max-entries="10000"/>
                          <expiration max-idle="100000"/>
                      </invalidation-cache>
                      <replicated-cache name="timestamps" mode="ASYNC"/>
                  </cache-container>
              </subsystem>
      

       

      I don't see what is wrong with above config

        • 1. Re: Wildfly 9.0.0  startup issues with domain:infinispan:3.0 subsystem, works fine with domain:infinispan:2.0
          pferraro

          That's not much to go on...  You said "after a few minutes it starts to spit errors".  Are there other stacktraces in the log that might be more informative?

          • 2. Re: Wildfly 9.0.0  startup issues with domain:infinispan:3.0 subsystem, works fine with domain:infinispan:2.0
            thomash

            I agree that is not much That is why I came to this forum.

            The stacktrace I showed in the post is the first one: there is this ERROR ("Timeout after [300]"), than a few lines of logging of a module that apparently was deployed correctly, which is not relevant here.

            After that, wildfly starts undeploying stuff and stopping.

            There are a bunch of other similar messages after that, but I suspect it is the first one that causes some sort of cascading effect.

            This is what follows the stacktrace of my first post (the very next lines)

            nov:03,11:07:18,030 ERROR (:Controller Boot Thread:) [org.jboss.as.controller.client] WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler@2f266b09 for operation {"operation" => "add-deployer-chains","address" => []} at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException
            nov:03,11:07:21,738 INFO  (:Thread-2:) [org.jboss.as.server] WFLYSRV0220: Server shutdown has been requested.
            nov:03,11:07:21,784 INFO  (:MSC service thread 1-32:) [org.jboss.as.connector.subsystems.datasources] WFLYJCA0010: Unbound data source [java:jboss/datasources/H2-bms-DS]
            nov:03,11:07:21,817 INFO  (:MSC service thread 1-31:) [org.wildfly.extension.undertow] WFLYUT0008: Undertow HTTP listener http-2 suspending
            nov:03,11:07:21,822 INFO  (:MSC service thread 1-7:) [org.wildfly.extension.undertow] WFLYUT0008: Undertow HTTPS listener https suspending
            nov:03,11:07:21,829 INFO  (:MSC service thread 1-12:) [org.wildfly.extension.undertow] WFLYUT0019: Host default-host stopping
            nov:03,11:07:21,846 INFO  (:MSC service thread 1-7:) [org.wildfly.extension.undertow] WFLYUT0007: Undertow HTTPS listener https stopped, was bound to /172.16.10.126:8443
            nov:03,11:07:21,851 INFO  (:MSC service thread 1-31:) [org.wildfly.extension.undertow] WFLYUT0007: Undertow HTTP listener http-2 stopped, was bound to /172.16.10.126:8081
            nov:03,11:07:23,032 ERROR (:Controller Boot Thread:) [org.jboss.as.controller.management-operation] WFLYCTL0190: Step handler org.jboss.as.security.SecurityDomainAdd$1@60141bff for operation {"operation" => "add","address" => [("subsystem" => "security"),("security-domain" => "jboss-ejb-policy")],"cache-type" => "default"} at address [
                ("subsystem" => "security"),
                ("security-domain" => "jboss-ejb-policy")
            ] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException
              at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:396) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1384) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1332) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1307) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1180) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.AbstractOperationContext.handleContainerStabilityFailure(AbstractOperationContext.java:964) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:590) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:354) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:330) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1183) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:453) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:327) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:313) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.server.ServerService.boot(ServerService.java:384) [wildfly-server-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.server.ServerService.boot(ServerService.java:359) [wildfly-server-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:271) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_45]
            
            
            nov:03,11:07:23,054 ERROR (:Controller Boot Thread:) [org.jboss.as.controller.client] WFLYCTL0190: Step handler org.jboss.as.security.SecurityDomainAdd$1@60141bff for operation {"operation" => "add","address" => [("subsystem" => "security"),("security-domain" => "jboss-ejb-policy")],"cache-type" => "default"} at address [
                ("subsystem" => "security"),
                ("security-domain" => "jboss-ejb-policy")
            ] failed handling operation rollback -- java.util.concurrent.TimeoutException
            

             

            It goes on and on like this.

            ie.

            nov:03,11:07:28,057 ERROR (:Controller Boot Thread:) [org.jboss.as.controller.management-operation] WFLYCTL0190: Step handler org.jboss.as.security.SecurityDomainAdd$1@457bef94 for operation {"operation" => "add","address" => [("subsystem" => "security"),("security-domain" => "other")],"cache-type" => "default"} at address [
                ("subsystem" => "security"),
                ("security-domain" => "other")
            ] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException
              at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:396) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1384) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1332) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1307) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1180) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.AbstractOperationContext.handleContainerStabilityFailure(AbstractOperationContext.java:964) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:590) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:354) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:330) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1183) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:453) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:327) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:313) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.server.ServerService.boot(ServerService.java:384) [wildfly-server-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.server.ServerService.boot(ServerService.java:359) [wildfly-server-1.0.0.Final.jar:1.0.0.Final]
              at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:271) [wildfly-controller-1.0.0.Final.jar:1.0.0.Final]
              at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_45]
            
            (...)
            nov:03,11:07:33,131 ERROR (:Controller Boot Thread:) [org.jboss.as.controller.client] WFLYCTL0190: Step handler org.jboss.as.messaging.HornetQServerAdd$3@53cc2ead for operation {"operation" => "add","address" => [("subsystem" => "messaging"),("hornetq-server" => "default")],"journal-file-size" => 102400L,"clustered" => undefined,"persistence-enabled" => undefined,"scheduled-thread-pool-max-size" => undefined,"thread-pool-max-size" => undefined,"security-domain" => undefined,"security-enabled" => undefined,"security-invalidation-interval" => undefined,"override-in-vm-security" => undefined,"wild-card-routing-enabled" => undefined,"management-address" => undefined,"management-notification-address" => undefined,"cluster-user" => undefined,"cluster-password" => undefined,"jmx-management-enabled" => undefined,"jmx-domain" => undefined,"statistics-enabled" => undefined,"message-counter-enabled" => undefined,"message-counter-sample-period" => undefined,"message-counter-max-day-history" => undefined,"connection-ttl-override" => undefined,"async-connection-execution-enabled" => undefined,"transaction-timeout" => undefined,"transaction-timeout-scan-period" => undefined,"message-expiry-scan-period" => undefined,"message-expiry-thread-priority" => undefined,"id-cache-size" => undefined,"persist-id-cache" => undefined,"remoting-interceptors" => undefined,"remoting-incoming-interceptors" => undefined,"remoting-outgoing-interceptors" => undefined,"backup" => undefined,"allow-failback" => undefined,"failback-delay" => undefined,"failover-on-shutdown" => undefined,"shared-store" => undefined,"persist-delivery-count-before-delivery" => undefined,"page-max-concurrent-io" => undefined,"create-bindings-dir" => undefined,"create-journal-dir" => undefined,"journal-type" => undefined,"journal-buffer-timeout" => undefined,"journal-buffer-size" => undefined,"journal-sync-transactional" => undefined,"journal-sync-non-transactional" => undefined,"log-journal-write-rate" => undefined,"journal-min-files" => undefined,"journal-compact-percentage" => undefined,"journal-compact-min-files" => undefined,"journal-max-io" => undefined,"max-saved-replicated-journal-size" => undefined,"perf-blast-pages" => undefined,"run-sync-speed-test" => undefined,"server-dump-interval" => undefined,"memory-warning-threshold" => undefined,"memory-measure-interval" => undefined,"check-for-live-server" => undefined,"backup-group-name" => undefined,"replication-clustername" => undefined} at address [
                ("subsystem" => "messaging"),
                ("hornetq-server" => "default")
            ] failed handling operation rollback -- java.util.concurrent.TimeoutException
            
            (...)
            
            nov:03,11:07:38,170 ERROR (:Controller Boot Thread:) [org.jboss.as.controller.client] WFLYCTL0190: Step handler org.jboss.as.domain.management.security.SecurityRealmAddHandler$ServiceInstallStepHandler@647d33f3 for operation {"address" => [("core-service" => "management"),("security-realm" => "ApplicationRealm")],"operation" => "add","map-groups-to-roles" => undefined} at address [
                ("core-service" => "management"),
                ("security-realm" => "ApplicationRealm")
            ] failed handling operation rollback -- java.util.concurrent.TimeoutException
            
            (...)
            nov:03,11:07:53,263 ERROR (:Controller Boot Thread:) [org.jboss.as.controller.client] WFLYCTL0190: Step handler org.jboss.as.connector.subsystems.datasources.DataSourceEnable$1@18606b4d for operation {"operation" => "add","jndi-name" => "java:jboss/datasources/derby-bms-DS","enabled" => true,"use-java-context" => true,"address" => [("subsystem" => "datasources"),("xa-data-source" => "derby-bms-DS")],"driver-name" => "derby","user-name" => "bms","password" => "bms","no-recovery" => undefined,"recovery-username" => "bms","recovery-password" => "bms","xa-datasource-class" => undefined,"new-connection-sql" => undefined,"url-delimiter" => undefined,"url-selector-strategy-class-name" => undefined,"max-pool-size" => undefined,"min-pool-size" => undefined,"initial-pool-size" => undefined,"pool-prefill" => undefined,"pool-use-strict-min" => undefined,"interleaving" => undefined,"capacity-incrementer-class" => undefined,"capacity-decrementer-class" => undefined,"no-tx-separate-pool" => undefined,"pad-xid" => undefined,"same-rm-override" => undefined,"wrap-xa-resource" => undefined,"security-domain" => undefined,"reauth-plugin-class-name" => undefined,"flush-strategy" => undefined,"allow-multiple-users" => undefined,"connection-listener-class" => undefined,"prepared-statements-cache-size" => undefined,"share-prepared-statements" => undefined,"track-statements" => undefined,"allocation-retry" => undefined,"allocation-retry-wait-millis" => undefined,"blocking-timeout-wait-millis" => undefined,"idle-timeout-minutes" => undefined,"query-timeout" => undefined,"use-try-lock" => undefined,"set-tx-query-timeout" => undefined,"transaction-isolation" => undefined,"check-valid-connection-sql" => undefined,"exception-sorter-class-name" => undefined,"stale-connection-checker-class-name" => undefined,"valid-connection-checker-class-name" => undefined,"background-validation-millis" => undefined,"background-validation" => undefined,"use-fast-fail" => undefined,"validate-on-match" => undefined,"xa-resource-timeout" => undefined,"spy" => undefined,"use-ccm" => undefined,"connectable" => undefined,"statistics-enabled" => undefined,"tracking" => undefined,"recovery-security-domain" => undefined,"recovery-plugin-class-name" => undefined,"url-property" => undefined,"reauth-plugin-properties" => undefined,"exception-sorter-properties" => undefined,"stale-connection-checker-properties" => undefined,"valid-connection-checker-properties" => undefined,"recovery-plugin-properties" => undefined,"connection-listener-property" => undefined,"capacity-incrementer-properties" => undefined,"capacity-decrementer-properties" => undefined} at address [
                ("subsystem" => "datasources"),
                ("xa-data-source" => "derby-bms-DS")
            ] failed handling operation rollback -- java.util.concurrent.TimeoutException
            
            
            • 3. Re: Wildfly 9.0.0  startup issues with domain:infinispan:3.0 subsystem, works fine with domain:infinispan:2.0
              pferraro

              You are absolutely sure there are no other errors in the log?  like anything specifically relating to Infinispan?

              Do you have some bootstrap code that uses this interserver cache container?  Might there be a problem there?

              • 4. Re: Wildfly 9.0.0  startup issues with domain:infinispan:3.0 subsystem, works fine with domain:infinispan:2.0
                ggmoura

                The error happened to me in my case was the database service that was not starting.  The postgresql. look this