how to increase default timeout of 300s?

[lafr]

When I have deployed some ear's and restart the WildFly server (WF 9.0 build from current sources of github master) I get this message after about 5 minutes:

16.11. 23:12:10,446 ERROR [org.jboss.as.controller.management-operation#awaitServiceContainerStability] WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[

    ("core-service" => "management"),

    ("management-interface" => "http-interface")

]'

A lot of more error messages and the the server hangs and cannot be shutdown. Only kill -9 helps.

Where to tune this timeout value of 300 seconds to a bigger value?

Using web-console or editing standalone.xml?

[jaysensharma]

Hello Frank,

    Looks like you may need to set the system property "jboss.as.management.blocking.timeout" to a larger value than 5 minutes  if your deployment or management operations are taking more than 5 minutes.

Example:

/host=master/server-config=server-one/system-property=jboss.as.management.blocking.timeout:add(boot-time=true,value=750)

   Or can be set at the server-group level.

[1] https://github.com/wildfly/wildfly/pull/6237    [Related to [WFLY-2741][WFLY-3058] as part of these JIRAs this system property was added]

[lafr]

I added "-Djboss.as.management.blocking.timeout=600" to my startup script. I also increased the default transaction timeout from 300 to 3600. This has an effect, but does not solve the problem.

Startup "hangs" after some minutes and does not continue. After about 5 minutes of doing nothing (no CPU activity, no messages on console) the timeout limit of 600 is reached and messages like in my OP appear.

I'll attach server.log from this try. Last activity at 18.11. 08:50:03,700, starting abort at 18.11. 08:55:55,463, and a stack trace taken during this idle period.

After killing the server (kill -9) and erasing one of the two ear in deployments folder startup is successful with otherwise unmodified configuration.

Then I redeployed the ear file but removed all war files from deployments folder. This startup could make it in time (< 300 s) but also hangs after about 3:30 min. Completely blocked, Ctrl-C or kill show no effect.

With the timeout value set after 600 secs, without after 300 secs.

So there seems to be a problem in my application or in the camunda BPM module used here. When 2 ears are deployed in parallel on startup they block each other somehow.

[salamont]

Hi, we have exactly the same problem. Did you solve it somehow?

Thanks

Tomas

[ctomc]

I would recommend using https://github.com/wolfc/jboss-as-divine-spells

to tell you help you find out the underlying issue.

Issue itself could be caused by wrong resource lookup in @Resource, maybe a typo

[salamont]

Hi Tomaz,

Many thanks, this is really a cool tool. However, the particular results are quite confusing for me. It seems that everything is indicated ok, isn't it? (please, see the output below)

It also isn't a typo or anything that trivial. But it seems we have identified the problem meanwhile. We are running on TCP (unfortunately, no other option), currently with 9 servers (3 nodes x 3 server groups). And it seems it is too much. If we stop a part of the cluster, everything works without much troubles. If 8 or 9 servers are up and running it starts to make exactly the same troubles as described in the original post.

As I have spent considerable time on research, I know that this behavior is described several times over the web (e.g. here ). I think that the problem is simply in the TCP transport and what I am thinking about now is how to split the cluster into three separate clusters following the particular server groups (which would be perfectly enough for us). Could you please advise, what is the best way how to do that? My idea is that it should be enough to create separate socket-binding-groups, however given that TCPPING is involved (within stack definition), I am afraid we also need three separate profiles. Is that a good approach? Thank you.

Best,

Tomas

Missing/failed Dependencies (condensed reverse dependencies)

jboss.clustering.nodes.ejb.dist (DOWN)

  jboss.clustering.registry.ejb.dist.factory (DOWN) ->  jboss.clustering.nodes.ejb.dist (DOWN) jboss.clustering.group.ejb.dist (DOWN)

    jboss.clustering.registry.ejb.dist (DOWN) ->  jboss.clustering.registry.ejb.dist.factory (DOWN) jboss.clustering.registry.ejb.dist.entry (***MISSING***)

  jboss.clustering.group.ejb.dist (DOWN) ->  jboss.clustering.nodes.ejb.dist (DOWN)

jboss.clustering.registry.ejb.dist.entry ***MISSING***

jboss.clustering.nodes.hibernate (DOWN)

  jboss.clustering.nodes.hibernate.default (DOWN) ->  jboss.clustering.nodes.hibernate (DOWN)

    jboss.clustering.registry.hibernate.default.factory (DOWN) ->  jboss.clustering.nodes.hibernate.default (DOWN) jboss.infinispan.hibernate.default (DOWN) jboss.clustering.group.hibernate.default (DOWN)

      jboss.clustering.registry.hibernate.default (DOWN) ->  jboss.clustering.registry.hibernate.default.factory (DOWN) jboss.clustering.registry.hibernate.default.entry (***MISSING***)

    jboss.clustering.group.hibernate.default (DOWN) ->  jboss.infinispan.hibernate.default (DOWN) jboss.clustering.nodes.hibernate.default (DOWN)

  jboss.clustering.nodes.hibernate.entity (DOWN) ->  jboss.clustering.nodes.hibernate (DOWN)

    jboss.clustering.group.hibernate.entity (DOWN) ->  jboss.infinispan.hibernate.entity (DOWN) jboss.clustering.nodes.hibernate.entity (DOWN)

      jboss.clustering.registry.hibernate.entity.factory (DOWN) ->  jboss.clustering.group.hibernate.entity (DOWN) jboss.infinispan.hibernate.entity (DOWN) jboss.clustering.nodes.hibernate.entity (DOWN)

        jboss.clustering.registry.hibernate.entity (DOWN) ->  jboss.clustering.registry.hibernate.entity.factory (DOWN) jboss.clustering.registry.hibernate.entity.entry (***MISSING***)

  jboss.clustering.group.hibernate.local-query (DOWN) ->  jboss.clustering.nodes.hibernate (DOWN)

    jboss.clustering.registry.hibernate.local-query.factory (DOWN) ->  jboss.clustering.group.hibernate.local-query (DOWN)

      jboss.clustering.registry.hibernate.local-query (DOWN) ->  jboss.clustering.registry.hibernate.local-query.factory (DOWN) jboss.clustering.registry.hibernate.local-query.entry (***MISSING***)

  jboss.clustering.nodes.hibernate.timestamps (DOWN) ->  jboss.clustering.nodes.hibernate (DOWN)

    jboss.clustering.group.hibernate.timestamps (DOWN) ->  jboss.clustering.nodes.hibernate.timestamps (DOWN) jboss.infinispan.hibernate.timestamps (DOWN)

      jboss.clustering.registry.hibernate.timestamps.factory (DOWN) ->  jboss.infinispan.hibernate.timestamps (DOWN) jboss.clustering.group.hibernate.timestamps (DOWN) jboss.clustering.nodes.hibernate.timestamps (DOWN)

        jboss.clustering.registry.hibernate.timestamps (DOWN) ->  jboss.clustering.registry.hibernate.timestamps.factory (DOWN) jboss.clustering.registry.hibernate.timestamps.entry (***MISSING***)

jboss.jgroups.channel.hibernate.factory (DOWN)

  jboss.jgroups.channel.hibernate (DOWN) ->  jboss.jgroups.channel.hibernate.factory (DOWN)

    jboss.infinispan.hibernate.transport (DOWN) ->  jboss.jgroups.channel.hibernate (DOWN) jboss.jgroups.channel.hibernate.factory (DOWN)

      jboss.infinispan.hibernate.config (DOWN) ->  jboss.infinispan.hibernate.transport (DOWN)

        jboss.infinispan.hibernate (DOWN) ->  jboss.infinispan.hibernate.config (DOWN)

          jboss.infinispan.hibernate.local-query (DOWN) ->  jboss.infinispan.hibernate (DOWN) jboss.infinispan.hibernate.local-query.config (DOWN)

            jboss.infinispan.hibernate.default (DOWN) ->  jboss.infinispan.hibernate.local-query (DOWN)

          jboss.infinispan.hibernate.local-query.config (DOWN) ->  jboss.infinispan.hibernate (DOWN)

          jboss.infinispan.hibernate.entity (DOWN) ->  jboss.infinispan.hibernate (DOWN) jboss.infinispan.hibernate.entity.config (DOWN)

          jboss.infinispan.hibernate.entity.config (DOWN) ->  jboss.infinispan.hibernate (DOWN)

          jboss.infinispan.hibernate.timestamps (DOWN) ->  jboss.infinispan.hibernate (DOWN) jboss.infinispan.hibernate.timestamps.config (DOWN)

          jboss.infinispan.hibernate.timestamps.config (DOWN) ->  jboss.infinispan.hibernate (DOWN)

jboss.clustering.registry.hibernate.default.entry ***MISSING***

jboss.clustering.registry.hibernate.entity.entry ***MISSING***

jboss.clustering.registry.hibernate.local-query.entry ***MISSING***

jboss.clustering.registry.hibernate.timestamps.entry ***MISSING***

jboss.clustering.nodes.server (DOWN)

  jboss.clustering.nodes.server.default (DOWN) ->  jboss.clustering.nodes.server (DOWN)

    jboss.clustering.registry.server.default.factory (DOWN) ->  jboss.clustering.nodes.server.default (DOWN) jboss.clustering.group.server.default (DOWN) jboss.infinispan.server.default (DOWN)

      jboss.clustering.registry.server.default (DOWN) ->  jboss.clustering.registry.server.default.factory (DOWN) jboss.clustering.registry.server.default.entry (***MISSING***)

    jboss.clustering.group.server.default (DOWN) ->  jboss.clustering.nodes.server.default (DOWN) jboss.infinispan.server.default (DOWN)

jboss.jgroups.channel.server.factory (DOWN)

  jboss.infinispan.server.transport (DOWN) ->  jboss.jgroups.channel.server.factory (DOWN) jboss.jgroups.channel.server (DOWN)

    jboss.infinispan.server.config (DOWN) ->  jboss.infinispan.server.transport (DOWN)

      jboss.infinispan.server (DOWN) ->  jboss.infinispan.server.config (DOWN)

        jboss.infinispan.server.default (DOWN) ->  jboss.infinispan.server (DOWN) jboss.infinispan.server.default.config (DOWN)

        jboss.infinispan.server.default.config (DOWN) ->  jboss.infinispan.server (DOWN)

  jboss.jgroups.channel.server (DOWN) ->  jboss.jgroups.channel.server.factory (DOWN)

jboss.clustering.registry.server.default.entry ***MISSING***

jboss.clustering.registry.web.dist.entry ***MISSING***

  jboss.clustering.registry.web.dist (DOWN) ->  jboss.clustering.registry.web.dist.entry (***MISSING***) jboss.clustering.registry.web.dist.factory (DOWN)

jboss.clustering.nodes.web.dist (DOWN)

  jboss.clustering.registry.web.dist.factory (DOWN) ->  jboss.clustering.nodes.web.dist (DOWN) jboss.clustering.group.web.dist (DOWN)

  jboss.clustering.group.web.dist (DOWN) ->  jboss.clustering.nodes.web.dist (DOWN)

jboss.infinispan.web.redirectcache (DOWN)

  jboss.clustering.registry.web.redirectcache.factory (DOWN) ->  jboss.infinispan.web.redirectcache (DOWN) jboss.clustering.nodes.web.redirectcache (DOWN) jboss.clustering.group.web.redirectcache (DOWN)

    jboss.clustering.registry.web.redirectcache (DOWN) ->  jboss.clustering.registry.web.redirectcache.factory (DOWN) jboss.clustering.registry.web.redirectcache.entry (***MISSING***)

  jboss.clustering.group.web.redirectcache (DOWN) ->  jboss.clustering.nodes.web.redirectcache (DOWN) jboss.infinispan.web.redirectcache (DOWN)

jboss.clustering.nodes.web.redirectcache (DOWN)

jboss.clustering.registry.web.redirectcache.entry ***MISSING***

jboss.xts.handlers ***MISSING***

  jboss.ws.xts-integration (DOWN) ->  jboss.xts.handlers (***MISSING***)

No services that have not been started found.