WFLYCTL0348: Timeout after [300] seconds waiting for service container stability.
Hi,
Sometimes receive timeout error when starting server or hot deploying 33MB war file to WF 13 instance running in standalone-full mode.
For hot deploy I have put logger for org.jboss.as.server.deployment on TRACE, but I cannot find an indication on why it is getting stuck.
As for start-up I am getting lots of reconnection attempts for activemq.
What fixes things is deleting everything in /standalone/data, standalone/tmp and .isDeployed or .deployedFailed from /standalone/deployments and restarting server.
Stack trace for server start:
2019-02-20 12:09:58,747 DEBUG [org.jboss.weld.BootstrapTracker] (MSC service thread 1-8) END bootstrap (39464 ms)
2019-02-20 12:10:00,457 DEBUG [org.apache.activemq.artemis.ra] (ViSixdScheduler_Worker-9) Creating Connection Factory on the resource adapter for transport=[TransportConfiguration(name=aebb083f-34f7-11e9-92f1-6ae220524153, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?bufferPooling=false&serverId=0] with ha=false
2019-02-20 12:10:00,457 DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl] (ViSixdScheduler_Worker-9) Trying reconnection attempt 0/1
2019-02-20 12:10:00,457 DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl] (ViSixdScheduler_Worker-9) Trying to connect with connectorFactory = org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnectorFactory@a9d42d, connectorConfig=TransportConfiguration(name=aebb083f-34f7-11e9-92f1-6ae220524153, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?bufferPooling=false&serverId=0
2019-02-20 12:10:00,457 DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl] (ViSixdScheduler_Worker-9) Reconnection successful
2019-02-20 12:10:00,457 DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl] (ViSixdScheduler_Worker-9) ClientSessionFactoryImpl received backup update for live/backup pair = TransportConfiguration(name=aebb083f-34f7-11e9-92f1-6ae220524153, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?bufferPooling=false&serverId=0 / null but it didn't belong to TransportConfiguration(name=aebb083f-34f7-11e9-92f1-6ae220524153, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?bufferPooling=false&serverId=0
2019-02-20 12:10:00,457 DEBUG [org.apache.activemq.artemis.ra] (ViSixdScheduler_Worker-9) Creating Recovery Connection Factory on the resource adapter for transport=[TransportConfiguration(name=aebb0847-34f7-11e9-92f1-6ae220524153, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?bufferPooling=false&serverId=0]
2019-02-20 12:10:00,457 DEBUG [org.apache.activemq.artemis.ra] (ViSixdScheduler_Worker-9) registering recovery for factory : ActiveMQConnectionFactory [serverLocator=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=aebb0847-34f7-11e9-92f1-6ae220524153, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?bufferPooling=false&serverId=0], discoveryGroupConfiguration=null], clientID=null, consumerWindowSize = 1048576, dupsOKBatchSize=1048576, transactionBatchSize=1048576, readOnly=false]
2019-02-20 12:10:01,107 DEBUG [org.apache.activemq.artemis.core.server.impl.QueueImpl] (Thread-4 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$3@146eb85)) QueueImpl[name=jms.queue.conditionalRuleQueue, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=2368e543-3422-11e9-a784-ccda20524153]]@1b9e92c doing deliver. messageReferences=0
2019-02-20 12:10:42,508 DEBUG [org.apache.activemq.artemis.service.extensions.xa.recovery] (Periodic Recovery) Trying to connect recovery on XARecoveryConfig [transportConfiguration=[TransportConfiguration(name=, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?serverId=0], discoveryConfiguration=null, username=null, password=****, JNDI_NAME=java:/JmsXA] of [XARecoveryConfig [transportConfiguration=[TransportConfiguration(name=, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?serverId=0], discoveryConfiguration=null, username=null, password=****, JNDI_NAME=java:/JmsXA]]
2019-02-20 12:10:42,508 DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl] (Periodic Recovery) Trying reconnection attempt 0/1
2019-02-20 12:10:42,518 DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl] (Periodic Recovery) Trying to connect with connectorFactory = org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnectorFactory@1054dfd, connectorConfig=TransportConfiguration(name=, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?serverId=0
2019-02-20 12:10:42,518 DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl] (Periodic Recovery) Reconnection successful
2019-02-20 12:10:42,518 DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl] (Periodic Recovery) ClientSessionFactoryImpl received backup update for live/backup pair = TransportConfiguration(name=, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?serverId=0 / null but it didn't belong to TransportConfiguration(name=, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?serverId=0
2019-02-20 12:10:42,518 DEBUG [org.apache.activemq.artemis.service.extensions.xa.recovery] (Periodic Recovery) looking for recover at ClientSessionImpl [name=c7cd081d-34f7-11e9-92f1-6ae220524153, username=null, closed=false, factory = org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl@672f45, metaData=()]@171104e configuration [XARecoveryConfig [transportConfiguration=[TransportConfiguration(name=, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?serverId=0], discoveryConfiguration=null, username=null, password=****, JNDI_NAME=java:/JmsXA]]
2019-02-20 12:10:52,518 DEBUG [org.apache.activemq.artemis.service.extensions.xa.recovery] (Periodic Recovery) looking for recover at ClientSessionImpl [name=c7cd081d-34f7-11e9-92f1-6ae220524153, username=null, closed=false, factory = org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl@672f45, metaData=()]@171104e configuration [XARecoveryConfig [transportConfiguration=[TransportConfiguration(name=, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?serverId=0], discoveryConfiguration=null, username=null, password=****, JNDI_NAME=java:/JmsXA]]
2019-02-20 12:12:52,524 DEBUG [org.apache.activemq.artemis.service.extensions.xa.recovery] (Periodic Recovery) looking for recover at ClientSessionImpl [name=c7cd081d-34f7-11e9-92f1-6ae220524153, username=null, closed=false, factory = org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl@672f45, metaData=()]@171104e configuration [XARecoveryConfig [transportConfiguration=[TransportConfiguration(name=, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?serverId=0], discoveryConfiguration=null, username=null, password=****, JNDI_NAME=java:/JmsXA]]
2019-02-20 12:13:02,525 DEBUG [org.apache.activemq.artemis.service.extensions.xa.recovery] (Periodic Recovery) looking for recover at ClientSessionImpl [name=c7cd081d-34f7-11e9-92f1-6ae220524153, username=null, closed=false, factory = org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl@672f45, metaData=()]@171104e configuration [XARecoveryConfig [transportConfiguration=[TransportConfiguration(name=, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?serverId=0], discoveryConfiguration=null, username=null, password=****, JNDI_NAME=java:/JmsXA]]
2019-02-20 12:13:37,824 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[
("core-service" => "management"),
("management-interface" => "http-interface")
]'
Stack trace for hot deploy looks as follows:
2019-02-20 16:24:40,956 DEBUG [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 2) Deployment scan of [C:\WildFly13\standalone\deployments] found update action [{
"operation" => "full-replace-deployment",
"address" => [],
"name" => "sixD-dashboard.war",
"content" => [{"url" => "file:/C:/WildFly13/standalone/deployments/sixD-dashboard.war"}],
"persistent" => false,
"owner" => [
("subsystem" => "deployment-scanner"),
("scanner" => "default")
],
"enabled" => true
}]
2019-02-20 16:24:42,375 DEBUG [org.jboss.as.repository] (DeploymentScanner-threads - 1) Deleting C:\WildFly13\standalone\data\content\content8006780446048340969.tmp recursively
2019-02-20 16:24:42,377 DEBUG [org.jboss.as.repository] (DeploymentScanner-threads - 1) Deleting C:\WildFly13\standalone\data\content\7c\258a3f11f655414ba332d689abce52ec5887ca\tmp recursively
2019-02-20 16:24:42,378 INFO [org.jboss.as.repository] (DeploymentScanner-threads - 1) WFLYDR0001: Content added at location C:\WildFly13\standalone\data\content\7c\258a3f11f655414ba332d689abce52ec5887ca\content
2019-02-20 16:24:42,416 DEBUG [org.vaadin.dontpush.server.SocketCommunicationManager] (ServerService Thread Pool -- 173) Removed SocketCommunicationManager 3964378f-ca5b-4e2f-ae92-3ff2a534b6e0 from active map.
2019-02-20 16:24:45,209 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0028: Stopped deployment sixD-dashboard.war (runtime-name: sixD-dashboard.war) in 2816ms
2019-02-20 16:24:45,214 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0027: Starting deployment of "sixD-dashboard.war" (runtime-name: "sixD-dashboard.war")
2019-02-20 16:26:05,358 DEBUG [org.apache.activemq.artemis.service.extensions.xa.recovery] (Periodic Recovery) looking for recover at ClientSessionImpl [name=a49877b6-34f8-11e9-9568-6ae220524153, username=null, closed=false, factory = org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl@bc8694, metaData=()]@18808cd configuration [XARecoveryConfig [transportConfiguration=[TransportConfiguration(name=, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?serverId=0], discoveryConfiguration=null, username=null, password=****, JNDI_NAME=java:/JmsXA]]
2019-02-20 16:26:15,442 DEBUG [org.apache.activemq.artemis.service.extensions.xa.recovery] (Periodic Recovery) looking for recover at ClientSessionImpl [name=a49877b6-34f8-11e9-9568-6ae220524153, username=null, closed=false, factory = org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl@bc8694, metaData=()]@18808cd configuration [XARecoveryConfig [transportConfiguration=[TransportConfiguration(name=, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?serverId=0], discoveryConfiguration=null, username=null, password=****, JNDI_NAME=java:/JmsXA]]
2019-02-20 16:28:15,444 DEBUG [org.apache.activemq.artemis.service.extensions.xa.recovery] (Periodic Recovery) looking for recover at ClientSessionImpl [name=a49877b6-34f8-11e9-9568-6ae220524153, username=null, closed=false, factory = org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl@bc8694, metaData=()]@18808cd configuration [XARecoveryConfig [transportConfiguration=[TransportConfiguration(name=, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?serverId=0], discoveryConfiguration=null, username=null, password=****, JNDI_NAME=java:/JmsXA]]
2019-02-20 16:28:25,447 DEBUG [org.apache.activemq.artemis.service.extensions.xa.recovery] (Periodic Recovery) looking for recover at ClientSessionImpl [name=a49877b6-34f8-11e9-9568-6ae220524153, username=null, closed=false, factory = org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl@bc8694, metaData=()]@18808cd configuration [XARecoveryConfig [transportConfiguration=[TransportConfiguration(name=, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?serverId=0], discoveryConfiguration=null, username=null, password=****, JNDI_NAME=java:/JmsXA]]
2019-02-20 16:29:45,224 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'full-replace-deployment' at address '[]'