JBoss EAP 6.0.1 domain mode deployment hangs
kclair-rei May 29, 2013 4:53 PMI'm running JBoss EAP 6.0.1 in domain mode on RHEL.
When I deploy an application through the web console in the domain controller and then try to start the application, these are the last INFO level logs I see (some info x'ed out):
[Server:cluster1] 13:14:55,072 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-3) JBAS010400: Bound data source [java:jboss/OnlineDataStore]
[Server:cluster1] 13:14:55,073 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-4) JBAS010400: Bound data source [java:jboss/DataWarehouse]
[Server:cluster1] 13:14:55,079 INFO [org.jboss.as.remoting] (MSC service thread 1-7) JBAS017100: Listening on xx.xx.xx.xx:4447
[Server:cluster1] 13:14:55,080 INFO [org.jboss.as.server.deployment] (MSC service thread 1-9) JBAS015876: Starting deployment of "online-web-99999-SNAPSHOT.war"
[Server:cluster1] 13:15:01,445 INFO [org.jboss.as.osgi] (MSC service thread 1-14) JBAS011907: Register module: Module "deployment.online-web-99999-SNAPSHOT.war:main" from Service Module Loader
[Host Controller] 13:15:24,525 INFO [org.jboss.as.host.controller] (proxy-threads - 1) JBAS010919: Registering server cluster1
After that there is nothing - the context never registers, and the application is not accessible.
The TRACE level logs after this show a lot of messaging activity.
The DEBUG level logs after this show a lot of activity from org.jboss.osgi.resolver and after that consistent PeriodicRecovery activity.
I'm not sure how to figure out what the application deployment is doing.
When I try to get a thread dump using jstack, I need to use the -F option. The thread dump can be viewed here:
There are quite a few blocked threads that look like they are waiting on unix processes and file reads. I can see that it created a deployment directory in $JBOSS_HOME/domain/servers/cluster1/tmp/vfs/ where the jars are unpacked, so it appears to be reading the war file without a problem.
strace shows that the processes are either stuck on FUTEX() or wait4().
I'd love some advice about where else to look to figure out what is going on.