JBOSS 5.1.0 occasionally does not fully startup
We are seeing intermittent issues where JBOSS does not seem to fully startup (5.1.0 on CentOS 5.6 x86_64) The server is up and listening on port 8080, but browsing to that port does not get a response. I.e., I can telnet to the port to confirm there is a listener but get no response to say, browsing to the port. We will restart JBOSS using the init.d script. Sometimes it takes a couple restarts to get it to startup properly. I have modified the startup script to save a copy of the consle.log and boot.log on startup (and remove the tmp dir) contents. I have have reviewed these logs but see no obvious errors. The only notable difference between a good startup and hung one is the last log entry:
bad:
...
console.log.20130214163804:14:19:52,304 INFO [Http11AprProtocol] Starting Coyote HTTP/1.1 on http-0.0.0.0-8080
console.log.20130214163804:14:19:52,396 INFO [Http11AprProtocol] Starting Coyote HTTP/1.1 on http-0.0.0.0-8081
console.log.20130214163804:14:19:52,435 INFO [Http11AprProtocol] Starting Coyote HTTP/1.1 on http-0.0.0.0-8443
good:
...
console.log.20130217095512:16:39:26,879 INFO [Http11AprProtocol] Starting Coyote HTTP/1.1 on http-0.0.0.0-8080
console.log.20130217095512:16:39:27,002 INFO [Http11AprProtocol] Starting Coyote HTTP/1.1 on http-0.0.0.0-8081
console.log.20130217095512:16:39:27,149 INFO [Http11AprProtocol] Starting Coyote HTTP/1.1 on http-0.0.0.0-8443
console.log.20130217095512:16:39:27,169 INFO [AjpAprProtocol] Starting Coyote AJP/1.3 on ajp-0.0.0.0-8009
That's what I see most of the hangs (stopping after the port 8443 msg). On a couple hangs, the last entry was the port 8081 message (instead of 8443). The boot.log for a good vs bad startup looks the same (same msgs, number of lines, etc). E.g., The last messages from the last hang today is:
22:06:44,256 DEBUG [ServiceConfigurator] BlockingMode set to run in jboss.system:service=ThreadPool
22:06:44,256 DEBUG [ServiceController] Creating service jboss.system:service=ThreadPool
22:06:44,257 DEBUG [ServiceController] starting service jboss.system:service=ThreadPool
22:06:44,257 DEBUG [ServiceCreator] About to create xmbean object: jboss.system:type=Log4jService,service=Logging with code: org.jboss.logging.Log4jService with descriptor: resource:xmdesc/
Log4jService-xmbean.xml
22:06:44,278 DEBUG [ServiceCreator] Created mbean: jboss.system:type=Log4jService,service=Logging
22:06:44,279 DEBUG [ServiceConfigurator] ConfigurationURL set to resource:jboss-log4j.xml in jboss.system:type=Log4jService,service=Logging
22:06:44,279 DEBUG [ServiceConfigurator] Log4jQuietMode set to true in jboss.system:type=Log4jService,service=Logging
22:06:44,279 DEBUG [ServiceConfigurator] RefreshPeriod set to 60 in jboss.system:type=Log4jService,service=Logging
22:06:44,280 DEBUG [ServiceConfigurator] DefaultJBossServerLogThreshold set to INFO in jboss.system:type=Log4jService,service=Logging
22:06:44,280 DEBUG [ServiceController] Creating service jboss.system:type=Log4jService,service=Logging
22:06:44,280 DEBUG [Log4jService] Creating jboss.system:type=Log4jService,service=Logging
22:06:44,287 DEBUG [Log4jService$URLWatchTimerTask] Configuring from URL: resource:jboss-log4j.xml
Looking for tips on how to narrow this down further or if there are any known issues related to this I can look into.