Mysterious Shutdowns of JBoss application server

[gopi1828]

Hi,

I am using jboss AS 4.2.1 with jdk1.5.0_15 on Linux 2.6.9-67.0.7.ELsmp. My JBoss server is shutting down for every week and when i check the logs, i dont see any exception in the logs and but instead the shutdown messanges in the logs are same as when we give a stop singnal to that JBoss server.

Please help me in resolving this issue.

[ajp-xxx.xx.com %2Fxxx.xxx.xxx.xxx-18009-6] bizcode.BusinessCodeManager (BusinessCodeManager.java:158) - Unable to locate CodeValue for CodeType: mkttyp and Code: -1
2010-02-11 12:18:07,309 ERROR [JBoss Shutdown Hook] scheduler.SchedulerFactory (SchedulerFactory.java:104) - Exception getting Quartz Scheduler
java.lang.IllegalStateException: Shutdown in progress
    at java.lang.Shutdown.add(Shutdown.java:81)
    at java.lang.Runtime.addShutdownHook(Runtime.java:190)
    at org.quartz.plugins.management.ShutdownHookPlugin.initialize(ShutdownHookPlugin.java:141)
    at org.quartz.impl.StdSchedulerFactory.instantiate(StdSchedulerFactory.java:987)
    at org.quartz.impl.StdSchedulerFactory.getScheduler(StdSchedulerFactory.java:1152)
    at com.imany.foundation.scheduler.SchedulerFactory.getQuartzScheduler(SchedulerFactory.java:101)
    at com.xxxxx.foundation.webapp.bootstrap.BootstrapServlet.stopScheduler(BootstrapServlet.java:206)
    at com.xxxxx.foundation.webapp.bootstrap.BootstrapServlet.destroy(BootstrapServlet.java:103)
    at org.apache.catalina.core.StandardWrapper.unload(StandardWrapper.java:1382)
    at org.apache.catalina.core.StandardWrapper.stop(StandardWrapper.java:1727)
    at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4511)
    at org.apache.catalina.core.ContainerBase.destroy(ContainerBase.java:1134)
    at org.apache.catalina.core.StandardContext.destroy(StandardContext.java:4615)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:297)
    at org.jboss.mx.server.RawDynamicInvoker.invoke(RawDynamicInvoker.java:164)
    at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
    at org.jboss.web.tomcat.service.TomcatDeployer.performUndeployInternal(TomcatDeployer.java:454)
    at org.jboss.web.tomcat.service.TomcatDeployer.performUndeploy(TomcatDeployer.java:425)
    at org.jboss.web.AbstractWebDeployer.stop(AbstractWebDeployer.java:422)
    at org.jboss.web.WebModule.stopModule(WebModule.java:100)
    at org.jboss.web.WebModule.stopService(WebModule.java:66)
    at org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:315)
    at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:247)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
    at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
    at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
    at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
    at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
    at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
    at $Proxy0.stop(Unknown Source)
    at org.jboss.system.ServiceController.stop(ServiceController.java:508)
    at sun.reflect.GeneratedMethodAccessor4572.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
    at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
    at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
    at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
    at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
    at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
    at $Proxy51.stop(Unknown Source)
    at org.jboss.web.AbstractWebContainer.stop(AbstractWebContainer.java:498)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
    at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
    at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
    at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
    at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
    at org.jboss.mx.interceptor.DynamicInterceptor.invoke(DynamicInterceptor.java:97)
    at org.jboss.system.InterceptorServiceMBeanSupport.invokeNext(InterceptorServiceMBeanSupport.java:238)
    at org.jboss.ws.integration.jboss42.DeployerInterceptor.stop(DeployerInterceptor.java:111)
    at org.jboss.deployment.SubDeployerInterceptorSupport$XMBeanInterceptor.stop(SubDeployerInterceptorSupport.java:196)
    at org.jboss.deployment.SubDeployerInterceptor.invoke(SubDeployerInterceptor.java:99)
    at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
    at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
    at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
    at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
    at $Proxy52.stop(Unknown Source)
    at org.jboss.deployment.MainDeployer.stop(MainDeployer.java:667)
    at org.jboss.deployment.MainDeployer.stop(MainDeployer.java:659)
    at org.jboss.deployment.MainDeployer.undeploy(MainDeployer.java:638)
    at org.jboss.deployment.MainDeployer.shutdown(MainDeployer.java:516)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
    at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
    at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
    at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
    at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
    at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
    at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
    at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
    at org.jboss.system.server.ServerImpl$ShutdownHook.shutdownDeployments(ServerImpl.java:1058)
    at org.jboss.system.server.ServerImpl$ShutdownHook.shutdown(ServerImpl.java:1033)
    at org.jboss.system.server.ServerImpl$ShutdownHook.run(ServerImpl.java:996)
2010-02-11 12:18:07,314 ERROR [JBoss Shutdown Hook] util.FoundationException (FoundationException.java:366) - <Exception getting Quartz Scheduler>
   
2010-02-11 12:18:07,315 ERROR [JBoss Shutdown Hook] bootstrap.BootstrapServlet (BootstrapServlet.java:211) - Exception on scheduler shutdown
<id 67ce63b4126be09a1d2> com.xxx.foundation.util.FoundationException: <id 67ce63b4126be09a1d2> Shutdown in progress;
    ---> nested java.lang.IllegalStateException: Shutdown in progress
    at com.xxxx.foundation.scheduler.SchedulerFactory.getQuartzScheduler(SchedulerFactory.java:105)
    at com.xxxx.foundation.webapp.bootstrap.BootstrapServlet.stopScheduler(BootstrapServlet.java:206)
    at com.imany.foundation.webapp.bootstrap.BootstrapServlet.destroy(BootstrapServlet.java:103)
    at org.apache.catalina.core.StandardWrapper.unload(StandardWrapper.java:1382)
    at org.apache.catalina.core.StandardWrapper.stop(StandardWrapper.java:1727)
    at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4511)
    at org.apache.catalina.core.ContainerBase.destroy(ContainerBase.java:1134)
    at org.apache.catalina.core.StandardContext.destroy(StandardContext.java:4615)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:297)
    at org.jboss.mx.server.RawDynamicInvoker.invoke(RawDynamicInvoker.java:164)
    at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
    at org.jboss.web.tomcat.service.TomcatDeployer.performUndeployInternal(TomcatDeployer.java:454)
    at org.jboss.web.tomcat.service.TomcatDeployer.performUndeploy(TomcatDeployer.java:425)
    at org.jboss.web.AbstractWebDeployer.stop(AbstractWebDeployer.java:422)
    at org.jboss.web.WebModule.stopModule(WebModule.java:100)
    at org.jboss.web.WebModule.stopService(WebModule.java:66)
    at org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:315)
    at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:247)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
    at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
    at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
    at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
    at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
    at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
    at $Proxy0.stop(Unknown Source)
    at org.jboss.system.ServiceController.stop(ServiceController.java:508)
    at sun.reflect.GeneratedMethodAccessor4572.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
    at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
    at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
    at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
    at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
    at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
    at $Proxy51.stop(Unknown Source)
    at org.jboss.web.AbstractWebContainer.stop(AbstractWebContainer.java:498)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
    at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
    at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
    at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
    at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
    at org.jboss.mx.interceptor.DynamicInterceptor.invoke(DynamicInterceptor.java:97)
    at org.jboss.system.InterceptorServiceMBeanSupport.invokeNext(InterceptorServiceMBeanSupport.java:238)
    at org.jboss.ws.integration.jboss42.DeployerInterceptor.stop(DeployerInterceptor.java:111)
    at org.jboss.deployment.SubDeployerInterceptorSupport$XMBeanInterceptor.stop(SubDeployerInterceptorSupport.java:196)
    at org.jboss.deployment.SubDeployerInterceptor.invoke(SubDeployerInterceptor.java:99)
    at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
    at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
    at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
    at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
    at $Proxy52.stop(Unknown Source)
    at org.jboss.deployment.MainDeployer.stop(MainDeployer.java:667)
    at org.jboss.deployment.MainDeployer.stop(MainDeployer.java:659)
    at org.jboss.deployment.MainDeployer.undeploy(MainDeployer.java:638)
    at org.jboss.deployment.MainDeployer.shutdown(MainDeployer.java:516)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
    at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
    at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
    at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
    at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
    at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
    at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
    at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
    at org.jboss.system.server.ServerImpl$ShutdownHook.shutdownDeployments(ServerImpl.java:1058)
    at org.jboss.system.server.ServerImpl$ShutdownHook.shutdown(ServerImpl.java:1033)
    at org.jboss.system.server.ServerImpl$ShutdownHook.run(ServerImpl.java:996)

[peterj]

Gopi, welcome to the JBoss forums!

A search on "jboss unexpected shutdown" should reveal a number of discussions about this same topic. Here is one that you can start with:

http://community.jboss.org/message/247680#247680

[gopi1828]

Peter,

I am using nohup to start the JBoss instance and there are 3 other environments for the same application hosted on the same physical sever, but i am not facing this kind of issue. I already considered using -Xrs option, but this is not happening in other environments. Is there any other way to work on this issue.

Before shutting down, i am getting some warnings like below

2010-02-17 23:44:59,678 WARN  [ajp- server-name.xxx.com%2Fxxx.xxx.xxx.xxx-28009-5] hibernate3.CriteriaConverter (CriteriaConv
erter.java:1007) - HbnKeyInfo not found in Hibernate keys-map for order-by key "lastModifiedTime". Caller should ensure that H
bnKeyInfo is specified in Hibernate keys-map for every order-by field that is intended to be used in com.xxxxx.foundation.api.
collaboration.WorkSession

2010-02-17 21:43:48,265 WARN  [JMS SessionPool Worker-60] util.JBossCacheWrapper (JBossCacheWrapper.java:108) - Cache.put: fai
lure acquiring lock: fqn=/xxxxx/UserPreference, caller=GlobalTransaction:<null>:31, lock=read owners=[GlobalTransaction:<null>
:22, GlobalTransaction:<null>:23, GlobalTransaction:<null>:18, GlobalTransaction:<null>:19, GlobalTransaction:<null>:24, Globa
lTransaction:<null>:21, GlobalTransaction:<null>:20, GlobalTransaction:<null>:25, GlobalTransaction:<null>:26, GlobalTransacti
on:<null>:27, GlobalTransaction:<null>:28], write owner=GlobalTransaction:<null>:17 (org.jboss.cache.lock.LockStrategyReadUnco
mmitted@38663561)

[peterj]

I do not think that those warning have anything to do with the shutdown.

Is there a log entry in the serverlog that indicates that the app server received a shutdown request? Something like this:

2010-03-16 14:19:55,469 INFO  [org.jboss.bootstrap.microcontainer.ServerImpl] (JBoss Shutdown Hook) Runtime shutdown hook called, forceHalt: true

If so, then something external to the app server is requesting it to shut down. Tracking down what is doing that request is the hard part. I do not recall any definitive advice on how to do this.

[lauri]

By default jmx port of jboss is not secure.

If you have your server looking to internet (i.e. -b 0.0.0.0 on startup), you should be aware that anyone in interenet can run

$JBOSS_HOME/bin/shutdown.sh -S -s yourip

and your jboss will be shutting down.

Two options here

1) read and follow

http://community.jboss.org/wiki/SecureJBoss

2)

a) make jboss bound to localhost (-b 127.0.0.0),

b) remove all administration consoles (/jmx-console, /web-console, /admin-consle) or secure them (http://community.jboss.org/wiki/SecureTheJmxConsole)

c) setup webserver to redirect requests to AJP port of jboss (search for mod_jk or mod_proxy_ajp).