4 Replies Latest reply on Mar 25, 2010 12:32 PM by Lauri

    Mysterious Shutdowns of JBoss application server

    gopi v Newbie

      Hi,

       

      I am using jboss AS 4.2.1 with jdk1.5.0_15 on Linux 2.6.9-67.0.7.ELsmp. My JBoss server is shutting down for every week and when i check the logs, i dont see any exception in the logs and but instead the shutdown messanges in the logs are same as when we give a stop singnal to that JBoss server.

       

      Please help me in resolving this issue.

       

      [ajp-xxx.xx.com %2Fxxx.xxx.xxx.xxx-18009-6] bizcode.BusinessCodeManager (BusinessCodeManager.java:158) - Unable to locate CodeValue for CodeType: mkttyp and Code: -1
      2010-02-11 12:18:07,309 ERROR [JBoss Shutdown Hook] scheduler.SchedulerFactory (SchedulerFactory.java:104) - Exception getting Quartz Scheduler
      java.lang.IllegalStateException: Shutdown in progress
          at java.lang.Shutdown.add(Shutdown.java:81)
          at java.lang.Runtime.addShutdownHook(Runtime.java:190)
          at org.quartz.plugins.management.ShutdownHookPlugin.initialize(ShutdownHookPlugin.java:141)
          at org.quartz.impl.StdSchedulerFactory.instantiate(StdSchedulerFactory.java:987)
          at org.quartz.impl.StdSchedulerFactory.getScheduler(StdSchedulerFactory.java:1152)
          at com.imany.foundation.scheduler.SchedulerFactory.getQuartzScheduler(SchedulerFactory.java:101)
          at com.xxxxx.foundation.webapp.bootstrap.BootstrapServlet.stopScheduler(BootstrapServlet.java:206)
          at com.xxxxx.foundation.webapp.bootstrap.BootstrapServlet.destroy(BootstrapServlet.java:103)
          at org.apache.catalina.core.StandardWrapper.unload(StandardWrapper.java:1382)
          at org.apache.catalina.core.StandardWrapper.stop(StandardWrapper.java:1727)
          at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4511)
          at org.apache.catalina.core.ContainerBase.destroy(ContainerBase.java:1134)
          at org.apache.catalina.core.StandardContext.destroy(StandardContext.java:4615)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:585)
          at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:297)
          at org.jboss.mx.server.RawDynamicInvoker.invoke(RawDynamicInvoker.java:164)
          at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
          at org.jboss.web.tomcat.service.TomcatDeployer.performUndeployInternal(TomcatDeployer.java:454)
          at org.jboss.web.tomcat.service.TomcatDeployer.performUndeploy(TomcatDeployer.java:425)
          at org.jboss.web.AbstractWebDeployer.stop(AbstractWebDeployer.java:422)
          at org.jboss.web.WebModule.stopModule(WebModule.java:100)
          at org.jboss.web.WebModule.stopService(WebModule.java:66)
          at org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:315)
          at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:247)
          at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:585)
          at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
          at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
          at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
          at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
          at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
          at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
          at $Proxy0.stop(Unknown Source)
          at org.jboss.system.ServiceController.stop(ServiceController.java:508)
          at sun.reflect.GeneratedMethodAccessor4572.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:585)
          at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
          at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
          at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
          at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
          at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
          at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
          at $Proxy51.stop(Unknown Source)
          at org.jboss.web.AbstractWebContainer.stop(AbstractWebContainer.java:498)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:585)
          at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
          at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
          at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
          at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
          at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
          at org.jboss.mx.interceptor.DynamicInterceptor.invoke(DynamicInterceptor.java:97)
          at org.jboss.system.InterceptorServiceMBeanSupport.invokeNext(InterceptorServiceMBeanSupport.java:238)
          at org.jboss.ws.integration.jboss42.DeployerInterceptor.stop(DeployerInterceptor.java:111)
          at org.jboss.deployment.SubDeployerInterceptorSupport$XMBeanInterceptor.stop(SubDeployerInterceptorSupport.java:196)
          at org.jboss.deployment.SubDeployerInterceptor.invoke(SubDeployerInterceptor.java:99)
          at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
          at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
          at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
          at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
          at $Proxy52.stop(Unknown Source)
          at org.jboss.deployment.MainDeployer.stop(MainDeployer.java:667)
          at org.jboss.deployment.MainDeployer.stop(MainDeployer.java:659)
          at org.jboss.deployment.MainDeployer.undeploy(MainDeployer.java:638)
          at org.jboss.deployment.MainDeployer.shutdown(MainDeployer.java:516)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:585)
          at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
          at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
          at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
          at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
          at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
          at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
          at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
          at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
          at org.jboss.system.server.ServerImpl$ShutdownHook.shutdownDeployments(ServerImpl.java:1058)
          at org.jboss.system.server.ServerImpl$ShutdownHook.shutdown(ServerImpl.java:1033)
          at org.jboss.system.server.ServerImpl$ShutdownHook.run(ServerImpl.java:996)
      2010-02-11 12:18:07,314 ERROR [JBoss Shutdown Hook] util.FoundationException (FoundationException.java:366) - <Exception getting Quartz Scheduler>
         
      2010-02-11 12:18:07,315 ERROR [JBoss Shutdown Hook] bootstrap.BootstrapServlet (BootstrapServlet.java:211) - Exception on scheduler shutdown
      <id 67ce63b4126be09a1d2> com.xxx.foundation.util.FoundationException: <id 67ce63b4126be09a1d2> Shutdown in progress;
          ---> nested java.lang.IllegalStateException: Shutdown in progress
          at com.xxxx.foundation.scheduler.SchedulerFactory.getQuartzScheduler(SchedulerFactory.java:105)
          at com.xxxx.foundation.webapp.bootstrap.BootstrapServlet.stopScheduler(BootstrapServlet.java:206)
          at com.imany.foundation.webapp.bootstrap.BootstrapServlet.destroy(BootstrapServlet.java:103)
          at org.apache.catalina.core.StandardWrapper.unload(StandardWrapper.java:1382)
          at org.apache.catalina.core.StandardWrapper.stop(StandardWrapper.java:1727)
          at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4511)
          at org.apache.catalina.core.ContainerBase.destroy(ContainerBase.java:1134)
          at org.apache.catalina.core.StandardContext.destroy(StandardContext.java:4615)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:585)
          at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:297)
          at org.jboss.mx.server.RawDynamicInvoker.invoke(RawDynamicInvoker.java:164)
          at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
          at org.jboss.web.tomcat.service.TomcatDeployer.performUndeployInternal(TomcatDeployer.java:454)
          at org.jboss.web.tomcat.service.TomcatDeployer.performUndeploy(TomcatDeployer.java:425)
          at org.jboss.web.AbstractWebDeployer.stop(AbstractWebDeployer.java:422)
          at org.jboss.web.WebModule.stopModule(WebModule.java:100)
          at org.jboss.web.WebModule.stopService(WebModule.java:66)
          at org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:315)
          at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:247)
          at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:585)
          at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
          at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
          at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
          at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
          at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
          at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
          at $Proxy0.stop(Unknown Source)
          at org.jboss.system.ServiceController.stop(ServiceController.java:508)
          at sun.reflect.GeneratedMethodAccessor4572.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:585)
          at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
          at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
          at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
          at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
          at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
          at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
          at $Proxy51.stop(Unknown Source)
          at org.jboss.web.AbstractWebContainer.stop(AbstractWebContainer.java:498)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:585)
          at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
          at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
          at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
          at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
          at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
          at org.jboss.mx.interceptor.DynamicInterceptor.invoke(DynamicInterceptor.java:97)
          at org.jboss.system.InterceptorServiceMBeanSupport.invokeNext(InterceptorServiceMBeanSupport.java:238)
          at org.jboss.ws.integration.jboss42.DeployerInterceptor.stop(DeployerInterceptor.java:111)
          at org.jboss.deployment.SubDeployerInterceptorSupport$XMBeanInterceptor.stop(SubDeployerInterceptorSupport.java:196)
          at org.jboss.deployment.SubDeployerInterceptor.invoke(SubDeployerInterceptor.java:99)
          at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
          at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
          at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
          at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
          at $Proxy52.stop(Unknown Source)
          at org.jboss.deployment.MainDeployer.stop(MainDeployer.java:667)
          at org.jboss.deployment.MainDeployer.stop(MainDeployer.java:659)
          at org.jboss.deployment.MainDeployer.undeploy(MainDeployer.java:638)
          at org.jboss.deployment.MainDeployer.shutdown(MainDeployer.java:516)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:585)
          at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
          at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
          at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
          at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
          at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
          at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
          at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
          at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
          at org.jboss.system.server.ServerImpl$ShutdownHook.shutdownDeployments(ServerImpl.java:1058)
          at org.jboss.system.server.ServerImpl$ShutdownHook.shutdown(ServerImpl.java:1033)
          at org.jboss.system.server.ServerImpl$ShutdownHook.run(ServerImpl.java:996)

        • 1. Re: Mysterious Shutdowns of JBoss application server
          Peter Johnson Master

          Gopi, welcome to the JBoss forums!

           

          A search on "jboss unexpected shutdown" should reveal a number of discussions about this same topic. Here is one that you can start with:

          http://community.jboss.org/message/247680#247680

          • 2. Re: Mysterious Shutdowns of JBoss application server
            gopi v Newbie

            Peter,

             

            I am using nohup to start the JBoss instance and there are 3 other environments for the same application hosted on the same physical sever, but i am not facing this kind of issue. I already considered using -Xrs option, but this is not happening in other environments. Is there any other way to work on this issue.

             

            Before shutting down, i am getting some warnings like below

             

            2010-02-17 23:44:59,678 WARN  [ajp- server-name.xxx.com%2Fxxx.xxx.xxx.xxx-28009-5] hibernate3.CriteriaConverter (CriteriaConv
            erter.java:1007) - HbnKeyInfo not found in Hibernate keys-map for order-by key "lastModifiedTime". Caller should ensure that H
            bnKeyInfo is specified in Hibernate keys-map for every order-by field that is intended to be used in com.xxxxx.foundation.api.
            collaboration.WorkSession

             

            2010-02-17 21:43:48,265 WARN  [JMS SessionPool Worker-60] util.JBossCacheWrapper (JBossCacheWrapper.java:108) - Cache.put: fai
            lure acquiring lock: fqn=/xxxxx/UserPreference, caller=GlobalTransaction:<null>:31, lock=read owners=[GlobalTransaction:<null>
            :22, GlobalTransaction:<null>:23, GlobalTransaction:<null>:18, GlobalTransaction:<null>:19, GlobalTransaction:<null>:24, Globa
            lTransaction:<null>:21, GlobalTransaction:<null>:20, GlobalTransaction:<null>:25, GlobalTransaction:<null>:26, GlobalTransacti
            on:<null>:27, GlobalTransaction:<null>:28], write owner=GlobalTransaction:<null>:17 (org.jboss.cache.lock.LockStrategyReadUnco
            mmitted@38663561)

            • 3. Re: Mysterious Shutdowns of JBoss application server
              Peter Johnson Master

              I do not think that those warning have anything to do with the shutdown.

               

              Is there a log entry in the serverlog that indicates that the app server received a shutdown request? Something like this:

               

              2010-03-16 14:19:55,469 INFO  [org.jboss.bootstrap.microcontainer.ServerImpl] (JBoss Shutdown Hook) Runtime shutdown hook called, forceHalt: true

               

              If so, then something external to the app server is requesting it to shut down. Tracking down what is doing that request is the hard part. I do not recall any definitive advice on how to do this.

              • 4. Re: Mysterious Shutdowns of JBoss application server
                Lauri Newbie

                By default jmx port of jboss is not secure.

                If you have your server looking to internet (i.e. -b 0.0.0.0 on startup), you should be aware that anyone in interenet can run

                $JBOSS_HOME/bin/shutdown.sh -S -s yourip

                and your jboss will be shutting down.

                 

                Two options here

                1) read and follow

                http://community.jboss.org/wiki/SecureJBoss

                2)

                a) make jboss bound to localhost (-b 127.0.0.0),

                b) remove all administration consoles (/jmx-console, /web-console, /admin-consle) or secure them (http://community.jboss.org/wiki/SecureTheJmxConsole)

                c) setup webserver to redirect requests to AJP port of jboss (search for mod_jk or mod_proxy_ajp).