1 2 Previous Next 26 Replies Latest reply on Jul 28, 2009 6:41 PM by jfrazier

    Few Issues

      Hey All,

      We are wanting to implement JOPR/JON in our production/testing environemnts however we are running into some issues. First few issues were corrected by running JOPR 2.2.1 instead of JON 2.2 (pending release of JON 2.3), these were ldap login issues and having a symlinked directory in the path for JBoss.

      However now we are not able to monitor the JbossAS instance's JVM for things like GC/etc.

      I added the -Djboss.platform.mbeanserver to the java opts (running Sun JDK 1.6.0_12), it starts throwing this error in the logs:

      22 Jul 2009 18:14:44,338 ERROR Connector:748 - invalid Object Name
      javax.management.InstanceNotFoundException: jboss.remoting:service=invoker,transport= socket,host=pdxuat8app02vm.corp.unicru.com,port=4446,dataType=invocation,enableTcpNoDelay=true,marshaller=org.jboss.invocation.unified.marshall.InvocationMarshaller,socketTimeout=600000,unmarshaller=org.jboss.invocation.unified.marshall.InvocationUnMarshaller is not registered.
       at org.jboss.mx.server.registry.BasicMBeanRegistry.get(BasicMBeanRegistry.java:523)
       at org.jboss.mx.server.MBeanServerImpl.unregisterMBean(MBeanServerImpl.java:383)
       at org.jboss.remoting.transport.Connector.stop(Connector.java:744)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:995)
       at $Proxy0.stop(Unknown Source)
       at org.jboss.system.ServiceController.stop(ServiceController.java:508)
       at sun.reflect.GeneratedMethodAccessor5382.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
       at $Proxy4.stop(Unknown Source)
       at org.jboss.deployment.SARDeployer.stop(SARDeployer.java:336)
       at org.jboss.deployment.MainDeployer.stop(MainDeployer.java:667)
       at org.jboss.deployment.MainDeployer.undeploy(MainDeployer.java:638)
       at org.jboss.deployment.MainDeployer.shutdown(MainDeployer.java:516)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.system.server.ServerImpl$ShutdownHook.shutdownDeployments(ServerImpl.java:1058)
       at org.jboss.system.server.ServerImpl$ShutdownHook.shutdown(ServerImpl.java:1033)
       at org.jboss.system.server.ServerImpl$ShutdownHook.run(ServerImpl.java:996)


      Here is what we are running:

      Type: JBossAS Server (Server)
      Description: JBoss Enterprise Application Platform
      Version: 4.3.0.GA
      Parent: Linux 'pdxuat8app02vm'
      Partition Name: uatcluster
      Build Date: January 3 2008
      Version Name: EAP
      Start Date: Thu Jul 16 03:52:52 EDT 2009
      Architecture: x86_64
      OS Name: Linux Distribution
      Name: Red Hat Enterprise Linux Server
      OS Version: 2.6.18-128.1.1.el5
      Distribution Version: release 5.1 (Tikanga)


      We also have an issue with an agent on a different machine exiting for unknown reasons. It looks like in the log it shutdowns based on a user request but we dont know why someone would be doing that, is there a way to turn on more logging to catch exactly why the agent is "exiting".

      Any help would be greatly appreciated and if more info is needed please let me know and id be more then willing to provide more information.

        • 1. Re: Few Issues
          mazz

          Unless your JBossAS instance successfully exposes the standard platform MBeans, you won't be able to monitor its JVM for things like GC/etc.

          There are several JBossAS issues related to this (that is, getting your JBoss app server to expose the standard platform MBeans). Look at these JBossAS JIRA issues, these are all probably directly or indirectly related:

          https://jira.jboss.org/jira/browse/JBAS-5220
          https://jira.jboss.org/jira/browse/JBAS-3381
          https://jira.jboss.org/jira/browse/JBPAPP-2031
          https://jira.jboss.org/jira/browse/JBAS-2427
          https://jira.jboss.org/jira/browse/JBPAPP-2031

          Unsure when all of these were fixed, check the jiras and match them with your JBossAS version.

          FWIW: to run JBossAS 4 with the platform MBeans exposed via JNP too, pass in these extra variables when you start your JBossAS app server instance:

          -Djboss.platform.mbeanserver
          -Djavax.management.builder.initial=org.jboss.system.server.jmx.MBeanServerBuilderImpl
          



          • 2. Re: Few Issues

            Thank you so much for the quick reply. I cleared all the logs and started over again adding that extra option you provide as all the JIRA's look like they should be fixed in the version we are running.

            It looks like the error i posted is at shutdown rather then startup and happens when we dont have those java options set. So that one is a non issue. However the application still isnt coming up with those options set.

            Looks like this is the error:

            org.jboss.deployment.DeploymentInfo@885cf7d1 { url=file:/vol/app/atao/atao1/jboss405/jboss-as/server/production/deploy/atao.ear/ }
             deployer: org.jboss.deployment.EARDeployer@8f3d27
             status: Deployment FAILED reason: - nested throwable: (java.lang.NullPointerException)
             state: FAILED


            Im assuming i should just open a Jboss Support case about this rather then here?


            Also about the last part of my message, is there an easy way to tell why the agent "exited" on one of our machines as it seems to do it daily and the logs look like a clean shutdown but we arent telling it to shutdown as we want it always running.

            2009-07-22 01:41:19,198 INFO [MeasurementManager.sender-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement collection for [8] metrics took 1ms - sending report to Server...
            2009-07-22 01:41:43,187 INFO [InventoryManager.availability-1] (rhq.core.pc.inventory.InventoryManager)- Sending availability report to Server...
            2009-07-22 01:41:56,574 INFO [Thread-7] (org.rhq.enterprise.agent.AgentShutdownHook)- {AgentShutdownHook.exit.shutting-down}Shutting down...
            2009-07-22 01:41:56,574 INFO [Thread-7] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.shutting-down}Agent is being shut down...
            2009-07-22 01:41:56,575 INFO [RHQ Primary Server Switchover Thread] (org.rhq.enterprise.agent.AgentMain)- {PrimaryServerSwitchoverThread.stopped}The primary server switchover thread has stopped.
            2009-07-22 01:41:56,576 INFO [Thread-7] (rhq.core.pc.content.ContentManager)- Shutting down Content Manager...
            2009-07-22 01:41:56,576 INFO [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementManager)- Shutting down measurement collection...
            2009-07-22 01:41:56,585 ERROR [ResourceContainer.invoker.daemon-70] (rhq.core.pc.event.EventManager)- Failed to remove poller with PollerKey[resourceId=502071, eventType=SnmpTrap] from thread pool.
            2009-07-22 01:41:56,585 WARN [ResourceContainer.invoker.daemon-70] (plugins.platform.content.yum.YumServer)- Stop ignored: not running
            2009-07-22 01:41:56,908 INFO [Thread-7] (org.rhq.enterprise.communications.ServiceContainer)- {ServiceContainer.shutting-down}Service container shutting down...
            2009-07-22 01:41:56,919 INFO [Thread-7] (org.rhq.enterprise.communications.ServiceContainer)- {ServiceContainer.shutdown}Service container shut down - no longer accepting incoming commands
            2009-07-22 01:41:56,919 INFO [Thread-7] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.shut-down}Agent has been shut down
            2009-07-22 01:41:56,927 INFO [Thread-7] (org.rhq.enterprise.agent.AgentShutdownHook)- {AgentShutdownHook.wait}The agent will wait for [0] threads to die
            2009-07-22 01:41:56,928 INFO [Thread-7] (org.rhq.enterprise.agent.AgentShutdownHook)- {AgentShutdownHook.exit.shutdown-complete}Shutdown complete - agent will now exit.


            • 3. Re: Few Issues
              mazz

              That deployment error does not look related to anything in the Jopr world - definitely open a support case on that one against the JBossAS product you have.

              As for why the agent exited, that's very weird. The agent should never "just exit" without something to cause it. It looks like the shutdown hook is getting triggered, so I suspect someone or something is sending a kill signal to the agent (e.g kill -13 {agent-pid}) or there is a cron job that is running that is killing the agent (either via kill or by "rhq-agent-wrapper stop"). The only other thing I can think of is someone scheduled a recurring operation in the Jopr UI that executes an "exit" operation that kills the agent periodically.

              To get more debug log information, execute the debug prompt command to turn on debug in the agent, or set RHQ_AGENT_DEBUG=true then start the agent. Please see this FAQ for more information on turning on debug in the agent:

              http://jopr.org/confluence/display/JOPR2/FAQ#FAQ-HowdoIgetdebugmessagesfromtheJoprAgent%3F

              • 4. Re: Few Issues

                Thanks a bunch Mazz!

                I will open a case with support on the main issue as it doesnt fail when the -Djboss.platform.mbeanserver isnt passed in the java options.

                Only 3 people have access to the JOPR UI, none of them have done anything as im the main person trying to get it working.

                No Cron jobs it exits at seemingly random times, no scheduled tasks in JOPR itself, no cron jobs. The agent is being run as root so that regular users cant kill the agent while testing.

                We log all user activity and havent found anything going on on the system itself when the agent exts. I have turned on debugging and will wait for the next failure to see what happens.

                Thanks again for your quick responses!

                • 5. Re: Few Issues
                  mazz

                  Correct me if I'm wrong (I'm not an expert here) but isn't it true that under certain high-load circumstances, some operating systems kill processes to help maintain stability? Is it possible your OS is killing the agent process in an attempt to keep the system up? How is the load on this box?

                  Anyway, just throwing that out there... the debug logs might tell you more the next time it happens.

                  • 6. Re: Few Issues

                    It exited again this morning.

                    Here is the load average according to JOPR (im assuming this would be the last time it was able to poll that before it exited):

                    System Load 0 0.27% 0.67% 0.43% 0.29%

                    The log is pretty big but here is from shutdown on:

                    2009-07-23 05:06:02,124 INFO [Thread-7] (org.rhq.enterprise.agent.AgentShutdownHook)- {AgentShutdownHook.exit.shutting-down}Shutting down...
                    2009-07-23 05:06:02,124 INFO [Thread-7] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.shutting-down}Agent is being shut down...
                    2009-07-23 05:06:02,124 DEBUG [RHQ VM Health Check Thread] (org.rhq.enterprise.agent.VMHealthCheckThread)- {VMHealthCheckThread.stopped}VM Health check thread has been stopped
                    2009-07-23 05:06:02,125 INFO [RHQ Primary Server Switchover Thread] (org.rhq.enterprise.agent.AgentMain)- {PrimaryServerSwitchoverThread.stopped}The primary server switchover thread has stopped.
                    2009-07-23 05:06:02,126 INFO [Thread-7] (rhq.core.pc.content.ContentManager)- Shutting down Content Manager...
                    2009-07-23 05:06:02,127 INFO [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementManager)- Shutting down measurement collection...
                    2009-07-23 05:06:02,127 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502086].
                    2009-07-23 05:06:02,127 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502086].
                    2009-07-23 05:06:02,128 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502088].
                    2009-07-23 05:06:02,128 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502088].
                    2009-07-23 05:06:02,128 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [503067].
                    2009-07-23 05:06:02,128 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [503067].
                    2009-07-23 05:06:02,128 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502090].
                    2009-07-23 05:06:02,128 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502090].
                    2009-07-23 05:06:02,128 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502091].
                    2009-07-23 05:06:02,129 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502091].
                    2009-07-23 05:06:02,129 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502089].
                    2009-07-23 05:06:02,129 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502089].
                    2009-07-23 05:06:02,129 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502083].
                    2009-07-23 05:06:02,129 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502083].
                    2009-07-23 05:06:02,129 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502084].
                    2009-07-23 05:06:02,129 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502084].
                    2009-07-23 05:06:02,130 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502085].
                    2009-07-23 05:06:02,130 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502085].
                    2009-07-23 05:06:02,130 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [503066].
                    2009-07-23 05:06:02,130 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [503066].
                    2009-07-23 05:06:02,130 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502087].
                    2009-07-23 05:06:02,130 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502087].
                    2009-07-23 05:06:02,131 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502082].
                    2009-07-23 05:06:02,131 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502082].
                    2009-07-23 05:06:02,131 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502092].
                    2009-07-23 05:06:02,131 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502092].
                    2009-07-23 05:06:02,131 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502080].
                    2009-07-23 05:06:02,131 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502080].
                    2009-07-23 05:06:02,132 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502094].
                    2009-07-23 05:06:02,132 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502094].
                    2009-07-23 05:06:02,132 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502098].
                    2009-07-23 05:06:02,132 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502098].
                    2009-07-23 05:06:02,133 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502551].
                    2009-07-23 05:06:02,133 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502551].
                    2009-07-23 05:06:02,133 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502550].
                    2009-07-23 05:06:02,133 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502550].
                    2009-07-23 05:06:02,133 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502552].
                    2009-07-23 05:06:02,133 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502552].
                    2009-07-23 05:06:02,133 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502555].
                    2009-07-23 05:06:02,133 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502555].
                    2009-07-23 05:06:02,134 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502553].
                    2009-07-23 05:06:02,134 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502553].
                    2009-07-23 05:06:02,134 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502554].
                    2009-07-23 05:06:02,134 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502554].
                    2009-07-23 05:06:02,134 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502556].
                    2009-07-23 05:06:02,134 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502556].
                    2009-07-23 05:06:02,135 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502096].
                    2009-07-23 05:06:02,135 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502096].
                    2009-07-23 05:06:02,135 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502097].
                    2009-07-23 05:06:02,135 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502097].
                    2009-07-23 05:06:02,135 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502099].
                    2009-07-23 05:06:02,135 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502099].
                    2009-07-23 05:06:02,135 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502095].
                    2009-07-23 05:06:02,135 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502095].
                    2009-07-23 05:06:02,135 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502077].
                    2009-07-23 05:06:02,136 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502077].
                    2009-07-23 05:06:02,136 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502078].
                    2009-07-23 05:06:02,136 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502078].
                    2009-07-23 05:06:02,136 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502079].
                    2009-07-23 05:06:02,136 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502079].
                    2009-07-23 05:06:02,137 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502073].
                    2009-07-23 05:06:02,137 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502073].
                    2009-07-23 05:06:02,137 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502572].
                    2009-07-23 05:06:02,137 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502572].
                    2009-07-23 05:06:02,137 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502571].
                    2009-07-23 05:06:02,137 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502571].
                    2009-07-23 05:06:02,138 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502574].
                    2009-07-23 05:06:02,138 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502574].
                    2009-07-23 05:06:02,138 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502566].
                    2009-07-23 05:06:02,138 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502566].
                    2009-07-23 05:06:02,138 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502586].
                    2009-07-23 05:06:02,138 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502586].
                    2009-07-23 05:06:02,138 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [503052].
                    2009-07-23 05:06:02,138 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [503052].
                    2009-07-23 05:06:02,139 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502588].
                    2009-07-23 05:06:02,139 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502588].
                    2009-07-23 05:06:02,139 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502587].
                    2009-07-23 05:06:02,139 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502587].
                    2009-07-23 05:06:02,139 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502589].
                    2009-07-23 05:06:02,139 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502589].
                    2009-07-23 05:06:02,139 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [503055].
                    2009-07-23 05:06:02,139 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [503055].
                    2009-07-23 05:06:02,140 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502590].
                    2009-07-23 05:06:02,140 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502590].
                    2009-07-23 05:06:02,140 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [503057].
                    2009-07-23 05:06:02,140 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [503057].
                    2009-07-23 05:06:02,140 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502591].
                    2009-07-23 05:06:02,140 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502591].
                    2009-07-23 05:06:02,140 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502592].
                    2009-07-23 05:06:02,140 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502592].
                    2009-07-23 05:06:02,141 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [503058].
                    2009-07-23 05:06:02,141 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [503058].
                    2009-07-23 05:06:02,141 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502594].
                    2009-07-23 05:06:02,141 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502594].
                    2009-07-23 05:06:02,141 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502593].
                    2009-07-23 05:06:02,141 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502593].
                    2009-07-23 05:06:02,141 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [503053].
                    2009-07-23 05:06:02,141 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [503053].
                    2009-07-23 05:06:02,141 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [503054].
                    2009-07-23 05:06:02,141 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [503054].
                    2009-07-23 05:06:02,142 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502595].
                    2009-07-23 05:06:02,142 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502595].
                    2009-07-23 05:06:02,142 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502596].
                    2009-07-23 05:06:02,142 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502596].
                    2009-07-23 05:06:02,142 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [503056].
                    2009-07-23 05:06:02,142 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [503056].
                    2009-07-23 05:06:02,142 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502597].
                    2009-07-23 05:06:02,142 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502597].
                    2009-07-23 05:06:02,143 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502598].
                    2009-07-23 05:06:02,143 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502598].
                    2009-07-23 05:06:02,143 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502599].
                    2009-07-23 05:06:02,143 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502599].
                    2009-07-23 05:06:02,143 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [503051].
                    2009-07-23 05:06:02,143 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [503051].
                    2009-07-23 05:06:02,143 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [503050].
                    2009-07-23 05:06:02,143 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [503050].
                    2009-07-23 05:06:02,143 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502561].
                    2009-07-23 05:06:02,143 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502561].
                    2009-07-23 05:06:02,144 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502564].
                    2009-07-23 05:06:02,144 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502564].
                    2009-07-23 05:06:02,144 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502565].
                    2009-07-23 05:06:02,144 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502565].
                    2009-07-23 05:06:02,144 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502577].
                    2009-07-23 05:06:02,144 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502577].
                    2009-07-23 05:06:02,144 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502568].
                    2009-07-23 05:06:02,144 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502568].
                    2009-07-23 05:06:02,145 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502578].
                    2009-07-23 05:06:02,145 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502578].
                    2009-07-23 05:06:02,145 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502569].
                    2009-07-23 05:06:02,145 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502569].
                    2009-07-23 05:06:02,145 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502559].
                    2009-07-23 05:06:02,145 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502559].
                    2009-07-23 05:06:02,145 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502573].
                    2009-07-23 05:06:02,145 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502573].
                    2009-07-23 05:06:02,145 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502575].
                    2009-07-23 05:06:02,145 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502575].
                    2009-07-23 05:06:02,146 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502560].
                    2009-07-23 05:06:02,146 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502560].
                    2009-07-23 05:06:02,146 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502567].
                    2009-07-23 05:06:02,146 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502567].
                    2009-07-23 05:06:02,146 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502579].
                    2009-07-23 05:06:02,146 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502579].
                    2009-07-23 05:06:02,146 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502580].
                    2009-07-23 05:06:02,146 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502580].
                    2009-07-23 05:06:02,146 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [503060].
                    2009-07-23 05:06:02,146 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [503060].
                    2009-07-23 05:06:02,147 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502563].
                    2009-07-23 05:06:02,147 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502563].
                    2009-07-23 05:06:02,147 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502582].
                    2009-07-23 05:06:02,147 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502582].
                    2009-07-23 05:06:02,147 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [503061].
                    2009-07-23 05:06:02,147 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [503061].
                    2009-07-23 05:06:02,147 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502570].
                    2009-07-23 05:06:02,147 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502570].
                    2009-07-23 05:06:02,147 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502583].
                    2009-07-23 05:06:02,148 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502583].
                    2009-07-23 05:06:02,148 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502585].
                    2009-07-23 05:06:02,148 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502585].
                    2009-07-23 05:06:02,148 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502584].
                    2009-07-23 05:06:02,148 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502584].
                    2009-07-23 05:06:02,148 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502557].
                    2009-07-23 05:06:02,148 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502557].
                    2009-07-23 05:06:02,148 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502576].
                    2009-07-23 05:06:02,148 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502576].
                    2009-07-23 05:06:02,149 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [503059].
                    2009-07-23 05:06:02,149 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [503059].
                    2009-07-23 05:06:02,149 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502562].
                    2009-07-23 05:06:02,149 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502562].
                    2009-07-23 05:06:02,149 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502558].
                    2009-07-23 05:06:02,149 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502558].
                    2009-07-23 05:06:02,149 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502581].
                    2009-07-23 05:06:02,149 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502581].
                    2009-07-23 05:06:02,150 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502074].
                    2009-07-23 05:06:02,150 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502074].
                    2009-07-23 05:06:02,150 ERROR [ResourceContainer.invoker.daemon-79] (rhq.core.pc.event.EventManager)- Failed to remove poller with PollerKey[resourceId=502071, eventType=SnmpTrap] from thread pool.
                    2009-07-23 05:06:02,151 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502071].
                    2009-07-23 05:06:02,151 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502071].
                    2009-07-23 05:06:02,151 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [503072].
                    2009-07-23 05:06:02,151 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [503072].
                    2009-07-23 05:06:02,151 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502093].
                    2009-07-23 05:06:02,151 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502093].
                    2009-07-23 05:06:02,151 WARN [ResourceContainer.invoker.daemon-79] (plugins.platform.content.yum.YumServer)- Stop ignored: not running
                    2009-07-23 05:06:02,151 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Successfully deactivated resource with id [502070].
                    2009-07-23 05:06:02,151 DEBUG [Thread-7] (rhq.core.pc.inventory.InventoryManager)- Set component state to STOPPED for resource with id [502070].
                    2009-07-23 05:06:02,258 DEBUG [Thread-7] (org.rhq.plugins.mysql.MySqlPluginLifecycleListener)- Deregistered JDBC driver: class sun.jdbc.odbc.JdbcOdbcDriver
                    2009-07-23 05:06:02,259 DEBUG [Thread-7] (org.rhq.plugins.mysql.MySqlPluginLifecycleListener)- Deregistered JDBC driver: class com.mysql.jdbc.Driver
                    2009-07-23 05:06:02,259 DEBUG [Thread-7] (org.rhq.plugins.mysql.MySqlPluginLifecycleListener)- MySqlPluginLifecycleListener completed shutdown.
                    2009-07-23 05:06:02,259 DEBUG [Thread-7] (org.rhq.plugins.oracle.OraclePluginLifecycleListener)- OraclePluginLifecycleListener completed shutdown.
                    2009-07-23 05:06:02,259 DEBUG [Thread-7] (org.rhq.plugins.postgres.PostgresPluginLifecycleListener)- PostgresPluginLifecycleListener completed shutdown.
                    2009-07-23 05:06:02,259 DEBUG [Thread-7] (org.rhq.plugins.database.DatabasePluginLifecycleListener)- DatabasePluginLifecycleListener completed shutdown.
                    2009-07-23 05:06:02,436 DEBUG [Thread-7] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.plugin-container-shutdown}The plugin container has been shutdown
                    2009-07-23 05:06:02,437 DEBUG [Thread-7] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.notifying-server-of-shutdown}Agent is notifying the server that it is shutting down.
                    2009-07-23 05:06:02,455 DEBUG [RHQ Server Polling Thread] (enterprise.communications.command.client.ServerPollingThread)- {ServerPollingThread.stopped}Server polling thread has been stopped
                    2009-07-23 05:06:02,458 DEBUG [Thread-7] (org.jboss.remoting.InvokerRegistry)- removed org.jboss.remoting.transport.http.HTTPClientInvoker@1490eb5 from registry
                    2009-07-23 05:06:02,458 DEBUG [Thread-7] (enterprise.communications.command.client.ClientCommandSender)- {ClientCommandSender.spooling}Persisting commands with guaranteed delivery to the spool file
                    2009-07-23 05:06:02,458 DEBUG [Thread-7] (enterprise.communications.command.client.ClientCommandSender)- {ClientCommandSender.spool-done}Persisted [0] commands to the spool file for later delivery to [remoting endpoint [servlet://pdxsa801.corp.unicru.com:7080/jboss-remoting-servlet-invoker/ServerInvokerServlet]]; [0] volatile commands remain in the queue and will be lost if the sender does not start again
                    2009-07-23 05:06:02,459 DEBUG [Thread-7] (enterprise.communications.command.client.ClientCommandSender)- {ClientCommandSender.no-longer-sending}Client is no longer actively sending commands to [remoting endpoint [servlet://pdxsa801.corp.unicru.com:7080/jboss-remoting-servlet-invoker/ServerInvokerServlet]]
                    2009-07-23 05:06:02,459 DEBUG [Thread-7] (enterprise.communications.command.client.ClientCommandSender)- {ClientCommandSender.notifying-state-listeners}Notifying command client sender state listeners of a state change - sender is sending=[false]
                    2009-07-23 05:06:02,459 DEBUG [Thread-7] (enterprise.communications.command.client.CommandQueue)- {CommandQueue.disabled}Command Queue is not throttling - commands will be dequeued as fast as possible
                    2009-07-23 05:06:02,459 DEBUG [Thread-7] (enterprise.communications.command.client.SendThrottle)- {SendThrottle.disabled}Send throttling is disabled - commands will be sent as fast as possible
                    2009-07-23 05:06:02,459 INFO [Thread-7] (org.rhq.enterprise.communications.ServiceContainer)- {ServiceContainer.shutting-down}Service container shutting down...
                    2009-07-23 05:06:02,459 DEBUG [Thread-7] (org.jboss.remoting.ServerInvoker)- SocketServerInvoker[10.3.39.4:16163] stopped
                    2009-07-23 05:06:02,460 DEBUG [AcceptorThread#0:16163] (jboss.remoting.transport.socket.SocketServerInvoker)- SocketServerInvoker[10.3.39.4:16163] caught exception in run()
                    java.net.SocketException: Socket closed
                     at java.net.PlainSocketImpl.socketAccept(Native Method)
                     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                     at java.net.ServerSocket.implAccept(ServerSocket.java:453)
                     at java.net.ServerSocket.accept(ServerSocket.java:421)
                     at org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:520)
                     at java.lang.Thread.run(Thread.java:619)
                    2009-07-23 05:06:02,474 DEBUG [Thread-7] (enterprise.communications.command.server.CommandServiceDirectory)- {CommandServiceDirectory.removed-command-support}Command [echo] is no longer supported; removed [rhq.remoting:type=command,subsystem=RHQ,id=1]
                    2009-07-23 05:06:02,474 DEBUG [Thread-7] (enterprise.communications.command.server.CommandServiceDirectory)- {CommandServiceDirectory.removed-command-support}Command [identify] is no longer supported; removed [rhq.remoting:type=command,subsystem=RHQ,id=2]
                    2009-07-23 05:06:02,474 DEBUG [Thread-7] (enterprise.communications.command.server.CommandServiceDirectory)- {CommandServiceDirectory.removed-command-support}Command [remotepojo] is no longer supported; removed [rhq.remoting:type=command,subsystem=RHQ,id=3]
                    2009-07-23 05:06:02,475 INFO [Thread-7] (org.rhq.enterprise.communications.ServiceContainer)- {ServiceContainer.shutdown}Service container shut down - no longer accepting incoming commands
                    2009-07-23 05:06:02,475 INFO [Thread-7] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.shut-down}Agent has been shut down
                    2009-07-23 05:06:02,484 INFO [Thread-7] (org.rhq.enterprise.agent.AgentShutdownHook)- {AgentShutdownHook.wait}The agent will wait for [0] threads to die
                    2009-07-23 05:06:02,484 INFO [Thread-7] (org.rhq.enterprise.agent.AgentShutdownHook)- {AgentShutdownHook.exit.shutdown-complete}Shutdown complete - agent will now exit.
                    



                    • 7. Re: Few Issues
                      mazz

                      What do the logs say before that.

                      2009-07-23 05:06:02,124 INFO [Thread-7] (org.rhq.enterprise.agent.AgentShutdownHook)- {AgentShutdow
                      nHook.exit.shutting-down}Shutting down...


                      What happened BEFORE the shutdown hook was invoked... that's what we need to know.

                      • 8. Re: Few Issues

                        Here is a good chunk of log from before that. Sorry for not including more. Would it be better if i uploaded the full log somewhere? I emptied out the agent.log before it was started with debug on so it only contains one run instead of multiple.

                        2009-07-23 05:00:12,636 DEBUG [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Running Availability Scan...
                        2009-07-23 05:00:12,683 DEBUG [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Built availability report for [1] resources with a size of [126] bytes in [47]ms
                        2009-07-23 05:00:12,683 INFO [InventoryManager.availability-1] (rhq.core.pc.inventory.InventoryManager)- Sending availability report to Server...
                        2009-07-23 05:00:12,683 DEBUG [InventoryManager.availability-1] (rhq.core.pc.inventory.InventoryManager)- Availability report content: AV:[pdxcfgapp802vm.corp.unicru.com][1][changesOnly]
                        2009-07-23 05:00:15,982 INFO [MeasurementManager.sender-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement collection for [8] metrics took 2ms - sending report to Server...
                        2009-07-23 05:00:45,981 DEBUG [MeasurementManager.sender-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement report contains no data - not sending to Server.
                        2009-07-23 05:00:46,010 DEBUG [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurements not collected for inactive resource component: Resource[id=503072, type=SSHD, key=sshd, name=OpenSSH sshd, parent=pdxcfgapp802vm.corp.unicru.com, version=1.0]
                        2009-07-23 05:00:46,014 INFO [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Starting configuration update check
                        2009-07-23 05:00:46,015 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502073, type=RHQ Agent, key=pdxcfgapp802vm.corp.unicru.com RHQ Agent, name=pdxcfgapp802vm.corp.unicru.com RHQ Agent, parent=pdxcfgapp802vm.corp.unicru.com, version=1.2.1]
                        2009-07-23 05:00:46,016 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502098, type=VM Class Loading System, key=java.lang:type=ClassLoading, name=Class Loading, parent=RHQ Agent JVM]
                        2009-07-23 05:00:46,016 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502096, type=VM Memory System, key=java.lang:type=Memory, name=Memory Subsystem, parent=RHQ Agent JVM]
                        2009-07-23 05:00:46,017 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502099, type=java.util.logging, key=java.util.logging:type=Logging, name=Logging, parent=RHQ Agent JVM]
                        2009-07-23 05:00:46,021 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502095, type=Threading, key=java.lang:type=Threading, name=Threading, parent=RHQ Agent JVM]
                        2009-07-23 05:00:46,021 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502079, type=Environment Setup Script, key=environment-setup-script, name=rhq-agent-env.sh, parent=pdxcfgapp802vm.corp.unicru.com RHQ Agent, version=1.2.1]
                        2009-07-23 05:00:46,022 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502586, type=JBM JMS Queue, key=jboss.messaging.destination:name=textSearchRegisterQueue,service=Queue, name=textSearchRegisterQueue JMS Queue (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:46,078 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=textSearchRegisterQueue,service=Queue] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/tmp/deploy/tmp1133445629687147088atao-services-4.2.0.45592.1.jar-contents/jboss-service.xml ].
                        2009-07-23 05:00:46,084 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=503052, type=JBM JMS Topic, key=jboss.messaging.destination:name=testDurableTopic,service=Topic, name=testDurableTopic JMS Topic (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:46,128 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=testDurableTopic,service=Topic] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/deploy/jboss-messaging.sar/destinations-service.xml ].
                        2009-07-23 05:00:46,134 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502587, type=JBM JMS Queue, key=jboss.messaging.destination:name=emailQueue,service=Queue, name=emailQueue JMS Queue (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:46,187 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=emailQueue,service=Queue] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/tmp/deploy/tmp1133445629687147088atao-services-4.2.0.45592.1.jar-contents/jboss-service.xml ].
                        2009-07-23 05:00:46,192 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502588, type=JBM JMS Queue, key=jboss.messaging.destination:name=ExpiryQueue,service=Queue, name=ExpiryQueue JMS Queue (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:46,234 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=ExpiryQueue,service=Queue] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/deploy/jboss-messaging.sar/destinations-service.xml ].
                        2009-07-23 05:00:46,239 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502589, type=JBM JMS Queue, key=jboss.messaging.destination:name=B,service=Queue, name=B JMS Queue (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:46,276 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=B,service=Queue] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/deploy/jboss-messaging.sar/destinations-service.xml ].
                        2009-07-23 05:00:46,286 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=503055, type=JBM JMS Topic, key=jboss.messaging.destination:name=testDistributedTopic,service=Topic, name=testDistributedTopic JMS Topic (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:46,322 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=testDistributedTopic,service=Topic] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/deploy/jboss-messaging.sar/destinations-service.xml ].
                        2009-07-23 05:00:46,327 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502590, type=JBM JMS Queue, key=jboss.messaging.destination:name=DLQ,service=Queue, name=DLQ JMS Queue (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:46,371 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=DLQ,service=Queue] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/deploy/jboss-messaging.sar/destinations-service.xml ].
                        2009-07-23 05:00:46,377 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502591, type=JBM JMS Queue, key=jboss.messaging.destination:name=D,service=Queue, name=D JMS Queue (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:46,416 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=D,service=Queue] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/deploy/jboss-messaging.sar/destinations-service.xml ].
                        2009-07-23 05:00:46,422 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=503057, type=JBM JMS Topic, key=jboss.messaging.destination:name=TopicWithOwnRedeliveryDelay,service=Topic, name=TopicWithOwnRedeliveryDelay JMS Topic (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:46,467 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=TopicWithOwnRedeliveryDelay,service=Topic] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/deploy/jboss-messaging.sar/destinations-service.xml ].
                        2009-07-23 05:00:46,473 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502592, type=JBM JMS Queue, key=jboss.messaging.destination:name=ex,service=Queue, name=ex JMS Queue (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:46,509 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=ex,service=Queue] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/deploy/jboss-messaging.sar/destinations-service.xml ].
                        2009-07-23 05:00:46,514 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=503058, type=JBM JMS Topic, key=jboss.messaging.destination:name=assetCatalogUpdateTopic,service=Topic, name=assetCatalogUpdateTopic JMS Topic (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:46,560 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=assetCatalogUpdateTopic,service=Topic] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/tmp/deploy/tmp1133445629687147088atao-services-4.2.0.45592.1.jar-contents/jboss-service.xml ].
                        2009-07-23 05:00:46,566 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502593, type=JBM JMS Queue, key=jboss.messaging.destination:name=QueueWithOwnRedeliveryDelay,service=Queue, name=QueueWithOwnRedeliveryDelay JMS Queue (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:46,605 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=QueueWithOwnRedeliveryDelay,service=Queue] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/deploy/jboss-messaging.sar/destinations-service.xml ].
                        2009-07-23 05:00:46,611 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502594, type=JBM JMS Queue, key=jboss.messaging.destination:name=testQueue,service=Queue, name=testQueue JMS Queue (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:46,650 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=testQueue,service=Queue] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/deploy/jboss-messaging.sar/destinations-service.xml ].
                        2009-07-23 05:00:46,656 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=503053, type=JBM JMS Topic, key=jboss.messaging.destination:name=testTopic,service=Topic, name=testTopic JMS Topic (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:46,691 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=testTopic,service=Topic] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/deploy/jboss-messaging.sar/destinations-service.xml ].
                        2009-07-23 05:00:46,697 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=503054, type=JBM JMS Topic, key=jboss.messaging.destination:name=TopicWithOwnDLQAndExpiryQueue,service=Topic, name=TopicWithOwnDLQAndExpiryQueue JMS Topic (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:46,735 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=TopicWithOwnDLQAndExpiryQueue,service=Topic] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/deploy/jboss-messaging.sar/destinations-service.xml ].
                        2009-07-23 05:00:46,741 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502595, type=JBM JMS Queue, key=jboss.messaging.destination:name=C,service=Queue, name=C JMS Queue (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:46,780 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=C,service=Queue] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/deploy/jboss-messaging.sar/destinations-service.xml ].
                        2009-07-23 05:00:46,787 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502596, type=JBM JMS Queue, key=jboss.messaging.destination:name=QueueWithOwnDLQAndExpiryQueue,service=Queue, name=QueueWithOwnDLQAndExpiryQueue JMS Queue (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:46,824 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=QueueWithOwnDLQAndExpiryQueue,service=Queue] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/deploy/jboss-messaging.sar/destinations-service.xml ].
                        2009-07-23 05:00:46,829 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502597, type=JBM JMS Queue, key=jboss.messaging.destination:name=JAFQueue,service=Queue, name=JAFQueue JMS Queue (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:46,872 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=JAFQueue,service=Queue] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/tmp/deploy/tmp1133445629687147088atao-services-4.2.0.45592.1.jar-contents/jboss-service.xml ].
                        2009-07-23 05:00:46,877 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=503056, type=JBM JMS Topic, key=jboss.messaging.destination:name=securedTopic,service=Topic, name=securedTopic JMS Topic (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:46,912 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=securedTopic,service=Topic] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/deploy/jboss-messaging.sar/destinations-service.xml ].
                        2009-07-23 05:00:46,918 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502598, type=JBM JMS Queue, key=jboss.messaging.destination:name=PrivateDLQ,service=Queue, name=PrivateDLQ JMS Queue (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:46,954 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=PrivateDLQ,service=Queue] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/deploy/jboss-messaging.sar/destinations-service.xml ].
                        2009-07-23 05:00:46,960 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502599, type=JBM JMS Queue, key=jboss.messaging.destination:name=A,service=Queue, name=A JMS Queue (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:46,998 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=A,service=Queue] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/deploy/jboss-messaging.sar/destinations-service.xml ].
                        2009-07-23 05:00:47,004 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=503050, type=JBM JMS Queue, key=jboss.messaging.destination:name=PrivateExpiryQueue,service=Queue, name=PrivateExpiryQueue JMS Queue (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:47,045 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=PrivateExpiryQueue,service=Queue] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/deploy/jboss-messaging.sar/destinations-service.xml ].
                        2009-07-23 05:00:47,051 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=503051, type=JBM JMS Queue, key=jboss.messaging.destination:name=testDistributedQueue,service=Queue, name=testDistributedQueue JMS Queue (JBM), parent=JBoss Messaging]
                        2009-07-23 05:00:47,085 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.messaging.destination:name=testDistributedQueue,service=Queue] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/deploy/jboss-messaging.sar/destinations-service.xml ].
                        2009-07-23 05:00:47,090 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502559, type=Datasource, key=jboss.jca:name=ataoDS,service=DataSourceBinding, name=ataoDS Datasource, parent=pdxcfgapp802vm.corp.unicru.com JBossEAP 4.3.0.GA production (0.0.0.0:1099)]
                        2009-07-23 05:00:47,118 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.jca:name=ataoDS,service=DataSourceBinding] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/deploy/atao-ds.xml ].
                        2009-07-23 05:00:47,120 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502560, type=ConnectionFactory, key=jboss.jca:name=JmsXA,service=ConnectionFactoryBinding, name=JmsXA Connection Factory, parent=pdxcfgapp802vm.corp.unicru.com JBossEAP 4.3.0.GA production (0.0.0.0:1099)]
                        2009-07-23 05:00:47,155 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.jca:name=JmsXA,service=ConnectionFactoryBinding] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/deploy/hajndi-jms-ds.xml ].
                        2009-07-23 05:00:47,158 DEBUG [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Checking for updated resource configuration on: Resource[id=502558, type=Datasource, key=jboss.jca:name=DefaultDS,service=DataSourceBinding, name=DefaultDS Datasource, parent=pdxcfgapp802vm.corp.unicru.com JBossEAP 4.3.0.GA production (0.0.0.0:1099)]
                        2009-07-23 05:00:47,193 DEBUG [ResourceContainer.invoker.daemon-79] (rhq.plugins.jbossas.util.DeploymentUtility)- Descriptor file for [jboss.jca:name=DefaultDS,service=DataSourceBinding] is [/vol/app/atao/atao2/jboss405/jboss-as/server/production/deploy/hsqldb-ds.xml ].
                        2009-07-23 05:00:47,196 INFO [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Configuration update check completed in 1181ms
                        2009-07-23 05:01:12,695 DEBUG [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Running Availability Scan...
                        2009-07-23 05:01:12,733 DEBUG [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Built availability report for [1] resources with a size of [126] bytes in [38]ms
                        2009-07-23 05:01:12,733 INFO [InventoryManager.availability-1] (rhq.core.pc.inventory.InventoryManager)- Sending availability report to Server...
                        2009-07-23 05:01:12,733 DEBUG [InventoryManager.availability-1] (rhq.core.pc.inventory.InventoryManager)- Availability report content: AV:[pdxcfgapp802vm.corp.unicru.com][1][changesOnly]
                        2009-07-23 05:01:15,982 INFO [MeasurementManager.sender-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement collection for [17] metrics took 12ms - sending report to Server...
                        2009-07-23 05:01:45,981 DEBUG [MeasurementManager.sender-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement report contains no data - not sending to Server.
                        2009-07-23 05:02:12,746 DEBUG [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Running Availability Scan...
                        2009-07-23 05:02:12,787 DEBUG [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Built availability report for [1] resources with a size of [126] bytes in [41]ms
                        2009-07-23 05:02:12,787 INFO [InventoryManager.availability-1] (rhq.core.pc.inventory.InventoryManager)- Sending availability report to Server...
                        2009-07-23 05:02:12,787 DEBUG [InventoryManager.availability-1] (rhq.core.pc.inventory.InventoryManager)- Availability report content: AV:[pdxcfgapp802vm.corp.unicru.com][1][changesOnly]
                        2009-07-23 05:02:15,983 INFO [MeasurementManager.sender-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement collection for [8] metrics took 2ms - sending report to Server...
                        2009-07-23 05:02:45,982 DEBUG [MeasurementManager.sender-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement report contains no data - not sending to Server.
                        2009-07-23 05:03:12,799 DEBUG [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Running Availability Scan...
                        2009-07-23 05:03:12,838 DEBUG [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Built availability report for [1] resources with a size of [126] bytes in [37]ms
                        2009-07-23 05:03:12,838 INFO [InventoryManager.availability-1] (rhq.core.pc.inventory.InventoryManager)- Sending availability report to Server...
                        2009-07-23 05:03:12,838 DEBUG [InventoryManager.availability-1] (rhq.core.pc.inventory.InventoryManager)- Availability report content: AV:[pdxcfgapp802vm.corp.unicru.com][1][changesOnly]
                        2009-07-23 05:03:15,983 INFO [MeasurementManager.sender-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement collection for [5] metrics took 1ms - sending report to Server...
                        2009-07-23 05:03:45,982 DEBUG [MeasurementManager.sender-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement report contains no data - not sending to Server.
                        2009-07-23 05:04:12,850 DEBUG [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Running Availability Scan...
                        2009-07-23 05:04:12,899 DEBUG [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Built availability report for [1] resources with a size of [126] bytes in [49]ms
                        2009-07-23 05:04:12,899 INFO [InventoryManager.availability-1] (rhq.core.pc.inventory.InventoryManager)- Sending availability report to Server...
                        2009-07-23 05:04:12,899 DEBUG [InventoryManager.availability-1] (rhq.core.pc.inventory.InventoryManager)- Availability report content: AV:[pdxcfgapp802vm.corp.unicru.com][1][changesOnly]
                        2009-07-23 05:04:15,981 INFO [MeasurementManager.sender-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement collection for [8] metrics took 1ms - sending report to Server...
                        2009-07-23 05:04:45,982 DEBUG [MeasurementManager.sender-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement report contains no data - not sending to Server.
                        2009-07-23 05:05:12,911 DEBUG [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Running Availability Scan...
                        2009-07-23 05:05:12,949 DEBUG [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Built availability report for [1] resources with a size of [126] bytes in [38]ms
                        2009-07-23 05:05:12,949 INFO [InventoryManager.availability-1] (rhq.core.pc.inventory.InventoryManager)- Sending availability report to Server...
                        2009-07-23 05:05:12,949 DEBUG [InventoryManager.availability-1] (rhq.core.pc.inventory.InventoryManager)- Availability report content: AV:[pdxcfgapp802vm.corp.unicru.com][1][changesOnly]
                        2009-07-23 05:05:15,983 INFO [MeasurementManager.sender-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement collection for [5] metrics took 1ms - sending report to Server...
                        2009-07-23 05:05:45,980 DEBUG [MeasurementManager.sender-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement report contains no data - not sending to Server.
                        2009-07-23 05:05:46,323 DEBUG [ResourceContainer.invoker.daemon-79] (org.rhq.plugins.jbossas.JBossASTomcatConnectorComponent)- Finding metrics for: jboss.web:name=ajp-0.0.0.0-8009,type=ThreadPool:currentThreadsBusy
                        2009-07-23 05:05:46,325 DEBUG [ResourceContainer.invoker.daemon-79] (org.rhq.plugins.jbossas.JBossASTomcatConnectorComponent)- Finding metrics for: jboss.web:name=ajp-0.0.0.0-8009,type=ThreadPool:currentThreadCount
                        2009-07-23 05:05:46,326 DEBUG [ResourceContainer.invoker.daemon-79] (org.rhq.plugins.jbossas.JBossASTomcatConnectorComponent)- Finding metrics for: jboss.web:name=http-0.0.0.0-8080,type=ThreadPool:currentThreadCount
                        2009-07-23 05:05:46,328 DEBUG [ResourceContainer.invoker.daemon-79] (org.rhq.plugins.jbossas.JBossASTomcatConnectorComponent)- Finding metrics for: jboss.web:name=http-0.0.0.0-8080,type=ThreadPool:currentThreadsBusy
                        2009-07-23 05:05:46,462 DEBUG [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurements not collected for inactive resource component: Resource[id=503072, type=SSHD, key=sshd, name=OpenSSH sshd, parent=pdxcfgapp802vm.corp.unicru.com, version=1.0]
                        


                        • 9. Re: Few Issues
                          mazz

                          Doesn't look like anything happened. The agent is running along fine and BAM! someone triggered its shutdown hook. It has to be an external influence here, nothing in the logs looks like the agent is in a bad state.

                          • 10. Re: Few Issues

                            Ya after looking at those logs i posted i thought the same thing. I even checked the server and there is nothing in its logs around the time the agent goes down. I double checked everything and no scheduled tasks/no crons noone even logged into the box around the time the agent went down.

                            Ive got some more digging to do on my end. I suspect someone is as they are scared of what jopr/jon will tell us about our application.

                            Thanks for all your quick responses.

                            • 11. Re: Few Issues

                              I did some testing and someone would have had to issue the command from the JOPR UI as killing the process makes the log just stop not shutdown cleanly.

                              I noticed that there is what appears to be an access log for the server located at: jopr-server-2.2.1/jbossas/server/default/log/rt/_ROOT_rt.log

                              Is there any way to make that more apache style logging or get a date/time there or is the long number after the page the timestamp that has to be converted? Im thinking someone is doing it from the console and removing their traces by deleting the operations history item afterward.

                              • 12. Re: Few Issues
                                mazz

                                FYI: Check your agent logs for anything related to the "VM Health Check Thread". I didn't see anything related to that in your logs that you attached, so I don't think this is involved, but specifically search for "Health" in your agent logs just in case. If this was the cause, it prints out messages to stdout and the log, so I would think you would have noticed it if this was the cause. See http://jopr.org/confluence/display/RHQ/Design-AgentVMHealthCheck for more info.

                                • 13. Re: Few Issues
                                  mazz

                                  that rt log is used by the response time subsystem in the jboss plugin. The timestamp is epoch millis - convert to local time using normal epoch millis-to-date conversion.

                                  • 14. Re: Few Issues

                                     

                                    "mazz" wrote:
                                    FYI: Check your agent logs for anything related to the "VM Health Check Thread". I didn't see anything related to that in your logs that you attached, so I don't think this is involved, but specifically search for "Health" in your agent logs just in case. If this was the cause, it prints out messages to stdout and the log, so I would think you would have noticed it if this was the cause. See http://jopr.org/confluence/display/RHQ/Design-AgentVMHealthCheck for more info.


                                    Here are all the entries that have the word Health.. As well as something that looks odd right after it started.

                                    2009-07-22 22:55:46,024 DEBUG [RHQ VM Health Check Thread] (org.rhq.enterprise.agent.VMHealthCheckThread)- {VMHealthCheckThread.started}VM health check thread has started and will check every [5,000] milliseconds
                                    
                                    2009-07-22 22:56:15,995 DEBUG [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement collection is falling behind... Missed requested time by [30012ms]
                                    2009-07-22 22:56:15,996 DEBUG [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement collection is falling behind... Missed requested time by [30012ms]
                                    2009-07-22 22:56:15,996 DEBUG [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement collection is falling behind... Missed requested time by [30011ms]
                                    2009-07-22 22:56:15,996 DEBUG [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement collection is falling behind... Missed requested time by [30010ms]
                                    2009-07-22 22:56:15,996 DEBUG [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement collection is falling behind... Missed requested time by [30009ms]
                                    2009-07-22 22:56:15,996 DEBUG [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement collection is falling behind... Missed requested time by [30007ms]
                                    2009-07-22 22:56:15,997 DEBUG [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement collection is falling behind... Missed requested time by [30006ms]
                                    2009-07-22 22:56:15,997 DEBUG [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement collection is falling behind... Missed requested time by [30003ms]
                                    2009-07-22 22:56:15,997 DEBUG [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement collection is falling behind... Missed requested time by [30003ms]
                                    2009-07-22 22:56:15,997 DEBUG [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement collection is falling behind... Missed requested time by [30003ms]
                                    2009-07-22 22:56:15,997 DEBUG [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement collection is falling behind... Missed requested time by [30003ms]
                                    2009-07-22 22:56:15,997 DEBUG [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement collection is falling behind... Missed requested time by [30003ms]
                                    
                                    2009-07-23 05:06:02,124 DEBUG [RHQ VM Health Check Thread] (org.rhq.enterprise.agent.VMHealthCheckThread)- {VMHealthCheckThread.stopped}VM Health check thread has been stopped
                                    




                                    1 2 Previous Next