Agent doesn't send reports to server
kbielecki May 9, 2012 5:51 AMAgent was running for a long time without problem. Then suddenly all resources discovered by agent went down.
I've tried to restart agent, RHQ server, Jboss, install agent one more time, uninventory resources but reports are not send to server.
Strange is that I have values in Platform Utilization but I can't connect to Jboss or Apache. Remaining agents are running without any problems.
RHQ 4.3
OS: Solaris 10
2012-05-09 11:11:39,132 WARN [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Failed to synchronize local inventory with Server inventory for Resource [10129] and its descendants: null 2012-05-09 11:11:39,132 WARN [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Exception caught while running server discovery java.lang.RuntimeException: java.util.ConcurrentModificationException at org.rhq.core.pc.inventory.InventoryManager.synchInventory(InventoryManager.java:1072) at org.rhq.core.pc.inventory.InventoryManager.handleReport(InventoryManager.java:1006) at org.rhq.core.pc.inventory.AutoDiscoveryExecutor.call(AutoDiscoveryExecutor.java:121) at org.rhq.core.pc.inventory.AutoDiscoveryExecutor.run(AutoDiscoveryExecutor.java:92) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793) at java.util.HashMap$KeyIterator.next(HashMap.java:828) at org.rhq.core.pc.inventory.InventoryManager.mergeResource(InventoryManager.java:2708) at org.rhq.core.pc.inventory.InventoryManager.mergeUnknownResources(InventoryManager.java:2610) at org.rhq.core.pc.inventory.InventoryManager.synchInventory(InventoryManager.java:1041) ... 12 more 2012-05-09 11:11:39,139 INFO [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Found 1 servers. 2012-05-09 11:11:39,139 INFO [InventoryManager.discovery-1] (rhq.core.pc.inventory.RuntimeDiscoveryExecutor)- Running runtime discovery scan rooted at [platform] 2012-05-09 11:12:00,811 INFO [ResourceDiscoveryComponent.invoker.daemon-2] (org.rhq.plugins.apache.ApacheServerDiscoveryComponent)- SNMPConf directive not found. Skipping SNMP configuration. 2012-05-09 11:12:01,357 ERROR [InventoryManager.discovery-1] (rhq.core.pc.inventory.RuntimeDiscoveryExecutor)- Error in runtime discovery java.lang.NullPointerException at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.discoverForResource(RuntimeDiscoveryExecutor.java:239) at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.discoverForResource(RuntimeDiscoveryExecutor.java:277) at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.discoverForResource(RuntimeDiscoveryExecutor.java:277) at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.discoverForResourceRecursive(RuntimeDiscoveryExecutor.java:158) at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.runtimeDiscover(RuntimeDiscoveryExecutor.java:140) at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.call(RuntimeDiscoveryExecutor.java:99) at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.run(RuntimeDiscoveryExecutor.java:88) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) 2012-05-09 11:12:01,368 ERROR [InventoryManager.discovery-1] (rhq.core.pc.inventory.RuntimeDiscoveryExecutor)- Error in runtime discovery java.lang.NullPointerException at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.discoverForResource(RuntimeDiscoveryExecutor.java:239) at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.discoverForResource(RuntimeDiscoveryExecutor.java:277) at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.discoverForResource(RuntimeDiscoveryExecutor.java:277) at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.discoverForResourceRecursive(RuntimeDiscoveryExecutor.java:158) at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.runtimeDiscover(RuntimeDiscoveryExecutor.java:140) at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.call(RuntimeDiscoveryExecutor.java:99) at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.run(RuntimeDiscoveryExecutor.java:88) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) 2012-05-09 11:12:01,378 ERROR [InventoryManager.discovery-1] (rhq.core.pc.inventory.RuntimeDiscoveryExecutor)- Error in runtime discovery or: 2012-05-09 11:27:12,224 INFO [InventoryManager.discovery-1] (core.pc.drift.sync.DriftSyncManager)- Finished server sync for drift definitions. 0 added and 0 deleted in 30 ms 2012-05-09 11:27:12,224 INFO [InventoryManager.discovery-1] (core.pc.drift.sync.DriftSyncManager)- Starting drift content sync... 2012-05-09 11:27:12,224 INFO [InventoryManager.discovery-1] (core.pc.drift.sync.DriftSyncManager)- Finished drift content sync in 0 ms 2012-05-09 11:27:12,292 INFO [InventoryManager.discovery-1] (core.pc.drift.sync.DriftSyncManager)- Starting server sync for drift definitions... 2012-05-09 11:27:12,323 INFO [InventoryManager.discovery-1] (core.pc.drift.sync.DriftSyncManager)- Finished server sync for drift definitions. 0 added and 0 deleted in 31 ms 2012-05-09 11:27:12,323 INFO [InventoryManager.discovery-1] (core.pc.drift.sync.DriftSyncManager)- Starting drift content sync... 2012-05-09 11:27:12,323 INFO [InventoryManager.discovery-1] (core.pc.drift.sync.DriftSyncManager)- Finished drift content sync in 0 ms 2012-05-09 11:27:12,688 INFO [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Found 1 servers. 2012-05-09 11:33:31,558 WARN [Content.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Cannot get a resource container for an invalid resource ID=0 2012-05-09 11:33:31,559 WARN [Content.discovery-1] (rhq.core.pc.content.ContentDiscoveryRunner)- Exception received from component while attempting content retrieval org.rhq.core.clientapi.agent.PluginContainerException: Resource component container could not be retrieved for resource: 0 at org.rhq.core.pc.util.ComponentUtil.getComponent(ComponentUtil.java:102) at org.rhq.core.pc.content.ContentManager.performContentDiscovery(ContentManager.java:418) at org.rhq.core.pc.content.ContentDiscoveryRunner.call(ContentDiscoveryRunner.java:123) at org.rhq.core.pc.content.ContentDiscoveryRunner.run(ContentDiscoveryRunner.java:90) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) 2012-05-09 11:33:31,560 WARN [Content.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Cannot get a resource container for an invalid resource ID=0
I don't have any idea what I can check?