Inventory sync issue when plugins lose resource types

[genman]

In RHQ 4.9, I observed the following issue:

• Deploy a custom plugin
• The agent creates resources of a type
• Remove the type from the plugin descriptor
• The resource still exists in inventory
• The agent says INFO  [InventoryManager.discovery-1] (InventoryManager)- Got unknown resource: 965889
• Then, the agent will attempt to sync. The server receives lots of full availability reports, over and over again.

23:59:56,855 INFO  [org.rhq.enterprise.server.util.concurrent.AvailabilityReportSerializer] (http-/0.0.0.0:7080-192) tid=534; agent=xxx: releasing write lock after being locked for millis=14851

• Over enough agents, the server uses a lot of CPU and crash.

The solution appears to be to remove the unknown resources.

Is this still a potential bug in 4.10 or master? Is there some sort of process that might have failed on the server side to remove these resources?

Unfortunately I don't have detailed logs.

[mazz]

This sounds very familiar and something I think we fixed. I can't say for sure (perhaps search Bugzilla) but I'm more than 50% sure this has been fixed. Though, if it is what I'm thinking of, it was a while ago when we fixed it.

[mazz]

Maybe not so long ago afterall? I thought there was something else we did in this area a while ago, but I found this that was fixed: https://bugzilla.redhat.com/show_bug.cgi?id=1073201

[jayshaughnessy]

I'm not sure this is the same thing as the fix for 1073201, which had to do with Ignored Types, I think.  (although those changes still may affect this behavior in 4.10).  But when you remove a type from plugin and redeploy that plugin I think all of the associated resources of that type are supposed to get removed. The type doesn't even exist anymore, so how can they remain?  Something seems off.

[genman]

Yes, I agree. Ordinarily the removed resource type would get removed. Unfortunately I haven't figured out what might have caused this. It happened back in April 5 when I wasn't doing a deployment. The resource type still exists in the database, but not in the descriptor, so I suspect the delete timed out or failed.

11062

UserInfo

SERVICE

CONFIGURATION

NEITHER

1

0

(null)

xxx

1377021067803

1377021067803

0

(null)

(null)

(null)

(null)

0

We did have a large number of resources of that type (>200), so my suspicion is the deletion/inventory removal somehow timed out. I suppose I should look at the code to see where it might have happened.