4 Replies Latest reply on Apr 24, 2014 8:30 PM by genman

    Inventory sync issue when plugins lose resource types

    genman
    genman Apr 23, 2014 8:20 PM

      In RHQ 4.9, I observed the following issue:

      • Deploy a custom plugin
      • The agent creates resources of a type
      • Remove the type from the plugin descriptor
      • The resource still exists in inventory
      • The agent says INFO  [InventoryManager.discovery-1] (InventoryManager)- Got unknown resource: 965889
      • Then, the agent will attempt to sync. The server receives lots of full availability reports, over and over again.

      23:59:56,855 INFO  [org.rhq.enterprise.server.util.concurrent.AvailabilityReportSerializer] (http-/0.0.0.0:7080-192) tid=534; agent=xxx: releasing write lock after being locked for millis=14851

      • Over enough agents, the server uses a lot of CPU and crash.

       

      The solution appears to be to remove the unknown resources.


      Is this still a potential bug in 4.10 or master? Is there some sort of process that might have failed on the server side to remove these resources?


      Unfortunately I don't have detailed logs.

        • 1. Re: Inventory sync issue when plugins lose resource types
          mazz
          mazz Apr 23, 2014 8:44 PM (in response to genman)

          This sounds very familiar and something I think we fixed. I can't say for sure (perhaps search Bugzilla) but I'm more than 50% sure this has been fixed. Though, if it is what I'm thinking of, it was a while ago when we fixed it.

            • Actions
          • 2. Re: Inventory sync issue when plugins lose resource types
            mazz
            mazz Apr 23, 2014 8:46 PM (in response to genman)

            Maybe not so long ago afterall? I thought there was something else we did in this area a while ago, but I found this that was fixed: https://bugzilla.redhat.com/show_bug.cgi?id=1073201

              • Actions
            • 3. Re: Inventory sync issue when plugins lose resource types
              jayshaughnessy
              jayshaughnessy Apr 24, 2014 9:17 AM (in response to genman)

              I'm not sure this is the same thing as the fix for 1073201, which had to do with Ignored Types, I think.  (although those changes still may affect this behavior in 4.10).  But when you remove a type from plugin and redeploy that plugin I think all of the associated resources of that type are supposed to get removed. The type doesn't even exist anymore, so how can they remain?  Something seems off.

              1 of 1 people found this helpful
                • Actions
              • 4. Re: Inventory sync issue when plugins lose resource types
                genman
                genman Apr 24, 2014 8:30 PM (in response to jayshaughnessy)

                Yes, I agree. Ordinarily the removed resource type would get removed. Unfortunately I haven't figured out what might have caused this. It happened back in April 5 when I wasn't doing a deployment. The resource type still exists in the database, but not in the descriptor, so I suspect the delete timed out or failed.

                 

                11062UserInfoSERVICECONFIGURATIONNEITHER10(null)xxx137702106780313770210678030(null)(null)(null)(null)0

                 

                We did have a large number of resources of that type (>200), so my suspicion is the deletion/inventory removal somehow timed out. I suppose I should look at the code to see where it might have happened.

                  • Actions