3 Replies Latest reply on Aug 6, 2014 6:48 AM by Stian Lund

    Upgrade 4.6 to 4.12 fails: Database Plugin versions are not updated.

    Stian Lund Expert

      Hi,

      after fixing the other bug I think this one is a harder nut to crack.

       

      Upgrade using 'rhqctl upgrade' exits with no errors using the command:

      ./rhqctl upgrade --from-agent-dir /opt/rhq/rhq-agent --from-server-dir /opt/rhq/rhq-server --storage-data-root-dir /opt/rhq/rhq-data
      

       

      The server starts OK but all the agents fail in their download of the new plugins from the server.

      From agent.log:

       

      2014-08-05 11:33:43,637 INFO  [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.downloading}Downloading the plugin [rhq-script-plugin-4.6.0.jar]...
      2014-08-05 11:33:43,702 ERROR [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.download-failure}Failed to download the plugin [Script]. Cause: java.lang.ClassNotFoundException: [Warning] org.rhq.enterprise.server.xmlschema.generated.serverplugin.perspective.PerspectivePluginDescriptorType from [Module "deployment.rhq.ear.rhq-server.jar:main" from Service Module Loader]
      2014-08-05 11:33:43,702 WARN  [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.download-failure-will-retry}Failed to download the plugin [rhq-script-plugin-4.6.0.jar]. This was attempt #[1]. Will retry again in [56,746]ms. Cause: [org.rhq.core.util.exception.WrappedRemotingException:org.rhq.enterprise.server.xmlschema.generated.serverplugin.perspective.PerspectivePluginDescriptorType from [Module "deployment.rhq.ear.rhq-server.jar:main" from Service Module Loader]]
      2014-08-05 11:34:40,515 ERROR [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.download-failure}Failed to download the plugin [Script]. Cause: java.lang.ClassNotFoundException: [Warning] org.rhq.enterprise.server.xmlschema.generated.serverplugin.perspective.PerspectivePluginDescriptorType from [Module "deployment.rhq.ear.rhq-server.jar:main" from Service Module Loader]
      2014-08-05 11:34:40,516 WARN  [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.download-failure-will-retry}Failed to download the plugin [rhq-script-plugin-4.6.0.jar]. This was attempt #[2]. Will retry again in [52,890]ms. Cause: [org.rhq.core.util.exception.WrappedRemotingException:org.rhq.enterprise.server.xmlschema.generated.serverplugin.perspective.PerspectivePluginDescriptorType from [Module "deployment.rhq.ear.rhq-server.jar:main" from Service Module Loader]]
      2014-08-05 11:35:33,455 ERROR [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.download-failure}Failed to download the plugin [Script]. Cause: java.lang.ClassNotFoundException: [Warning] org.rhq.enterprise.server.xmlschema.generated.serverplugin.perspective.PerspectivePluginDescriptorType from [Module "deployment.rhq.ear.rhq-server.jar:main" from Service Module Loader]
      2014-08-05 11:35:33,456 WARN  [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.download-failure-will-not-retry}Failed to download the plugin [rhq-script-plugin-4.6.0.jar]. This was attempt #[3] - will no longer retry. This plugin will not be deployed in the agent. Cause: org.rhq.core.util.exception.WrappedRemotingException:org.rhq.enterprise.server.xmlschema.generated.serverplugin.perspective.PerspectivePluginDescriptorType from [Module "deployment.rhq.ear.rhq-server.jar:main" from Service Module Loader]
      2014-08-05 11:35:33,456 INFO  [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.downloading}Downloading the plugin [rhq-platform-plugin-4.6.0.jar]...
      2014-08-05 11:35:33,505 ERROR [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.download-failure}Failed to download the plugin [Platforms]. Cause: java.lang.ClassNotFoundException: [Warning] org.rhq.enterprise.server.xmlschema.generated.serverplugin.perspective.PerspectivePluginDescriptorType from [Module "deployment.rhq.ear.rhq-server.jar:main" from Service Module Loader]
      2014-08-05 11:35:33,506 WARN  [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.download-failure-will-retry}Failed to download the plugin [rhq-platform-plugin-4.6.0.jar]. This was attempt #[1]. Will retry again in [59,442]ms. Cause: [org.rhq.core.util.exception.WrappedRemotingException:org.rhq.enterprise.server.xmlschema.generated.serverplugin.perspective.PerspectivePluginDescriptorType from [Module "deployment.rhq.ear.rhq-server.jar:main" from Service Module Loader]]
      
      

       

      From server.log:

       

      12:53:28,434 INFO  [org.rhq.enterprise.server.core.plugin.ServerPluginScanner] (http-/0.0.0.0:7080-34) Filesystem has a server plugin [alert-email] at the file [/opt/rhq/rhq-server-4.12.0/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-serverplugins/alert-email-4.12.0.jar] which is different than where the DB thinks it should be [/opt/rhq/rhq-server-4.12.0/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-serverplugins/alert-email-4.6.0.jar]
      12:53:28,434 INFO  [org.rhq.enterprise.server.core.plugin.ServerPluginScanner] (http-/0.0.0.0:7080-34) It appears that the server plugin [ServerPlugin [id=0, name=alert-email, md5=53f2f5f6ee0e283627471cad049e6fef]] in the database may be obsolete. If so, it will be updated later.
      12:53:28,434 INFO  [org.rhq.enterprise.server.core.plugin.ServerPluginScanner] (http-/0.0.0.0:7080-34) Filesystem has a server plugin [DiskContentSourcePlugin] at the file [/opt/rhq/rhq-server-4.12.0/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-serverplugins/rhq-serverplugin-disk-4.12.0.jar] which is different than where the DB thinks it should be [/opt/rhq/rhq-server-4.12.0/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-serverplugins/rhq-serverplugin-disk-4.6.0.jar]
      12:53:28,434 INFO  [org.rhq.enterprise.server.core.plugin.ServerPluginScanner] (http-/0.0.0.0:7080-34) It appears that the server plugin [ServerPlugin [id=0, name=DiskContentSourcePlugin, md5=7c60222ffb7397b431e1c50cdab214fb]] in the database may be obsolete. If so, it will be updated later.
      
      

       

      The database (Oracle 11) contains just the old versions:

       

      
      
      
      snmptrapd                  
      Receiver for SNMP Traps      
      4.6.0     
       
      
      
      jboss-as-7                 
      JBoss Application Server 7.x 
      4.4.0-SNAPSHOT 
      
      
      hudson                     
      Hudson                       
      4.6.0     
       
      
      
      Twitter                    
      Twitter Plugin               
      4.6.0     
       
      
      
      Script                     
      Script                       
      4.6.0     
       
      
      
      Platforms                  
      Operating System Platforms   
      4.6.0     
       
      
      
      PerfTest                   
      Performance Test Plugin      
      4.6.0     
       
      
      
      NetworkServices            
      Network Services             
      4.6.0     
       
      
      
      JMX                        
      Generic JMX                  
      4.6.0     
       
      
      
      Tomcat                     
      Tomcat Server                
      4.6.0     
       
      
      
      RHQAgent                   
      RHQ Agent                    
      4.6.0     
       
      
      
      JBossAS5                   
      JBoss Application Server 5.x/6.x 4.6.0     
       
      
      
      JBossCache3                
      JBossCache 3.x Services      
      4.6.0     
       
      
      
      JBossAS                    
      JBoss Application Server 3.x/4.x 4.6.0     
       
      
      
      
      
      
      

       

      And so on... not sure what the 4.4 snapshot is doing there though, it's been deleted.

       

      Any ideas how to fix without manually having to change every row in the RHQ_PLUGINS table, which I don't think is even possible... it contains md5 sums for one.

       

      Stian

        • 1. Re: Upgrade 4.6 to 4.12 fails: Database Plugin versions are not updated.
          Stian Lund Expert

          The log file grew very large after upgrade because it gets constantly filled with agent upgrade failures, but I've done a clean restart to see what errors pop up. THere are definitely some errors and nullpointers that look related to plugins.

           

          14:23:37,205 ERROR [org.rhq.enterprise.server.core.plugin.PluginDeploymentScanner] (pool-6-thread-1) Scan failed. Cause: java.lang.ClassNotFoundException:org.rhq.enterprise.server.xmlschema.generated.serverplugin.perspective.PerspectivePluginDescriptorType from [Module "deployment.rhq.ear.rhq-server.jar:main" from Service Module Loader]
          ...
          14:23:38,444 ERROR [org.jboss.as.ejb3.invocation] (pool-6-thread-1) JBAS014134: EJB Invocation failed on component ServerPluginManagerBean for method public abstract org.rhq.core.domain.plugin.ServerPlugin org.rhq.enterprise.server.plugin.ServerPluginManagerLocal.getServerPluginRelationships(org.rhq.core.domain.plugin.ServerPlugin): javax.ejb.EJBException: java.lang.NullPointerException
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:189) [jboss-as-ejb3-7.4.0.Final-redhat-4.jar:7.4.0.Final-redhat-4]
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:274) [jboss-as-ejb3-7.4.0.Final-redhat-4.jar:7.4.0.Final-redhat-4]
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:339) [jboss-as-ejb3-7.4.0.Final-redhat-4.jar:7.4.0.Final-redhat-4]
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:238) [jboss-as-ejb3-7.4.0.Final-redhat-4.jar:7.4.0.Final-redhat-4]
          
          
          
          ...
          Caused by: java.lang.NullPointerException
              at org.rhq.enterprise.server.plugin.ServerPluginManagerBean.getServerPluginRelationships(ServerPluginManagerBean.java:135) [rhq-server.jar:4.12.0]
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_45]
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_45]
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_45]
          ...
          14:23:38,466 WARN  [org.rhq.enterprise.server.plugin.pc.MasterServerPluginContainer] (pool-6-thread-1) Failed to preload server plugin [alert-irc] from URL [file:/opt/rhq/rhq-server-4.12.0/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-serverplugins/alert-irc-4.12.0.jar]: java.lang.RuntimeException: Failed to get plugin config/schedules from the database
          ...
          
          
          
          

           

          This looks most relevant I think:

           

          14:23:54,493 WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (http-/0.0.0.0:7080-10) SQL Error: 2289, SQLState: 42000
          14:23:54,493 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (http-/0.0.0.0:7080-10) ORA-02289: sequence does not exist
          
          14:23:54,497 ERROR [org.jboss.as.ejb3.invocation] (http-/0.0.0.0:7080-10) JBAS014134: EJB Invocation failed on component AgentManagerBean for method public abstract org.rhq.core.domain.install.remote.AgentInstall org.rhq.enterprise.server.core.AgentManagerLocal.updateAgentInstall(org.rhq.core.domain.auth.Subject,org.rhq.core.domain.install.remote.AgentInstall): javax.ejb.EJBException: javax.persistence.PersistenceException: org.hibernate.exception.SQLGrammarException: could not extract ResultSet
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:189) [jboss-as-ejb3-7.4.0.Final-redhat-4.jar:7.4.0.Final-redhat-4]
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:274) [jboss-as-ejb3-7.4.0.Final-redhat-4.jar:7.4.0.Final-redhat-4]
          ...
          Caused by: javax.persistence.PersistenceException: org.hibernate.exception.SQLGrammarException: could not extract ResultSet
              at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387) [hibernate-entitymanager-4.2.10.Final.jar:4.2.10.Final]
          
          ...
          Caused by: java.sql.SQLSyntaxErrorException: ORA-02289: sequence does not exist
          
              at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:445) [ojdbc6-11.2.0.3.0.jar:11.2.0.3.0]
              at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396) [ojdbc6-11.2.0.3.0.jar:11.2.0.3.0]
              at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:879) [ojdbc6-11.2.0.3.0.jar:11.2.0.3.0]
          
          
          
          
          
              at org.jboss.as.ejb3....tx.CMTTxInterceptor.required(CMTTxInterceptor.java:339) [jboss-as-ejb3-7.4.0.Final-redhat-4.jar:7.4.0.Final-redhat-4]
          
          
          

           

          Attaching server logs.

          • 2. Re: Upgrade 4.6 to 4.12 fails: Database Plugin versions are not updated.
            Elias Ross Master

            You have to fix the sequence number.

             

            alter sequence rhq_agent_install_seq rename to rhq_agent_install_id_seq

             

            not sure this is it though:

            https://docs.jboss.org/author/display/RHQ/Release+Notes+4.11

            • 3. Re: Re: Upgrade 4.6 to 4.12 fails: Database Plugin versions are not updated.
              Stian Lund Expert

              Hello Elias, thanks for the reply - it appears the name of the Sequence in 4.12 also is wrong, possibly. Anyhow, I renamed it but this was not the issue.

              And they can't really expect us to read every release note from old to current version, can they?

               

              The problem was an old leftover agent plugin called CorePerspective that was a part of RHQ earlier but was removed sometime between 4.6 and the latest.

               

              I still had it registered in RHQ, and when it tried to do an update on all the plugins, either on startup or using the UI, this was causing the ClassNotFound and then a NullPointer in the update causing it to abort the process.

               

              After manually deleting this plugin reference in the DB table and restarting RHQ the plugin update started and finished.

               

              I think it was John that gave me the clue when he said Perspective was an old plugin that was no longer used.

               

              The error message for those interested (and for Google to find) was:

               

              14:48:10,350 ERROR [org.rhq.enterprise.server.core.plugin.PluginDeploymentScanner] (pool-6-thread-1) Scan failed. Cause: java.lang.ClassNotFoundException:org.rhq.enterprise.server.xmlschema.generated.serverplugin.perspective.PerspectivePluginDescriptorType from [Module "deployment.rhq.ear.rhq-server.jar:main" from Service Module Loader]

               

              And then it caused I think:

               

              14:48:11,478 ERROR [org.jboss.as.ejb3.invocation] (pool-6-thread-1) JBAS014134: EJB Invocation failed on component ServerPluginManagerBean for method public abstract org.rhq.core.domain.plugin.ServerPlugin org.rhq.enterprise.server.plugin.ServerPluginManagerLocal.getServerPluginRelationships(org.rhq.core.domain.plugin.ServerPlugin): javax.ejb.EJBException: java.lang.NullPointerException

                  at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:189) [jboss-as-ejb3-7.4.0.Final-redhat-4.jar:7.4.0.Final-redhat-4]

               

              So it's all sorted now, and I assume this is such a special case that making a BZ would be no use. How many are still running 4.5 or 4.6?

               

              Stian