3 Replies Latest reply on Aug 6, 2014 6:48 AM by pathduck

Upgrade 4.6 to 4.12 fails: Database Plugin versions are not updated.

pathduck Aug 5, 2014 7:00 AM

Hi,

after fixing the other bug I think this one is a harder nut to crack.

Upgrade using 'rhqctl upgrade' exits with no errors using the command:

./rhqctl upgrade --from-agent-dir /opt/rhq/rhq-agent --from-server-dir /opt/rhq/rhq-server --storage-data-root-dir /opt/rhq/rhq-data

The server starts OK but all the agents fail in their download of the new plugins from the server.

From agent.log:

2014-08-05 11:33:43,637 INFO  [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.downloading}Downloading the plugin [rhq-script-plugin-4.6.0.jar]...
2014-08-05 11:33:43,702 ERROR [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.download-failure}Failed to download the plugin [Script]. Cause: java.lang.ClassNotFoundException: [Warning] org.rhq.enterprise.server.xmlschema.generated.serverplugin.perspective.PerspectivePluginDescriptorType from [Module "deployment.rhq.ear.rhq-server.jar:main" from Service Module Loader]
2014-08-05 11:33:43,702 WARN  [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.download-failure-will-retry}Failed to download the plugin [rhq-script-plugin-4.6.0.jar]. This was attempt #[1]. Will retry again in [56,746]ms. Cause: [org.rhq.core.util.exception.WrappedRemotingException:org.rhq.enterprise.server.xmlschema.generated.serverplugin.perspective.PerspectivePluginDescriptorType from [Module "deployment.rhq.ear.rhq-server.jar:main" from Service Module Loader]]
2014-08-05 11:34:40,515 ERROR [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.download-failure}Failed to download the plugin [Script]. Cause: java.lang.ClassNotFoundException: [Warning] org.rhq.enterprise.server.xmlschema.generated.serverplugin.perspective.PerspectivePluginDescriptorType from [Module "deployment.rhq.ear.rhq-server.jar:main" from Service Module Loader]
2014-08-05 11:34:40,516 WARN  [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.download-failure-will-retry}Failed to download the plugin [rhq-script-plugin-4.6.0.jar]. This was attempt #[2]. Will retry again in [52,890]ms. Cause: [org.rhq.core.util.exception.WrappedRemotingException:org.rhq.enterprise.server.xmlschema.generated.serverplugin.perspective.PerspectivePluginDescriptorType from [Module "deployment.rhq.ear.rhq-server.jar:main" from Service Module Loader]]
2014-08-05 11:35:33,455 ERROR [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.download-failure}Failed to download the plugin [Script]. Cause: java.lang.ClassNotFoundException: [Warning] org.rhq.enterprise.server.xmlschema.generated.serverplugin.perspective.PerspectivePluginDescriptorType from [Module "deployment.rhq.ear.rhq-server.jar:main" from Service Module Loader]
2014-08-05 11:35:33,456 WARN  [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.download-failure-will-not-retry}Failed to download the plugin [rhq-script-plugin-4.6.0.jar]. This was attempt #[3] - will no longer retry. This plugin will not be deployed in the agent. Cause: org.rhq.core.util.exception.WrappedRemotingException:org.rhq.enterprise.server.xmlschema.generated.serverplugin.perspective.PerspectivePluginDescriptorType from [Module "deployment.rhq.ear.rhq-server.jar:main" from Service Module Loader]
2014-08-05 11:35:33,456 INFO  [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.downloading}Downloading the plugin [rhq-platform-plugin-4.6.0.jar]...
2014-08-05 11:35:33,505 ERROR [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.download-failure}Failed to download the plugin [Platforms]. Cause: java.lang.ClassNotFoundException: [Warning] org.rhq.enterprise.server.xmlschema.generated.serverplugin.perspective.PerspectivePluginDescriptorType from [Module "deployment.rhq.ear.rhq-server.jar:main" from Service Module Loader]
2014-08-05 11:35:33,506 WARN  [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.download-failure-will-retry}Failed to download the plugin [rhq-platform-plugin-4.6.0.jar]. This was attempt #[1]. Will retry again in [59,442]ms. Cause: [org.rhq.core.util.exception.WrappedRemotingException:org.rhq.enterprise.server.xmlschema.generated.serverplugin.perspective.PerspectivePluginDescriptorType from [Module "deployment.rhq.ear.rhq-server.jar:main" from Service Module Loader]]

From server.log:

12:53:28,434 INFO  [org.rhq.enterprise.server.core.plugin.ServerPluginScanner] (http-/0.0.0.0:7080-34) Filesystem has a server plugin [alert-email] at the file [/opt/rhq/rhq-server-4.12.0/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-serverplugins/alert-email-4.12.0.jar] which is different than where the DB thinks it should be [/opt/rhq/rhq-server-4.12.0/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-serverplugins/alert-email-4.6.0.jar]
12:53:28,434 INFO  [org.rhq.enterprise.server.core.plugin.ServerPluginScanner] (http-/0.0.0.0:7080-34) It appears that the server plugin [ServerPlugin [id=0, name=alert-email, md5=53f2f5f6ee0e283627471cad049e6fef]] in the database may be obsolete. If so, it will be updated later.
12:53:28,434 INFO  [org.rhq.enterprise.server.core.plugin.ServerPluginScanner] (http-/0.0.0.0:7080-34) Filesystem has a server plugin [DiskContentSourcePlugin] at the file [/opt/rhq/rhq-server-4.12.0/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-serverplugins/rhq-serverplugin-disk-4.12.0.jar] which is different than where the DB thinks it should be [/opt/rhq/rhq-server-4.12.0/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-serverplugins/rhq-serverplugin-disk-4.6.0.jar]
12:53:28,434 INFO  [org.rhq.enterprise.server.core.plugin.ServerPluginScanner] (http-/0.0.0.0:7080-34) It appears that the server plugin [ServerPlugin [id=0, name=DiskContentSourcePlugin, md5=7c60222ffb7397b431e1c50cdab214fb]] in the database may be obsolete. If so, it will be updated later.

The database (Oracle 11) contains just the old versions:




snmptrapd                  
Receiver for SNMP Traps      
4.6.0     
 


jboss-as-7                 
JBoss Application Server 7.x 
4.4.0-SNAPSHOT 


hudson                     
Hudson                       
4.6.0     
 


Twitter                    
Twitter Plugin               
4.6.0     
 


Script                     
Script                       
4.6.0     
 


Platforms                  
Operating System Platforms   
4.6.0     
 


PerfTest                   
Performance Test Plugin      
4.6.0     
 


NetworkServices            
Network Services             
4.6.0     
 


JMX                        
Generic JMX                  
4.6.0     
 


Tomcat                     
Tomcat Server                
4.6.0     
 


RHQAgent                   
RHQ Agent                    
4.6.0     
 


JBossAS5                   
JBoss Application Server 5.x/6.x 4.6.0     
 


JBossCache3                
JBossCache 3.x Services      
4.6.0     
 


JBossAS                    
JBoss Application Server 3.x/4.x 4.6.0

And so on... not sure what the 4.4 snapshot is doing there though, it's been deleted.

Any ideas how to fix without manually having to change every row in the RHQ_PLUGINS table, which I don't think is even possible... it contains md5 sums for one.

Stian

1. Re: Upgrade 4.6 to 4.12 fails: Database Plugin versions are not updated.

pathduck Aug 5, 2014 8:34 AM (in response to pathduck)

The log file grew very large after upgrade because it gets constantly filled with agent upgrade failures, but I've done a clean restart to see what errors pop up. THere are definitely some errors and nullpointers that look related to plugins.

14:23:37,205 ERROR [org.rhq.enterprise.server.core.plugin.PluginDeploymentScanner] (pool-6-thread-1) Scan failed. Cause: java.lang.ClassNotFoundException:org.rhq.enterprise.server.xmlschema.generated.serverplugin.perspective.PerspectivePluginDescriptorType from [Module "deployment.rhq.ear.rhq-server.jar:main" from Service Module Loader]
...
14:23:38,444 ERROR [org.jboss.as.ejb3.invocation] (pool-6-thread-1) JBAS014134: EJB Invocation failed on component ServerPluginManagerBean for method public abstract org.rhq.core.domain.plugin.ServerPlugin org.rhq.enterprise.server.plugin.ServerPluginManagerLocal.getServerPluginRelationships(org.rhq.core.domain.plugin.ServerPlugin): javax.ejb.EJBException: java.lang.NullPointerException
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:189) [jboss-as-ejb3-7.4.0.Final-redhat-4.jar:7.4.0.Final-redhat-4]
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:274) [jboss-as-ejb3-7.4.0.Final-redhat-4.jar:7.4.0.Final-redhat-4]
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:339) [jboss-as-ejb3-7.4.0.Final-redhat-4.jar:7.4.0.Final-redhat-4]
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:238) [jboss-as-ejb3-7.4.0.Final-redhat-4.jar:7.4.0.Final-redhat-4]

...

Caused by: java.lang.NullPointerException
    at org.rhq.enterprise.server.plugin.ServerPluginManagerBean.getServerPluginRelationships(ServerPluginManagerBean.java:135) [rhq-server.jar:4.12.0]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_45]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_45]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_45]
...
14:23:38,466 WARN  [org.rhq.enterprise.server.plugin.pc.MasterServerPluginContainer] (pool-6-thread-1) Failed to preload server plugin [alert-irc] from URL [file:/opt/rhq/rhq-server-4.12.0/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-serverplugins/alert-irc-4.12.0.jar]: java.lang.RuntimeException: Failed to get plugin config/schedules from the database
...

This looks most relevant I think:

14:23:54,493 WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (http-/0.0.0.0:7080-10) SQL Error: 2289, SQLState: 42000
14:23:54,493 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (http-/0.0.0.0:7080-10) ORA-02289: sequence does not exist

14:23:54,497 ERROR [org.jboss.as.ejb3.invocation] (http-/0.0.0.0:7080-10) JBAS014134: EJB Invocation failed on component AgentManagerBean for method public abstract org.rhq.core.domain.install.remote.AgentInstall org.rhq.enterprise.server.core.AgentManagerLocal.updateAgentInstall(org.rhq.core.domain.auth.Subject,org.rhq.core.domain.install.remote.AgentInstall): javax.ejb.EJBException: javax.persistence.PersistenceException: org.hibernate.exception.SQLGrammarException: could not extract ResultSet
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:189) [jboss-as-ejb3-7.4.0.Final-redhat-4.jar:7.4.0.Final-redhat-4]
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:274) [jboss-as-ejb3-7.4.0.Final-redhat-4.jar:7.4.0.Final-redhat-4]
...
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.SQLGrammarException: could not extract ResultSet
    at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387) [hibernate-entitymanager-4.2.10.Final.jar:4.2.10.Final]

...
Caused by: java.sql.SQLSyntaxErrorException: ORA-02289: sequence does not exist

    at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:445) [ojdbc6-11.2.0.3.0.jar:11.2.0.3.0]
    at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396) [ojdbc6-11.2.0.3.0.jar:11.2.0.3.0]
    at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:879) [ojdbc6-11.2.0.3.0.jar:11.2.0.3.0]





    at org.jboss.as.ejb3....tx.CMTTxInterceptor.required(CMTTxInterceptor.java:339) [jboss-as-ejb3-7.4.0.Final-redhat-4.jar:7.4.0.Final-redhat-4]

Attaching server logs.

logs.zip 114.1 KB

2. Re: Upgrade 4.6 to 4.12 fails: Database Plugin versions are not updated.

genman Aug 5, 2014 2:52 PM (in response to pathduck)

You have to fix the sequence number.

alter sequence rhq_agent_install_seq rename to rhq_agent_install_id_seq

not sure this is it though:
https://docs.jboss.org/author/display/RHQ/Release+Notes+4.11
Actions
3. Re: Re: Upgrade 4.6 to 4.12 fails: Database Plugin versions are not updated.

pathduck Aug 6, 2014 6:48 AM (in response to genman)

Hello Elias, thanks for the reply - it appears the name of the Sequence in 4.12 also is wrong, possibly. Anyhow, I renamed it but this was not the issue.
And they can't really expect us to read every release note from old to current version, can they?

The problem was an old leftover agent plugin called CorePerspective that was a part of RHQ earlier but was removed sometime between 4.6 and the latest.

I still had it registered in RHQ, and when it tried to do an update on all the plugins, either on startup or using the UI, this was causing the ClassNotFound and then a NullPointer in the update causing it to abort the process.

After manually deleting this plugin reference in the DB table and restarting RHQ the plugin update started and finished.

I think it was John that gave me the clue when he said Perspective was an old plugin that was no longer used.

The error message for those interested (and for Google to find) was:

14:48:10,350 ERROR [org.rhq.enterprise.server.core.plugin.PluginDeploymentScanner] (pool-6-thread-1) Scan failed. Cause: java.lang.ClassNotFoundException:org.rhq.enterprise.server.xmlschema.generated.serverplugin.perspective.PerspectivePluginDescriptorType from [Module "deployment.rhq.ear.rhq-server.jar:main" from Service Module Loader]

And then it caused I think:

14:48:11,478 ERROR [org.jboss.as.ejb3.invocation] (pool-6-thread-1) JBAS014134: EJB Invocation failed on component ServerPluginManagerBean for method public abstract org.rhq.core.domain.plugin.ServerPlugin org.rhq.enterprise.server.plugin.ServerPluginManagerLocal.getServerPluginRelationships(org.rhq.core.domain.plugin.ServerPlugin): javax.ejb.EJBException: java.lang.NullPointerException
at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:189) [jboss-as-ejb3-7.4.0.Final-redhat-4.jar:7.4.0.Final-redhat-4]

So it's all sorted now, and I assume this is such a special case that making a BZ would be no use. How many are still running 4.5 or 4.6?

Stian
Actions

Go to original post