Upgrade 4.6 to 4.12 fails with missing keystores during Jboss startup

[pathduck]

Hi,

I've started upgrading our installations to RHQ 4.12 and have encountered an error when upgrading one of our older 4.6 installations to 4.12.

The installer fails at the point where it (I assume) tries to connect to Jboss to do deploy and setup. It looks like it's trying to initialize SSL and needs to find a keystore and truststore. The log states the keystore is created but then says it can't find them - even though they are in the correct location.

I've upgraded a 4.11 to 4.12 without any problems on another install so pretty sure I'm doing it the right way. But we also have 4.5.1 in production so probably it will fail there as well.

Standalone.xml does not have the HTTPS connector configured in in the web subsystem either. I think maybe this will be configured after inital connection is made.

I'll open a BZ but I need to know if this is expected or a known problem, and if I am doing something stupid.

This is the command I use for the upgrade:

./rhqctl upgrade --from-agent-dir /opt/rhq/rhq-agent --from-server-dir /opt/rhq/rhq-server --storage-data-root-dir /opt/rhq/rhq-data

Thing is, normally I would try to do a rhqctl install instead if upgrade fails, and would assume the installer upgrades the Database regardless. However when running install RHQ starts OK but all the agent plugins fail to upgrade since the table is still refering to version 4.6 of the plugins. So I am stuck

This is on Redhat 6.5 using Java 1.7_45.

Attaching logs from RHQ.

09:14:31,064 INFO  [org.rhq.enterprise.server.installer.ServerInstallUtil] Default transaction timeout set to 600 seconds.
09:14:31,110 INFO  [org.rhq.enterprise.server.installer.ServerInstallUtil] Logging category org.rhq set to [${rhq.server.log-level:INFO}]
09:14:32,023 INFO  [org.rhq.enterprise.server.installer.ServerInstallUtil] New keystore created [/opt/rhq/rhq-server-4.12.0/jbossas/standalone/configuration/rhq.keystore] with cert domain name of [CN=d32apvl001.oera-t.local, OU=RHQ, O=rhq-project.org, C=US]
09:14:32,027 INFO  [org.rhq.enterprise.server.installer.ServerInstallUtil] Creating https connector...
09:14:32,032 WARN  [org.rhq.enterprise.server.installer.ServerInstallUtil] Missing both keystore [rhq.keystore] and truststore [rhq.truststore]
09:14:32,279 ERROR [org.rhq.enterprise.server.installer.Installer] The installer will now exit due to previous errors: org.rhq.common.jbossas.client.controller.FailureException: Failed to add new connector [https]: {"JBAS014653: Composite operation failed and was rolled back. Steps that failed:" => {"Operation step-1" => {"JBAS014671: Failed services" => {"jboss.web.connector.https" => "org.jboss.msc.service.StartException in service jboss.web.connector.https: JBAS018007: Error starting web connector
    Caused by: LifecycleException:  JBWEB000023: Protocol handler initialization failed"}}}}
        at org.rhq.common.jbossas.client.controller.WebJBossASClient.addConnector(WebJBossASClient.java:176) [rhq-jboss-as-dmr-client-4.12.0.jar:4.12.0]
        at org.rhq.enterprise.server.installer.ServerInstallUtil.setupWebConnectors(ServerInstallUtil.java:1489) [rhq-installer-util-4.12.0.jar:4.12.0]
        at org.rhq.enterprise.server.installer.InstallerServiceImpl.install(InstallerServiceImpl.java:348) [rhq-installer-util-4.12.0.jar:4.12.0]
        at org.rhq.enterprise.server.installer.Installer.doInstall(Installer.java:116) [rhq-installer-util-4.12.0.jar:4.12.0]
        at org.rhq.enterprise.server.installer.Installer.main(Installer.java:57) [rhq-installer-util-4.12.0.jar:4.12.0]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_45]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_45]
        at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_45]
        at org.jboss.modules.Module.run(Module.java:292) [jboss-modules.jar:1.3.0.Final-redhat-2]
        at org.jboss.modules.Main.main(Main.java:455) [jboss-modules.jar:1.3.0.Final-redhat-2]

09:14:32,599 INFO  [org.rhq.server.control.command.Upgrade] The server installer is running
09:14:32,680 INFO  [org.xnio] XNIO Version 3.0.9.GA-redhat-1
09:14:32,687 INFO  [org.xnio.nio] XNIO NIO Implementation Version 3.0.9.GA-redhat-1
09:14:32,701 INFO  [org.jboss.remoting] JBoss Remoting version (unknown)
09:15:03,250 INFO  [org.rhq.server.control.command.Upgrade] Still waiting for server to start...
09:15:33,373 INFO  [org.rhq.server.control.command.Upgrade] Still waiting for server to start...
09:16:03,522 INFO  [org.rhq.server.control.command.Upgrade] Still waiting for server to start...
09:16:33,630 INFO  [org.rhq.server.control.command.Upgrade] It has been over [2] minutes - you may want to ensure your server startup is proceeding as expected. You can check the log at [/opt/rhq/rhq-server-4.12.0/logs/server.log].

[pathduck]

Solved.

It appears the value of "rhq.server.tomcat.security.keystore.file" was set to just the value 'rhq.keystore'.

The same for the corresponding truststore value.

I believe in 4.5.1 this value is 'conf/rhq.keystore'. So not sure if it affects upgrade from 4.5.1 to 4.12 - we are running 4.5.1 in production.

While in 4.12 it expects a value = ${jboss.server.config.dir}/rhq.keystore

So to fix I edited the values in rhq-server.properties for the old installation before running the upgrade, ensuring the value got copied over correctly.

So it should look like this:

rhq.server.tomcat.security.keystore.file=${jboss.server.config.dir}/rhq.keystore
...
rhq.server.tomcat.security.truststore.file=${jboss.server.config.dir}/rhq.truststore