4 Replies Latest reply on Aug 23, 2013 10:34 PM by lukasz74nj

    Could not complete Cassandra schema installation (RHQ 4.8.0)

    lukasz74nj

      Hi,

       

      I'm getting below exeption when running "rhqctl install" script to setup an initial installation of RHQ:

       

      11:59:11,831 INFO  [org.rhq.enterprise.server.installer.InstallerServiceImpl] Cassandra schema exists but installer was told to overwrite it - a the existing  schema will be created now.

      11:59:11,841 INFO  [org.rhq.cassandra.schema.VersionManager] Preparing to drop RHQ schema

      11:59:11,842 INFO  [org.rhq.cassandra.schema.AbstractManager] Initializing session to connect to X.X.X.X

      11:59:12,041 INFO  [org.rhq.cassandra.schema.AbstractManager] Shutting down connections

      11:59:12,042 ERROR [org.rhq.enterprise.server.installer.InstallerServiceImpl] Could not complete Cassandra schema installation: java.lang.NullPointerException: java.lang.NullPointerException

              at org.rhq.cassandra.schema.AbstractManager.shutdown(AbstractManager.java:251) [rhq-cassandra-schema-4.8.0.jar:4.8.0]

              at org.rhq.cassandra.schema.VersionManager.drop(VersionManager.java:142) [rhq-cassandra-schema-4.8.0.jar:4.8.0]

              at org.rhq.cassandra.schema.SchemaManager.drop(SchemaManager.java:93) [rhq-cassandra-schema-4.8.0.jar:4.8.0]

              at org.rhq.enterprise.server.installer.InstallerServiceImpl.prepareDatabase(InstallerServiceImpl.java:486) [rhq-installer-util-4.8.0.jar:4.8.0]

              at org.rhq.enterprise.server.installer.InstallerServiceImpl.install(InstallerServiceImpl.java:258) [rhq-installer-util-4.8.0.jar:4.8.0]

              at org.rhq.enterprise.server.installer.Installer.doInstall(Installer.java:115) [rhq-installer-util-4.8.0.jar:4.8.0]

              at org.rhq.enterprise.server.installer.Installer.main(Installer.java:56) [rhq-installer-util-4.8.0.jar:4.8.0]

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25]

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25]

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]

              at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]

              at org.jboss.modules.Module.run(Module.java:262) [jboss-modules.jar:1.2.0.CR1]

              at org.jboss.modules.Main.main(Main.java:329) [jboss-modules.jar:1.2.0.CR1]

       

      As I wanted to drop the schema, I set the "rhq.autoinstall.database" property to "overwrite" in rhq-server.properties.

       

      How to fix this? RHQ version is 5.8.0.

       

      So far, the RHQ setup process has been very flustrating for me. Also, I had to define below JVM property (in both scripts and Java Service Wrapper configuration files) to fix connection reset errors when using Oracle (the Oracle driver requires a random number generator to encrypt user credentials and on a server system there is not enough entropy to generate random numbers using the default generator):

       

      -Djava.security.egd=file:///dev/urandom

       

      It wouldn't hurt if the above property was defined by default. Otherwise, RHQ will not work on a server system when using Oracle (at least not with the 11.2.0.3 version of the JDBC driver).

       

      Thanks,

      Lukasz

        • 1. Re: Could not complete Cassandra schema installation (RHQ 4.8.0)
          lukasz74nj

          When I ran it the second time, it succeeded. However, now when I try to access the RQH Web Console using the following URL, the page is blank (for a half of second it says that the RHQ is initializing and then goes blank):

           

          http://<rhq-host>:7080

           

          At the same time, there lots of errors in the server log, i.e:

           

          12:49:57,070 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/jboss-remoting-servlet-invoker].[ServerInvokerServlet]] (http-/0.0.0.0:7080-1) JBWEB000235: Allocate exception for servlet ServerInvokerServlet: javax.servlet.ServletException: Could not find the remoting servlet invoker: jboss.remoting:service=invoker,rhq.communications.connector.rhqtype=server,* - need to wait for remoting to be initialized later

          • 2. Re: Could not complete Cassandra schema installation (RHQ 4.8.0)
            nstefan

            Hello Lukasz,

             

            The error from your first post has been fixed and will be released in RHQ 4.9.0 (latest commit). And you used the correct workaround, to re-run the installer. The error comes from the new storage cluster and not the actual SQL database. That first error is totally unrelated to other issues that you might have encountered later with the Oracle JDBC.

             

            Could you please provide more details about the issue with Oracle JDBC driver and password encoding? I am not sure I understand exactly what the problem was/is. In general terms adding an optional property that can be turned on by users along with proper documentation should not be hard.

             

            For your second error, server not starting, could you please provide a more detailed server log? The error that you pasted in the second post is too generic to be able to debug the issue.

             

             

            Thank you,

            Stefan

            • 3. Re: Could not complete Cassandra schema installation (RHQ 4.8.0)
              lukasz74nj

              Hi Stefan,

               

              Thanks for your response and clarifications.

               

              Regarding the Oracle JDBC driver issue, it has been explained well here:

               

              http://www.usn-it.de/index.php/2009/02/20/oracle-11g-jdbc-driver-hangs-blocked-by-devrandom-entropy-pool-empty/

               

              Basically, by default JDBC driver uses a blocking device (/dev/random) as a source of random number generator. However, on a server system, since there is not enough entropy - unless someone installs a microphone inside a chasis , the device will block so will the JDBC driver, causing timeouts or connection reset errors. Fortunately, there is an easy fix, at the expense of the quality of random data when using a non-blocking device (/dev/urandom):

               

              -Djava.security.egd=file:///dev/urandom

               

              Regarding my original issue, I will provide more details shortly.

               

              Thanks,

              Lukasz

              • 4. Re: Could not complete Cassandra schema installation (RHQ 4.8.0)
                lukasz74nj

                Hi Stefan,

                 

                I ended up reinstalling everything as documented all the steps along the way - and it worked this time! Well, at least I can access RHQ web Console. I hope below instructions can be of use to others trying to setup RHQ with Oracle database.

                 

                Thanks,

                Lukasz

                 

                 

                Preparation:

                 

                1. Extracted RHQ archive to <RHQ-HOME>

                 

                2. Edited database (Oracle) and SMTP settings under <RHQ-HOME>/bin/rhq-server.properties

                 

                3. Edited rhq.agent.basedir under <RHQ-HOME>/bin/rhqctl.properties to point to <RHQ-HOME>/rhq-agent directory

                 

                4. Set RHQ_CONTROL_ADDITIONAL_JAVA_OPTS variable in <RHQ-HOME>/bin/rhqctl

                 

                RHQ_CONTROL_ADDITIONAL_JAVA_OPTS="-Djava.security.egd=file:///dev/urandom"

                 

                5. Added the following JVM argument to <RHQ-HOME>/bin/wrapper/rhq-server-wrapper.conf

                 

                wrapper.java.additional.23=-Djava.security.egd=file:///dev/urandom

                 

                6. Added the following JVM argument to <RHQ-HOME>/bin/wrapper/rhq-storage-wrapper.conf

                 

                wrapper.java.additional.18=-Djava.security.egd=file:///dev/urandom

                 

                7. Installed Oracle JDBC driver jar under <RHQ-HOME>/modules/org/rhq/oracle/main

                 

                8. Edited JDBC driver's path in <RHQ-HOME>/modules/org/rhq/oracle/main/modules.xml

                 

                 

                Installation:

                 

                $ cd <RHQ-HOME>/bin

                $ ./rhqctl install --storage-data-root-dir <RHQ-HOME>/data

                 

                All went fine except below error was reported (not sure if it is harmless or not):

                 

                22:05:22,961 ERROR [org.jboss.remoting.remote.connection] JBREM000200: Remote connection failed: java.io.IOException: JBREM000202: Abrupt close on Remoting connection 6b19ae3e to localhost.localdomain/127.0.0.1:6999

                 

                $ ./rhqctl status

                 

                22:09:28,944 INFO  [org.jboss.modules] JBoss Modules version 1.2.0.CR1

                RHQ Storage Node               (pid 10400  ) IS running

                RHQ Server                     (pid 10779  ) IS running

                JBossAS Java VM child process  (pid 10779  ) IS running

                RHQ Agent                      (pid 12016  ) IS running

                 

                 

                Notes:

                 

                - there are repeated XA recovery errors in the <RHQ-HOME>/logs/server.log (these could be related to the fact that I reused the database schema from previous installation attempts):

                 

                22:12:04,079 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016027: Local XARecoveryModule.xaRecovery got XA exception XAException.XAER_RMERR: javax.transaction.xa.XAException

                        at oracle.jdbc.xa.OracleXAResource.recover(OracleXAResource.java:709)

                        at org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.recover(XAManagedConnection.java:358)

                        at org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.recover(XAResourceWrapperImpl.java:165)

                        at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoveryEnd(XARecoveryModule.java:683) [jbossjts-jacorb-4.17.3.Final.jar:4.17.3.Final (revision: 74343b48951c0fdab92316e56bfcaed605d620f6)]

                        at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecoveryForRecoveryHelpers(XARecoveryModule.java:490) [jbossjts-jacorb-4.17.3.Final.jar:4.17.3.Final (revision: 74343b48951c0fdab92316e56bfcaed605d620f6)]

                        at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:378) [jbossjts-jacorb-4.17.3.Final.jar:4.17.3.Final (revision: 74343b48951c0fdab92316e56bfcaed605d620f6)]

                        at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:166) [jbossjts-jacorb-4.17.3.Final.jar:4.17.3.Final (revision: 74343b48951c0fdab92316e56bfcaed605d620f6)]

                        at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) [jbossjts-jacorb-4.17.3.Final.jar:4.17.3.Final (revision: 74343b48951c0fdab92316e56bfcaed605d620f6)]

                        at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [jbossjts-jacorb-4.17.3.Final.jar:4.17.3.Final (revision: 74343b48951c0fdab92316e56bfcaed605d620f6)]

                 

                - no errors in <RHQ-HOME>/logs/rhq-storage.log

                 

                - no errors in <RHQ-HOME>/rhq-agent/logs/agent.log (except initial problems regarding missing plugins, before these were automatically downloaded)