12 Replies Latest reply on Feb 6, 2009 12:41 PM by nsyed4

    Failed to start the agent

    nsyed4

      Environment: Linux Fedora 8 32 bit O.S Java SE 6

      I built RHQ Server and Agent as described in the build instructions given at http://support.rhq-project.org/display/RHQ/Building+RHQ:
      1. Used Maven: mvn install -Penterprise,dev -Dmaven.test.skip=true -Ddbsetup
      2. Used Oracle 10g: Edited pom.xml to use oracle jdbc driver
      3. Started the RHQ Server: <rhq-trunk>/modules/enterprise/server/container/target/rhq-server-1.2.0-SNAPSHOT/bin/rhq-server.sh start
      4. Tried Starting the RHQ Agent: <rhq-trunk>/modules/enterprise/agent/target/rhq-agent/bin/rhq-agent.sh
      Please Note that there is no version appened to the rhq-agent directory but there is to the server (rhq-server-1.2.0-SNAPSHOT).

      Recieved the following error:
      Failed to start the agent
      java.lang.IllegalStateException: Neither a native not a Java platform was discovered - this should never happen. Known platform types are [ResourceTpe[id=0, category=Platform, name=platform-a,plugin=PerfTest]].
      at org.rhq.core.pc.inventory.InventoryManager.discoverPlatform(InventoryManager.java:1308)

      Agent will be restarted soon to see if the failure condition cleared up

      I restarted the agent under debug mode and these are some interesting logs:
      2009-02-05 11:41:48,545 DEBUG [main] (org.rhq.core.pc.PluginContainer)- Starting and configuring container service: PluginComponentFactory
      2009-02-05 11:41:48,545 DEBUG [main] (org.rhq.core.pc.PluginContainer)- Starting and configuring container service: InventoryManager
      2009-02-05 11:41:48,545 INFO [main] (rhq.core.pc.inventory.InventoryManager)- Initializing Inventory Manager...
      2009-02-05 11:41:48,547 DEBUG [main] (rhq.core.pc.inventory.InventoryManager)- Executing platform scan...
      2009-02-05 11:41:48,739 DEBUG [main] (rhq.core.pc.plugin.PluginComponentFactory)- Loading class: org.rhq.plugins.perftest.PerfTestDiscoveryComponent
      2009-02-05 11:41:48,741 DEBUG [main] (rhq.core.pc.plugin.PluginComponentFactory)- Loaded class: class org.rhq.plugins.perftest.PerfTestDiscoveryComponent
      2009-02-05 11:41:48,741 DEBUG [main] (rhq.core.pc.plugin.PluginComponentFactory)- Loading and creating the discovery component [org.rhq.plugins.perftest.PerfT
      estDiscoveryComponent] for resource type [platform-a]
      2009-02-05 11:41:48,741 DEBUG [main] (rhq.core.pc.plugin.PluginComponentFactory)- Created and cached the discovery component [org.rhq.plugins.perftest.PerfTes
      tDiscoveryComponent] for resource type [platform-a]
      2009-02-05 11:41:48,747 ERROR [main] (org.rhq.plugins.perftest.ScenarioManager)- Cannot find scenario name. Make sure the property on.perftest.scenario is set
      .
      2009-02-05 11:41:48,749 FATAL [main] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.startup-error}The agent encountered an error during startup and must abo
      rt
      java.lang.IllegalStateException: Neither a native nor a Java platform was discovered - this should never happen. Known platform types are [ResourceType[id=0,
      category=Platform, name=platform-a, plugin=PerfTest]].
      at org.rhq.core.pc.inventory.InventoryManager.discoverPlatform(InventoryManager.java:1308)
      at org.rhq.core.pc.inventory.InventoryManager.executePlatformScan(InventoryManager.java:264)
      at org.rhq.core.pc.inventory.InventoryManager.initialize(InventoryManager.java:186)
      at org.rhq.core.pc.PluginContainer.startContainerService(PluginContainer.java:307)
      at org.rhq.core.pc.PluginContainer.initialize(PluginContainer.java:231)
      at org.rhq.enterprise.agent.AgentMain.startPluginContainer(AgentMain.java:1724)
      at org.rhq.enterprise.agent.AgentMain.start(AgentMain.java:622)
      at org.rhq.enterprise.agent.AgentMain.main(AgentMain.java:397)

      I searched for on.perftest.scenario and found out that in the file rhq-home/modules/plugins/perfterst/src/main/resources/META-INF/rhq-plugin.xml there was information asking to start the agent with -Don.perftest.scenario=scenario_name where scenario_name is one of the configurations for number of servers and services, used for performance testing.

      I started with the perftest option but I recieved the same error.

      Then,I put some debug statements in the source code of the file org.rhq.core.pc.inventoryInventoryManager, and found out that the java.util.Set allDiscoveredPlatforms is empty, since discoveredResources is null.



        • 1. Re: Failed to start the agent
          nsyed4

          Thanks to John M., that I checked in the plugins directory:

          rhq-agent-plugin-1.2.0-SNAPSHOT.jar rhq-hudson-plugin-1.2.0-SNAPSHOT.jar rhq-oracle-plugin-1.2.0-SNAPSHOT.jar
          rhq-apache-plugin-1.2.0-SNAPSHOT.jar rhq-iis-plugin-1.2.0-SNAPSHOT.jar rhq-perftest-plugin-1.2.0-SNAPSHOT.jar
          rhq-apt-plugin-1.2.0-SNAPSHOT.jar rhq-jmx-plugin-1.2.0-SNAPSHOT.jar rhq-snmptrapd-plugin-1.2.0-SNAPSHOT.jar
          rhq-database-plugin-1.2.0-SNAPSHOT.jar rhq-mysql-plugin-1.2.0-SNAPSHOT.jar rhq-sshd-plugin-1.2.0-SNAPSHOT.jar
          rhq-hosts-plugin-1.2.0-SNAPSHOT.jar rhq-netservices-plugin-1.2.0-SNAPSHOT.jar

          I had the platform plugin missing, so I copied it from a binary RHQ agent that I downloaded from internet to the plugins directory (the version was 1.1.0):
          rhq-platform-plugin-1.1.0.jar

          But this did not help either.



            • 2. Re: Failed to start the agent
              mazz

              So, you have to find out why that platform plugin was missing.

              Here's how the plugins get deployed across your environment:

              1) the server starts and loads in the plugins it has deployed inside of it (see jbossas/server/default/deploy/rhq.ear/rhq-downloads/rhq-plugins - this is on the server-side).
              2) when an agent starts, it asks the server, "what plugins do you have for me?"
              3) the server tells the agent, "I have plugin A, B and C"
              4) the agent compares what plugins it has in its /plugins directory and if it has plugin D, it removes it. If Its missing A, B or C (or has an old one), the agent will download them from the server
              5) at this point the agent starts up

              So, I suspect you are missing the platform plugin from within your server's rhq-downloads/rhq-plugins directory.

              Something must have gone wrong when you ran your build. See what happens if you "mvn -Pdev install" from within the modules/plugins/platform directory.

              • 3. Re: Failed to start the agent
                nsyed4

                I ran "mvn -Pdev install" from <rhq-home>/modules/plugins/platform. This is what I get:
                [INFO] Scanning for projects...
                [WARNING] No SCM providers configured.
                WAGON_VERSION: 1.0-beta-2
                ...
                [INFO] Executing tasks
                [echo] *** Updating /export/home/nsyed/rhq/modules/plugins/platform/C:/Projects/rhq-trunk/dev-container/jbossas/server/default/deploy/rhq.ear/rhq-downloads/rhq-plugins/rhq-platform-plugin-1.2.0-SNAPSHOT.jar...
                [jar] Building jar: /export/home/nsyed/rhq/modules/plugins/platform/C:/Projects/rhq-trunk/dev-container/jbossas/server/default/deploy/rhq.ear/rhq-downloads/rhq-plugins/rhq-platform-plugin-1.2.0-SNAPSHOT.jar

                I also found a directory named C: in <rhq-home>/modules/plugins/platform

                In the end:
                [INFO] [install:install]
                [INFO] Installing /export/home/nsyed/rhq/modules/plugins/platform/target/rhq-platform-plugin-1.2.0-SNAPSHOT.jar to /export/home/nsyed/.m2/repository/org/rhq/rhq-platform-plugin/1.2.0-SNAPSHOT/rhq-platform-plugin-1.2.0-SNAPSHOT.jar
                [INFO] ------------------------------------------------------------------------
                [INFO] BUILD SUCCESSFUL
                [INFO] ------------------------------------------------------------------------
                [INFO] Total time: 12 seconds
                [INFO] Finished at: Thu Feb 05 14:30:32 CST 2009
                [INFO] Final Memory: 17M/31M
                [INFO] ------------------------------------------------------------------------





                • 4. Re: Failed to start the agent
                  nsyed4

                  I do have the platform plugin in the server's rhq-downloads/rhq-plugins directory

                  • 5. Re: Failed to start the agent
                    mazz

                     

                    Building jar: /export/home/nsyed/rhq/modules/plugins/platform/C:/Projects/rhq-trunk/dev-container/jbossas/server/default/deploy/rhq.ear/rhq-downloads/rhq-plugins/rhq-platform-plugin-1.2.0-SNAPSHOT.jar


                    That certainly doesn't look right. Clearly, something is misconfigured.

                    I suspect you need to override the location of your dev-container in the Maven ~/.m2/settings.xml. That C: location is the default I think, and obviously, doesn't work for you, being that you are on UNIX :) Usually, the build will work fine if you do not have a ~/.m2/settings.xml - but I assume you created a ~/.m2/settings.xml file based on the default one located in <rhq-trunk>/etc/m2. You need to adjust the rhq.containerDir property to the place where you want the dev-container to go.

                    See http://support.rhq-project.org/display/RHQ/Advanced+Build+Notes#AdvancedBuildNotes-{{settings.xml}}


                    • 6. Re: Failed to start the agent
                      mazz

                      You might also have to set the property rhq.rootDir in settings.xml too - its been a while and I can't remember when that is used - but its default value in etc/m2/settings.xml has C: so its possible this is where you got it from and may have to be changed to match your directory structure.

                      • 7. Re: Failed to start the agent
                        nsyed4

                        Ok. I configured the settings file with correct entries.
                        Built the Project again.
                        Started the server successfully.
                        Started the agent successfully except that the prompt looks like
                        >
                        instead of
                        sending>

                        These are the warnings and the errors that I received in the server log:
                        2009-02-05 15:51:07,593 WARN [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.recovery.xarecovery1] Local XARecoveryModule.xaRecovery got XA exception javax.transaction.xa.XAException, XAException.XAER_RMERR
                        2009-02-05 15:51:28,230 ERROR [org.rhq.enterprise.server.discovery.DiscoveryBossBean] Reported resource [Resource[id=0, type=SnmpTrapd, key=Trapd (tnglab1), n
                        ame=Trapd (tnglab1)]] has an unknown type [ResourceType[id=0, category=Server, name=SnmpTrapd, plugin=snmptrapd]]. The Agent most likely has a plugin named 's nmptrapd' installed that is not installed on the Server. Resource will be ignored...
                        2009-02-05 15:51:28,234 ERROR [org.rhq.enterprise.server.discovery.DiscoveryBossBean] Reported resource [Resource[id=0, type=SSHD, key=sshd, name=OpenSSH sshd
                        , version=1.0]] has an unknown type [ResourceType[id=0, category=Server, name=SSHD, plugin=OpenSSH]]. The Agent most likely has a plugin named 'OpenSSH' insta
                        lled that is not installed on the Server. Resource will be ignored...
                        2009-02-05 15:51:28,242 ERROR [org.rhq.enterprise.server.discovery.DiscoveryBossBean] Reported resource [Resource[id=0, type=Hosts, key=hosts, name=Hosts Serv
                        ice, version=1.0]] has an unknown type [ResourceType[id=0, category=Server, name=Hosts, plugin=Hosts]]. The Agent most likely has a plugin named 'Hosts' installed that is not installed on the Server. Resource will be ignored...
                        2009-02-05 15:53:17,670 WARN [com.arjuna.ats.jta.logging.loggerI18N]

                        (the same xaRecovery warning keeps on getting displayed.)


                        The server has the above mentioned plug-ins and the agent downloaded it from there.... still it says the plugin named is not installed on the server.

                        Is this an issue?

                        Thanks.




                        • 8. Re: Failed to start the agent
                          nsyed4

                          I think that is an issue because it is also ignoring my JMX resources -
                          2009-02-05 16:12:49,659 ERROR [org.rhq.enterprise.server.discovery.DiscoveryBossBean] Reported resource [Resource[id=0, type=JMX Server, key=10001, name=simplembean-1.0.0.jar (10001)]] has an unknown type [ResourceType[id=0, category=Server, name=JMX Server, plugin=JMX]]. The Agent most likely has a plugin named 'JMX' installed that is not installed on the Server. Resource will be ignored...

                          • 9. Re: Failed to start the agent
                            mazz

                            1. The prompt is OK. Read this: http://jira.rhq-project.org/browse/RHQ-718

                            2. Regarding "XAException.XAER_RMERR" - see http://jira.rhq-project.org/browse/RHQ-1368 for what you need to do in oracle

                            3. "Agent most likely has a plugin named 'ABC' installed that is not installed on the Server." -- your build sounds like its very screwed up at the moment. I suggest you "mvn -Pdev,enterprise clean" and then rebuild again with -Ddbsetup (to clean out your database and rebuild it).

                            • 10. Re: Failed to start the agent
                              nsyed4

                              I did a Project Clean and tried building the project with the following command:
                              mvn install -Penterprise,dev -Dmaven.test.skip=true -Ddbsetup


                              I get the following error:
                              java.sql.SQLException: No suitable driver found for jdbc:oracle:thin@:xyz01c:1521/xyz.world

                              I was receiving the above error in my earlier builds too, I eliminated it by writing a <user.home>/.m2/settings.xml file.

                              I am using ojdbc14.jar but http://jira.rhq-project.org/browse/RHQ-1208 suggested to use ojdbc5.jar for Java 6.

                              I have the driver in the following location:
                              /export/home/nsyed/.m2/repository/ojdbc14/ojdbc14/10.2.0.4.0/ojdbc14-10.2.0.4.0.jar

                              and this is how my pom.xml is:

                              <rhq.groupId>org.rhq</rhq.groupId>
                              <rhq.product.name>RHQ</rhq.product.name>
                              <rhq.earName>rhq.ear</rhq.earName>
                              <!-- the below path is relative to {rhq-trunk} -->
                              <rhq.defaultDevContainerPath>dev-container</rhq.defaultDevContainerPath>
                              <!-- set the below prop to true to enable misc extra build output -->
                              <rhq.debug>false</rhq.debug>
                              ...........................
                              <ojdbc14.version>10.2.0.4.0</ojdbc14.version>
                              ......
                              <!-- misc properties -->
                              <!-- <server.quartzDelegate>org.quartz.impl.jdbcjobstore.PostgreSQLDelegate</server.quartzDelegate> -->
                              <!-- defaults for datasource used by dbsetup, unit+integration tests, and dev container build - these may be overridden in ~/.m2/settings.xml -->
                              <rhq.test.ds.connection-url>jdbc:oracle:thin@:xyz01:1521/xyz.world</rhq.test.ds.connection-url>
                              <rhq.test.ds.driver-class>oracle.jdbc.driver.OracleDriver</rhq.test.ds.driver-class>
                              <rhq.test.ds.xa-datasource-class>oracle.jdbc.xa.client.OracleXADataSource</rhq.test.ds.xa-datasource-class>
                              <rhq.test.ds.user-name>jon2</rhq.test.ds.user-name>
                              <rhq.test.ds.password>jon2</rhq.test.ds.password>
                              <rhq.test.ds.type-mapping>Oracle10g</rhq.test.ds.type-mapping>
                              <rhq.test.ds.server-name>xyz01</rhq.test.ds.server-name>
                              <rhq.test.ds.port>1521</rhq.test.ds.port>
                              <rhq.test.ds.db-name>xyz.world</rhq.test.ds.db-name>
                              <!-- default port numbers for RHQ server -->
                              <rhq.server.http.port>7080</rhq.server.http.port>
                              <rhq.server.https.port>7443</rhq.server.https.port>
                              ...........................


                              oracle-driver


                              <!-- Oracle JDBC driver - required only if you want to run the tests, or dbsetup, against Oracle.
                              Add a top level dependency if this profile is active to allow for db-required tests in any module.
                              The driver can not be packaged for OpenSource distributions due to licensing issues, and as such,
                              this profile should not be active by default. -->

                              ojdbc14
                              ojdbc14
                              ${ojdbc14.version}
                              compile

                              <!--

                              com.oracle
                              ojdbc5
                              ${ojdbc5.version}
                              provided

                              -->


                              ......................................................


                              Do I still need to override the oracle
                              in the <rhq.home>/etc/m2/settings.xml

                              Please let me know if you want me to post it as another topic.














                              • 11. Re: Failed to start the agent
                                ips

                                To use Oracle w/ RHQ/Jopr, you need to do a few things:

                                1) Download the ojdbc14 or ojdbc5 jar from Oracle and stick it in your local repo or some other internal Maven repo that is configured in your settings.xml.
                                2) Enable the ojdbc-driver profile (using mvn -Pojdbc-driver or via settings.xml)
                                3) Set the necessary properties to configure the RHQ/Jopr builds to use Oracle rather than Postgres, e.g.:

                                 <activeProfiles>
                                 ...
                                 <activeProfile>oracle</activeProfile>
                                 </activeProfiles>
                                
                                 <profile>
                                 <id>oracle</id>
                                 <properties>
                                 <rhq.test.ds.connection-url>jdbc:oracle:thin:@127.0.0.1:1521:xe</rhq.test.ds.connection-url>
                                 <rhq.test.ds.user-name>rhqadmin</rhq.test.ds.user-name>
                                 <rhq.test.ds.password>rhqadmin</rhq.test.ds.password>
                                 <rhq.test.ds.type-mapping>Oracle10g</rhq.test.ds.type-mapping>
                                 <rhq.test.ds.driver-class>oracle.jdbc.driver.OracleDriver</rhq.test.ds.driver-class>
                                 <rhq.test.ds.xa-datasource-class>oracle.jdbc.xa.client.OracleXADataSource</rhq.test.ds.xa-datasource-class>
                                 <rhq.test.ds.server-name>127.0.0.1</rhq.test.ds.server-name>
                                 <rhq.test.ds.port>1521</rhq.test.ds.port>
                                 <rhq.test.ds.db-name>xe</rhq.test.ds.db-name>
                                 </properties>
                                 </profile>


                                4) Do a full clean build of RHQ, then Jopr.

                                Fyi, here is another forum post related to the ojdbc jar:
                                http://www.jboss.com/index.html?module=bb&op=viewtopic&t=145651

                                • 12. Re: Failed to start the agent
                                  nsyed4

                                  The driver issue is fixed. I figured out that I need to have ~/.m2/settings.xml
                                  This file is a copy of <rhq.home>/etc/m2/settings.xml

                                  I activated the profile through the settings file.