7 Replies Latest reply on Mar 10, 2009 8:44 AM by akarl

    2.2 Embedded Agent

    akarl

      I just installed the 2.2 beta and I am having trouble getting the embedded agent to connect to the instance. To be clear, I am running this in a VMWare instance of Ubuntu although I do not believe that to be the problem.

      I have used all of the installation defaults for the server other than enabling the embedded agent per the following configuration.

      rhq.server.embedded-agent.enabled=true
      rhq.server.embedded-agent.name=jopr-test-embedded
      rhq.server.embedded-agent.reset-configuration=true
      rhq.server.embedded-agent.disable-native-system=false


      The embedded-agent.log shows the following error message indicating that is was not able to connect to the server to register.

      I have not connected any other agents to the server although that is something I plan to try.

      07:51:24,654 INFO [Embedded RHQ Agent Main] (org.rhq.enterprise.communications.ServiceContainer)- {ServiceContainer.global-concurrency-limit-disabled}Global concurrency limit has been disabled - there is no limit to the number of incoming commands allowed
      07:51:26,227 INFO [Embedded RHQ Agent Main] (org.rhq.enterprise.communications.ServiceContainer)- {ServiceContainer.started}Service container started - ready to accept incoming commands
      07:51:26,228 INFO [Embedded RHQ Agent Main] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.no-auto-detect}Neither server auto-detection nor polling is enabled - the client will be allowed to start sending commands immediately, but when the server is offline, be prepared for alot of errors to be logged
      07:51:27,651 ERROR [RHQ Agent Registration Thread] (enterprise.communications.command.client.ClientCommandSenderTask)- {ClientCommandSenderTask.send-failed}Failed to send command [Command: type=[remotepojo]; cmd-in-response=[false]; config=[{rhq.agent-name=jopr-test-embedded, rhq.send-throttle=true}]; params=[{invocation=NameBasedInvocation[registerAgent], targetInterfaceName=org.rhq.core.clientapi.server.core.CoreServerService}]]. Cause: org.jboss.remoting.CannotConnectException:Can not connect http client invoker. -> java.net.ConnectException:Connection refused. Cause: org.jboss.remoting.CannotConnectException: Can not connect http client invoker.


      This error repeats as the agent retries.

        • 1. Re: 2.2 Embedded Agent
          mazz

          Go to the Administration > Manage Servers UI page and confirm that your public endpoint address for your server is routable (i.e. is there a DNS mapping from that address host to a valid, routable IP)

          • 2. Re: 2.2 Embedded Agent
            mazz

            That might not be able to help - because you are failing in the request to register the agent.

            Go into <jopr-install-dir>/jbossas/server/default/deploy/rhq-agent.sar/META-INF/embedded-agent/log4j.xml and change it to be debug for the root category:

            <root>
             <priority value="DEBUG"/>
             <appender-ref ref="FILE"/>
             <!-- <appender-ref ref="CONSOLE"/> -->
             </root>
            


            Restart your Jopr server and look at the logs for what IP the embedded agent is configured for (it should show up at the start of the embedded-agent.log file somewhere).

            • 3. Re: 2.2 Embedded Agent
              akarl

              I see an IP address only once. The rest of the time my dns name is being used which should be fine. I did check and ensure that my dns name (jopr-test.domainname) does point back to 127.0.0.1.

              The message below looks odd to me. Why is it using port 80? My server is set up to use port 7080. Now, I do know that when I first set up the server I attempted to use port 80 but then decided to change back to 7080 which I did via the server configuration file. Could this be something left over from that initial configuration attempt? If so, where is that config being stored so I can change it?

              10:48:04,707 DEBUG [Embedded RHQ Agent Main] (enterprise.communications.command.client.ClientCommandSender)- {ClientCommandSender.loading-command-spool}Loading [0] commands from the spool file and queueing them up to be sent to [remoting endpoint [servlet://jopr-test.nighthawkrad.net:80/jboss-remoting-servlet-invoker/ServerInvokerServlet]]
              10:48:04,708 DEBUG [Embedded RHQ Agent Main] (enterprise.communications.command.client.ClientCommandSender)- {ClientCommandSender.unspooled}[0] commands were unspooled and will be resent to [remoting endpoint [servlet://jopr-test.nighthawkrad.net:80/jboss-remoting-servlet-invoker/ServerInvokerServlet]]


              also see lots of urls such as in the following line but these look normal and correct.
              10:48:05,876 DEBUG [RHQ Agent Registration Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.agent-registration-attempt}Agent will now attempt to register with the server [AgentRegistrationRequest: [name=[jopr-test-embedded]; address=jopr-test.nighthawkrad.net]; port=26163]; remote-endpoint=socket://jopr-test.nighthawkrad.net:26163/?rhq.communications.connector.rhqtype=agent&numAcceptThreads=1&maxPoolSize=303&clientMaxPoolSize=304&socketTimeout=60000&enableTcpNoDelay=true&backlog=200]; regenerate-token=false]; original-token=<was null>]; agent-version=1.2.0.Beta1(3273)]]


              • 4. Re: 2.2 Embedded Agent
                mazz

                Agent configurations are persisted via Java Preferences API.

                There is alot of information on this - search the forums and docs (http://www.redhat.com/docs/en-US/JBoss_ON/)

                In short, if you are on Windows, SUN's Java Preferences implementation puts them in the Registry - on Linux, in ~/.java.

                I'm surprised "rhq.server.embedded-agent.reset-configuration=true" didn't do this, I haven't looked at the embedded agent much lately, can't remember how the configu would all work - plus, I'd need to know exactly what files you modified (the "80" versus "7080"). If you changed rhq-server.properties, you have to restart the server. To confirm your changes took effect by looking at the port/address info on that Administration>ManageServers page.

                • 5. Re: 2.2 Embedded Agent
                  akarl

                  I did in fact change the rhq-server.properties file and restarted. The http port changed appropriately however I just checked the manage servers page and it was still using the old address (80) was still there. I changed this then restarted. After a few minutes, I now have my embedded agent in the auto discovery list. So, if the intention is that changing the port in rhq-server.properties is all I should have needed to do, then there is a bug. Otherwise, you can chalk this up to my user error.

                  Thanks for the tip on where to turn on DEBUG level logging for the agent. That should be useful for users who screw up their configs like me.

                  • 6. Re: 2.2 Embedded Agent
                    mazz

                    Now that you have the embedded agent running, I would like you bring your attention to this caveat:

                    http://www.redhat.com/docs/en-US/JBoss_ON/html/Installation_Guide/JON_Agent_Guide-Running_the_JON_Agent-Running_Embedded_in_a_JON_Server.html

                    There is a problem trying to monitor the Jopr Server's EJB services from the embedded agent. If you want to monitor the Jopr Server (and specifically its EJB services), you need to use a standalone agent.

                    See this JIRA for more information why this is:

                    http://jira.rhq-project.org/browse/RHQ-1001

                    If you continue using the embedded agent, please keep us posted with your results. Is it working for you? If not, what isn't working?

                    • 7. Re: 2.2 Embedded Agent
                      akarl

                      I am seeing some of the monitoring data working for EJBs from the embedded agent but I can verify that RHQ-1001 still exists and is exactly as described. When using a non embedded agent, I do not get the ClassNotFoundExceptions noted.

                      I will note that the installation instructions for the new agent should include ensuring that write permissions exist for all of the agent subdirectories. For awhile, I sat at a prompt where the agent was supposedly downloading plugins. I found that I had not set write permissions on the plugins directory and that it was spitting errors into the log but not into the console. Just a minor usability/instructional thing.

                      I will continue to run both agents side by side for any differences.