RHQ 4.9 Agent cannot discover running software.

[brandondash]

edit: Moved the questions/issues to the top. Bolded those items that are new since the last edit.

1. Apache Web Server and PostgreSQL can only be detected when they are run via the console window or when their services are run by the account that I'm logged in as.
2. All 4 of the above software packages are marked as unavailable (red exclamation) in the rhq-server interface.
3. The documentation states more than once that the bundled service wrapper only works with 32-bit Java, yet my rhq-server is running via a Windows service without visible problems; that is, I can log in to http://localhost:7080 and get to the console, as well as see the agent I have registered and its discovered resources. Is there some other functionality that breaks that isn't readily apparent?
4. PostgreSQL installed with all default values can be discovered. PostgreSQL installed with modified values cannot be discovered. What?

Tomcat and JBoss can only be discovered when run via command line. Under no circumstance can they be discovered when run as windows services. There is an outstanding bug against this.

Hi guys,

I’m trying to set up a local RHQ monitoring server on a self-contained test bed for my web server. The server hosts:

• 3 instances of Apache Tomcat 7.x
• 1 instance of JBoss EAP 6.1
• 1 instance of AWS
• 1 instance of PostgreSQL

I kept the JBoss service turned off to avoid port issues with RHQ-server (it would appear it has a hardcoded check for port 9999 use on install?)

After unzipping the rhq bundle and also the jboss native bundle for a properly working service, I installed the RHQ server (java 1.7.25 and postgres 9.2.x backing), storage, and agent. I installed with the flag that does not allow auto-start of the agent so I could configure the first run. Here is the console output from the initial agent run:

C:\rhq-agent\bin>rhq-agent.bat -L -u
RHQ 4.9.0 [bb4fa70] (Tue Sep 10 16:05:44 EDT 2013)
Answer the following questions to setup this RHQ Agent instance.
- After each prompt, a default value will appear in square brackets.
If you press the ENTER key without providing any value,
the new preference value will be set to that default value.
- If you wish to rely on the system internal default value and
not define any preference value, enter '!*'.
- If you wish to stop before finishing all the questions but still
retain those preferences you already set, enter '!+'.
- If you wish to cancel before finishing all the questions and revert
all preferences back to their original values, enter '!-'.
- If you need help for a particular preference, enter '!?'.

Agent Name [EA-APP.dhcp.saic.com] : EA-APP
Agent Hostname or IP Address [!*] : 127.0.0.1
Agent Port [16163] :
RHQ Server Hostname or IP Address [10.172.42.232] : 127.0.0.1
RHQ Server Port [7080] :
The setup has been completed for the preferences at node [/rhq-agent/default].

After all of that, I ran the command line ‘discovery’ function. It could only find 4 resources:

• - EA-APP (windows)
• - Cron (cron)
• - RHQ Agent (RHQAgent)
• - Samba Server (Samba)

Note that not only are none of my software pieces not showing up, not even the rhq-server is showing up. The postgres isn't showing up that the rhq-server needed just to install in the first place. It's quite vexing.

MORE INFO:

• If I run rhq-server via the console command, the agent can see it. If I run the rhq-server via windows service the agent cannot see it. I know the server is running because I can muck around in a browser via localhost:7080
• If I run a basic tomcat via console, the agent can see it. If I run any of my tomcats via service, the agent cannot see them. This is either a function of running as a service, or perhaps running by using CATALINA_HOME/BASE rather than the out-of-the-box configuration.
• Given the above two bullets, perhaps rhq-agent needs additional configuration to interact with software running via windows services?

Regards,

brandon-

[pathduck]

Hi Brandon.

RHQ should be able to discover these resources, but have you checked if the plugins for these resource types are enabled under Agent Plugins (Administration) ?

It is really strange it dos not discover the RHQ Server itself since it runs locally - it should be there along with the agent.

You could also find some info in the RHQ Agent or Server logs on why the resources are not added to discovery.

Caveat; I am not experienced with running RHQ on Windows so there might be some special things you need to do, so the forum regulars will probably be better help.

Stian

[brandondash]

Thanks for the response, Stian,

I double checked, and both the server and agent have the proper Plugins installed.

The more I troubleshoot it, the more it is obvious this is a windows-only issue. When running the Tomcat JVMs (for example) by hand, the agent can see them; however, when running  them via a windows service the agent cannot find them.

The PostgreSQL issue is something altogether different I think. When I install PostgreSQL with default values then the agent can find it. When I muck with the values (e.g. service account name) then it can't find it.

I'm dreading that I may have to reverse engineer these plugins to find out exactly what they look for when hunting for their respective processes.

[jayshaughnessy]

This sounds to me like a security issue.  Most likely the win service running Tomcat is logged on as the system default account, or another account that the RHQ agent has no permission to see.  Run the agent as administrator or otherwise ensure that the service runs as the same user as the agent,

[brandondash]

Hi Jay,

I've been running the agent via command line as administrator the whole time.

Following your advice:

• I ran the AWS service as a non-system account and the agent still failed to detect it.
• I ran the AWS service as the same account as  the agent and it DID detect it.

So that's progress after a fashion. Security prevents me from running any of these services as an administrator (for obvious reasons) so now the followup question becomes "How do I solve the security issue where the agent and service are being run as different accounts?"

regards,

brandon-

[jayshaughnessy]

I'm not exactly sure but are you definitely starting your command window with "Run as Administrator".  This is different than a command window for a user that is an Administrator.  By default the command window still does not start with admin privs.  Discovery is done by looking for processes.  If the agent does not have view access to the service process it can't be discovered (or managed).

[brandondash]

I am both running as a user who is part of the administrators group, and selecting "run as administrator" from the context menu.

I am currently investigating SDDL to see if I can change the security settings of the services in question, but I have no idea if this is the proper way forward.

[tsegismont]

Hi Brandon,

I think you hit Bug 1035348 - Tomcat servers not discovered on Windows when installed as a service. In this case, you should see some errors in the agent logs. This should be fixed in RHQ 4.10 (the patch is waiting approval from the JBoss Web team).

I'm not aware of such an issue with EAP6. Can you file a bug?

What did you change in Postgres config?

I don't know what AWS is.

Regards,

Thomas

[brandondash]

Hi Thomas,

I'll be happy to log a JBoss EAP bug. If there is a problem with Tomcat it stands to reason there would be a problem with JBoss being that JBoss uses a forked version of Tomcat code under the hood.

re:Postgres deltas

• Install location
• Data directory
• Service account name
• DBA name

AWS is Apache Web Server (httpd).

[tsegismont]

You don't have a problem with EAP, but a problem with RHQ, which is unable to discover the EAP6 server started as a Windows service. So unless I misunderstood, file a Bugzilla against RHQ, not EAP.

The Tomcat and EAP6 plugins issues are not related. The plugins are independent, and AS7 plugin does not rely on JBoss Web connectors to talk with the server to be managed.

[genman]

I'm not a Windows guy but there may a way out.

On UNIX, I run Agent as rhq user and JBoss/Tomcat and others as different users. Since Java prevents you from managing processes via JMX as a different user (out of the box), to get it to work I have to manually setup JMX ports for any Java process I want. See:

http://stackoverflow.com/questions/834581/remote-jmx-connection

Once this is done RHQ agent may or may not be able to discover and connect. If it can't discover, one way around is to manually add the servers you want monitored. For RHQ Server it would be basically doing a manual add of the JBoss 7/EAP or Tomcat process. It may take a bit of work to figure out the connection settings.