running JBoss EAP 4.3CP02 instance always shown as DOWN

[yanfengzhang816]

I just set up a Jopr 2.2 server and added several JBoss EAP 4.3 CP instances to it. Most of them looks fine on Jopr, but there is one running JBoss EAP instance that is always shown as DOWN in Jopr.

The configuration of the "DOWN" instance is exactly the same as another EAP instance which looks fine on the Jopr server. Also there is no error or warning message in the agent log file. And other resources (cpu/file/...) on the same box (a RHEL 5.2 server) all looks fine.

I also used avail command in the agent shell, the line for the EAP 4.3 CP02 is listed below:
+ Resource[id=500558, type=JBossAS Server, key=/home/jbossas/jboss/server/all, name=oasis-ejbc1 JBossEAP 4.3.0.GA_CP02 all (0.0.0.0:1099), parent=oasis-ejbc1.mybest.com, version=4.3.0.GA_CP02] (sync=SYNCHRONIZED, state=STARTED, avail=DOWN, sched=12/15)

I've re-installed the agent and restarted the JBoss EAP server several times, the JBoss EAP instance is always shown as down.

What could possiblly cause this problem? Any idea?

Thanks in advance.

[mazz]

Take a look at the agent log file and see if you can find any warning or error messages.

Its possible that this one EAP instance requires username/password credentials - the agent's jboss plugin needs to know those creds to monitor the EAP instance. Go to the GUI, traverse to that EAP resource and see if you get a yellow error box at the top that said you need to edit the connection properties.

If that isn't it, it might be that the EAP's binding address is different than what the plugin thinks it is and thus can't talk to it. Again, all you would need to do in this case is edit the connection properties in the GUI to tell the agent and its plugin how to connect to the EAP instance.

When in doubt, check agent.log and find out what the problem is via log messages.

[rafaelcba]

Hello Mazz!

I have many JBoss AS instances inventoried on Jon Server. Someone instances appear UP others as Down. I already verify these points you mentioned above (credentials for jboss connection and IP address jboss bind) it's ok.

On our env. the agent is bound at eth0 and JBoss instaces is bound on virtual IPs (eth0:1; eth0:2; etc). But I think no problem with this because on others platforms the same scenario works fine.

I don't know more what can I see. There aren't errors or warnings on logs (agent an server). Others resources appear ok for that platform, just JBoss AS appear Down.

I already restart the agent with -l option for cleanup it but doesn't work. The JBoss appear down yet.

Any idea?

[ips]

I'd suggest enabling DEBUG logging on the Agent (see http://jopr.org/confluence/display/JOPR2/FAQ#FAQ-HowdoIgetdebugmessagesfromtheJoprAgent%3F ), then run the 'avail' command from the Agent prompt to trigger an availability scan. Then take a look at your agent.log file and see if you can tell why the JBAS server's availability is being reported as DOWN.

-Ian

[mazz]

-l wouldn't help. The issue is the jboss plugin running in the agent is not able to communicate with the JBossAS instance, in all probability.

Go to the GUI's Inventory tab for your EAP resource that is down. Go to the "Connection" properties. what does the JNP URL say? Is it correct? Can you use the EAP's "bin/twiddle.sh" utility to talk to the EAP instance? (run twiddle on the same box where your agent and EAP instance are running). Like this (assume my JNP URL is jnp://127.0.0.1:1099 - yours might be different, look at the Inventory tab for your resource to see what the plugin will use - also, this shows that my EAP instance needs user/pass credentials, your may not or the creds may be different):

>twiddle.sh --server=jnp://127.0.0.1:1099 --user=admin --password=admin query *:*

This should return a list of all MBeans in your EAP instance. If it does not, then this would explain why the plugin cannot see that the instance is UP.

----

What you can try is to turn on agent debug, and then see if you get any other helpful debug messages in agent.log.

If you have the agent prompt command (i.e. running the agent in foreground and you have the agent prompt up), type "debug -f log4j-debug.xml" to enable debug mode. Or, if you want to restart the agent with debug already on, do this:

RHQ_AGENT_DEBUG=true
export RHQ_AGENT_DEBUG
rhq-agent.sh

(the above assumes you are on UNIX, if you are on Windows, its similar, just "set RHQ_AGENT_DEBUG=true" and run the agent).

After that, wait about 60 seconds (the agent will ask all its plugins to check the availability for all the resources every minute). Then, see what type of messages the plugin logs in the log file.

[rafaelcba]

Hello Ian,

Following your instructions after enable the DEBUG level on agent I get this:

On agent log:

2009-05-11 12:17:45,322 INFO [RHQ Agent Prompt Input Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.prompt-command-invoked}Prompt command invoked: [avail]
2009-05-11 12:17:45,323 DEBUG [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Running Availability Scan...
2009-05-11 12:17:45,331 DEBUG [ResourceContainer.invoker.daemon-1] (org.rhq.plugins.jbossas.JBossASServerComponent)- Availability check for JBAS resource with configPath [/java/servers/jboss/server/server-homologa-verdados-2] is trying to connect to a different running JBAS which is installed at [/java/servers/jboss-eap-4.2-INEP/server/server-homologa-verdados-2]. Returning AvailabilityType.DOWN for the former resource.
2009-05-11 12:17:45,340 DEBUG [ResourceContainer.invoker.daemon-1] (org.rhq.plugins.jbossas.JBossASServerComponent)- Availability check for JBAS resource with configPath [/java/servers/jboss/server/server-homologa-geral-1] is trying to connect to a different running JBAS which is installed at [/java/servers/jboss-eap-4.2-INEP/server/server-homologa-geral-1]. Returning AvailabilityType.DOWN for the former resource.
2009-05-11 12:17:45,344 DEBUG [InventoryManager.availability-1] (rhq.core.pc.inventory.InventoryManager)- Starting component for Resource[id=503667, type=Apache HTTP Server, key=/etc/httpd, name=inepvm57 Apache 2.2.3 (/etc/httpd/), parent=inepvm57.inep.gov.br, version=2.2.3](current state = null, new plugin config = false)...
2009-05-11 12:17:45,345 INFO [ResourceContainer.invoker.daemon-1] (org.rhq.plugins.apache.ApacheServerComponent)- Initializing server component for server [/etc/httpd]...

On agent console:

DOWN: [platform_name] JBossEAP 4.2.0.GA_CP01 server-homologa-verdados-2 (IP:1099)
UP: CPU 3
UP: GRUB Boot Loader
DOWN: sit0
UP: [platform_name] File System (local) /boot
UP: [platform_name] File System (local) /
UP: [platform_name] File System (remote) /mnt/pdfs
UP: Hosts Service
UP: CPU 2
DOWN: inepvm57 JBossEAP 4.2.0.GA_CP01 server-homologa-geral-1 (IP:1099)

No errors...

But jboss is UP and accepting connections on port 1099:

[root@inepvm57 tools]# jps -ml
16465 org.jboss.Main --configuration=server-homologa-verdados-2 --host=IP2
16647 sun.tools.jps.Jps -ml
16292 org.rhq.enterprise.agent.AgentMain
10276 org.jboss.Main --configuration=server-homologa-geral-1 --host=IP1


:(

[ips]

The below debug message explains why the Resource is returning DOWN:

2009-05-11 12:17:45,340 DEBUG [ResourceContainer.invoker.daemon-1] (org.rhq.plugins.jbossas.JBossASS
erverComponent)- Availability check for JBAS resource with configPath [/java/servers/jboss/server/se
rver-homologa-geral-1] is trying to connect to a different running JBAS which is installed at [/java
/servers/jboss-eap-4.2-INEP/server/server-homologa-geral-1]. Returning AvailabilityType.DOWN for the
former resource.

It looks like at the time the JBAS Resource was discovered, its configuration directory was detected as "/java/servers/jboss/server/server-homologa-geral-1", but the configuration path the server instance itself is reporting is "/java/servers/jboss-eap-4.2-INEP/server/server-homologa-geral-1". Jopr uses the JBAS config dir as the "resource key" that uniquely identifies a JBAS instance from other JBAS instances on the same platform. If the jboss-as plugin notices the key has changed for a JBAS Resource, it reports the Resource as DOWN, since it assumes it may no longer be talking to the same JBAS instance that was originally discovered. Unfortunately, Jopr currently does not support resource keys changing once a Resource has been inventoried. Therefore, in order to get things working again for this JBAS instance, you will need to uninventory the Resource via the JON GUI (go to the Browse Resources > Servers page). Then issue the "discovery -f" command from the Agent prompt so that the Agent will rediscover the JBAS instance. Go the Dashboard in the GUI and re-import the Resource into inventory. Hopefully, now its Resource key will be "/java/servers/jboss-eap-4.2-INEP/server/server-homologa-geral-1", and the Resource will report itself as UP.

[rafaelcba]

I did these actions (univentory and rediscover resources) but the JON Server don't update the JBAS keys. The JBAS appear as DOWN yet.

2009-05-11 12:17:45,340 DEBUG [ResourceContainer.invoker.daemon-1] (org.rhq.plugins.jbossas.JBossASS
erverComponent)- Availability check for JBAS resource with configPath [/java/servers/jboss/server/se
rver-homologa-geral-1] is trying to connect to a different running JBAS which is installed at [/java
/servers/jboss-eap-4.2-INEP/server/server-homologa-geral-1]. Returning AvailabilityType.DOWN for the
former resource.

The path /java/servers/jboss is just a symbolic link to
/java/servers/jboss-eap-4.2-INEP

If I remove this symbolic link and start JBoss with the real path (/java
/servers/jboss-eap-4.2-INEP) the agent discovery it as a new resource because the "key" is new and all works...

The resource's key is persisted on RHQ DB or on agent file system? I think the operation
Univentory from JON-Server don't remove the resource key because this action (univentory and discovery -f) don't work for me. There is one way to purge resource key when it is uninventoried? Thus jon could see the resource as new.

The interesting is this problem not occur on others platform that are clone this that has the problem. I don't understand this behavior in some platforms. The scenario is the same.

[yanfengzhang816]

"mazz" wrote:
Take a look at the agent log file and see if you can find any warning or error messages.

Its possible that this one EAP instance requires username/password credentials - the agent's jboss plugin needs to know those creds to monitor the EAP instance. Go to the GUI, traverse to that EAP resource and see if you get a yellow error box at the top that said you need to edit the connection properties.

If that isn't it, it might be that the EAP's binding address is different than what the plugin thinks it is and thus can't talk to it. Again, all you would need to do in this case is edit the connection properties in the GUI to tell the agent and its plugin how to connect to the EAP instance.

When in doubt, check agent.log and find out what the problem is via log messages.

Hi. Thanks for the response.

I checked the connection configuration and there is no problem. Also there is no warning/error message in logging.

I checked debug level logging and found the following :

2009-05-12 15:58:55,720 DEBUG [ResourceContainer.invoker.daemon-4] (org.rhq.plugins.jbossas.JBossASServerComponent)- Availability check for JBAS resource with configPath [/home/jbossas/jboss/server/all] is trying to connect to a different running JBAS which is installed at [/home/jbossas/jboss43cp02/server/all]. Returning AvailabilityType.DOWN for the former resource.

In my environment, I set up a symbol-link named jboss which is linked to directory jboss43cp02. And I use jboss/bin/run.sh to start jboss.

So how can I solve this problem? Any further suggestions?

Thanks

[yanfengzhang816]

Some supplemental info:

As I said, there are several JBoss EAP servers in my enironment. I use symbol-link on all these servers. Only one has the problem (always shown as DOWN), and the others looks fine.

Strange....

[mazz]

Sounds like the symbolic links are the source of the problem.

What does your Jboss EAP servers say is their "config" directory? When you start EAP up, you get INFO logs like this - see what the paths say for like server home and the like for your EAP server that is showing DOWN:

08:23:29,859 INFO [Server] Home Dir: C:\jboss\jboss-eap-4.3\jboss-as
08:23:29,859 INFO [Server] Home URL: file:/C:/jboss/jboss-eap-4.3/jboss-as/
08:23:29,859 INFO [Server] Patch URL: null
08:23:29,859 INFO [Server] Server Name: default
08:23:29,859 INFO [Server] Server Home Dir: C:\jboss\jboss-eap-4.3\jboss-as\server\default
08:23:29,859 INFO [Server] Server Home URL: file:/C:/jboss/jboss-eap-4.3/jboss-as/server/default/
08:23:29,859 INFO [Server] Server Log Dir: C:\jboss\jboss-eap-4.3\jboss-as\server\default\log
08:23:29,859 INFO [Server] Server Temp Dir: C:\jboss\jboss-eap-4.3\jboss-as\server\default\tmp

[rafaelcba]

Man it's true! Now this make sense...

On Jon-Server the JBAS instance has the key: "/java/servers/jboss/server/server-desenv-geral-20" with my symbolic link: "/java/servers/jboss/"

But on boot log during server init. the real path is used:

/java/servers/jboss-eap-4.2-INEP/server/server-desenv-geral-20

20:14:09,574 INFO [Server] Home Dir: /java/servers/jboss-eap-4.2-INEP
20:14:09,574 INFO [Server] Home URL: file:/java/servers/jboss-eap-4.2-INEP/
20:14:09,574 DEBUG [Server] Library URL: file:/java/servers/jboss-eap-4.2-INEP/lib/
20:14:09,575 INFO [Server] Patch URL: null
20:14:09,575 INFO [Server] Server Name: server-desenv-geral-20
20:14:09,575 INFO [Server] Server Home Dir: /java/servers/jboss-eap-4.2-INEP/server/server-desenv-geral-20
20:14:09,575 INFO [Server] Server Home URL: file:/java/servers/jboss-eap-4.2-INEP/server/server-desenv-geral-20/
20:14:09,575 INFO [Server] Server Log Dir: /java/servers/jboss-eap-4.2-INEP/server/server-desenv-geral-20/log
20:14:09,575 DEBUG [Server] Server Data Dir: /java/servers/jboss-eap-4.2-INEP/server/server-desenv-geral-20/data
20:14:09,575 INFO [Server] Server Temp Dir: /java/servers/jboss-eap-4.2-INEP/server/server-desenv-geral-20/tmp
20:14:09,575 DEBUG [Server] Server Config URL: file:/java/servers/jboss-eap-4.2-INEP/server/server-desenv-geral-20/conf/
20:14:09,575 DEBUG [Server] Server Library URL: file:/java/servers/jboss-eap-4.2-INEP/server/server-desenv-geral-20/lib/

20:14:10,103 DEBUG [ServerInfo] java.vm.specification.vendor: Sun Microsystems Inc.
20:14:10,109 DEBUG [ServerInfo] java.runtime.version: 1.6.0_13-b03
20:14:10,109 DEBUG [ServerInfo] user.name: root
20:14:10,109 DEBUG [ServerInfo] jboss.bind.address: host_ip
20:14:10,109 DEBUG [ServerInfo] jboss.home.dir: /java/servers/jboss-eap-4.2-INEP
20:14:10,109 DEBUG [ServerInfo] user.language: pt
20:14:10,109 DEBUG [ServerInfo] sun.boot.library.path: /java/jdk1.6.0_13/jre/lib/amd64
20:14:10,109 DEBUG [ServerInfo] jboss.home.url: file:/java/servers/jboss-eap-4.2-INEP/
20:14:10,109 DEBUG [ServerInfo] java.version: 1.6.0_13
20:14:10,109 DEBUG [ServerInfo] user.timezone: America/Sao_Paulo
20:14:10,109 DEBUG [ServerInfo] java.net.preferIPv4Stack: true
20:14:10,109 DEBUG [ServerInfo] jboss.server.home.dir: /java/servers/jboss-eap-4.2-INEP/server/server-desenv-geral-20

How can we resolve this issue? The problem occur in some platforms but not on others.

[yanfengzhang816]

"mazz" wrote:
Sounds like the symbolic links are the source of the problem.

What does your Jboss EAP servers say is their "config" directory? When you start EAP up, you get INFO logs like this - see what the paths say for like server home and the like for your EAP server that is showing DOWN:

08:23:29,859 INFO [Server] Home Dir: C:\jboss\jboss-eap-4.3\jboss-as
08:23:29,859 INFO [Server] Home URL: file:/C:/jboss/jboss-eap-4.3/jboss-as/
08:23:29,859 INFO [Server] Patch URL: null
08:23:29,859 INFO [Server] Server Name: default
08:23:29,859 INFO [Server] Server Home Dir: C:\jboss\jboss-eap-4.3\jboss-as\server\default
08:23:29,859 INFO [Server] Server Home URL: file:/C:/jboss/jboss-eap-4.3/jboss-as/server/default/
08:23:29,859 INFO [Server] Server Log Dir: C:\jboss\jboss-eap-4.3\jboss-as\server\default\log
08:23:29,859 INFO [Server] Server Temp Dir: C:\jboss\jboss-eap-4.3\jboss-as\server\default\tmp

hmmm, it is true!
The boot.log file of Jboss EAP said that the server home is the "real path", and I use symbol link to specify the sever home.
I deleted the symbol link and use the real path to start

[mazz]

We've had several reports of things like this.

See https://jira.jboss.org/jira/browse/JOPR-206.

We think it is fixed now. If you run out of the symlink directory, things should work. The JBossAS resource key assigned by Jopr should be the symlink directory but we can still work if you hvae multiple JBossAS servers running on the same box on different ports.