JOPR falsely reports JBossAS not available on 3 platforms

[dreedatcmrg]

I'm running a couple dozen identical Windows vBoxes with JBoss set up the same on all of them. JOPR's Auto-Discovery found all of them, but there are three where JBossAS shows up as not "available" (red) and no metric data is shown for JBossAS, although the file system, CPU, etc. displays okay (green) in JOPR for these platforms where JBossAS is reported as not available. JOPR's report must be erroneous, because my application is being served fine on all of these platforms.

How can I troubleshoot JOPR's reporting JBossAS as not available when it really is available?

Thanks,
Dave

[ips]

A common cause of JBossAS Resources being unavailable is that their JNP invoker is secured (i.e. requires a username and password). The JNP invoker is what the Jopr Agent connects to in order to remotely access the AS instance's JMX server.

http://www.jboss.org/community/docs/DOC-12189

describes how to secure the JMX invoker. This should allow you to figure out whether yours are secured or not. For a description of configuring usernames and passwords, see:

http://www.jboss.org/community/docs/DOC-12190

-Ian

[dreedatcmrg]

Thank you, Ian, for your quick and detailed response.

I found, however, that none of mine are secured. The jmx-invoker-service.xml is identical on the platforms where JBoss is reported as available and those where it is reported as not available. The out-of-the-box default xml file has not been modified in either case, so the snippet that would require authentication remains commented out in all cases.

Any other possible causes?

Thanks,
Dave

[mazz]

Best thing to do is look in your agent log file and see what its telling you. You might find some error messages.

If still nothing, turn on debug in the agent and see what the logs tell you.

To turn on debug in the agent (if you are running a new build from trunk) use the agent's "debug" prompt command. Or, start the agent with the env var set: RHQ_AGENT_DEBUG=true (this is the only way to do it with the Jopr GA distro). The Jopr FAQ tells you more info on agent debug.

[dreedatcmrg]

Thank you, Mazz.

Without turning on debug, I am getting an error message that JBossAS "refused" the connection, but I'm not sure what to do about it since it appears that JBoss is configured the same on the problem servers as on those that are reporting okay in JOPR:

2008-12-15 11:23:09,443 INFO [ResourceContainer.invoker.daemon-1] (org.rhq.plugins.jbossas.JBossASServerComponent)- Loading JBoss connection [jnp://172.25.1.178:1199] with install path [D:\jboss-4.0.3SP1]...
2008-12-15 11:23:11,740 WARN [ResourceContainer.invoker.daemon-1] (org.rhq.plugins.jbossas.JBossASServerComponent)- Could not establish connection to the JBoss AS instance [1] times for resource [D:\jboss-4.0.3SP1\server\ATG_DRP1]
org.mc4j.ems.connection.EmsConnectException: Could not connect [jnp://172.25.1.178:1199] javax.naming.CommunicationException [Root exception is java.rmi.ConnectException: Connection refused to host: cmatg6a; nested exception is:
java.net.ConnectException: Connection refused: connect]
at org.mc4j.ems.impl.jmx.connection.support.providers.AbstractConnectionProvider.connect(AbstractConnectionProvider.java:103)
at org.rhq.plugins.jbossas.JBossASServerComponent.loadConnection(JBossASServerComponent.java:1126)
at org.rhq.plugins.jbossas.JBossASServerComponent.start(JBossASServerComponent.java:250)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:450)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectException: Connection refused to host: cmatg6a; nested exception is:
java.net.ConnectException: Connection refused: connect]
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:707)
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:572)
at javax.naming.InitialContext.lookup(InitialContext.java:351)
at org.mc4j.ems.impl.jmx.connection.support.providers.JBossConnectionProvider.doConnect(JBossConnectionProvider.java:70)
at org.mc4j.ems.impl.jmx.connection.support.providers.AbstractConnectionProvider.connect(AbstractConnectionProvider.java:99)
... 11 more
Caused by: java.rmi.ConnectException: Connection refused to host: cmatg6a; nested exception is:
java.net.ConnectException: Connection refused: connect
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:574)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:185)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:171)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:94)
at org.jnp.server.NamingServer_Stub.lookup(Unknown Source)
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:610)
... 15 more
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at java.net.Socket.connect(Socket.java:469)
at java.net.Socket.(Socket.java:366)
at java.net.Socket.(Socket.java:179)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:569)
... 20 more

Any suggestions as to how to get JBoss to stop refusing the connection?

Thanks,
Dave

[ccrouch]

It looks like you could have a DNS resolution problem. Try using twiddle to connect to these rogue JBAS instances e.g. jnp://172.25.1.178:1199 or jnp://cmatg6a:1199. If twiddle can connect then Jopr should be able to too.

[mazz]

This looks like a generic connection problem.

Caused by: java.rmi.ConnectException: Connection refused to host: cmatg6a

Is that hostname DNS-resolvable? Is it resolving to the correct IP?

org.mc4j.ems.connection.EmsConnectException: Could not connect [jnp://172.25.1.178:1199]

Weird that the JNP uses an IP but the connection error is showing a hostname.

But, I think if you track down what's up with "cmatg6a" and "172.25.1.178", that's the source of the connection issues.

[dreedatcmrg]

Mazz, thank you!

I should have mentioned that all 16 servers in the clustered environment are dual-homed Windows boxes on VMWare. And that we have had some Windows networking hostname resolution issues on these virtual boxes.

I'm told now that some of the app servers in the cluster resolve their hostname lookups differently, and that it is a Windows OS issue. The three problem servers are among them, and a modification was made to these to force the 172.x.x.x IP address, but there may also be some other servers to which this modification was made but that work okay with JOPR.

When configuring the Agent, I assigned the Agent Name "cmatg6a" instead of the fully-qualified domain name which showed up as the default in the setup dialog. This may explain the "cmatg6a" hostname in the logs.

The dialog also defaulted to a 192.x.x.x address for these problem boxes, but I changed that to the 172.x.x.x address during configuration.

Since I always used the 172.x.x.x address when setting up and configuring JOPR server and JOPR agent, shouldn't it therefore use the 172.x.x.x for all purposes?

Thanks,
Dave

[mazz]

Well, there are two things here.

One is the agent connecting to the server and the server connecting to the agent. Those are the IP addresses you give at agent startup time.

This is different, obviously, from the IP addresses the Jopr plugin uses to talk to your managed JBossAS instances.

The Jopr plugin manages the JBossAS instances via the JNP port you configure for the JBossAS resources (you see these values in the Inventory tab's connection properties).

The agent's IP is different - that's the IP the server uses to talk to the agent. You also need to tell the agent what IP the server can be found at.

Unsure what errors are related to the server<->agent comm layer - but it looks like for your issue, the problem lies with the Jopr plugin talking to your managed JBossAS instances... because I see the Jopr plugin code in the stack trace:

at org.rhq.plugins.jbossas.JBossASServerComponent.loadConnection(JBossASServerComponent.java:1126)

[dreedatcmrg]

Thank you, Mazz. I'm so new to JOPR that I didn't realize there was a separate plug-in being used for JBossAS.

Under the Type: JBossAS Server (Server) Inventory tab > Connection Info I have Naming Provider URL: jnp://172.25.1.178:1199, and under Operations I have Binding Address: 172.25.1.178

I tried changing the Naming Provider URL to a 192.x.x.x address instead, but saw no change in the results. Is there a different approach I should try?

Thanks,
Dave