Custom plugin for WildFly 8 using jboss-as-7-jmx fails; ClassNotFoundException

[genman]

See also Bug 1099220: https://bugzilla.redhat.com/show_bug.cgi?id=1099220

I fixed the plugin to use the new remoting interface. But even if I use http-jmx-remoting, the plugin fails with:

2014-05-19 21:12:45,134 ERROR [Remoting "endpoint" I/O-1] (org.xnio.listener)- XNIO001007: A channel event listener threw an exception
java.lang.NoClassDefFoundError: Could not initialize class org.jboss.sasl.digest.DigestMD5Client
        at org.jboss.sasl.digest.DigestMD5ClientFactory.createSaslClient(DigestMD5ClientFactory.java:41)
        at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities$1.run(ClientConnectionOpenListener.java:428)
        at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities$1.run(ClientConnectionOpenListener.java:426)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:426)
        at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:242)
        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
        at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:196)
        at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:110)
        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
        at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092)
        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
        at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
        at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:87)
        at org.xnio.nio.WorkerThread.run(WorkerThread.java:539)

I'm guessing the answer is here: https://access.redhat.com/site/solutions/871183

But anyway, since I'm putting some effort getting this to work with WildFly 8, can you throw me a bone?

[mazz]

The discovery and connection class for the jboss-as-7 plugin is loading its classes using a hard-coded class path of <JBOSS_HOME>/bin/client/jboss-client.jar. If the JMX connection requires other classes or libraries that are not in the managed app server's jboss-client.jar file, the connection will fail. See:

https://github.com/rhq-project/rhq/blob/RHQ_4_10_0/modules/plugins/jboss-as-7-jmx/src/main/java/org/rhq/modules/plugins/jbossas7/jmx/ApplicationMBeansDiscoveryComponent.java#L259

[genman]

That's not the problem. It can load the client jar file. Have you tested with WildFly 8.1 yet?

The problem is the JMX remoting library (from the client jar) loads the the saas library from the AS plugin, which then can't find JBoss logging, since the saas library can't see jboss-client.jar.

I'm thinking just forget about loading the client-api.jar and just put the remoting dependency in there. It seems to me the right thing to do, although I guess if anything is loading the client.jar it should be the JBoss 7 plugin, not the jboss-as-7-jmx plugin.

[genman]

What I propose (changes)

1. Have the JBoss AS 7 plugin depend on the JMX plugin. Reason: Avoid having to add EMS classes to JBoss AS plugin again.
2. ...also have additional remoting jars and logging jars as needed by remoting.
3. Don't load jboss-client.jar in the client's plugin. Reason: Classloader issues. Can't conflict
4. Use 'mc4j.ems.UseContextClassLoader' setting in the rhq-jboss-as-7-jmx-util classes. Reason: Load the right classes.
5. Fix JMXRemotingConnectionProvider to actually use the context class loader. It can't find the http-jmx-remoting implementation from the JMX plugin. Requires changes to EMS library. Might be easier to just create a custom version of this class for the JMX plugin.

It is very ugly.

[tsegismont]

Hi Elias,

I started the jboss-as7-jmx project and as documented in the ApplicationMBeansDiscoveryComponent class, "It is a tool that that plugin developers may use while writing their own plugin to monitor their application MBeans". So, as you're supposed to write your own plugin, you can really do anything you want, even just copy and paste the jboss-as7-jmx code to make what fit you needs. That being said, let me try to answer your different questions.

The configurations I've tested actually involved AS7 or EAP6 standalone servers or host controllers co-located with the RHQ agent. That's why ApplicationMBeansDiscoveryComponent does not address the remote server use case.

For RHQ4.12, we can change the implementaiton of ApplicationMBeansDiscoveryComponent#discoverResources. The client JAR file setup code looks like this:

File clientJarFile = new File(serverPluginConfiguration.getHomeDir(), "bin" + File.separator + "client"
    + File.separator + "jboss-client.jar");
if (!clientJarFile.isFile()) {
    LOG.warn(clientJarFile + " does not exist.");
    return Collections.emptySet();
}
pluginConfig.setSimpleValue(CLIENT_JAR_LOCATION, clientJarFile.getAbsolutePath());

We could extract this part to a protected ApplicationMBeansDiscoveryComponent#getClientJarFile method. Plugin developers could then implement their own way to provide the client JAR file. How could they do this? One option could be to embed it in their custom plugin, another one to deploy it to the machines the RHQ agent runs on.

Before RHQ4.12 is released you can implement something similar in your own plugin.

I have not tested this tool with Wildfly. AS7 and EAP6 only have the "jmx-remoting" protocol that's why it's hardcoded. But this can be made configurable too.

Thomas

[genman]

Thomas, thanks for the library, I do like the approach, hence my willingness to contribute back. Yes, I can fork what's there, but what I'd like to see is if I can get the whole thing working in a cleaner way with the following use cases:

• JBossAS 7/EAP
• Wildfly 8
• And Wildfly 8/JBoss remotely...

In the case of WildFly, I don't really think there is a client jar file that I can simply load to get it to work. So exposing a method might be useful, but I think it'd be easier to fix the dependency structure slightly, and probably patch the EMS library (or just override that one class) so it all works in a natural way.

[tsegismont]

Elias Ross a écrit:

In the case of WildFly, I don't really think there is a client jar file that I can simply load to get it to work.

What do you mean? I just downloaded a Wildfly 8.1.0.CR2. Here's what I get:

[tsegismont@stetson client]$ pwd
/home/tsegismont/Temp/wildfly-8.1.0.CR2/bin/client
[tsegismont@stetson client]$ ls
jboss-cli-client.jar  jboss-client.jar  README-CLI-JCONSOLE.txt  README-EJB-JMS.txt
[tsegismont@stetson client]$ pwd
/home/tsegismont/Temp/wildfly-8.1.0.CR2/bin/client
[tsegismont@stetson client]$ ll
total 11076
-rw-r--r--. 1 tsegismont tsegismont 4425260 12 mai   13:35 jboss-cli-client.jar
-rw-r--r--. 1 tsegismont tsegismont 6902323 12 mai   13:35 jboss-client.jar
-rw-r--r--. 1 tsegismont tsegismont     915 12 mai   13:34 README-CLI-JCONSOLE.txt
-rw-r--r--. 1 tsegismont tsegismont    1546 12 mai   13:34 README-EJB-JMS.txt

[genman]

I'm not saying the client jar is missing, when you connect to JMX, it doesn't work properly because the same classes were being loaded from different jars. Sorry I wasn't being clear about that.

Here's what I come up with as what works for me. I kind of went back and forth on a few things, but I figured out a clean way to make it work cleanly:

https://github.com/genman/rhq/tree/BZ1099220

Note my changes require a patched EMS as well, here's the very trivial patch:

https://github.com/genman/ems/tree/P136

What it does it basically force the JMXConnectorFactory to load classes only from the client library. What was going on before wasn't really working because of delegation.

[tsegismont]

I'll try the mbeans-on-as7-plugin with Wildfly 8 collocated with the RHQ Agent and see how it goes.

[genman]

Sure, if you could look at my patch, it'd be nice. I think it's a lot cleaner solution than what's there anyway, even if the existing .jar works.

[tsegismont]

Hi Elias,

I figured out how to tackle the classloading issue in ApplicationMBeansDiscoveryComponent. The changes look like:

if (connectionSettings.getControlProperties() == null) {
    connectionSettings.setControlProperties(new Properties());
}
connectionSettings.getControlProperties().setProperty(ConnectionFactory.COPY_JARS_TO_TEMP, String.valueOf(TRUE));
connectionSettings.getControlProperties().setProperty(ConnectionFactory.JAR_TEMP_DIR, pluginTempDir);

if (connectionSettings.getAdvancedProperties() == null) {
    connectionSettings.setAdvancedProperties(new Properties());
}
connectionSettings.getAdvancedProperties().setProperty(ConnectionFactory.USE_CONTEXT_CLASSLOADER, String.valueOf(FALSE));

And:

private static class CustomConnectionTypeDescriptor extends JSR160ConnectionTypeDescriptor {
    private final File clientJarFile;

    public CustomConnectionTypeDescriptor(File clientJarFile) {
        this.clientJarFile = clientJarFile;
    }

    @Override
    public String[] getConnectionClasspathEntries() {
        return new String[] { clientJarFile.getName() };
    }

    @Override
    public boolean isUseChildFirstClassLoader() {
        return true;
    }
}

Note the addition of isUseChildFirstClassLoader method.

And we also need the changes you sent with respect to server type detection (different port and JMX service URL on Widlfly).

I won't be able to push this today but it will be in master very soon.

I just wanted to keep you informed. Thanks for putting the issue on the radar.

Cheers,

Thomas

[tsegismont]

The fix is in master. See Bug 1099220 – jbossas7/jmx/ApplicationMBeansDiscoveryComponent does not work with WildFly8

[raylite3]

Many thanks to both for investigating and fixing the issue.

Can we expect this fix (to ApplicationMBeansDiscoveryComponent) to be available in 4.12 ? And is there a date for 4.12 ?

[tsegismont]

The fix is in master so it's going to be in 4.12. As for the release date, I can't say more than before the end of summer, so it could be very soon or in a couple of months. I think pilhuhn will announce it as soon as we know for sure.