How to monitor a MBean on WebSphere

[rafaelri]

Hi all,

I am trying to monitor a custom MBean on WebSphere. I've followed the steps described at http://management-platform.blogspot.com/2008/11/monitoring-custom-jmx-mbeans-with-jopr.html but it seems like discovery is not working neither the manual addition of the service.
I've packaged the rhq-plugin.xml file inside a jar (named: rhq-gife-jmx-1.0.jar) and deployed it inside rhq-plugins folder. As I mentioned, it is loaded fine but does not show on resources tab.
I even tried to run a manual update and discovery on agent console but with no success.
I am pasting here the contents of my rhq-plugin.xml:

<?xml version="1.0" encoding="UTF-8" ?>

<plugin name="GifeJmx"
 displayName="Gife JMX Plugin"
 version="1.0"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns="urn:xmlns:rhq-plugin"
 xmlns:c="urn:xmlns:rhq-configuration">

 <depends plugin="JMX"/>

 <service name="GifeManagement"
 description="GifeManagement"
 discovery="org.rhq.plugins.jmx.MBeanResourceDiscoveryComponent"
 class="org.rhq.plugins.jmx.MBeanResourceComponent"
 supportsManualAdd="true">

 <runs-inside>
 <parent-resource-type name="JMX Server" plugin="JMX"/>
 </runs-inside>

 <plugin-configuration>
 <c:simple-property name="objectName" readOnly="true" default="gife:cell=%cell%,name=PontuacaoManagement,type=PontuacaoManagement,node=%node%,process=%process%"/>
 <c:simple-property name="nameTemplate" default="PontuacaoManagement on #{server}"/>
 <c:simple-property name="descriptionTemplate" default="PontuacaoManagement on #{server}"/>
 <c:simple-property name="cell" type="string" description="Cell of the PontuacaoManagement"/>
 <c:simple-property name="server" type="string" description="Server of the PontuacaoManagement"/>
 <c:simple-property name="node" type="string" description="Server of the PontuacaoManagement"/>
 </plugin-configuration>

 <operation name="atualizarUltimaPontuacao"
 displayName="atualizarUltimaPontuacao"
 description="atualizarUltimaPontuacao">
 <parameters>
 <c:simple-property name="chave" type="string" required="true"/>
 <c:simple-property name="fila" type="long" required="true"/>
 <c:simple-property name="pontuacao" type="integer" required="true"/>
 </parameters>
 </operation>
 <operation name="obterUltimaPontuacao"
 displayName="obterUltimaPontuacao"
 description="obterUltimaPontuacao">
 <parameters>
 <c:simple-property name="chave" type="string" required="true"/>
 <c:simple-property name="fila" type="long" required="true"/>
 </parameters>
 <results>
 <c:simple-property name="operationResult" type="integer"/>
 </results>
 </operation>

 <metric property="NumeroPontuacoes"
 displayName="NumeroPontuacoes"
 defaultOn="true"
 displayType="summary"
 defaultInterval="30000"
 description="Numero Aleatorio"/>
 </service>


</plugin>

Am I missing some configuration or some entry on the xml file for the plugin? I've opened the JMX plugin file and saw an entry related to the WebSphere template but I couldn't figure out what I am missing, but I really feel like I am missing something since I've not specified the JMX url.

best regards,
Rafael Ribeiro

[mazz]

You do not need to specify the JMX URL in the plugin descriptor because you are reusing the connection metadata info through the use of injection into the JMX plugin resource type heirarchy (that's what the <runs-inside> does).

The "JMX Server" server resource type provides the metadata for the connection URL.

Now, its probably the case that the JMX plugin canNOT automatically discovery WebLogic servers. I don't know this part of the code well, but I suspect that is what is happening (typically when people say, "my agent can't auto-discovery my running application", the answer is usually, "its because the plugin doesn't know how to auto-detect that kind of application")

But you should be able to manually add it. I see in your plugin metadata, you specify, "supportsManualAdd="true"" which is good - you are telling the system that this kind of resource can be added explicitly by the user (this feature is used for this exact use-case - when you know you have a resource that can be monitored, but the plugin has no way of auto-discovering it - you are essentially giving a hint to the agent plugin as to the location of the resource).

Watch the end of this demo:

http://support.rhq-project.org/display/RHQ/Plugins+-+Demos+-+Monitoring+Hibernate

and you will see that I manually add a JMX Server - you will see the drop down includes WebLogic - this information is predefined for you in the JMX plugin metadata...which you can see here:

http://svn.rhq-project.org/repos/rhq/trunk/modules/plugins/jmx/src/main/resources/META-INF/rhq-plugin.xml

That's why your plugin metadata doesn't need to specify it, because you are extending the JMX plugin and it already has the info.

So:

1) in the UI, traverse to your platform resource's Inventory tab
2) Select the drop down value "JMX Server" in the manual add option area
3) Continue on from there - add JMX Server, select WebLogic, blah blah

You may have to go into your new manually added resource's Inventory tab and configure ITS plugin configuration - especially if your application is requiring authentication credentials.

[mazz]

Just noticed you mentioned WebSphere, not WebLogic.

Unsure if the JMX plugin is configured to work with WebSphere or if the underlying EMS library can connect to a WebSphere JMX MBeanServer.

Take a look at the plugin descriptor and code for the JMX plugin and see.

Feel free to contribute to the RHQ project with WebSphere support if it isn't supported today :)

[rafaelri]

Mazz,

I am about to connect with my custom plugin. As soon as I get it running I'll post everything over here.
What I'll risk saying for now is that indeed there is already support and it seems to be working. ;)

[rafaelri]

First thing I had to do although it seems to me like a hack... I had to place ws_runtime jar inside lib folder of Jopr.... I'll try to find a more elegant solution later on.
Now it is throwing a

An unexpected error occurred in the Agent. Failed to add resource with type [GifeManagement] and parent resource id [500550]. - Cause: java.lang.NullPointerException:null

Before it was complaining about some missing WAS classes on classpath.

[rafaelri]

Full stacktrace for this NPE:

2008-12-04 16:09:40,585 ERROR [WorkerThread#0[10.202.70.173:1693]] (rhq.core.pc.inventory.InventoryManager)- Manual add failed for resource of type 'GifeManagement' and parent resource id [500550].
java.lang.NullPointerException
 at org.rhq.plugins.jmx.MBeanResourceDiscoveryComponent.performDiscovery(MBeanResourceDiscoveryComponent.java:129)
 at org.rhq.plugins.jmx.MBeanResourceDiscoveryComponent.discoverResources(MBeanResourceDiscoveryComponent.java:81)
 at org.rhq.core.pc.inventory.InventoryManager.manuallyAddResource(InventoryManager.java:394)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.rhq.enterprise.communications.command.impl.remotepojo.server.RemotePojoInvocationCommandService.execute(RemotePojoInvocationCommandService.java:184)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at com.sun.jmx.mbeanserver.StandardMetaDataImpl.invoke(StandardMetaDataImpl.java:414)
 at com.sun.jmx.mbeanserver.MetaDataImpl.invoke(MetaDataImpl.java:220)
 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:815)
 at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:784)
 at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201)
 at $Proxy1.execute(Unknown Source)
 at org.rhq.enterprise.communications.command.server.CommandProcessor.handleIncomingInvocationRequest(CommandProcessor.java:358)
 at org.rhq.enterprise.communications.command.server.CommandProcessor.invoke(CommandProcessor.java:253)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734)
 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:560)
 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:383)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:165)
2008-12-04 16:09:47,711 INFO [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Collection of measurements for [org.rhq.plugins.platform.WindowsPlatformComponent@129bc5b] took [3501ms]
2008-12-04 16:09:47,711 INFO [MeasurementManager.sender-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement collection for [5] metrics took 3501ms - sending report to Server...
2008-12-04 16:09:59,384 INFO [ResourceContainer.invoker.daemon-9] (org.mc4j.ems.connection.ConnectionFactory)- Connection library dependancy [AppServer/lib/*] not found in directory: c:\ibm\WebSphere\AppServer\plugins
2008-12-04 16:09:59,384 INFO [ResourceContainer.invoker.daemon-9] (org.mc4j.ems.connection.ConnectionFactory)- Connection library dependancy [AppServer/deploytool/itp/plugins/com.ibm.etools.jsse/ibmjsse.jar] not found in directory: c:\ibm\WebSphere\AppServer\plugins
2008-12-04 16:09:59,384 INFO [ResourceContainer.invoker.daemon-9] (org.mc4j.ems.connection.ConnectionFactory)- Connection library dependancy [AppServer/java/jre/lib/ext/mail.jar] not found in directory: c:\ibm\WebSphere\AppServer\plugins
2008-12-04 16:09:59,384 INFO [ResourceContainer.invoker.daemon-9] (org.mc4j.ems.connection.ConnectionFactory)- Connection library dependancy [AppServer/java/jre/lib/ibmcertpathprovider.jar] not found in directory: c:\ibm\WebSphere\AppServer\plugins
2008-12-04 16:09:59,384 INFO [ResourceContainer.invoker.daemon-9] (org.mc4j.ems.connection.ConnectionFactory)- Connection library dependancy [AppServer/java/jre/lib/ext/ibmjceprovider.jar] not found in directory: c:\ibm\WebSphere\AppServer\plugins
2008-12-04 16:09:59,384 INFO [ResourceContainer.invoker.daemon-9] (org.mc4j.ems.connection.ConnectionFactory)- Connection library dependancy [AppServer/deploytool/itp/plugins/org.apache.xerces_4.0.13/xercesImpl.jar] not found in directory: c:\ibm\WebSphere\AppServer\plugins
2008-12-04 16:09:59,384 INFO [ResourceContainer.invoker.daemon-9] (org.mc4j.ems.connection.ConnectionFactory)- Connection library dependancy [AppServer/deploytool/itp/plugins/org.apache.xerces_4.0.13/xmlParserAPIs.jar] not found in directory: c:\ibm\WebSphere\AppServer\plugins
2008-12-04 16:09:59,384 INFO [ResourceContainer.invoker.daemon-9] (org.mc4j.ems.connection.ConnectionFactory)- Discovered libraries in 0 ms

[mazz]

"rafaelri" wrote:
First thing I had to do although it seems to me like a hack... I had to place ws_runtime jar inside lib folder of Jopr.... I'll try to find a more elegant solution later on.
Now it is throwing a

An unexpected error occurred in the Agent. Failed to add resource with type [GifeManagement] and parent resource id [500550]. - Cause: java.lang.NullPointerException:null

Before it was complaining about some missing WAS classes on classpath.

This is why I added this new config setting to the JMX plugin:

http://jira.rhq-project.org/browse/RHQ-1094

This is in trunk. Now, you can just point to the client .jar file that is found in your managed product (analogous to the client jars you find in JBossAS deployments). I demonstrate this feature in the hibernate demo. Hibernate had the same issue - it uses this JMX plugin to get to the hibernate statistics MBean, but I need the hibernate.jar classes because Hibernate's Statistics class is only found there (and could be different depending on which hibernate I'm monitoring).

In the JBossAS case, the EMS libraries knows which client jars it needs - but I'm suspecting it doesn't know about which WebSphere ones it needs, so you have to tell it yourself. That's where my new configuration setting comes in.

[rafaelri]

I had to create a custom AbstractConnectionTypeDescriptor called Websphere61ConnectionTypeDescriptor. WebsphereConnectionTypeDescriptor that comes with MC4J uses old jar in its classpath. Another problem I faced was that I had to repackage rhq-jmx-plugin-1.1.0.GA.jar so every jar that was inside org-mc4j-ems-1.2.5.jar is now on lib folder of rhq-jmx-plugin-1.1.0.GA.jar.
After all those changes, it seems like it is connecting now but it is throwing another weird NPE.
Ahh... those changes to packaging were necessary to avoid NoClassDefFound Errors... since the classloader that loaded the MC4J jars were unable to see WAS classes. Actually, the one that loaded org-mc4j-ems-impl.jar that was oddly placed inside org-mc4j-ems-1.2.5.jar.

[rafaelri]

First thing I noticed... WebsphereConnectionProvider wasnt able of handling a secure WAS installation. So I did some changes to this class, follows the new code:

package org.mc4j.ems.impl.jmx.connection.support.providers;

import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.net.URI;
import java.util.Properties;
import javax.management.MBeanServer;
import javax.management.j2ee.Management;
import javax.naming.Context;
import javax.naming.NamingException;
import org.mc4j.ems.connection.settings.ConnectionSettings;
import org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy;

public class WebsphereConnectionProvider extends AbstractConnectionProvider
{

 public WebsphereConnectionProvider()
 {
 }

 protected void doConnect()
 throws Exception
 {
 ClassLoader contextClassLoader = Thread.currentThread().getContextClassLoader();
 try {
 Context ctx = null;
 Thread.currentThread().setContextClassLoader(getClass().getClassLoader());
 Class adminClientClass = Class.forName("com.ibm.websphere.management.AdminClient", true, getClass().getClassLoader());
 Class adminClientFactoryClass = Class.forName("com.ibm.websphere.management.AdminClientFactory");
 Properties props = new Properties();
 URI serverUrl = new URI(connectionSettings.getServerUrl());
 if((serverUrl.getScheme().equalsIgnoreCase("http")) || (serverUrl.getScheme().equalsIgnoreCase("https")))
 {
 System.setProperty("javax.net.debug", "ssl,handshake,data,trustmanager");
 System.setProperty("java.protocol.handler.pkgs", "com.sun.net.ssl.internal.www.protocol");
 props.put(getConstant(adminClientClass, "CONNECTOR_TYPE"), getConstant(adminClientClass, "CONNECTOR_TYPE_SOAP"));
 } else
 {
 props.put(getConstant(adminClientClass, "CONNECTOR_TYPE"), getConstant(adminClientClass, "CONNECTOR_TYPE_RMI"));
 }
 String username = connectionSettings.getPrincipal();
 String password = connectionSettings.getCredentials();
 boolean security = ((username != null) && (!"".equals(username)));
 props.setProperty(getConstant(adminClientClass, "CONNECTOR_SECURITY_ENABLED"), Boolean.toString(security));
 if (security) {
 props.put(getConstant(adminClientClass, "USERNAME"), username);
 props.put(getConstant(adminClientClass, "PASSWORD"), password);
 }
 props.put(getConstant(adminClientClass, "CONNECTOR_HOST"), serverUrl.getHost());
 props.put(getConstant(adminClientClass, "CONNECTOR_PORT"), String.valueOf(serverUrl.getPort()));
 Method createMethod = adminClientFactoryClass.getMethod("createAdminClient", new Class[] {
 java.util.Properties.class
 });
 Object adminClient = createMethod.invoke(null, new Object[] {
 props
 });
 statsProxy = new GenericMBeanServerProxy(adminClient);
 mbeanServer = statsProxy.buildServerProxy();
 } finally {
 Thread.currentThread().setContextClassLoader(contextClassLoader);
 }
 }

 public String getConstant(Class clazz, String name)
 throws Exception
 {
 Field field = clazz.getField(name);
 return (String)field.get(null);
 }

 private Management retrieveMEJB(Context ic)
 {
 try
 {
 Object objref = ic.lookup("ejb/mgmt/MEJB");
 return mejb;
 }
 catch(NamingException ne) { }
 catch(Exception ce) { }
 return null;
 }

 public MBeanServer getMBeanServer()
 {
 return mbeanServer;
 }

 public long getRoundTrips()
 {
 return statsProxy.getRoundTrips();
 }

 public long getFailures()
 {
 return statsProxy.getFailures();
 }

 protected GenericMBeanServerProxy statsProxy;
 protected MBeanServer mbeanServer;
 private Management mejb;
 private static final String MEJB_JNDI = "ejb/mgmt/MEJB";
}

But anyway... I guess (only a guess) jopr is not properly setting connectionSettings object.
connectionSettings.getPrincipal() always returns null.

Apart from this I had to place several classes on lib folder under agent install dir:

AppServer/lib/bootstrap.jar
AppServer/deploytool/itp/plugins/com.ibm.websphere.v61_6.1.200/ws_runtime.jar
AppServer/runtimes/com.ibm.ws.admin.client_6.1.0.jar
appserver/plugins/com.ibm.ws.security.crypto_6.1.0.jar

This solved NoClassDefFoundErrors (probably the new feature you cited could solve my issue) but for now I did like this.
Apart from this, I did (but I am not sure whether this is mandatory) a new ConnectionTypeDescriptor ...
follows the code as well:

package org.mc4j.ems.connection.support.metadata;

public class Websphere61ConnectionTypeDescriptor extends AbstractConnectionTypeDescriptor {

 public Websphere61ConnectionTypeDescriptor()
 {
 }

 public boolean isMEJBCompliant()
 {
 return true;
 }

 public String getDefaultServerUrl()
 {
 return "http://localhost:8880";
 }

 public String getDefaultPrincipal()
 {
 return "wasadmin";
 }

 public String getDefaultCredentials()
 {
 return "";
 }

 public String getDefaultJndiName()
 {
 return null;
 }

 public String getDefaultInitialContext()
 {
 return null;
 }

 public String getConnectionType()
 {
 return "WebSphere6.1";
 }

 public String getConnectionMessage()
 {
 return "Websphere 6.1 Connection";
 }

 public String[] getConnectionClasspathEntries()
 {
 return (new String[] {
 "AppServer/deploytool/itp/plugins/com.ibm.websphere.v61_6.1.200/ws_runtime.jar","AppServer/deploytool/itp/plugins/com.ibm.websphere.webservice.rt.v5.1.1_6.1.2.v200803090003/lib/commons-logging-api.jar"
 });
 }

 public String getConnectionNodeClassName()
 {
 return "org.mc4j.ems.impl.jmx.connection.support.providers.WebsphereConnectionProvider";
 }

 public String getDisplayName()
 {
 return "WebSphere 6.1";
 }

 public String getRecongnitionPath()
 {
 return "AppServer/bin/ProfileManagement/plugins/com.ibm.websphere.v61_6.1.200/ws_runtime.jar";
 }

}

Also changed rhq-plugin.xml for JMX plugin to add support for this new ConnectionTypeDescriptor.

I guess I am almost making it work...
But Mazz... do you have any records of any issue related to the principal and credentials not set on connectionSettings?

regards,

[mazz]

The only odd credentials problem is related when EMS uses the JBossAS JNP client code and when running two JBossAS servers on the same box but have different creds between them.

But this doesn't involve you because you aren't going to JBossAS.

You can submit bugs to the MC4J project if you are seeing things in EMS wrt WebSphere connectivity. But unsure how much time Greg would have to devote to fixing them (plus I don't think any of us here have access to a WebSphere installation). But if you have patches for the WebSphere connector code that are fully functional that work for you, I'm sure he would be happy to take those and merge them into the code base.

[rafaelri]

For sure I'll send this to MC4J later on.
But how can I open something like a bug report for this part related to jopr not properly setting the credentials and password?

[rafaelri]

For sure I'll send this to MC4J later on.
But how can I open something like a bug report for this part related to jopr not properly setting the credentials and password?

[mazz]

https://jira.jboss.org/jira/browse/JOPR

I know the JMX plugin properly sets the credentials for the JBossAS EMS connector - that's how we handle authenticated connections to JBossAS via JNP. So, unsure how that is different from the stuff you are seeing.

[mazz]

Actually, since this involves the JMX plugin, that's a plugin that comes from the upstream RHQ project, its JIRA is:

http://jira.rhq-project.org/

[rafaelri]

Mazz,

I had posted a message on MC4J forum but had no reply. I saw that Greg is usually here on Jopr forum, do you suggest any other way for me to contact him and provide the patch for MC4J EMS?

best regards