new JMX capabilities

[mazz]

I just found out that Byteman 1.4.0 will contain some features that allow rules to expose data over a JMX MBean.

I have some questions on this:

1) into what MBeanServer does Byteman install its MBeans? I assume the platform MBeanServer itself, (which would probably be ideal).

2) In order for remote management clients to access these JMX MBeans data, I assume we would have to pass in additional Java system properties to remote the MBeanServer (e.g.  for SUN VMs: com.sun.management.jmxremote - see http://download.oracle.com/javase/1.5.0/docs/guide/management/agent.html ) - in other words, I assume there is nothing built into Byteman that automatically remotes the JMX MBeans. Which leads me to my last question....

3) will the Submit Java client have an API to access these JMX MBeans over the normal Byteman protocol? (thus allowing a remote management client to access the JMX MBeans data without having to remote the MBeanServer in the Byteman agent (see question 2) and having to go over the remote JMX protocol).

[adinn]

Hi John,

Apologies for not getting back to you sooner. I've been busy with the day job (the JBossTS XTS project ;-).

Before answering your questions I'll just mention that the Byteman blog contains a new post showing the the JMX integration script getting loaded into JBoss AS and displaying JVM statistics in the JBoss JMX console.

John Mazzitelli wrote:

1) into what MBeanServer does Byteman install its MBeans? I assume the platform MBeanServer itself, (which would probably be ideal).

That's a very good question. The answer is it is very simple-minded  at the moment. Here's the relevant code from class JMXHelper whichgets called when the background thread which samples the counters starts running.

    /**
     * a getter called when the helper is activated which computes the mbean server to use
     */



    private static MBeanServer getMBeanServer()
    {
        ArrayList<MBeanServer> mbeanServers = MBeanServerFactory.findMBeanServer(null);
        MBeanServer mbeanServer;
        if (mbeanServers != null) {
            mbeanServer = mbeanServers.get(0);
        } else {
            mbeanServer = ManagementFactory.getPlatformMBeanServer();
        }
        if (mbeanServer == null) {
            mbeanServer = MBeanServerFactory.createMBeanServer();
        }


        return mbeanServer;
    }

As you can see, the answer is that the helper uses the first MBean it finds in the list returned by the server factory or, failing that the platform MBean server. As a last resort it will create one.

I am not sure this is the best strategy and would be happy to take advice from you or anyone else on a better one. Of course, this code is just an example. You can always redefine this method to do whatever you need and deploy your own helper jar.

John Mazzitelli wrote:

2) In order for remote management clients to access these JMX MBeans data, I assume we would have to pass in additional Java system properties to remote the MBeanServer (e.g.  for SUN VMs: com.sun.management.jmxremote - see http://download.oracle.com/javase/1.5.0/docs/guide/management/agent.html ) - in other words, I assume there is nothing built into Byteman that automatically remotes the JMX MBeans. Which leads me to my last question....

Yes, you need to do more in order to provide access to the MBean server. The sample script/helper are built just to show off the capabilities on the Byteman side. I hope it is clear that this provides enough leverage to support a very general monitoring capability via JMX. Of  course, JMX is not the only option for making the statistics available and I want to develop this in other directions, RHQ integration being an obvious way to go. IDE support for rule development and  management is another area that needs work.

The main motiive for implementing this example was as a 'proof of concept' to investigate the viability of  developing a production quality monitoring capability in JBoss AS based on Byteman. Byteman is pencilled in for inclusion in EAP 6 for a variety of reasonsd and this is definitely one of  them. The features which make Byteman attractive in ths respect are the ability to inject counting rule almost anywhere in JVM, app server or application code and the simplicity and speed with which rules can be loaded, adapted and then reloaded into a live system. There's a long way to go develop  this into a general purpose tool but I think this is a very promising start.

John Mazzitelli wrote:

3) will the Submit Java client have an API to access these JMX MBeans over the normal Byteman protocol? (thus allowing a remote management client to access the JMX MBeans data without having to remote the MBeanServer in the Byteman agent (see question 2) and having to go over the remote JMX protocol).

Well, as it stands the Submit class doesn't know anything about the JMXHelper. In fact, it doesn't yet even provide information about heloper classes in general like, say, identifying which ones are currently activated, which rules are installed for a given helper etc. I think it would be very useful if Submit were extended to provide a general capability which allowed information gathered by activated helper classes to be made available to external clients and conversely allowed external clients to be able to upload information to be communicated to a specfic helper class. One major reason for sticking with using Submit, or, at least, comunicating via the agent  listener socket even if we provide a different client, is that it only  creates a single access point which needs to be secured. Given how much Byteman is allowed to mess with yourt app and/or runtime  it  is has to be very easy to control precisely which clients are allowed to submit  requests to the agent or to helper classes installed by the agent.

We could just extend Submit with a very simple protocol where, say, it is willing merely to fetch a block of free-format text from an active helper class identified by name and return it to the client or to pass a block of configuration property settings provided by the client to the helper. We could provide a more complex protocol, say one which allows for targetted execution of specified helper class methods with some structured argument passing mechanism but I would prefer to experiment with something quite simple to start with. If  you have a use case and want either to develop a prototype or suggest requirements for me to develop something I am very happy to collaborate on this. I would also welcome suggestions from other Byteman users out there who are reading this.

[mazz]

> I am not sure this is the best strategy

IMO, that is not a good way to do it. Just picking the first MBeanServer you get back is nondeterministic (depending on when that Byteman gets executed, I may or may not have invoked code that created N MBeanServers) - and from a user's perspective, we need to be assured of what MBeanServer is being used to populate these MBeans (because JMX clients need to know and on the server side I may want to reuse an MBeanServer I'm already using - or I may want to put my own MBeans in the same MBeanServer used by Byteman).

At minimum, you should default to the platform MBeanServer as that is the one most people will want to use. You could have a org.jboss.byteman.jmx.domain-name sysprop that overrides that default by defining the domain name of the MBeanServer (JBoss/Remoting and the new Hibernate stuff does something very similar).  Example code could be:

   String mbeanServerDomainToLookFor = System.getProperty("org.jboss.byteman.jmx.domain-name");
   if (mbeanServerDomainToLookFor == null) {
      return ManagementFactory.getPlatformMBeanServer();
   } else {
      for (Iterator i = MBeanServerFactory.findMBeanServer(null).iterator(); i.hasNext();){
         MBeanServer server = (MBeanServer) i.next();
         if (server.getDefaultDomain().equals(mbeanServerDomainToLookFor)) {
            return server;
         }
      }
      return ManagementFactory.getPlatformMBeanServer(); // fallback to default platform MBeanServer
   }

> You can always redefine this method to do whatever you need and deploy your own helper jar.

It would be best for this to be out of box rather than requiring users to write custom Java code to override it. I would recommend changing the base code so it doesn't just pick the first MBeanServer it sees and be more configurable/deterministic about which MBeanServer gets used.

[adinn]

John Mazzitelli wrote:

. . .

It would be best for this to be out of box rather than requiring users to write custom Java code to override it. I would recommend changing the base code so it doesn't just pick the first MBeanServer it sees and be more configurable/deterministic about which MBeanServer gets used.

That looks good enough to me. I knew there was a reason you were granted commit rights :-)

Please go ahead and tweak the helper class in trunk. Maybe also add a documentation note to two XXXMBean scripts in sample/scripts.

I can raise the JIRA for you if  you like ;-)

[mazz]

Yeah, raise a JIRA and assign it to me. Should be easy to implement this.

[whitingjr]

Hi,
  I am running a java HSQLDB database and interested in the rate of commits and Byteman is ideally suited. I have modified the JMXHelper to bind the PeriodicStats bean so remote JMX clients can connect and retrieve MBean attribute values.

This is outside JBossAS and currently the server on the IronJacama project (it will later) does not run an RMI registry.  To use Byteman in an application running in a separate vm the following steps are needed. These steps are very similar to those in Andrews recent http://bytemanblog.blogspot.com/2010/10/collecting-and-displaying-runtime.html blog. There are several additional steps and these are highlighted with '*'

• * Start an rmiregistry on port 9999, the rmiregistry included with the jdk will do

# rmiregistry 9999 &

• * Load a rule to enable a connector server the rule sets a boolean flag, see example rule

            * [optionally set the rmi host and port in the body of the action if rmiregistry was started on a different port]

Example Byteman rule snippet

# this rule changes the default field member on JMXHelper with a new
# value of true

RULE enable-rmi-server
CLASS JMXHelper
METHOD <init>(Rule)
AT EXIT
IF TRUE
DO setRMIServerRequired(true)
# setRmiHost("blah")
# setRmiPort("1234")
ENDRULE

• * Load the following two rules, specific to my use case, similar to the blog post

HELPER org.jboss.byteman.sample.helper.JMXHelper

# this rule is triggered when the periodic helper thread starts
# it returns a KeyInfo object identifying the stats counters
# updated by rules in this rule set

# Commit stats rule.
RULE CommitStatistics
INTERFACE org.hsqldb.TransactionManager
METHOD commitTransaction(Session)
AT EXIT
IF TRUE
DO debug("transaction committed");
incrementCounter("transaction commits")
ENDRULE

RULE return-key-info
CLASS JMXHelper
METHOD keyInfo() 
BIND keyInfo : KeyInfo = new KeyInfo("JVM Statistics in a Dynamic MBean")
IF TRUE
DO keyInfo.addKey("transaction commits", KeyInfo.KEY_TYPE_RATE, "Tx per second");
   keyInfo.addKey("transaction commits", KeyInfo.KEY_TYPE_CUMULATIVE, "Total tx");
   RETURN keyInfo
ENDRULE

To make JMXHelper work the following changes were introduced. These are extra fields to hold values supporting the changes.

    public static boolean isRMIServerRequired = false;
    private JMXConnectorServer connectorServer;
    private String rmiHost = DEFAULT_RMI_HOST;
    private String rmiPort = DEFAULT_RMI_PORT;
    private final String jmxUrlPattern = "service:jmx:rmi:///jndi/rmi://%1$s:%2$s/jmxrmi";
 
When the JMXHelper is initialised the boolean field member is used to check if the MBean should be put in the rmiregistry. The rule above configures the JMXHelper to additionally create a JMXConnectorServer to handle connections via the rmiregistry. The first line is copied from the existing JMXHelper for context.

mbeanServer.registerMBean(this, ObjectName.getInstance("org.jboss.byteman.sample.jmx:type=PeriodicStats"));
if (isRMIServerRequired())
{
   JMXServiceURL url = new JMXServiceURL(String.format( JMX_URL, getRmiHost(), getRmiPort()));
   connectorServer = JMXConnectorServerFactory.newJMXConnectorServer(url, null, mbeanServer);
   connectorServer.start();
}

This allows a JMXClient to access the MBean and get attributes belonging to the bean. For example

String jmxUrl = String.format(jmxUrlPattern , jmxHost, jmxPort);
JMXConnector connector = JMXConnectorFactory.connect(new JMXServiceURL(jmxUrl));
if (null != this.connector)
{
   MBeanServerConnection serverConnection = this.connector.getMBeanServerConnection();
   ObjectName name = ObjectName.getInstance("org.jboss.byteman.sample.jmx:type=PeriodicStats");
   Object beanAttrValue = serverConnection.getAttribute(name, "transaction commits : rate"); 
   if (null != beanAttrValue)
   {
   // do something with the attribute value
   }
}

I'll send a SVN patch with the changes.

Regards,
Jeremy

[adinn]

Hi Jeremy,

Thanks for the patch. It's  good to have someone who actually knows something about MXBeans than I do plugging the gaps in my code.

Jeremy Whiting wrote:

# this rule changes the default field member on JMXHelper with a new

# value of true

RULE enable-rmi-server
CLASS JMXHelper
METHOD <init>
AT EXIT
IF TRUE
DO setRMIServerRequired(true)
# setRmiHost("blah")
# setRmiPort("1234")
ENDRULE

I don't think this is the best way to obtain the configuration values from  the script. The problem  is that the JMXHelper constructor gets called every time a rule which uses it as a Helper is triggered. That's true with any helper class. Built-in methods are instance methods of the helper class. So, the only way you can call a built-in method in a rule body is by having an instance of the helper available when executing the rule.  In fact, when you compile a rule to bytecode the helper instance actually belongs to a subclass of the helper class. This subclass is generated by the rule engine and the bytecode bytecode which implements the rule body belongs to a method of this subclass called execute0().

So, this will work because the rule will get triggered when the helper used used by the background thread is created. However, this rule  will also get triggered inside a trigger call for any other rule using JMXHelper. That's rather wasteful.

I think a better way to achieve what you want is to follow the pattern for methods samplePeriod() and keyInfo(). The background thread explicitly calls these methods during startup to  obtains the default period and the key set. It expects the rule script to provide AT ENTRY rules which return different values in order to configure somethign other than th e default value. You should provide methods whci return default settings for the properties you need and call them during initialisation, That will allow the the script to attach a rule whcih configures a different value.

So, in JMXHelper add the following private methods:

private boolean rmiServerRequired()  {
        return false;
    } 

private String rmiHost()  {
        return "localhost";
    }

private int rmiPort()  {
        return 1234;
    }

These methods specify the default values for use by the background thread.

Next, you need to make sure that when the background thread starts it calls these methods to obtain the default values. The current initialisation happens in method initialise, naturally

    private void initialise()
    {
        // enable triggering and then install the MBean
        setTriggering(true);
        setPeriodMillisecs(samplePeriod());
        setSampleSetSize(sampleSetSize());
        setKeyInfo(keyInfo());
        mbeanServer = getMBeanServer();
        mbeanServer.registerMBean(this, ObjectName.getInstance("org.jboss.byteman.sample.jmx:type=PeriodicStats"));
        . . .

You need to extend this method as follows:

    private void initialise()
    {
        // enable triggering and then install the MBean
        setTriggering(true);
        setPeriodMillisecs(samplePeriod());
        setSampleSetSize(sampleSetSize());
        setKeyInfo(keyInfo());
        mbeanServer = getMBeanServer();
        mbeanServer.registerMBean(this, ObjectName.getInstance("org.jboss.byteman.sample.jmx:type=PeriodicStats"));
        // if a script wants to instal an rmi server it can inject code into
        // method rmiServerRequired() which returns true
        boolean rmiServerRequired = rmiServerRequired();
        if (rmiServerRequired) {
            // if the script wants a different host or port from the default it can inject
            // code into rmiHost() or rmiPort() to return the host or port it wants
            String rmiHost = rmiHost();
            String rmiPort = Integer.toString(rmiPort());
            JMXServiceURL url = new JMXServiceURL(String.format( JMX_URL, rmiHost, rmiPort); 
            connectorServer = JMXConnectorServerFactory.newJMXConnectorServer(url, null, mbeanServer);
            connectorServer.start();
        }
        . . .

So, now you can add rules to your script  to override the implementations of  methods rmiServerRequired(), rmiHost() and rmiPort() as follows

HELPER org.jboss.byteman.sample.helper.JMXHelper
 . . .
# this rule is triggered by the periodic helper thread when it initialises
# it returns true to force the thread to start up an rmi server
RULE require rmi server
CLASS JMXHelper
METHOD rmiServerRequired()
IF TRUE
DO RETURN TRUE
ENDRULE

# this rule is triggered by the periodic helper thread if
# method rmiServerRequired() returns true
# it returns a string identifying the rmi host to use
# RULE specify rmi host
# CLASS JMXHelper
# METHOD rmiHost()
# IF TRUE
# DO RETURN <fill in a String hostname here>
# ENDRULE

# this rule is triggered by the periodic helper thread if
# method rmiServerRequired() returns true
# it returns an integer identifying the rmi port to use
# RULE specify rmi port
# CLASS JMXHelper
# METHOD rmiHost()
# IF TRUE
# DO RETURN <fill in an integer port number here>
# ENDRULE

The first rule returns true instead of false frommethod rmiServerRequired(), causing method initialise to satrt the connector server. If you leave the  next two rules commented out it will use the default host, localhost, and the default  port, 1234. You can uncomment and fill in the <...> elements wiht a suitable host name or port number you can use a  different host or  port.

[whitingjr]

Hi Andrew,
  Yes what you have described is more efficient and should be used rather than adding a rules on JMXHelper.

Now the helper allows adding the bean to an rmi service I forgot to add a shutdown hook.
The shutdown hook makes sure the bean is removed prior to the jvm stopping. We need this because when a later jvm using a JMXHelper the adding of the MBean to the rmi service will fail without the cleanup.
So I am updating the JMXHelper code to add a shutdown hook.

Jeremy