7 Replies Latest reply on Jul 12, 2002 12:58 PM by adrian.brock

    Remote MBean discovery, deployment in JMX spec 1.5

    rwvanderlaan

      Hello there,

      I have some questions regarding future directions of JBossMX. Soon the JMX 1.5 specification will be out for public review and will taeckle issues like discovery of remote MBeans, unique identification of remote MBeans, remote event handling, deployment and more.

      Does the JBossMX team have any knowledge about the impact of the new specification on the current JBossMX implementation?

      Our company wants to prototype a distributed system using JBossMX. It has a gateway topology with one central control node running full JBoss. Furthermore it has several processing nodes, like advanced motion controllers and number crunchers, these all using a lightweight JMX implementation like JBossMX.

      Because of the distributed design of the system we need to overcome some shortcomings of the current JMX spec. We need a publishment and discovery mechanism for remote MBeans. Therefore we will use Jini as the underlying discovery protocol. A special MBeanRegistry bean will be equiped for this task. Furthermore we need a remote MBean deployment/instantiation and a remote event notification mechanism. We could use the MBeanServer RMI connector for all this, but there are some issues regarding the
      design we need.

      In order to create transparency in the location awareness of MBeans, a MBean event subscriber should lookup the broadcaster MBean using its local MBeanServer. Then it should add itself as a listener to the broadcaster MBean. Using the RMI connector a MBean could register itself as listener (the RMI connector incorporates the handling of remote MBean events), but this way the MBean has to know on which remote MBeanServer the broadcaster resides. This violates our location transparency. Furthermore the RMI connector implements a simple model for remote event notification. We would like to have a lease based subscription model with certain retry strategies for remote exceptions etcetera. The RMI connector might only be useful for remote instantiation of MBeans on our processing nodes. This because during this scenario there is knowledge about the location of the remote MBeanServer (this either e.g. using hostname or a description on a higher meta level).

      My idea (and I remember that Rickard Oberg also went this direction) is to implement a MBean for remote discovery and publishment of MBeans. I would like to dynamically generate MBean proxies that will live on every connected node.
      Then I might move the whole remote event notification part to the proxies. For MBeans implementing the broadcaster interface I will generate special broadcaster proxies. These proxies will handle calls for event subscriptions. They will keep track of subscribed listeners and will remotely export an object that will be later used as a callback mechanism to inform the local listeners. The remoteref of the exported object will be passed to the backend implementation of the proxy. There all remoterefs are kept (for event multiplexing). On the backend a surrogate eventlistener is subscribed to the real broadcaster MBean implementation. When the broadcaster notifies the surrogate listener, the listener
      will pass the event to the earlier described callback handler (this in a thread decoupled way). The callback handler will inform every subcribed listener.

      These are all ideas. Hopefully there are people that like to discuss more on these topics. Maybe if the JBoss team already has knowledge about the contents of the next JMX specification, they can clear some things up.

      From 14-7 till 5-8 I will be unable to reply due to my holidays.

      Kind regards,

      Richard van der Laan / [L]uminis
      Email : richard.vanderlaan@luminis.nl

        • 1. Re: Remote MBean discovery, deployment in JMX spec 1.5

          > Does the JBossMX team have any knowledge about the impact of the new specification on
          > the current JBossMX implementation?

          Yes, I am participating in the JSR-160 EG and we will support it once the new spec becomes public.

          -- Juha

          • 2. Re: Remote MBean discovery, deployment in JMX spec 1.5

            > Because of the distributed design of the system we
            > need to overcome some shortcomings of the current JMX
            > spec. We need a publishment and discovery mechanism
            > for remote MBeans. Therefore we will use Jini as the
            > underlying discovery protocol. A special
            > MBeanRegistry bean will be equiped for this task.

            Yes, you can replace the default registry implementation in JBossMX to implement your own specific one.

            > In order to create transparency in the location
            > awareness of MBeans, a MBean event subscriber should
            > lookup the broadcaster MBean using its local
            > MBeanServer. Then it should add itself as a listener
            > to the broadcaster MBean.

            Ok.

            > Using the RMI connector a
            > MBean could register itself as listener (the RMI
            > connector incorporates the handling of remote MBean
            > events), but this way the MBean has to know on which
            > remote MBeanServer the broadcaster resides. This
            > violates our location transparency. Furthermore the
            > RMI connector implements a simple model for remote
            > event notification. We would like to have a lease
            > based subscription model with certain retry
            > strategies for remote exceptions etcetera. The RMI
            > connector might only be useful for remote
            > instantiation of MBeans on our processing nodes. This
            > because during this scenario there is knowledge about
            > the location of the remote MBeanServer (this either
            > e.g. using hostname or a description on a higher meta
            > level).

            Ok, yes it sounds like as if you need your own modified version of the RMI connector in this case as well to handle the remote notification registration.

            Anything you'd be willing to contribute back to our project there?


            > My idea (and I remember that Rickard Oberg also went
            > this direction) is to implement a MBean for remote
            > discovery and publishment of MBeans. I would like to
            > dynamically generate MBean proxies that will live on
            > every connected node.

            Yes, this sounds about right. You might be able to reuse the MBeanProxy in this case, however it still needs the remote mbean connectivity added to its "backend", ie. allow it to talk over the wire. This will probably become as a form of an interceptor (as part of adding the interceptors to the proxy implementation) but currently you will need to tweak something by yourself.


            > Then I might move the whole remote event notification
            > part to the proxies. For MBeans implementing the
            > broadcaster interface I will generate special
            > broadcaster proxies. These proxies will handle calls
            > for event subscriptions. They will keep track of
            > subscribed listeners and will remotely export an
            > object that will be later used as a callback
            > mechanism to inform the local listeners.

            Hmm, I wonder.. internally we already generate proxies that handle event subscriptions, would it be possible to add a possibility to export them. We don't currently support replacing these proxies (they're "hidden" inside the MBeanServer implementation). Would it make sense to allow them to be replaced as well? Wonder if Adrian is reading this, he was the last one to touch this part of the code, what does he think?


            -- Juha

            • 3. Re: Remote MBean discovery, deployment in JMX spec 1.5
              marc.fleury

              > Hello there,
              >
              > I have some questions regarding future directions of
              > JBossMX. Soon the JMX 1.5 specification will be out
              > for public review and will taeckle issues like
              > discovery of remote MBeans, unique identification of
              > remote MBeans, remote event handling, deployment and
              > more.

              The JMX spec speaks of client/server mode. I.e. the client can discover and talk (event included) with the server. JSR77 does this trivially with the MEJB and poolers. Trivial, works. What you are talking about is different you are talking about federation of JBossMX nodes, there is no provision in the spec to provide cluster views of objects, which is what you are really talking about.

              > it has several processing nodes, like advanced motion
              > controllers and number crunchers, these all using a
              > lightweight JMX implementation like JBossMX.

              good

              > Because of the distributed design of the system we
              > need to overcome some shortcomings of the current JMX
              > spec. We need a publishment and discovery mechanism
              > for remote MBeans. Therefore we will use Jini as the
              > underlying discovery protocol. A special

              hmmmm, have looked at JavaGroups and how we federate EJBs? Jini is too pretty for the system level approach. But it is taste at this point. I was thinking of a cluster solution where nodes register themselves (for automated clustering). I am planning on using JG but maybe jini will do the trick.

              > MBeanRegistry bean will be equiped for this task.
              > Furthermore we need a remote MBean
              > deployment/instantiation and a remote event
              > notification mechanism. We could use the MBeanServer
              > RMI connector for all this, but there are some issues
              > regarding the
              > design we need.

              again, if you are looking for low tech (which is good) the RMI connector with some pooling (no callback) will marvelously simplify and do the work. Beware of wanking your brain too much on fancy schmancy stuff.

              That being said,


              > discovery and publishment of MBeans. I would like to
              > dynamically generate MBean proxies that will live on
              > every connected node.

              Ok there is a lot of the plumbing in place already. If you look at the proxy factory today, we are generalizing it to be a proxy generator for clients of MBeans (not just EJBs). What this means is that you throw at us the interfaces and we use Dynamic Proxies to assemble a front one with client interceptor behind it. In that interceptor you can have an "event interceptor" that takes care of registration and notification. This will be generic and reusable over many mbeans type. Then all you need to give the Proxy Factory (again there already) is the MBean interface, the event registration interface, and the stack of interceptors on the client site (surrogate node in your case) and include an event interceptor. Finally you pass an invoker to the stack and that will propagate the call to whomever you want.

              > Then I might move the whole remote event notification
              > part to the proxies. For MBeans implementing the
              > broadcaster interface I will generate special
              > broadcaster proxies. These proxies will handle calls
              > for event subscriptions. They will keep track of
              > subscribed listeners and will remotely export an
              > object that will be later used as a callback
              > mechanism to inform the local listeners. The
              > remoteref of the exported object will be passed to
              > the backend implementation of the proxy. There all
              > remoterefs are kept (for event multiplexing). On the

              bla bla bla bla, at the JMX level we need bydirectional invokers. Right now the invokers are mostly one-way with no callback. If you want a callback I would start by writing that invoker. The invoker "demultiplexes" (did you hurt your tongue?) by simply taking the MBean name (what we do alrealy on all the invokers). However take a look at JavaGroups for multicast and group notification. It is already there, all we need is an invoker. Whether that invoker is tied to the proxy (as in standard Proxy Factory objects) or you add a level of MBean indirection (i.e. the proxy really is local) is up to you.

              We need the JMX level client proxy to include the client side interceptors that is a generalization for 4.0.

              >
              > From 14-7 till 5-8 I will be unable to reply due to
              > my holidays.
              >
              > Kind regards,
              >
              > Richard van der Laan / [L]uminis
              > Email : richard.vanderlaan@luminis.nl

              dude, 2 things.

              1- Contribute. If you come here asking all these questions, you are obviously seriously in it. This is clearly work that is going on in JBoss right now, core work, and your input is welcome. Write that code, give us that code. If you don't plan on giving the code, don't come asking too many questions in the future.

              2- Get consulting from us. Get Mr Linfors to play with you for a couple of weeks/a month. You are serious and you need someone to do it with you. Don't be alone, already you are here talking because you are alone aren't you? You want us to maintain that code for you through JBoss maintenance it is going to be hell otherwise. What that means is that I recommend you hire one of us at JBoss Group to help on this topic, maybe a month worth of time. It is not "get help" it is a "don't do it alone, do it right, get an XP pair". Free discussion here is fine but you will need to talk to people who understand what you are talking about and point the poopoo when we see it. That's Mr Lindfors and myself, warjort, etc meaning JBoss Group.

              We want to see this work in JBoss and we need something back.

              Take care,

              marcf

              • 4. Re: Remote MBean discovery, deployment in JMX spec 1.5

                > again, if you are looking for low tech (which is good) the
                > RMI connector with some pooling (no callback) will
                > marvelously simplify and do the work.

                Actually I think Richard correctly identified the problem with the current JMX RMI connector implementation (when used directly), that is it does require you to have some knowledge as to *where* you are registering your notification subscription to. He's suggested solution of creating proxies for federated MBeans would work quite well in this case (the underlying connector can still be the RMI connector or any other connector), where you could therefore program your MBeans to register notification subscription as if they were local.

                In other words, you often want an API layer on top of the connector API that hides these details from your client (location for instance). This can be a proxy (as is used in JBoss today), or it can be the client API defined by JSR-160 (ie. 160 separates the JMX client and connector APIs where the client API may contain the factory methods for generating proxies -- still under debate).

                -- Juha

                • 5. Re: Remote MBean discovery, deployment in JMX spec 1.5
                  marc.fleury


                  > implementation (when used directly), that is it does
                  > require you to have some knowledge as to *where* you
                  > are registering your notification subscription to.

                  yes and no

                  > He's suggested solution of creating proxies for
                  > federated MBeans would work quite well in this case
                  > (the underlying connector can still be the RMI
                  > connector or any other connector), where you could

                  the point here is that the RMI connector could do "pooling" ala JSR77 which is the simplest way the EG came up with "events". The real solution J2EE based would be queues obviously where you register a listener.

                  In other words, the mechanism of how you get the event (callback or pooling) is under that API. All I am saying is that the simplest is that pooling mechanism. If you think about it that is pretty much the way SNMP works actually. It depends on what kind of program you are running and if you want tighter "push" then JMS is one simple option.

                  What we are discussing is inter-node messaging possibly clustered. This is a generic solution that needs to exist at the node level, independently of this problem.

                  • 6. Re: Remote MBean discovery, deployment in JMX spec 1.5

                    Ok, I think you are saying "polling" rather than "pooling"... polling makes sense wrt notifications, I'm not sure what you would gain by pooling (which is why i'm assuming you mean the former).

                    Just clarifying here, I found that a little confusing in this discussion.

                    But even with a polling mechanism implemented at the connector, I don't see how you would get around the location transparency requirement. Ie. you still need an API on top of the connector to hide the details of *where* you are polling from (the suggested generated proxies in this case that are local to MBeans in the same server, or possibly the 160 client API).

                    The generated proxies (light-weight ones) would therefore lead to a federated JMX namespace between a group of MBean servers.

                    -- Juha

                    -- Juha

                    • 7. Re: Remote MBean discovery, deployment in JMX spec 1.5

                      "We don't currently support replacing these proxies (they're "hidden" inside the MBeanServer implementation). Would it make sense to allow them to be replaced as well? Wonder if Adrian is reading this, he was the last one to touch this part of the code, what does he think?"

                      It wouldn't be very portable between JMX implementations.

                      Regards,
                      Adrian