14 Replies Latest reply on Apr 19, 2007 2:04 PM by jerrygauth

    JBCACHE-1025: JBossCache mbean registration fails on WebSphe

      Here's more information on this issue -

      At least two JBossCache customers have reported that JBC fails when the cache is started on WebSphere 6.1 due to an exception indicating that an attempt has been made to register an already registered mbean. This occurs on initial startup so it's not necessary to stop and restart the cache to trigger the problem.

      I've googled and found that similar WebSphere mbean registration problems have also been reported in forums for Spring and ActiveMQ. According to the ActiveMQ posting, the problem occurs because WebSphere modifies the registration name during the registration process (i.e., it appends some WebSphere goo). So code that only proceeds if an object isn't registered may fail since the registration is under a different name. Deregistration might also be affected but I'm not sure yet.

      I haven't confirmed the reported cause yet but I'm going to debug this issue on a WebSphere environment this afternoon. Assuming it's as reported, the most logical fix would be to retrieve the actual registration name during the registration process and then use this subsequently. This would require adding an instance variable to TreeCache to store the name as well as a getter/setter so that MBeanConfigurator can access the information. Alternative options would include modifying the registration method signature to pass the name or perhaps caching the information in MBeanConfigurator (ugh).

      The ActiveMQ posting suggests another alternative would be to use queryNames() instead of isRegistered() to determine whether an object is registered.

      I suppose the best alternative would be for WebSphere to modify their behavior. :)

      I'm planning to fix this initially for my employer as we're going to ship JBC 1.4.1 SPx with our next release for use on WebSphere. I can then apply whatever fix is appropriate to the JBC 1.4 code base. I know that the 2.0 code base is different so I'm not sure what would be appropriate there right now.

      I'll advise if my debugging turns up anything different but this is my current understanding of the problem and potential solutions.

        • 1. Re: JBCACHE-1025: JBossCache mbean registration fails on Web
          brian.stansberry

          In JBC 1.x, TreeCache inherits field serviceName from superclass ServiceMBeanSupport. That field stores the name the service was registered under when the JMX server calls ServiceMBeanSupport.preRegister(MBeanServer, ObjectName). And it seems MBeanConfigurator.registerInterceptors() is properly using that field. So it's odd that there is a mismatch.

          • 2. Re: JBCACHE-1025: JBossCache mbean registration fails on Web

            MBeanConfigurator is registering the cache's mbean using the object name that it retrieves as you noted. By default (e.g., no config file, no cluster), this name will be "jboss.cache:service=TreeCache-Group." If you retrieve the name of the registered object instance after registration, the returned name (on my server) is "jboss.cache:service=TreeCache-Group,cell=citgoNode01Cell,node=citgoNode01,process=server1"

            WebSphere appends the cell name, node name and server name to the object's registration name. It seems likely that you subsequently need to use this name if you want to determine whether an mbean has already been registered.

            The same issue applies to registration of the individual interceptors.

            • 3. Re: JBCACHE-1025: JBossCache mbean registration fails on Web

              It looks like WebSphere really mangles the registration names of interceptors by wrapping the interceptor name attribute within the WebSphere attributes.

              For example, when I register the TxInterceptor using the object name jboss.cache:service=TreeCache-Group,treecache-interceptor=TxInterceptor,
              the registration name in WebSphere becomes jboss.cache:service=TreeCache-Group,cell=citgoNode01Cell,treecache-interceptor=TxInterceptor,node=citgoNode01,process=server1


              • 4. Re: JBCACHE-1025: JBossCache mbean registration fails on Web
                brian.stansberry

                Where does ""jboss.cache:service=TreeCache-Group" come from? AFAICT the only place the serviceName field is set is in preRegister, and that should be the value WAS is passing in.

                Or is WAS passing in "jboss.cache:service=TreeCache-Group" to preRegister but actually registering it under something else? And then createService tries to register it as "jboss.cache:service=TreeCache-Group" and WAS barfs because the same object is registered under a different name?

                Re: the interceptors, the order of the attributes is not important; it's OK to mix them.

                • 5. Re: JBCACHE-1025: JBossCache mbean registration fails on Web

                  When a TreeCache instance is registered in JMX, the "cluster name" is used as the registration name. If the TreeCache has no cluster name (i.e., it's not clustered and the cluster name hasn't been explicitly set), a default name of TreeCache-Group is used.

                  Because of this, it appears that a best practice is to always provide a unique cluster name for a cache where statistics will be exposed. Otherwise the first cache instance to register itself with the default name will be the only one to have its statistics accessible via JMX.

                  If the cache has a "service name", then the service name is used for registration. If there is no service name, we concatenate the "jboss.cache:service=" prefix with the cache's cluster name.

                  I haven't looked closely at the origin of the service name but it seems likely that this name isn't available when running in non-JBoss environments. It's also presumably not available when running on JBoss without a service configuration file (e.g., when creating the cache in application code).

                  I'm not familiar with the preRegister process. Perhaps we should be using it when registering mbeans for which no service name has been provided?

                  I'll have to look closer at how the service name is provided for TreeCache.

                  re: the interceptors, my observation was essentially that it wouldn't be sufficient to store the cache's WAS registration suffix and simply append it to the interceptor registration names. It would be necessary to store the WAS registration names for each interceptor as well as for the cache itself. So it seems like we would need to maintain a map of registration names rather than simply a single variable. At least that's what seems necessary at first glance.

                  • 6. Re: JBCACHE-1025: JBossCache mbean registration fails on Web
                    brian.stansberry

                    OK, thanks for explaining where the name "jboss.cache:service=TreeCache-Group" is coming from. That tells me the when MBeanConfigurator.registerInterceptors() is invoked, preRegister() has *not* been called.

                    The preRegister() call is a standard JMX callback the MBeanServer is supposed to make when you register a bean that implements the callback interface. Nothing JBoss-specific about it. See javadoc for javax.management.MBeanRegistration.

                    So it sounds like you're deploying multiple caches and they are conflicting.

                    If these caches aren't meant to cluster with each other, they should have a unique ClusterName anyway.

                    Two possible approaches:

                    1) Add a TreeCache.setObjectName() and in TreeCache override ServiceMBeanSupport.getObjectName(Object name) to return it if available. Control the object name via configuration without depending on the JBoss deployer reading the "mbean" element's name attribute.

                    2) Programatically register the cache in JMX; don't count on registerInterceptors to do it. Use whatever name you like. Be sure you register it before calling createService().

                    • 7. Re: JBCACHE-1025: JBossCache mbean registration fails on Web

                      I've just discovered a related issue when JBossCache isn't configured as a JBoss service. This issue applies when running on JBossAS as well as other servers.

                      Suppose my cache's lifecycle is as follows.

                      TreeCache cache = new TreeCache();
                      cache.start();
                      ... // use the cache
                      cache.stop();
                      cache.destroy();

                      The start() method invocation will cause the cache's mbeans to be registered, assuming registration is enabled.

                      The destroy() method invocation should deregister the mbeans that were previously registered. However this isn't occurring when the cache is created as noted. When running on JBossAS, the following messages are emitted to the log and cache.destroyService() is never invoked so deregistration never occurs.

                      10:21:51,053 WARN [ServiceController] Ignoring request to stop nonexistent service: jboss.cache:service=TreeCache-Group
                      10:21:51,053 WARN [ServiceController] Ignoring request to destroy nonexistent service: jboss.cache:service=TreeCache-Group

                      It appears that the cache client must explicitly invoke cache.stopService() and cache.destroyService() to ensure that the cache is shut down properly.

                      • 8. Re: JBCACHE-1025: JBossCache mbean registration fails on Web

                        I've looked at preregistration and see that I can invoke TreeeCache.preregister() to preregister an mbean for which no JBoss service exists. Maybe this is something that should occur in MBeanConfigurator to ensure that any preregistration code is executed? Preregistration isn't necessary to obtain the actual object name used by the server since this can be obtain from the ObjectInstance returned by server.register().

                        I think the larger issue is that we need to store all of the actual registration names (for the cache and its interceptors) so that they can be used when checking for the existence of an mbean before registering it. I assume that deregistration requires the actual name as well.

                        Bottom line - I think we need to store a registration name map in cache
                        and provide a getter/setter so that the MBeanConfigurator methods can access it. Exposing the cache's object name is insufficient since it can't be used to derive the interceptors' object names on WebSphere.

                        • 9. Re: JBCACHE-1025: JBossCache mbean registration fails on Web
                          brian.stansberry

                          Re: the lifecycle, see http://www.jboss.com/index.html?module=bb&op=viewtopic&t=103608 for possibly relevant discussion.

                          You don't want to invoke preRegister(). That's the responsibility of the JMX server.

                          Re: maintaining a map of names, will this or something along these lines avoid that need?

                           if (registerCache)
                           {
                           ObjectName tmpObj = new ObjectName(tmpName);
                           if (!server.isRegistered(tmpObj))
                           server.registerMBean(cache, tmpObj);
                          
                           // ADDED: Use whatever name the cache is registered under as the
                           // base for the interceptor names
                           tmpName = cache.getServiceName().getCanonicalName();
                           }


                          • 10. Re: JBCACHE-1025: JBossCache mbean registration fails on Web

                            I don't think that will help. First, I think that cache.getServiceName() will return null if my cache instance isn't defined as a jboss service.

                            Maybe I can set the service name after registration for later use but the problem remains that each interceptor's registration name is modified by WebSphere such that I can't easily derive it from a service name.

                            For example,
                            jboss.cache:service=TreeCache-Cluster,treecache-interceptor=TxInterceptor,
                            the registration name in WebSphere becomes jboss.cache:service=TreeCache-Cluster,cell=citgoNode01Cell,treecache-interceptor=TxInterceptor,node=citgoNode01,process=server1

                            I suppose I could parse the bits that WebSphere adds, figure out which should be used as a prefix, which as a postfix, etc. but that seems cumbersome and probably susceptible to other implementation quirks. It seems like it would be safer to just store the actual object names and use them when needed.

                            This could probably be optimized by determining whether we need to do this (e.g., only when an mbean server mangles names) but this code only executes when the cache is started or destroyed so it might not be worthwhile to optimize it.

                            • 11. Re: JBCACHE-1025: JBossCache mbean registration fails on Web
                              brian.stansberry

                              If you call server.registerMBean(cache, tmpObj) and then a subsequent call to cache.getServiceName() returns null, then WAS's JMX impl is broken. The JMX server should invoke the preRegister() callback before it returns from server.registerMBean(). If it doesn't, it's broken. If it does invoke preRegister(), it should register the mbean under whatever name preRegister returns (which in the TreeCache case will be the name that's passed in as an arg to preRegister). If it registers it under some different name than what's returned by preRegister(), that's broken.

                              Another approach to the same thing while not counting on the preRegister() callback is to do this:

                              ObjectInstance oi = server.registerMBean(cache, tmpObj);
                              tmpName = oi.getObjectName().getCanonicalName().

                              Per the MBeanServer.registerMBean javadoc, the return value for that call is "An ObjectInstance, containing the ObjectName and the Java class name of the newly registered MBean.". If the WAS JMX doesn't even respect that, it's super broken.

                              • 12. Re: JBCACHE-1025: JBossCache mbean registration fails on Web

                                I don't know that cache.getServiceName() returns null on WebSphere. I wasn't familiar with the preregistration callback processing which I guess you're indicating should populate the name.

                                The ObjectInstance technique does work; that's how I've verified that WebSphere modifies the original name during the registration process.

                                I'll have to try getServiceName() and see what it returns. It will take me a while to try this as I don't have a personal install of WebSphere that I can use for testing.

                                • 13. Re: JBCACHE-1025: JBossCache mbean registration fails on Web
                                  brian.stansberry

                                  The ObjectInstance approach is cleaner anyway, so if you need to make mods, might as well do it that way.

                                  • 14. Re: JBCACHE-1025: JBossCache mbean registration fails on Web

                                    Good news. I was able to test this scenario on WebSphere today. It looks like the only thing we need to do in JBossCache is catch the registration error.

                                    From what I can see, WebSphere is doing the callback so cache.getServiceName() does return the actual registration name after the initial registration. This means that subsequent attempts to register the same instance will be detected. It also looks like deregistration of the cache mbean and interceptor mbeans works properly now (I need to verify again).

                                    So the typical failure mode is as follows -
                                    Someone in a standalone environment starts a cache instance using the default name (e.g., TreeCache-Group). On WebSphere, the instance is registered with a name like jboss.cache:service=TreeCache-Group, server=websphere....

                                    If the same instance tries to register again, we use the actual registration name and detect that this is an attempt to register again.

                                    If another instance (e.g., another standalone cache application) is then started and uses the default name, the code to detect a current registration won't detect that the name is already in use because it will use the default name (i.e., because serviceName is null at this point). The registration attempt will proceed and fail with the InstanceAlreadyExists exception because the actual name is already in use. So the fix is to simply catch the exception and report it as a warning.

                                    Note that this doesn't change the JMX availability of the cache's mbean. The mbean is registered to the first client to use the name.

                                    Unless I find anything further while looking at deregistration, I'll just fix this by handling the exception.