11 Replies Latest reply on Mar 8, 2008 1:25 AM by ron_sigal

    Duplicate mbean removal

    starksm64

      On shutdown of the jbossas server I'm seeing these spurious errors due to the Connector trying to unregister itself or some other mbean:

      13:00:10,081 INFO [TomcatDeployment] undeploy, ctxPath=/web-console, vfsUrl=management/console-mgr.sar/web-console.war
      13:00:10,092 ERROR [Connector] invalid Object Name
      javax.management.InstanceNotFoundException: jboss.remoting:service=invoker,transport= bisocket,host=succubus.starkinternational.com,port=4457,clientMaxPoolSize=50,clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper,dataType=jms,marshaller=org.jboss.jms.wireformat.JMSWireFormat,numberOfCallRetries=1,numberOfRetries=1,serverSocketClass=org.jboss.jms.server.remoting.ServerSocketWrapper,socket.check_connection=false,timeout=0,unmarshaller=org.jboss.jms.wireformat.JMSWireFormat is not registered.
       at org.jboss.mx.server.registry.BasicMBeanRegistry.get(BasicMBeanRegistry.java:529)
       at org.jboss.mx.server.MBeanServerImpl.unregisterMBean(MBeanServerImpl.java:383)
       at org.jboss.remoting.transport.Connector.stop(Connector.java:744)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668) at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:184)
       at $Proxy4.stop(Unknown Source)
       at org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:56)
       at org.jboss.system.microcontainer.ServiceControllerContextAction.uninstall(ServiceControllerContextAction.java:90)
       at org.jboss.dependency.plugins.AbstractControllerContextActions.uninstall(AbstractControllerContextActions.java:58)
       at org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:334)
       at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323)
       at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009)
       at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627)
       at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411)
       at org.jboss.system.ServiceController.doChange(ServiceController.java:659)
       at org.jboss.system.ServiceController.stop(ServiceController.java:481)
       at org.jboss.system.deployers.ServiceDeployer.stop(ServiceDeployer.java:156)
       at org.jboss.system.deployers.ServiceDeployer.undeploy(ServiceDeployer.java:136)
       at org.jboss.system.deployers.ServiceDeployer.undeploy(ServiceDeployer.java:46)
       at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.undeploy(AbstractSimpleRealDeployer.java:73)
       at org.jboss.deployers.plugins.deployers.DeployerWrapper.undeploy(DeployerWrapper.java:187)
       at org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:947)
       at org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:940)
       at org.jboss.deployers.plugins.deployers.DeployersImpl.uninstall(DeployersImpl.java:902)
       at org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:334)
       at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323)
       at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009)
       at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627)
       at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411)
       at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:420)
       at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:354)
       at org.jboss.deployers.plugins.main.MainDeployerImpl.shutdown(MainDeployerImpl.java:382)
       at org.jboss.system.server.profileservice.ProfileServiceBootstrap.shutdown(ProfileServiceBootstrap.java:151)
       at org.jboss.bootstrap.AbstractServerImpl.shutdownServer(AbstractServerImpl.java:482)
       at org.jboss.bootstrap.AbstractServerImpl$ShutdownHook.run(AbstractServerImpl.java:778)
      


      It should not be doing that. Registration is handled by the deployer layer. The Connector source is not in the jboss thirdparty repository (it should be as a jboss-remoting-sources.jar), so I can't check what is going on.


        • 1. Re: Duplicate mbean removal
          ron_sigal

          This is related to changes I made for JBREM-747 "org.jboss.remoting.transport.Connector should unregister server invoker from MBeanServer".

          Connector.init(), which is called from Connector.create(), used to have the code

           ObjectName objName = new ObjectName(invoker.getMBeanObjectName());
           if (!server.isRegistered(objName))
           {
           server.registerMBean(invoker, objName);
           invoker.setMBeanServer(server);
           }
          


          The problem was that when a Connector stops and restarts, it sees that it's already registered, doesn't pass the MBeanServer to the invoker, and the invoker is unable to see other MBeans. So I changed it to

           ObjectName objName = new ObjectName(invoker.getMBeanObjectName());
           if (!server.isRegistered(objName))
           {
           server.registerMBean(invoker, objName);
           }
           else
           {
           log.warn(objName + " is already registered with MBeanServer");
           }
           invoker.setMBeanServer(server);
          


          I also added

           if (server != null)
           {
           try
           {
           ObjectName objName = new ObjectName(invoker.getMBeanObjectName());
           server.unregisterMBean(objName);
           }
           catch (Exception e)
           {
           log.error("invalid Object Name", e);
           }
           }
          


          to Connector.stop().

          I'm confused. Judging from the original code, Connector clearly expects to register itself. And since Connector is registering itself, I thought it should also unregister itself.

          • 2. Re: Duplicate mbean removal
            starksm64

            Service lifecycle is independent of existence as an mbean. There still is not enough code here for me to understand what the invoker/connector relationship is. It looks like registration and the jboss service lifecycle are being mixed up here. The question is, what is actually creating the invoker mbean, the Connector?

            You can cycle an mbean service through start, stop, start and this should have no impact on whether the service exists as an mbean. The only time a service should be registering an mbean is if its acting as a container that fully owns the mbean; meaning its not exposed to the jboss service deployment layer.

            • 3. Re: Duplicate mbean removal
              ron_sigal

              Sorry, my explanation was off base.

              The Connector isn't registering itself - it's creating and registering an org.jboss.remoting.ServerInvoker (BisocketServerInvoker, in this case). As you say, the Connector is acting as a container for the ServerInvoker. The Connector appears in a *-service.xml file, and the ServerInvoker does not.

              So, registering and unregistering the ServerInvoker makes sense, I think. The question is, why is Connector.stop() not finding the ServerInvoker's object name? If, for some reason, there was a problem registering the ServerInvoker, the log should have the output of

               log.warn("Error registering invoker " + invoker + " with MBeanServer.", e);
              


              I was wondering if maybe JBossMessaging is programmatically stopping the Connector, leaving the ServerInvoker's object name unregistered, so that when the ServiceController stops the Connector, the object name isn't there. However, I see that the Connector is injected into two JBM MBeans (org.jboss.jms.server.ServerPeer and org.jboss.jms.server.connectionfactory.ConnectionFactory), and it doesn't look like either of those is doing it.

              • 4. Re: Duplicate mbean removal
                starksm64

                Ok, then need to figure out what is removing it. It must be something else trying to do a global cleanup of mbeans on shutdown.

                • 5. Re: Duplicate mbean removal
                  ron_sigal

                  By the way, I'm not sure what you mean by "the jboss thirdparty repository". There's a jboss-remoting-2.2.2.GA-src.tar.gz, for example, in http://repository.jboss.com/jboss/remoting/2.2.2.GA-brew/src/. Should it be somewhere else as well

                  Also, Remoting still lives in the CVS repository: cvs.forge.jboss.com/cvsroot/jboss/JBossRemoting .

                  • 6. Re: Duplicate mbean removal
                    ron_sigal

                    I told Tim and Clebert about this thread, just in case I missed something in JBossMessaging.

                    • 7. Re: Duplicate mbean removal
                      starksm64

                       

                      "ron.sigal@jboss.com" wrote:
                      By the way, I'm not sure what you mean by "the jboss thirdparty repository". There's a jboss-remoting-2.2.2.GA-src.tar.gz, for example, in http://repository.jboss.com/jboss/remoting/2.2.2.GA-brew/src/. Should it be somewhere else as well

                      Also, Remoting still lives in the CVS repository: cvs.forge.jboss.com/cvsroot/jboss/JBossRemoting .

                      That is not a location that is pulled down by the thirdparty mechanism, and the format is not usable in an eclipse sourcepath. There needs to be a zip/jar compatible archive that is declared in the component-info.xml as an archive, but not in the export list:

                       <component id="jboss/remoting"
                       licenseType="lgpl"
                       version="2.2.2.SP2"
                       projectHome="http://www.jboss.org/products/remoting"
                       description="a single API for most network based invocations and related service that uses pluggable transports and data marshallers">
                      
                       <artifact id="jboss-remoting.jar"/>
                       <artifact id="jboss-remoting-sources.jar"/>
                      
                       <import componentref="jboss/serialization">
                       <compatible version="1.0.0.GA"/>
                       <compatible version="1.0.1.GA"/>
                       <compatible version="1.0.2.GA"/>
                       <compatible version="1.0.3.GA"/>
                       <compatible version="1.0.3.GA-brew"/>
                       </import>
                       <export>
                       <include input="jboss-remoting.jar"/>
                       </export>
                       </component>
                      
                      </project>
                      

                      Looking at https://svn.jboss.org/repos/repository.jboss.org/jboss/remoting/2.2.2.SP2/lib/ I see a jboss-remoting-2_2_2_SP2.zip, but this is way too big to be just the source of the classes going into the jboss-remoting.jar. That is all the jboss-remoting-sources.jar should be.


                      • 8. Re: Duplicate mbean removal
                        ron_sigal

                        Ok, I added a jboss-remoting-sources.jar to https://svn.jboss.org/repos/repository.jboss.org/jboss/remoting/2.2.2.SP2/lib/ and updated component-info.xml.

                        Note that 2.2.2.SP2 is a directory I created manually, and I didn't realize it needed a source jar. But the jboss-remoting-src.jar's in the *-brew directories also include the whole Remoting directory. I'm not sure why that is, but I'll ask Fernando Nasser.

                        • 9. Re: Duplicate mbean removal
                          ron_sigal

                          The problem is related to the use of 0.0.0.0 as a bind address. See JIRA issue JBREM-909 "Connector.stop() cannot find invoker MBean when bind address is 0.0.0.0" (http://jira.jboss.com/jira/browse/JBREM-909).

                          • 10. Re: Duplicate mbean removal

                            I get the same error when shutting down
                            jboss4.3

                            "org.jboss.jms.wireformat.JMSWireFormat is not registered"
                            Is any fix for it available?

                            • 11. Re: Duplicate mbean removal
                              ron_sigal

                               

                              "snasto" wrote:
                              I get the same error when shutting down
                              jboss4.3

                              "org.jboss.jms.wireformat.JMSWireFormat is not registered"
                              Is any fix for it available?


                              That's not related to Remoting or the issue in this thread. Try the JBoss Messaging forum (http://www.jboss.com/index.html?module=bb&op=viewforum&f=238).