1 Reply Latest reply on Feb 5, 2008 1:04 PM by adrian.brock

    Callbacks and new classloader changes

    alesj

      I've changed the current MC artifacts in AS5_trunk instance with the one's from MC trunk, and I get a bunch of warnings (suppressed errors) at shutdown regarding the callbacks handling and classloader setting:

      23:36:28,187 WARN [AbstractKernelController] Cannot resolve callbacks.
      javax.management.InstanceNotFoundException: jboss.jca:service=JCAMetaDataRepository,name=DefaultJCAMetaDataRepository is not registered.
       at org.jboss.mx.server.registry.BasicMBeanRegistry.get(BasicMBeanRegistry.java:529)
       at org.jboss.mx.server.MBeanServerImpl.getClassLoaderFor(MBeanServerImpl.java:1062)
       at org.jboss.system.microcontainer.ServiceControllerContext.getClassLoader(ServiceControllerContext.java:167)
       at org.jboss.dependency.plugins.SecurityActions.setContextClassLoader(SecurityActions.java:46)
       at org.jboss.dependency.plugins.AbstractController.resolveCallbacks(AbstractController.java:1174)
       at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1006)
       at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:936)
       at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:463)
       at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:426)
       at org.jboss.system.ServiceController.shutdown(ServiceController.java:573)
       at org.jboss.system.server.jmx.JMXKernel.shutdownServices(JMXKernel.java:370)
       at org.jboss.system.server.jmx.JMXKernel.stop(JMXKernel.java:218)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:56)
       at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:110)
       at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)
       at org.jboss.kernel.plugins.dependency.KernelControllerContextAction$JoinpointDispatchWrapper.execute(KernelControllerContextAction.java:213)
       at org.jboss.kernel.plugins.dependency.ExecutionWrapper.execute(ExecutionWrapper.java:45)
       at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchExecutionWrapper(KernelControllerContextAction.java:108)
       at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchJoinPoint(KernelControllerContextAction.java:69)
       at org.jboss.kernel.plugins.dependency.LifecycleAction.uninstallActionInternal(LifecycleAction.java:249)
       at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.uninstallAction(KernelControllerContextAction.java:166)
       at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.uninstallAction(KernelControllerContextAction.java:46)
       at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleUninstallAction(SimpleControllerContextAction.java:79)
       at org.jboss.dependency.plugins.action.AccessControllerContextAction.uninstall(AccessControllerContextAction.java:131)
       at org.jboss.dependency.plugins.AbstractControllerContextActions.uninstall(AbstractControllerContextActions.java:58)
       at org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333)
       at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1330)
       at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009)
       at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:936)
       at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:463)
       at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:426)
       at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.undeployBean(AbstractKernelDeployer.java:384)
       at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.undeployBeans(AbstractKernelDeployer.java:363)
       at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.undeploy(AbstractKernelDeployer.java:162)
       at org.jboss.kernel.plugins.deployment.BasicKernelDeployer.undeploy(BasicKernelDeployer.java:88)
       at org.jboss.kernel.plugins.deployment.xml.BasicXMLDeployer.undeploy(BasicXMLDeployer.java:95)
       at org.jboss.kernel.plugins.deployment.BasicKernelDeployer.shutdown(BasicKernelDeployer.java:102)
       at org.jboss.bootstrap.microcontainer.ServerImpl.doShutdown(ServerImpl.java:135)
       at org.jboss.bootstrap.AbstractServerImpl.shutdownServer(AbstractServerImpl.java:520)
       at org.jboss.bootstrap.AbstractServerImpl$ShutdownHook.run(AbstractServerImpl.java:824)
      

      23:36:28,828 WARN [AbstractKernelController] Cannot resolve callbacks.
      javax.management.InstanceNotFoundException: jboss.aop:service=AspectManager is not registered.
       at org.jboss.mx.server.registry.BasicMBeanRegistry.get(BasicMBeanRegistry.java:529)
       at org.jboss.mx.server.MBeanServerImpl.getClassLoaderFor(MBeanServerImpl.java:1062)
       at org.jboss.system.microcontainer.ServiceControllerContext.getClassLoader(ServiceControllerContext.java:167)
       at org.jboss.dependency.plugins.SecurityActions.setContextClassLoader(SecurityActions.java:46)
       at org.jboss.dependency.plugins.AbstractController.resolveCallbacks(AbstractController.java:1174)
       at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1006)
       at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:936)
       at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:463)
       at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:426)
       at org.jboss.system.ServiceController.shutdown(ServiceController.java:573)
       at org.jboss.system.server.jmx.JMXKernel.shutdownServices(JMXKernel.java:370)
       at org.jboss.system.server.jmx.JMXKernel.stop(JMXKernel.java:218)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:56)
       at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:110)
       at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)
       at org.jboss.kernel.plugins.dependency.KernelControllerContextAction$JoinpointDispatchWrapper.execute(KernelControllerContextAction.java:213)
       at org.jboss.kernel.plugins.dependency.ExecutionWrapper.execute(ExecutionWrapper.java:45)
       at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchExecutionWrapper(KernelControllerContextAction.java:108)
       at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchJoinPoint(KernelControllerContextAction.java:69)
       at org.jboss.kernel.plugins.dependency.LifecycleAction.uninstallActionInternal(LifecycleAction.java:249)
       at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.uninstallAction(KernelControllerContextAction.java:166)
       at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.uninstallAction(KernelControllerContextAction.java:46)
       at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleUninstallAction(SimpleControllerContextAction.java:79)
       at org.jboss.dependency.plugins.action.AccessControllerContextAction.uninstall(AccessControllerContextAction.java:131)
       at org.jboss.dependency.plugins.AbstractControllerContextActions.uninstall(AbstractControllerContextActions.java:58)
       at org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333)
       at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1330)
       at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009)
       at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:936)
       at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:463)
       at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:426)
       at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.undeployBean(AbstractKernelDeployer.java:384)
       at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.undeployBeans(AbstractKernelDeployer.java:363)
       at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.undeploy(AbstractKernelDeployer.java:162)
       at org.jboss.kernel.plugins.deployment.BasicKernelDeployer.undeploy(BasicKernelDeployer.java:88)
       at org.jboss.kernel.plugins.deployment.xml.BasicXMLDeployer.undeploy(BasicXMLDeployer.java:95)
       at org.jboss.kernel.plugins.deployment.BasicKernelDeployer.shutdown(BasicKernelDeployer.java:102)
       at org.jboss.bootstrap.microcontainer.ServerImpl.doShutdown(ServerImpl.java:135)
       at org.jboss.bootstrap.AbstractServerImpl.shutdownServer(AbstractServerImpl.java:520)
       at org.jboss.bootstrap.AbstractServerImpl$ShutdownHook.run(AbstractServerImpl.java:824)
      

      23:36:30,015 WARN [AbstractKernelController] Cannot resolve callbacks.
      java.lang.IllegalStateException: MBeanServer not available.
       at org.jboss.system.microcontainer.ServiceControllerContext.getMBeanServer(ServiceControllerContext.java:122)
       at org.jboss.system.microcontainer.ServiceControllerContext.getClassLoader(ServiceControllerContext.java:167)
       at org.jboss.dependency.plugins.SecurityActions.setContextClassLoader(SecurityActions.java:46)
       at org.jboss.dependency.plugins.AbstractController.resolveCallbacks(AbstractController.java:1174)
       at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1006)
       at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:936)
       at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:463)
       at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:426)
       at org.jboss.system.ServiceController.shutdown(ServiceController.java:579)
       at org.jboss.system.server.jmx.JMXKernel.shutdownServices(JMXKernel.java:370)
       at org.jboss.system.server.jmx.JMXKernel.stop(JMXKernel.java:218)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:56)
       at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:110)
       at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)
       at org.jboss.kernel.plugins.dependency.KernelControllerContextAction$JoinpointDispatchWrapper.execute(KernelControllerContextAction.java:213)
       at org.jboss.kernel.plugins.dependency.ExecutionWrapper.execute(ExecutionWrapper.java:45)
       at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchExecutionWrapper(KernelControllerContextAction.java:108)
       at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchJoinPoint(KernelControllerContextAction.java:69)
       at org.jboss.kernel.plugins.dependency.LifecycleAction.uninstallActionInternal(LifecycleAction.java:249)
       at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.uninstallAction(KernelControllerContextAction.java:166)
       at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.uninstallAction(KernelControllerContextAction.java:46)
       at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleUninstallAction(SimpleControllerContextAction.java:79)
       at org.jboss.dependency.plugins.action.AccessControllerContextAction.uninstall(AccessControllerContextAction.java:131)
       at org.jboss.dependency.plugins.AbstractControllerContextActions.uninstall(AbstractControllerContextActions.java:58)
       at org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333)
       at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1330)
       at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009)
       at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:936)
       at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:463)
       at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:426)
       at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.undeployBean(AbstractKernelDeployer.java:384)
       at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.undeployBeans(AbstractKernelDeployer.java:363)
       at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.undeploy(AbstractKernelDeployer.java:162)
       at org.jboss.kernel.plugins.deployment.BasicKernelDeployer.undeploy(BasicKernelDeployer.java:88)
       at org.jboss.kernel.plugins.deployment.xml.BasicXMLDeployer.undeploy(BasicXMLDeployer.java:95)
       at org.jboss.kernel.plugins.deployment.BasicKernelDeployer.shutdown(BasicKernelDeployer.java:102)
       at org.jboss.bootstrap.microcontainer.ServerImpl.doShutdown(ServerImpl.java:135)
       at org.jboss.bootstrap.AbstractServerImpl.shutdownServer(AbstractServerImpl.java:520)
       at org.jboss.bootstrap.AbstractServerImpl$ShutdownHook.run(AbstractServerImpl.java:824)
      23:36:30,546 INFO [ServerImpl] Shutdown complete


      All these three cases are repeated a couple of times, probably for each controller state.

        • 1. Re: Callbacks and new classloader changes

          If you are seeing ServiceController.shutdown()
          then it means there are MBeans not getting undeployed in the proper way,
          i.e. uninstalled by the deployment or service that constructed them.

          The ServiceController.shutdown() is only really a hack to fix broken stuff
          in the case where the JVM is not really shutdown and would cause leaks otherwise.

          The same goes for the BasicKernelDeployer.shutdown()

          In this case, it looks like the issue is in
          system-jmx/src/main/org/jboss/system/microcontainer/jmx/ServiceControllerLifecycleCallback.java
          which implements the @JMX handling.

          It is only invoking the destroy() on the MBean.
          It is not removing the reference from the ServiceController, etc. via remove()

           public void uninstall(ControllerContext context) throws Exception
           {
           JMX jmx = readJmxAnnotation(context);
           ObjectName objectName = createObjectName(context, jmx);
          
           log.debug("Unregistering MBean " + objectName);
           serviceController.destroy(objectName);
          + serviceController.remove(objectName);
           }
          


          So the JMXKernel.shutdown() is trying to clean this up and failing.

          But rather than just fix this, I want to understand why the callbacks
          are being invoked on a context that doesn't exist
          (something wrong with dependencies?) and what those
          callbacks might be?

          I'd have more information if this was a better error message.
          Why is there no contextual information?

           // let's make sure we suppress any exceptions
           catch (Throwable t)
           {
          - log.warn("Cannot resolve callbacks.", t);
          + log.warn("Cannot resolve callbacks state=" + state + " isInstall=" + isInstallPhase + + " context="context + , t);
           }
          


          23:36:28,187 WARN [AbstractKernelController] Cannot resolve callbacks.
          javax.management.InstanceNotFoundException: jboss.jca:service=JCAMetaDataRepository,name=DefaultJCAM
          etaDataRepository is not registered.
           at org.jboss.mx.server.registry.BasicMBeanRegistry.get(BasicMBeanRegistry.java:529)
           at org.jboss.mx.server.MBeanServerImpl.getClassLoaderFor(MBeanServerImpl.java:1062)
           at org.jboss.system.microcontainer.ServiceControllerContext.getClassLoader(ServiceController
          Context.java:167)
           at org.jboss.dependency.plugins.SecurityActions.setContextClassLoader(SecurityActions.java:4
          6)
           at org.jboss.dependency.plugins.AbstractController.resolveCallbacks(AbstractController.java:
          1174)
           at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:
          1006)
           at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:
          936)
           at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:463)
           at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:426)
          


          Who, where, what, why, when? :-)