6 Replies Latest reply on Mar 26, 2010 5:43 AM by Maurice Zeijen

    JBoss won't shutdown when HornetQ Server and HornetQ RA are used on that same server

    Maurice Zeijen Master

      Is it a know problem that the JBoss AS 5.1 Server won't shut down if the HornetQ server is running on that server and the HornetQ RA is also being used to connect to that HornetQ instance? If not then I will file a Bug report with stack traces to point out the Lock that causes a Deadlock situation with the JBoss Shutdown Thread and a HornetQ Thread.

        • 1. Re: JBoss won't shutdown when HornetQ Server and HornetQ RA are used on that same server
          Andy Taylor Master

          can't say i have ever seen this happen, is the HornetQ instance standalone or embedded in the AS?

          1 of 1 people found this helpful
          • 2. Re: JBoss won't shutdown when HornetQ Server and HornetQ RA are used on that same server
            Maurice Zeijen Master

            The HornetQ is embedded in a JBoss AS 5.1 instance.

             

            The problem occurres in the org.hornetq.core.client.impl.FailoverManagerImpl class.

             

            The Deadlock occurres because of the failoverLock object

             

            The HornetQ Thread is in the 'createSession' method in the synchronized block of the failoverLock. This thread is probably from the RA because when the RA is nog being used then the problem doesn't occur.

             

            The JBoss Shutdown Thread is in the 'removeSession'  method and tries to entry the synchronized block of the failoverLock.

             

            Tomorrow I can provide Stacktraces of both threads if you want.

            • 3. Re: JBoss won't shutdown when HornetQ Server and HornetQ RA are used on that same server
              Clebert Suconic Master

              yes... First thing would be the analyze the stack trace from a  kill -3.

               

              It would be great if you could provide one.

              • 4. Re: JBoss won't shutdown when HornetQ Server and HornetQ RA are used on that same server
                Maurice Zeijen Master

                I did a shutdown of the JBoss Server and extracted the stack traces from both threads:

                 

                Daemon Thread [JBoss Shutdown Hook] 
                    FailoverManagerImpl.removeSession(ClientSessionInternal) line: 495    
                    ClientSessionImpl.doCleanup() line: 1614    
                    ClientSessionImpl.close() line: 811    
                    DelegatingSession.close() line: 144    
                    HornetQMessageHandler.teardown() line: 195    
                    HornetQActivation.teardown() line: 282    
                    HornetQActivation.stop() line: 245    
                    HornetQResourceAdapter.endpointDeactivation(MessageEndpointFactory, ActivationSpec) line: 155    
                    RARDeployment.endpointDeactivation(MessageEndpointFactory, ActivationSpec) line: 346    
                    RARDeployment.internalInvoke(String, Object[], String[]) line: 293    
                    RARDeployment(ServiceDynamicMBeanSupport).invoke(String, Object[], String[]) line: 156    
                    RawDynamicInvoker.invoke(String, Object[], String[]) line: 164    
                    MBeanServerImpl.invoke(ObjectName, String, Object[], String[]) line: 668    
                    JBoss42ActivationBridge.deactivate() line: 208    
                    JcaMessageAwareListener(BaseJcaInflow<T>).doStop() line: 242    
                    JcaMessageAwareListener(AbstractManagedLifecycle).stop() line: 216    
                    ManagedLifecycleController.stopInstances() line: 164    
                    ManagedLifecycleController.stopAndDestroy(boolean) line: 255    
                    ManagedLifecycleController.stop() line: 92    
                    EsbDeployment.stop() line: 132    
                    NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]    
                    NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39    
                    DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25    
                    Method.invoke(Object, Object...) line: 597    
                    ReflectionUtils.invoke(Method, Object, Object[]) line: 59    
                    ReflectMethodInfoImpl.invoke(Object, Object[]) line: 150    
                    BasicMethodJoinPoint.dispatch() line: 66    
                    KernelControllerContextAction$JoinpointDispatchWrapper.execute() line: 241    
                    KernelControllerContextAction$JoinpointDispatchWrapper(ExecutionWrapper).execute(AccessControlContext) line: 47    
                    KernelControllerContextAction.dispatchExecutionWrapper(KernelControllerContext, ExecutionWrapper) line: 109    
                    KernelControllerContextAction.dispatchJoinPoint(KernelControllerContext, Joinpoint) line: 70    
                    StartStopLifecycleAction(LifecycleAction).uninstallActionInternal(KernelControllerContext) line: 249    
                    StartStopLifecycleAction(InstallsAwareAction).uninstallAction(KernelControllerContext) line: 157    
                    StartStopLifecycleAction(InstallsAwareAction).uninstallAction(ControllerContext) line: 42    
                    StartStopLifecycleAction(SimpleControllerContextAction<T>).simpleUninstallAction(T) line: 79    
                    StartStopLifecycleAction(AccessControllerContextAction<S,T>).uninstall(ControllerContext) line: 131    
                    KernelControllerContextActions(AbstractControllerContextActions).uninstall(ControllerContext, ControllerState, ControllerState) line: 58    
                    AbstractKernelControllerContext(AbstractControllerContext).uninstall(ControllerState, ControllerState) line: 354    
                    AbstractKernelController(AbstractController).uninstall(ControllerContext, ControllerState, ControllerState) line: 1664    
                    AbstractKernelController(AbstractController).uninstallContext(ControllerContext, boolean) line: 1275    
                    AbstractKernelController(AbstractController).uninstallContext(ControllerContext, ControllerState, boolean) line: 1179    
                    AbstractKernelController(AbstractController).uninstallContext(ControllerContext, boolean) line: 1234    
                    AbstractKernelController(AbstractController).uninstallContext(ControllerContext, ControllerState, boolean) line: 1179    
                    AbstractKernelController(AbstractController).uninstallContext(ControllerContext, boolean) line: 1234    
                    AbstractKernelController(AbstractController).uninstallContext(ControllerContext, ControllerState, boolean) line: 1179    
                    AbstractKernelController(AbstractController).uninstallContext(ControllerContext, boolean) line: 1234    
                    AbstractKernelController(AbstractController).change(ControllerContext, ControllerState, boolean) line: 827    
                    AbstractKernelController(AbstractController).change(ControllerContext, ControllerState) line: 553    
                    ServiceController.doChange(KernelController, ServiceControllerContext, ControllerState, String) line: 688    
                    ServiceController.stop(ObjectName) line: 510    
                    ServiceDeployer.stop(ObjectName) line: 170    
                    ServiceDeployer.undeploy(DeploymentUnit, ServiceMetaData) line: 150    
                    ServiceDeployer.undeploy(DeploymentUnit, Object) line: 46    
                    ServiceDeployer(AbstractSimpleRealDeployer<T>).internalUndeploy(DeploymentUnit) line: 69    
                    ServiceDeployer(AbstractRealDeployer).undeploy(DeploymentUnit) line: 112    
                    DeployerWrapper.undeploy(DeploymentUnit) line: 196    
                    DeployersImpl.doUndeploy(Deployer, DeploymentUnit) line: 1469    
                    DeployersImpl.doUninstallParentLast(Deployer, DeploymentContext, boolean, boolean) line: 1376    
                    DeployersImpl.doUninstallParentLast(Deployer, DeploymentContext, boolean, boolean) line: 1369    
                    DeployersImpl.doUninstallParentLast(Deployer, DeploymentContext, boolean, boolean) line: 1356    
                    DeployersImpl.uninstall(ControllerContext, ControllerState, ControllerState) line: 1331    
                    DeploymentControllerContext(AbstractControllerContext).uninstall(ControllerState, ControllerState) line: 354    
                    AbstractKernelController(AbstractController).uninstall(ControllerContext, ControllerState, ControllerState) line: 1664    
                    AbstractKernelController(AbstractController).uninstallContext(ControllerContext, boolean) line: 1275    
                    AbstractKernelController(AbstractController).change(ControllerContext, ControllerState, boolean) line: 827    
                    AbstractKernelController(AbstractController).change(ControllerContext, ControllerState) line: 553    
                    DeployersImpl.process(List<DeploymentContext>, List<DeploymentContext>) line: 694    
                    MainDeployerImpl.process() line: 679    
                    MainDeployerAdapter.process() line: 117    
                    ProfileDeployAction.uninstall(Profile) line: 95    
                    ProfileDeployAction(AbstractProfileAction).uninstall(ProfileContext) line: 70    
                    AbstractProfileService.uninstall(ControllerContext, ControllerState, ControllerState) line: 375    
                    ProfileContext(AbstractControllerContext).uninstall(ControllerState, ControllerState) line: 354    
                    ScopedProfileServiceController(AbstractController).uninstall(ControllerContext, ControllerState, ControllerState) line: 1664    
                    ScopedProfileServiceController(AbstractController).uninstallContext(ControllerContext, boolean) line: 1275    
                    ScopedProfileServiceController(AbstractController).uninstallContext(ControllerContext, ControllerState, boolean) line: 1179    
                    ScopedProfileServiceController(AbstractController).uninstallContext(ControllerContext, boolean) line: 1234    
                    ScopedProfileServiceController(AbstractController).change(ControllerContext, ControllerState, boolean) line: 827    
                    ScopedProfileServiceController(AbstractController).change(ControllerContext, ControllerState) line: 553    
                    AbstractProfileService.deactivateProfile(ProfileKey) line: 402    
                    ProfileServiceBootstrap.deactivateProfiles(Collection<ProfileKey>) line: 388    
                    ProfileServiceBootstrap.shutdown(Server) line: 345    
                    ServerImpl(AbstractServerImpl).shutdownServer() line: 571    
                    AbstractServerImpl$ShutdownHook.run() line: 909  

                 

                Daemon Thread [Thread-0 (group:HornetQ-client-global-threads-32315681)]
                    Thread.sleep(long) line: not available [native method]    
                    FailoverManagerImpl.getConnectionWithRetry(int) line: 821    
                    FailoverManagerImpl.reconnectSessions(RemotingConnection, int) line: 758    
                    FailoverManagerImpl.failoverOrReconnect(Object, HornetQException) line: 682    
                    FailoverManagerImpl.handleConnectionFailure(Object, HornetQException) line: 548    
                    FailoverManagerImpl.access$600(FailoverManagerImpl, Object, HornetQException) line: 69    
                    FailoverManagerImpl$DelegatingFailureListener.connectionFailed(HornetQException) line: 1111    
                    RemotingConnectionImpl.callFailureListeners(HornetQException) line: 445    
                    RemotingConnectionImpl.fail(HornetQException) line: 250    
                    FailoverManagerImpl$Channel0Handler$1.run() line: 1079    
                    OrderedExecutorFactory$OrderedExecutor$1.run() line: 96    
                    ThreadPoolExecutor$Worker.runTask(Runnable) line: 886    
                    ThreadPoolExecutor$Worker.run() line: 908    
                    Thread.run() line: 619  
                

                 

                If you want I can create a Jira Issue for this problem.

                • 5. Re: JBoss won't shutdown when HornetQ Server and HornetQ RA are used on that same server
                  Andy Taylor Master

                  Yes, if you could raise a jira and refernce this thread that would be cool.

                  • 6. Re: JBoss won't shutdown when HornetQ Server and HornetQ RA are used on that same server
                    Maurice Zeijen Master

                    I created the following Jira Issue: HORNETQ-339

                     

                    I also created a workaround. You can find it in the attached zip file.

                     

                    It is a small Service that you deploy on the JBoss AS service. It makes it possible to create dependencies on the deploy cycle of the HornetQ service. To make it work you need to create a dependency to this service. When JBoss Server is being shut down then the dependency on this service makes sure that your service is stopped first. This is only needed when you require a JMX dependency. If your application is build for the JBoss MC then you can create a direct depedency on the HornetQ service.

                     

                    The JMX dependency that you need to make is to: com.smies.hornetq:service=HornetQMBeanHook

                     

                    For example in the JBoss ESB deployment you need to add the following line to the deployment.xml: <depends>com.smies.hornetq:service=HornetQMBeanHook</depends>. However in case of JBoss ESB then it is best when you make sure that all your packages have the <depends>jboss.esb:deployment=jbossesb.esb</depends> dependency and the jbossesb.esb package has the dependency on the HornetQMBeanHook

                     

                    This HornetQMBeanHook also does one extra thing namely it makes sure that the hornetq-ra.rar or the jms-ra.rar deployment is stopped before the HornetQServer is stopped.