1 Reply Latest reply on Apr 2, 2009 4:11 AM by alexander.meyer

    Slow server shutdown in cluster

      Hello,

      I have a problem with jboss server shutdown using a two-node jboss cluster (4.2.2.GA) with JBM 1.4.2.GA. In the cluster are several queues and topics deployed, all are configured as clustered and all messages are send with the ClusteredConnectionFactory (Failover and Loadbalancing true).

      If I shutdown the first node the server stops after a few seconds without any problem. At this time the JBM failover on node2 starts. If I try to stop the second node the shutdown stucks after undeploying a few EJB3.
      In the logfile appears a few warnings:

      2009-03-18 11:43:04,458 WARN [org.jboss.remoting.Client] unable to remove remote callback handler : Can not get connection to server. Problem establishing socket conn
      ection for InvokerLocator [bisocket://169.254.0.101:4457/?JBM_clientMaxPoolSize=2000&clientLeasePeriod=10000&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrappe
      r&dataType=jms&marshaller=org.jboss.jms.wireformat.JMSWireFormat&numberOfCallRetries=1&numberOfRetries=10&pingFrequency=214748364&pingWindowFactor=10&socket.check_connection=f
      alse&stopLeaseOnFailure=false&timeout=200000&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat&validatorPingPeriod=10000&validatorPingTimeout=5000]
      


      Thread dump:

      "JBoss Shutdown Hook" daemon prio=6 tid=0x2b0ba400 nid=0x4da4 runnable [0x2f9fd000..0x2f9ffa14]
       java.lang.Thread.State: RUNNABLE
       at java.net.PlainSocketImpl.socketConnect(Native Method)
       at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
       - locked <0x25779ec8> (a java.net.SocksSocketImpl)
       at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
       at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
       at java.net.Socket.connect(Socket.java:519)
       at org.jboss.remoting.transport.socket.SocketClientInvoker.createSocket(SocketClientInvoker.java:192)
       at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.createSocket(BisocketClientInvoker.java:429)
       at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:827)
       at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:569)
       at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:422)
       at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:133)
       at org.jboss.remoting.Client.invoke(Client.java:1640)
       at org.jboss.remoting.Client.invoke(Client.java:554)
       at org.jboss.remoting.Client.invoke(Client.java:542)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate.org$jboss$jms$client$delegate$ClientConnectionDelegate$stop$aop(ClientConnectionDelegate.java:258)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java)
       at org.jboss.jms.client.container.ConnectionAspect.handleStop(ConnectionAspect.java:157)
       at sun.reflect.GeneratedMethodAccessor214.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java)
       at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:111)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java)
       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate.stop(ClientConnectionDelegate.java)
       at org.jboss.jms.client.JBossConnection.stop(JBossConnection.java:134)
       at org.jboss.resource.adapter.jms.inflow.JmsActivation.teardownSessionPool(JmsActivation.java:603)
       at org.jboss.resource.adapter.jms.inflow.JmsActivation.teardown(JmsActivation.java:335)
       at org.jboss.resource.adapter.jms.inflow.JmsActivation.stop(JmsActivation.java:226)
       at org.jboss.resource.adapter.jms.JmsResourceAdapter.endpointDeactivation(JmsResourceAdapter.java:80)
       at org.jboss.resource.deployment.RARDeployment.endpointDeactivation(RARDeployment.java:287)
       at org.jboss.resource.deployment.RARDeployment.internalInvoke(RARDeployment.java:235)
       at org.jboss.system.ServiceDynamicMBeanSupport.invoke(ServiceDynamicMBeanSupport.java:156)
       at org.jboss.mx.server.RawDynamicInvoker.invoke(RawDynamicInvoker.java:164)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.ejb3.JmxClientKernelAbstraction.invoke(JmxClientKernelAbstraction.java:44)
       at org.jboss.ejb3.mdb.inflow.JBossMessageEndpointFactory.deactivate(JBossMessageEndpointFactory.java:323)
       at org.jboss.ejb3.mdb.inflow.JBossMessageEndpointFactory.stop(JBossMessageEndpointFactory.java:200)
       at org.jboss.ejb3.mdb.MessagingContainer.stopProxies(MessagingContainer.java:319)
       at org.jboss.ejb3.mdb.MessagingContainer.stop(MessagingContainer.java:312)
       at sun.reflect.GeneratedMethodAccessor215.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.jboss.ejb3.ServiceDelegateWrapper.stopService(ServiceDelegateWrapper.java:119)
       at org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:315)
       at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:247)
       at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
       at $Proxy0.stop(Unknown Source)
       at org.jboss.system.ServiceController.stop(ServiceController.java:508)
       at org.jboss.system.ServiceController.stop(ServiceController.java:499)
       at org.jboss.system.ServiceController.stop(ServiceController.java:499)
       at org.jboss.system.ServiceController.stop(ServiceController.java:499)
       at org.jboss.system.ServiceController.stop(ServiceController.java:499)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
       at $Proxy4.stop(Unknown Source)
       at org.jboss.deployment.SARDeployer.stop(SARDeployer.java:336)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
       at $Proxy51.stop(Unknown Source)
       at org.jboss.deployment.XSLSubDeployer.stop(XSLSubDeployer.java:202)
       at org.jboss.deployment.MainDeployer.stop(MainDeployer.java:667)
       at org.jboss.deployment.MainDeployer.undeploy(MainDeployer.java:638)
       at org.jboss.deployment.MainDeployer.shutdown(MainDeployer.java:516)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.system.server.ServerImpl$ShutdownHook.shutdownDeployments(ServerImpl.java:1058)
       at org.jboss.system.server.ServerImpl$ShutdownHook.shutdown(ServerImpl.java:1033)
       at org.jboss.system.server.ServerImpl$ShutdownHook.run(ServerImpl.java:996)
      


      After a few minutes (> 5) the shutdown of node2 is finished too. Is there a way to make this shutdown faster? Is it possible to disable failover before/while stopping the server?