Slow server shutdown in cluster
alexander.meyer Mar 18, 2009 6:55 AMHello,
I have a problem with jboss server shutdown using a two-node jboss cluster (4.2.2.GA) with JBM 1.4.2.GA. In the cluster are several queues and topics deployed, all are configured as clustered and all messages are send with the ClusteredConnectionFactory (Failover and Loadbalancing true).
If I shutdown the first node the server stops after a few seconds without any problem. At this time the JBM failover on node2 starts. If I try to stop the second node the shutdown stucks after undeploying a few EJB3.
In the logfile appears a few warnings:
2009-03-18 11:43:04,458 WARN [org.jboss.remoting.Client] unable to remove remote callback handler : Can not get connection to server. Problem establishing socket conn ection for InvokerLocator [bisocket://169.254.0.101:4457/?JBM_clientMaxPoolSize=2000&clientLeasePeriod=10000&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrappe r&dataType=jms&marshaller=org.jboss.jms.wireformat.JMSWireFormat&numberOfCallRetries=1&numberOfRetries=10&pingFrequency=214748364&pingWindowFactor=10&socket.check_connection=f alse&stopLeaseOnFailure=false&timeout=200000&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat&validatorPingPeriod=10000&validatorPingTimeout=5000]
Thread dump:
"JBoss Shutdown Hook" daemon prio=6 tid=0x2b0ba400 nid=0x4da4 runnable [0x2f9fd000..0x2f9ffa14] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333) - locked <0x25779ec8> (a java.net.SocksSocketImpl) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:519) at org.jboss.remoting.transport.socket.SocketClientInvoker.createSocket(SocketClientInvoker.java:192) at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.createSocket(BisocketClientInvoker.java:429) at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:827) at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:569) at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:422) at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:133) at org.jboss.remoting.Client.invoke(Client.java:1640) at org.jboss.remoting.Client.invoke(Client.java:554) at org.jboss.remoting.Client.invoke(Client.java:542) at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189) at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160) at org.jboss.jms.client.delegate.ClientConnectionDelegate.org$jboss$jms$client$delegate$ClientConnectionDelegate$stop$aop(ClientConnectionDelegate.java:258) at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java) at org.jboss.jms.client.container.ConnectionAspect.handleStop(ConnectionAspect.java:157) at sun.reflect.GeneratedMethodAccessor214.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121) at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java) at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:111) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105) at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java) at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105) at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java) at org.jboss.jms.client.delegate.ClientConnectionDelegate.stop(ClientConnectionDelegate.java) at org.jboss.jms.client.JBossConnection.stop(JBossConnection.java:134) at org.jboss.resource.adapter.jms.inflow.JmsActivation.teardownSessionPool(JmsActivation.java:603) at org.jboss.resource.adapter.jms.inflow.JmsActivation.teardown(JmsActivation.java:335) at org.jboss.resource.adapter.jms.inflow.JmsActivation.stop(JmsActivation.java:226) at org.jboss.resource.adapter.jms.JmsResourceAdapter.endpointDeactivation(JmsResourceAdapter.java:80) at org.jboss.resource.deployment.RARDeployment.endpointDeactivation(RARDeployment.java:287) at org.jboss.resource.deployment.RARDeployment.internalInvoke(RARDeployment.java:235) at org.jboss.system.ServiceDynamicMBeanSupport.invoke(ServiceDynamicMBeanSupport.java:156) at org.jboss.mx.server.RawDynamicInvoker.invoke(RawDynamicInvoker.java:164) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.ejb3.JmxClientKernelAbstraction.invoke(JmxClientKernelAbstraction.java:44) at org.jboss.ejb3.mdb.inflow.JBossMessageEndpointFactory.deactivate(JBossMessageEndpointFactory.java:323) at org.jboss.ejb3.mdb.inflow.JBossMessageEndpointFactory.stop(JBossMessageEndpointFactory.java:200) at org.jboss.ejb3.mdb.MessagingContainer.stopProxies(MessagingContainer.java:319) at org.jboss.ejb3.mdb.MessagingContainer.stop(MessagingContainer.java:312) at sun.reflect.GeneratedMethodAccessor215.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.ejb3.ServiceDelegateWrapper.stopService(ServiceDelegateWrapper.java:119) at org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:315) at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:247) at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978) at $Proxy0.stop(Unknown Source) at org.jboss.system.ServiceController.stop(ServiceController.java:508) at org.jboss.system.ServiceController.stop(ServiceController.java:499) at org.jboss.system.ServiceController.stop(ServiceController.java:499) at org.jboss.system.ServiceController.stop(ServiceController.java:499) at org.jboss.system.ServiceController.stop(ServiceController.java:499) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210) at $Proxy4.stop(Unknown Source) at org.jboss.deployment.SARDeployer.stop(SARDeployer.java:336) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210) at $Proxy51.stop(Unknown Source) at org.jboss.deployment.XSLSubDeployer.stop(XSLSubDeployer.java:202) at org.jboss.deployment.MainDeployer.stop(MainDeployer.java:667) at org.jboss.deployment.MainDeployer.undeploy(MainDeployer.java:638) at org.jboss.deployment.MainDeployer.shutdown(MainDeployer.java:516) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.system.server.ServerImpl$ShutdownHook.shutdownDeployments(ServerImpl.java:1058) at org.jboss.system.server.ServerImpl$ShutdownHook.shutdown(ServerImpl.java:1033) at org.jboss.system.server.ServerImpl$ShutdownHook.run(ServerImpl.java:996)
After a few minutes (> 5) the shutdown of node2 is finished too. Is there a way to make this shutdown faster? Is it possible to disable failover before/while stopping the server?