long time to reload messages in fail over state
Hi,
I have done some tests like the exemples distribueted. In my fail over test I have make 2 clustered istances of Jboss 4.2.2 + JBM 1.4.0 SP1 and I have sended 100 messages in to node 1. In my condition test the balancer have put all (!?) messages in node 2 and afther few second I have manualy killed node 2. Follow the log about this step:
12:55:36,518 INFO [FailoverCommandCenter] JBoss Messaging server failure detected - waiting for failover to complete... 12:55:36,705 INFO [TreeCache] viewAccepted(): [192.168.10.48:32790|4] [192.168.10.48:32790] 12:55:36,720 INFO [TreeCache] viewAccepted(): [192.168.10.48:32792|4] [192.168.10.48:32792] 12:55:36,726 INFO [TreeCache] viewAccepted(): [192.168.10.48:32783|4] [192.168.10.48:32783] 12:55:36,799 INFO [MessagingPostOffice] JBoss Messaging is failing over for failed node 48. If there are many messages to reload this may take some time... 13:27:08,793 INFO [MessageMdb] MESSAGE BEAN: IntProperty iteration: 63
You can see: the node 1 takes 32 minutes to start to consume messages. In this time the CPUs are at 0%... Is it normal?
When I have restarted node 2 it throws the follow exception:
14:23:21,934 INFO [TreeCache] viewAccepted(): [192.168.10.48:32783|5] [192.168.10.48:32783, 192.168.10.72:1961] 14:23:22,037 INFO [StateTransferGenerator_140] returning the state for tree rooted in /(1024 bytes)14:23:57,576 WARN [Client] unable to remove remote callback handler: Can not remove callback listener from target server with id of 3j011-vuslhj-f9l6ile8-1-f9l6pat0-8+3j011-vuslhj-f9l6ile8-1-f9l6payb-b as it does not exist as a registered callback listener. 14:24:06,520 ERROR [FailoverCommandCenter] Failover failed org.jboss.jms.exception.MessagingShutdownException: Cannot handle invocation since messaging server is not active (it is either starting up or shutting down) at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:133) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:769) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:573) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166) at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:163) at org.jboss.remoting.Client.invoke(Client.java:1634) at org.jboss.remoting.Client.invoke(Client.java:548) at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:167) at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java) at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83) at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java) at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java) at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java) at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:134) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121) at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java) at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java) at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:129) at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:62) at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:81) at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:452) 14:24:06,522 ERROR [ConnectionFailureListener] Caught exception in handling failure org.jboss.jms.exception.MessagingShutdownException: Cannot handle invocation since messaging server is not active (it is either starting up or shutting down) at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:133) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:769) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:573) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166) at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:163) at org.jboss.remoting.Client.invoke(Client.java:1634) at org.jboss.remoting.Client.invoke(Client.java:548) at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:167) at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java) at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83) at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java) at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java) at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java) at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:134) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121) at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java) at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java) at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:129) at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:62) at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:81) at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:452) ^[[5~^[[5;5~^[[5;5~^[[5~14:25:02,748 INFO [TreeCache] viewAccepted(): [192.168.10.48:32790|5] [192.168.10.48:32790, 192.168.10.72:1978] 14:25:05,280 INFO [TreeCache] viewAccepted(): [192.168.10.48:32792|5] [192.168.10.48:32792, 192.168.10.72:1983] 14:26:59,230 INFO [STDOUT] ------------------------------------------------------- GMS: address is 192.168.10.48:32796 -------------------------------------------------------
Is it a bug? Thank you.
