Client failover
Hello all,
I have a problem with the client failover sending JMS messages to the JBoss Cluster if I shutdown (not kill) one of the cluster nodes.
JBoss version 4.2.2 GA
JBM version 1.4.0 SP3
Java version 1.6.0_04
OS solaris 2.10
The cluster contains 2 nodes. JBM was installed on the clean JBoss AS installations.
The client is sending 1000 messages to the clustered queue.
If one of the cluster nodes is shut down (using shutdown.sh) the exception on the client side is sometimes thrown:
javax.jms.JMSException: Failed to route Reference[1369]:RELIABLE to sampleprocess at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:743) at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383) at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87) at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java) at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157) at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121) at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java) at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java) at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95) at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:795) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:573) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166) at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:163) at org.jboss.remoting.Client.invoke(Client.java:1634) at org.jboss.remoting.Client.invoke(Client.java:548) at org.jboss.remoting.Client.invoke(Client.java:536) at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189) at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160) at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:477) at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java) at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:632) at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java) at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java) at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105) at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java) at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105) at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java) at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java) at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:278) at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect39.invoke(ProducerAspect39.java) at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java) at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105) at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java) at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java) at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164) at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207) at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145) at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136) at com.mycompany.bpm.client.service.impl.BpmMessageServiceImpl.sendMessage(BpmMessageServiceImpl.java:162) at com.mycompany.bpm.client.service.impl.BpmMessageServiceImpl.sendTextMessage(BpmMessageServiceImpl.java:154) at com.mycompany.bpm.client.service.impl.BpmMessageServiceImplTest.sendTextMessageTest(BpmMessageServiceImplTest.java:16) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.junit.internal.runners.TestMethodRunner.executeMethodBody(TestMethodRunner.java:99) at org.junit.internal.runners.TestMethodRunner.runUnprotected(TestMethodRunner.java:81) at org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34) at org.junit.internal.runners.TestMethodRunner.runMethod(TestMethodRunner.java:75) at org.junit.internal.runners.TestMethodRunner.run(TestMethodRunner.java:45) at org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod(TestClassMethodsRunner.java:71) at org.junit.internal.runners.TestClassMethodsRunner.run(TestClassMethodsRunner.java:35) at org.junit.internal.runners.TestClassRunner$1.runUnprotected(TestClassRunner.java:42) at org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34) at org.junit.internal.runners.TestClassRunner.run(TestClassRunner.java:52) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:38) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:460) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)
There is also an exception on the server side:
javax.jms.JMSException: Failed to route Reference[1369]:RELIABLE to sampleprocess at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:743) at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383) at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87) at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java) at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157) at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121) at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java) at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java) at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95) at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:795) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:573) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166) 15:00:33,450 INFO [ServerPeer] ServerPeer[0] is Stopping. NOTE! Stopping the server peer cleanly will NOT cause failover to occur 15:00:34,017 INFO [ServerPeer] JMS ServerPeer[0] stopped
Queue definion in destinations-service.xml (on both of nodes)
<mbean code="org.jboss.jms.server.destination.QueueService" name="jboss.messaging.destination:service=Queue,name=sampleprocess" xmbean-dd="xmdesc/Queue-xmbean.xml"> <depends optional-attribute-name="ServerPeer">jboss.messaging:service=ServerPeer</depends> <depends>jboss.messaging:service=PostOffice</depends> <attribute name="Clustered">true</attribute> </mbean>
My jndi config
### JBossNS properties java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory java.naming.provider.url=jnp://node1:1099,jnp://node2:1099 java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces
The messages in the queue are consumed by MDBs.
I know that clean shutdown does not cause failover and the messages will be delivered as soon as I start the node again.
What I don't understand why the exception is occasionally thrown on the client side. Should it work like this or is it a bug? What should I do in this case? Should I reconnect like with JBossMQ or not?
Thanks in advance.
Regards
Dmitry
