1 Reply Latest reply on Dec 24, 2008 9:16 AM by jbabad

    Client failover

    Dmitry Lakhin Newbie

      Hello all,

      I have a problem with the client failover sending JMS messages to the JBoss Cluster if I shutdown (not kill) one of the cluster nodes.

      JBoss version 4.2.2 GA
      JBM version 1.4.0 SP3
      Java version 1.6.0_04
      OS solaris 2.10

      The cluster contains 2 nodes. JBM was installed on the clean JBoss AS installations.

      The client is sending 1000 messages to the clustered queue.
      If one of the cluster nodes is shut down (using shutdown.sh) the exception on the client side is sometimes thrown:

      javax.jms.JMSException: Failed to route Reference[1369]:RELIABLE to sampleprocess
       at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:743)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
       at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
       at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
       at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:795)
       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:573)
       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
       at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:163)
       at org.jboss.remoting.Client.invoke(Client.java:1634)
       at org.jboss.remoting.Client.invoke(Client.java:548)
       at org.jboss.remoting.Client.invoke(Client.java:536)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
       at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:477)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
       at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:632)
       at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
       at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
       at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
       at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:278)
       at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect39.invoke(ProducerAspect39.java)
       at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
       at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
       at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
       at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
       at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)
       at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
       at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136)
       at com.mycompany.bpm.client.service.impl.BpmMessageServiceImpl.sendMessage(BpmMessageServiceImpl.java:162)
       at com.mycompany.bpm.client.service.impl.BpmMessageServiceImpl.sendTextMessage(BpmMessageServiceImpl.java:154)
       at com.mycompany.bpm.client.service.impl.BpmMessageServiceImplTest.sendTextMessageTest(BpmMessageServiceImplTest.java:16)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
       at java.lang.reflect.Method.invoke(Unknown Source)
       at org.junit.internal.runners.TestMethodRunner.executeMethodBody(TestMethodRunner.java:99)
       at org.junit.internal.runners.TestMethodRunner.runUnprotected(TestMethodRunner.java:81)
       at org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34)
       at org.junit.internal.runners.TestMethodRunner.runMethod(TestMethodRunner.java:75)
       at org.junit.internal.runners.TestMethodRunner.run(TestMethodRunner.java:45)
       at org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod(TestClassMethodsRunner.java:71)
       at org.junit.internal.runners.TestClassMethodsRunner.run(TestClassMethodsRunner.java:35)
       at org.junit.internal.runners.TestClassRunner$1.runUnprotected(TestClassRunner.java:42)
       at org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34)
       at org.junit.internal.runners.TestClassRunner.run(TestClassRunner.java:52)
       at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:38)
       at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
       at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:460)
       at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673)
       at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386)
       at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)
      
      


      There is also an exception on the server side:
      javax.jms.JMSException: Failed to route Reference[1369]:RELIABLE to sampleprocess
       at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:743)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
       at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
       at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
       at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:795)
       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:573)
       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
      15:00:33,450 INFO [ServerPeer] ServerPeer[0] is Stopping. NOTE! Stopping the server peer cleanly will NOT cause failover to occur
      15:00:34,017 INFO [ServerPeer] JMS ServerPeer[0] stopped


      Queue definion in destinations-service.xml (on both of nodes)

      <mbean code="org.jboss.jms.server.destination.QueueService"
       name="jboss.messaging.destination:service=Queue,name=sampleprocess"
       xmbean-dd="xmdesc/Queue-xmbean.xml">
       <depends optional-attribute-name="ServerPeer">jboss.messaging:service=ServerPeer</depends>
       <depends>jboss.messaging:service=PostOffice</depends>
       <attribute name="Clustered">true</attribute>
       </mbean>
      


      My jndi config

      ### JBossNS properties
      java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
      java.naming.provider.url=jnp://node1:1099,jnp://node2:1099
      java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces
      


      The messages in the queue are consumed by MDBs.

      I know that clean shutdown does not cause failover and the messages will be delivered as soon as I start the node again.

      What I don't understand why the exception is occasionally thrown on the client side. Should it work like this or is it a bug? What should I do in this case? Should I reconnect like with JBossMQ or not?

      Thanks in advance.

      Regards

      Dmitry