New JBoss Messaging clustering

[jbabad]

Hi,

We're using JBoss 5.0.0.GA with JBM 1.4.1(GA).
I have a cluster of 2 Jboss instances running on my machine - node1 and node2.
They're started as follows:

./run.sh -c node1

./run.sh -c node2 -Djboss.service.binding.set=ports-01 -Djboss.messaging.ServerPeerID=1

The nodes are clustered and using an Oracle database as the message store.
JNDI port for node1 is 1099 and for node2 is 1199
I have a simple sender and receiver programs. Sender sends 10 messages at 5 second intervals, Receiver picks them up.

When I connect the Sender to JNDI for node1 and the Receiver to JNDI for node2
then I can CTRL-C node1 (I've set the FailoveronNodeLeave to true) and the receiver still gets the messages.
BUT...
If I have both the Sender and Receiver connecting to the JNDI for node1 and then stop node1, I see errors like:

12:23:48,879 ERROR [ExceptionUtil] SessionEndpoint[ua-xp0jy3pf-1-ix4gy3pf-hm24yq-s2s2o4c5] send [2b-37djy3pf-1-ix4gy3pf-hm24yq-s2s2o4c5]
javax.jms.JMSException: Failed to route Reference[20154309490524163]:RELIABLE to testDistributedQueue
 at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:757)
 at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:397)
 at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
 at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeTarget(SessionAdvised$send_7280680627620114891.java)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
 at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
 at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
 at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
 at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:908)
 at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:742)
 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:695)
 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:549)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:230)

or

2008-12-24 13:14:02,460 WARN [org.jgroups.blocks.ConnectionTable] (OOB-17,127.0.0.1:7900) connection table is not running, discarding message to 127.0.0.1:7901
2008-12-24 13:14:03,384 ERROR [org.jboss.jms.client.container.ClosedInterceptor] (Thread-40) ClosedInterceptor.ClientProducerDelegate[NO_ID_SET]: method send() did not go through, the interceptor is CLOSED
2008-12-24 13:14:03,385 ERROR [org.jboss.messaging.core.impl.clusterconnection.MessageSucker] (Thread-40) Failed to forward message
javax.jms.IllegalStateException: The object is closed
 at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:157)
 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
 at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
 at org.jboss.messaging.core.impl.clusterconnection.MessageSucker.onMessage(MessageSucker.java:269)
 at org.jboss.jms.client.container.ClientConsumer.callOnMessage(ClientConsumer.java:229)
 at org.jboss.jms.client.container.ClientConsumer$ListenerRunner.run(ClientConsumer.java:1043)
 at org.jboss.messaging.util.OrderedExecutorFactory$ChildExecutor.run(OrderedExecutorFactory.java:120)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
 at java.lang.Thread.run(Thread.java:619)

Any ideas would be appreciated.

Cheers,
Joseph

[jbabad]

Apologies for the subject line.... should have read: New to JBoss Messaging Clustering... :)

[gaohoward]

Hi,
try to let node1 use ports-01 and nod2 use ports-02. If still fail, can you enable the DEBUG or Trace log level and see if there is more infomation in the log? Normally, if you correctly setup the cluster, failover will be performed when you shutdown one node.

[clebert.suconic]

You are killing one of the nodes. The communication between them was interrupted. You're still supposed to see failures on logs from broken connections.


What should happen is the servers recovering from the failures. That's what failover is supposed to do... recover from failures, what means messages being received on client as the server didn't fail.

[jbabad]

Hi,

Thanks for the replies.
I realised that I forgot to mention that in the scenario when both sender and receiver programs are connecting to node1's JNDI and I then shut down the node, then the sender program crashes with the exception

Exception: Failed to route Reference[20162686882938884]:RELIABLE to testDistributedQueue

This is the same exception seen in the server log that I included initially.

I was hoping that the sender would transparently start sending the messages via the other node in the cluster.
When the sender and receiver connect to the different nodes' JNDI, it does seem to cope with shutting down the node and the sender continues to send messages.

Sender code is as follows - basically a setup followed by a loop attempting to send 10 messages (some of the code not included):

 public void setUpJms(String port) throws Exception {

 String destinationName = System.getProperty("example.queue.name");



 ConnectionFactory cf = null;

// try {
 // Create a connection to the clustered messaging instance

 Hashtable env = new Hashtable();
 env.put("java.naming.factory.initial","org.jnp.interfaces.NamingContextFactory");
 env.put("java.naming.provider.url","jnp://localhost:" + port);
 env.put("java.naming.factory.url.pkgs","org.jboss.naming:org.jnp.interfaces");
 ic = new InitialContext(env);

 cf = (ConnectionFactory) ic.lookup("/ClusteredConnectionFactory");

 Queue distributedQueue = (Queue) ic.lookup(destinationName);
 log("Distributed queue " + destinationName + " exists");

 // When connecting to a messaging cluster, the ConnectionFactory has the capability of
 // transparently creating physical connections to different cluster nodes, in a round
 // robin fashion ...

 connection = cf.createConnection();

 connection.start();

 // Send 2 messages to the queue

 session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

 messageProducer = session.createProducer(distributedQueue);

 displayProviderInfo(connection.getMetaData());

 return;

 public void sendMessage( String message) throws JMSException
 {

 TextMessage message1 = session.createTextMessage(message);

 messageProducer.send(message1);

 }

 public static void main(String[] args)
 {
 System.setProperty("example.queue.name", "/queue/testDistributedQueue");


 try {
 JmsSender sender = new JmsSender();
 sender.setUpJms(args[0]);

 for (int i=0; i<10; i++)
 {
 log("Sending message: " + i);
 sender.sendMessage("Hello" + i);
 Thread.sleep(5000);
 }
 sender.close();
 } catch (Exception e) {
 log("Exception: " + e.getMessage());
 }
 }

I then realised that I wasn't trapping the exception in the sendMessage method which meant that it filtered up to main and exited the program.

Obvious really, but still strange as this wasn't an issue when the sender and receiver programs were connecting to different nodes?

Trapping the exception in sendMessage meant that I could attempt to re-send the message. Going down this route I saw a couple of things.

Doing a CTRL-C shutdown I still saw messages either going missing or being duplicated. I still sometimes got the "Failed to Route" error, but at least I could continue to attempt to send messages.
Killing the JBoss instance (kill -9) seemed to fail-over OK with no loss of data

So,

are these observations consistent with what would be expected?
(Note in both these scenarios both the sender and receiver initially connect to the same node)
Why the difference when both programs are initially connected to a single node - when I need to trap the send exception - as opposed to when they are connected to different nodes and that exception doesn't appear to happen?


Cheers,
Joseph.

[timfox]

Probably you haven't deployed your queue on both nodes.

Regarding CTRL-C - there is a FAQ on this and many old discussions in JIRA and on the forums.