JBoss Messaging Assistance
jbossbill Oct 24, 2006 4:26 AMHi,
We are using JBoss Messaging 1.0.1.GA
We have an application where we have set up simple failover.
We connect to 2 JBoss Messaging Servers on start up and use one as the primary and send messages to that server until it goes down, then we send to the other server. For consuming messages we connect to both servers and set up a MessageListener to receive messages. We use an ExceptionListener to detect a problem with any of the connections and initiate reconnection in a background thread.
Our main application is running on the same machine as one of the messaging servers. Each are running in a different application server.
Our issue is whenever the JBoss Messaging Server on the same machine as our main application goes down for whatever reason it has an effect on the other Server. We believe that the JBoss Messaging Servers should be in-dependant of each other, so one going down should NOT effect the other Server. But we have noticed that when one server goes down, the other server produces a warning from the SimpleConnectionManager which detects a problem with the remote client and removes all connection resources for the client process.
We have come up with a test to replicate our issue in our application.
Following is the test code that we used:
import java.util.Hashtable;
import javax.jms.Connection;
import javax.jms.ConnectionFactory;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageConsumer;
import javax.jms.MessageListener;
import javax.jms.MessageProducer;
import javax.jms.Queue;
import javax.jms.Session;
import javax.jms.TextMessage;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
public class TestMultiSessionMessageProducer {
/**
* @param args
*/
public static void main(String[] args) {
TestMultiSessionMessageProducer ml = new TestMultiSessionMessageProducer();
try {
ml.test();
} catch (JMSException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
private void test() throws JMSException {
// JMS Server1
Hashtable properties = new Hashtable();
properties.put(Context.INITIAL_CONTEXT_FACTORY,
"org.jnp.interfaces.NamingContextFactory");
properties.put(Context.URL_PKG_PREFIXES,
"org.jboss.naming:org.jnp.interfaces");
properties.put(Context.PROVIDER_URL, "jnp://161.117.20.38:1099");
properties.put(Context.SECURITY_PRINCIPAL, "admin");
properties.put(Context.SECURITY_CREDENTIALS, "admin");
ConnectionFactory connectionFactory = null;
try {
Context context = new InitialContext(properties);
connectionFactory = (ConnectionFactory) context
.lookup("ConnectionFactory");
} catch (NamingException ne) {
throw new RuntimeException(ne);
}
Connection connection = connectionFactory.createConnection();
Session session = connection.createSession(false,
Session.AUTO_ACKNOWLEDGE);
Queue queue = session.createQueue("publish.request");
MessageProducer producer = session.createProducer(queue);
// JMS Server2
Hashtable properties2 = new Hashtable();
properties2.put(Context.INITIAL_CONTEXT_FACTORY,
"org.jnp.interfaces.NamingContextFactory");
properties2.put(Context.URL_PKG_PREFIXES,
"org.jboss.naming:org.jnp.interfaces");
properties2.put(Context.PROVIDER_URL, "jnp://localhost:1099");
properties2.put(Context.SECURITY_PRINCIPAL, "admin");
properties2.put(Context.SECURITY_CREDENTIALS, "admin");
ConnectionFactory connectionFactory2 = null;
try {
Context context2 = new InitialContext(properties2);
connectionFactory2 = (ConnectionFactory) context2
.lookup("ConnectionFactory");
} catch (NamingException ne) {
throw new RuntimeException(ne);
}
Connection connection2 = connectionFactory2.createConnection();
Session session2 = connection2.createSession(false,
Session.AUTO_ACKNOWLEDGE);
Queue queue2 = session2.createQueue("publish.request");
MessageProducer producer2 = session2.createProducer(queue2);
boolean serverOne = true;
try {
int j = 0;
while (++j < 13000) {
System.out.println(j);
StringBuffer sb = new StringBuffer();
sb.append("");
for (int i=0; i < 100; i++) {
sb.append("" + i + "");
}
sb.append("");
TextMessage msg = session
.createTextMessage(sb.toString());
TextMessage msg2 = session2
.createTextMessage(sb.toString());
if (serverOne) {
try {
producer.send(msg);
serverOne = false;
} catch (Exception e) {
// send msg to other server
producer2.send(msg2);
serverOne = false;
}
} else
try {
producer2.send(msg2);
serverOne = true;
} catch (Exception e) {
// send msg to other server
producer.send(msg);
serverOne = true;
}
}
} finally {
if(producer != null)
producer.close();
if(session != null)
session.close();
if(connection != null)
connection.close();
if(producer2 != null)
producer2.close();
if(session2 != null)
session2.close();
if(connection != null)
connection2.close();
}
}
}
These are the steps we used to cause our issue:
1. Setup two JBoss Messaging Servers on different machines, create a queue named 'publish.request' on each
2. Start both JBoss Messaging Servers
2. Run test code on one of the machines running a JBoss Messaging Server
3. Kill the JBoss Messaging Server running on the same machine as the test code
4. Wait until you notice a warning shown on the JBoss Messaging Server which was NOT killed:
"17:06:44,640 WARN [SimpleConnectionManager] A problem has been detected with the connection to remote client 4h39k1l-65jq6z-etnyf5al-1-etnyf86b-9. It is possible the client has exited without closing its connection(s) or there is a network problem. All connection resources corresponding to that client process will now be removed."
5. When the test code tries to send a message to the Server which was not killed it fails with the following exception thrown in the test code console output:
2006-10-24 17:06:47,015 ERROR org.jboss.jms.client.container.ExceptionInterceptor - Caught Exception:
org.jboss.aop.NotFoundInDispatcherException: Object with oid: -2147483645 was not found in the Dispatcher
at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:85)
at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:1008)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:857)
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:454)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:541)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:261)
at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:172)
at org.jboss.remoting.Client.invoke(Client.java:589)
at org.jboss.remoting.Client.invoke(Client.java:581)
at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
at org.jboss.jms.client.container.TransactionAspect.handleSend(TransactionAspect.java:176)
at org.jboss.aop.advice.org.jboss.jms.client.container.TransactionAspect16.invoke(TransactionAspect16.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:253)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:130)
at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:172)
at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:220)
at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:147)
at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:138)
at TestMultiSessionMessageProducer.test(TestMultiSessionMessageProducer.java:117)
at TestMultiSessionMessageProducer.main(TestMultiSessionMessageProducer.java:26)
2006-10-24 17:06:47,031 ERROR org.jboss.jms.client.container.ExceptionInterceptor - Linked exception is:
org.jboss.aop.NotFoundInDispatcherException: Object with oid: -2147483645 was not found in the Dispatcher
at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:85)
at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:1008)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:857)
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:454)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:541)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:261)
at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:172)
at org.jboss.remoting.Client.invoke(Client.java:589)
at org.jboss.remoting.Client.invoke(Client.java:581)
at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
at org.jboss.jms.client.container.TransactionAspect.handleSend(TransactionAspect.java:176)
at org.jboss.aop.advice.org.jboss.jms.client.container.TransactionAspect16.invoke(TransactionAspect16.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:253)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:130)
at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:172)
at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:220)
at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:147)
at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:138)
at TestMultiSessionMessageProducer.test(TestMultiSessionMessageProducer.java:117)
at TestMultiSessionMessageProducer.main(TestMultiSessionMessageProducer.java:26)
Any assistance would be most appreciated.
Cheers,
Bill