3 Replies Latest reply on Nov 8, 2006 10:16 AM by clebert.suconic

    JBoss Messaging Assistance

    jbossbill

      Hi,

      We are using JBoss Messaging 1.0.1.GA

      We have an application where we have set up simple failover.

      We connect to 2 JBoss Messaging Servers on start up and use one as the primary and send messages to that server until it goes down, then we send to the other server. For consuming messages we connect to both servers and set up a MessageListener to receive messages. We use an ExceptionListener to detect a problem with any of the connections and initiate reconnection in a background thread.
      Our main application is running on the same machine as one of the messaging servers. Each are running in a different application server.

      Our issue is whenever the JBoss Messaging Server on the same machine as our main application goes down for whatever reason it has an effect on the other Server. We believe that the JBoss Messaging Servers should be in-dependant of each other, so one going down should NOT effect the other Server. But we have noticed that when one server goes down, the other server produces a warning from the SimpleConnectionManager which detects a problem with the remote client and removes all connection resources for the client process.

      We have come up with a test to replicate our issue in our application.

      Following is the test code that we used:

      import java.util.Hashtable;

      import javax.jms.Connection;
      import javax.jms.ConnectionFactory;
      import javax.jms.JMSException;
      import javax.jms.Message;
      import javax.jms.MessageConsumer;
      import javax.jms.MessageListener;
      import javax.jms.MessageProducer;
      import javax.jms.Queue;
      import javax.jms.Session;
      import javax.jms.TextMessage;
      import javax.naming.Context;
      import javax.naming.InitialContext;
      import javax.naming.NamingException;

      public class TestMultiSessionMessageProducer {

      /**
      * @param args
      */
      public static void main(String[] args) {
      TestMultiSessionMessageProducer ml = new TestMultiSessionMessageProducer();

      try {
      ml.test();
      } catch (JMSException e) {
      // TODO Auto-generated catch block
      e.printStackTrace();
      }
      }

      private void test() throws JMSException {
      // JMS Server1
      Hashtable properties = new Hashtable();
      properties.put(Context.INITIAL_CONTEXT_FACTORY,
      "org.jnp.interfaces.NamingContextFactory");
      properties.put(Context.URL_PKG_PREFIXES,
      "org.jboss.naming:org.jnp.interfaces");
      properties.put(Context.PROVIDER_URL, "jnp://161.117.20.38:1099");
      properties.put(Context.SECURITY_PRINCIPAL, "admin");
      properties.put(Context.SECURITY_CREDENTIALS, "admin");

      ConnectionFactory connectionFactory = null;

      try {
      Context context = new InitialContext(properties);
      connectionFactory = (ConnectionFactory) context
      .lookup("ConnectionFactory");
      } catch (NamingException ne) {
      throw new RuntimeException(ne);
      }

      Connection connection = connectionFactory.createConnection();

      Session session = connection.createSession(false,
      Session.AUTO_ACKNOWLEDGE);
      Queue queue = session.createQueue("publish.request");
      MessageProducer producer = session.createProducer(queue);

      // JMS Server2
      Hashtable properties2 = new Hashtable();
      properties2.put(Context.INITIAL_CONTEXT_FACTORY,
      "org.jnp.interfaces.NamingContextFactory");
      properties2.put(Context.URL_PKG_PREFIXES,
      "org.jboss.naming:org.jnp.interfaces");
      properties2.put(Context.PROVIDER_URL, "jnp://localhost:1099");
      properties2.put(Context.SECURITY_PRINCIPAL, "admin");
      properties2.put(Context.SECURITY_CREDENTIALS, "admin");

      ConnectionFactory connectionFactory2 = null;

      try {
      Context context2 = new InitialContext(properties2);
      connectionFactory2 = (ConnectionFactory) context2
      .lookup("ConnectionFactory");
      } catch (NamingException ne) {
      throw new RuntimeException(ne);
      }

      Connection connection2 = connectionFactory2.createConnection();

      Session session2 = connection2.createSession(false,
      Session.AUTO_ACKNOWLEDGE);
      Queue queue2 = session2.createQueue("publish.request");
      MessageProducer producer2 = session2.createProducer(queue2);

      boolean serverOne = true;

      try {
      int j = 0;
      while (++j < 13000) {
      System.out.println(j);

      StringBuffer sb = new StringBuffer();
      sb.append("");
      for (int i=0; i < 100; i++) {
      sb.append("" + i + "");
      }
      sb.append("");
      TextMessage msg = session
      .createTextMessage(sb.toString());

      TextMessage msg2 = session2
      .createTextMessage(sb.toString());
      if (serverOne) {
      try {
      producer.send(msg);
      serverOne = false;
      } catch (Exception e) {
      // send msg to other server
      producer2.send(msg2);
      serverOne = false;
      }
      } else
      try {
      producer2.send(msg2);
      serverOne = true;
      } catch (Exception e) {
      // send msg to other server
      producer.send(msg);
      serverOne = true;
      }
      }
      } finally {
      if(producer != null)
      producer.close();
      if(session != null)
      session.close();
      if(connection != null)
      connection.close();

      if(producer2 != null)
      producer2.close();
      if(session2 != null)
      session2.close();
      if(connection != null)
      connection2.close();
      }
      }
      }

      These are the steps we used to cause our issue:

      1. Setup two JBoss Messaging Servers on different machines, create a queue named 'publish.request' on each
      2. Start both JBoss Messaging Servers
      2. Run test code on one of the machines running a JBoss Messaging Server
      3. Kill the JBoss Messaging Server running on the same machine as the test code
      4. Wait until you notice a warning shown on the JBoss Messaging Server which was NOT killed:

      "17:06:44,640 WARN [SimpleConnectionManager] A problem has been detected with the connection to remote client 4h39k1l-65jq6z-etnyf5al-1-etnyf86b-9. It is possible the client has exited without closing its connection(s) or there is a network problem. All connection resources corresponding to that client process will now be removed."

      5. When the test code tries to send a message to the Server which was not killed it fails with the following exception thrown in the test code console output:

      2006-10-24 17:06:47,015 ERROR org.jboss.jms.client.container.ExceptionInterceptor - Caught Exception:
      org.jboss.aop.NotFoundInDispatcherException: Object with oid: -2147483645 was not found in the Dispatcher
      at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:85)
      at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127)
      at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:1008)
      at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:857)
      at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:454)
      at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:541)
      at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:261)
      at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:172)
      at org.jboss.remoting.Client.invoke(Client.java:589)
      at org.jboss.remoting.Client.invoke(Client.java:581)
      at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111)
      at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
      at org.jboss.jms.client.container.TransactionAspect.handleSend(TransactionAspect.java:176)
      at org.jboss.aop.advice.org.jboss.jms.client.container.TransactionAspect16.invoke(TransactionAspect16.java)
      at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
      at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
      at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
      at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
      at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
      at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
      at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:253)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:130)
      at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
      at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
      at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
      at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
      at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
      at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
      at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:172)
      at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:220)
      at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:147)
      at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:138)
      at TestMultiSessionMessageProducer.test(TestMultiSessionMessageProducer.java:117)
      at TestMultiSessionMessageProducer.main(TestMultiSessionMessageProducer.java:26)
      2006-10-24 17:06:47,031 ERROR org.jboss.jms.client.container.ExceptionInterceptor - Linked exception is:
      org.jboss.aop.NotFoundInDispatcherException: Object with oid: -2147483645 was not found in the Dispatcher
      at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:85)
      at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127)
      at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:1008)
      at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:857)
      at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:454)
      at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:541)
      at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:261)
      at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:172)
      at org.jboss.remoting.Client.invoke(Client.java:589)
      at org.jboss.remoting.Client.invoke(Client.java:581)
      at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111)
      at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
      at org.jboss.jms.client.container.TransactionAspect.handleSend(TransactionAspect.java:176)
      at org.jboss.aop.advice.org.jboss.jms.client.container.TransactionAspect16.invoke(TransactionAspect16.java)
      at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
      at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
      at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
      at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
      at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
      at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
      at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:253)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:130)
      at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
      at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
      at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
      at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
      at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
      at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
      at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:172)
      at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:220)
      at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:147)
      at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:138)
      at TestMultiSessionMessageProducer.test(TestMultiSessionMessageProducer.java:117)
      at TestMultiSessionMessageProducer.main(TestMultiSessionMessageProducer.java:26)

      Any assistance would be most appreciated.

      Cheers,
      Bill

        • 1. Re: JBoss Messaging Assistance
          jbossbill

          Also, I'd like to note that our application works fine when used with ActiveMQ. We can stop either or both of the JMS Servers and re-start them without having either of them effect each other.

          We use the following properties when connecting to JBoss Messaging Server:

          Hashtable properties = new Hashtable();

          properties.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory");
          properties.put(Context.URL_PKG_PREFIXES, "org.jboss.naming:org.jnp.interfaces");
          properties.put(Context.PROVIDER_URL, "jnp://localhost:1099");
          properties.put(Context.SECURITY_PRINCIPAL, "username");
          properties.put(Context.SECURITY_CREDENTIALS, "password");

          If you require any further information, please ask for it.

          Cheers,
          Bill

          • 2. Connecting to two JBoss messaging servers from the one clien
            davidrh

            We would like to implement something similar to this to get a basic level of redundancy for generating and receiving messages. Is there any reason why this wouldn't work i.e. should the two connections be completely independant of each other?

            • 3. Re: JBoss Messaging Assistance
              clebert.suconic

              Take a look at your jboss-messaging.sar/messaging-server.xml

              Each server must have a unique serverID:

              That is used by routers to redirect to the appropriate callback client. If both have the same ID, a client with two identical connections might get lost.

              <constructor>
               <!-- ServerPeerID -->
               <arg type="java.lang.String" value="server.0" />
               <!-- DefaultQueueJNDIContext -->
               <arg type="java.lang.String" value="/queue" />
               <!-- DefaultTopicJNDIContext -->
               <arg type="java.lang.String" value="/topic" />
               </constructor>