1 2 Previous Next 20 Replies Latest reply on Feb 22, 2007 11:13 PM by bander

    Connecting to two JBoss messaging servers causes interferenc

    davidrh

      We are having problems getting JBoss to work when we have a connection from the one client to two JBoss messaging servers simultaneously. Everything works fine initially, but if one server goes down the client connections to the good server are also disconnected. This situation (i.e. one server going down) is the reason why we have multiple connections.

      Some background: In our application, we have implemented high availability in our messaging client by maintaining a connection to two (or more) JMS servers simultaneously. For dispatching of messages, we try the first JMS server and if that doesn't work, we try the second and so on. For receiving messages, we set up a message listener for each queue on all of the JMS servers. For receiving and sending, we always use the session that received the message to dispatch any subsequent messages. In this way, we always should have somewhere to send messages and each JMS server will always have listeners to receive messages. Any messages sent as part of receiving a message will occur in the same JMS transaction and therefore be committed or rolled back as one.

      We have ExceptionListeners registered on the connections to handle periodically attempting to reconnect to the JMS server that has failed. For sending messages we use Spring's JmsTemplate backed by a pool of sessions using Apache commons pooling. For receiving messages, we use message listeners.

      The problem that we are seeing is reproduced in this post http://www.jboss.org/index.html?module=bb&op=viewtopic&t=93258 which unfortunately has a bit of a vague subject and hasn't attracted any response. My apologies for reposting if this lack of response is not caused by the subject.

      We successfully employ this strategy with Active MQ and Oracle AQ so we think that our code is workable. Is there any inherent reason why you can't have one client connecting to two JBoss messaging servers at the same time, without the failure of one connection interfering with the other? Is there anything in the above approach to availability that would be inherently unsound with JBoss Messaging?

      Any guidance appreciated.

        • 1. Re: Connecting to two JBoss messaging servers causes interfe
          davidrh

          Also, we are running 1.0.1.GA on JBoss 4.0.4.GA deployed as standalone-messaging.

          • 2. Re: Connecting to two JBoss messaging servers causes interfe
            timfox

            Can you replicate this in a simple test case and send it to us?

            • 3. Re: Connecting to two JBoss messaging servers causes interfe
              davidrh

              Simple test class is attached below. It basically sends messages to one server until it gets an exception, at which point it starts sending them to a second server.

              I am running this test case on the server1 (localhost) machine. I run it and leave it for a few minutes sending messages to the server1 JBoss Messaging instance on localhost. I then shutdown the server1 instance and the messages begin to be sent to server2 (as expected). If you leave it run for a few minutes more, eventually sending to server2 will fail as well with the exception:

              2006-11-09 14:21:36,859 ERROR org.jboss.jms.client.container.ExceptionInterceptor - Caught RuntimeException
              org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for locator - InvokerLocator [socket://161.117.20.9:4457/?dataType=jms&marshaller=org.jboss.jms.server.remoting.JMSWireFormat&
              serializationtype=jboss&socket.check_connection=false&unmarshaller=org.jboss.jms.server.remoting.JMSWireFormat]
               at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:323)
               at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:125)
               at org.jboss.remoting.Client.invoke(Client.java:589)
               at org.jboss.remoting.Client.invoke(Client.java:581)
               at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.invoke(ClientConnectionFactoryDelegate.java:199)
               at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$getIdBlock_2799534920426283165.invokeNext(ClientConnectionFactoryDelegate$getIdBlock_2799534920426283165.java)
               at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
               at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$getIdBlock_2799534920426283165.invokeNext(ClientConnectionFactoryDelegate$getIdBlock_2799534920426283165.java)
               at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
               at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$getIdBlock_2799534920426283165.invokeNext(ClientConnectionFactoryDelegate$getIdBlock_2799534920426283165.java)
               at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.getIdBlock(ClientConnectionFactoryDelegate.java)
               at org.jboss.jms.message.MessageIdGenerator.getNextBlock(MessageIdGenerator.java:72)
               at org.jboss.jms.message.MessageIdGenerator.getId(MessageIdGenerator.java:86)
               at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:230)
               at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
               at java.lang.reflect.Method.invoke(Method.java:324)
               at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:130)
               at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
               at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
               at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
               at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
               at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
               at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
               at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
               at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
               at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
               at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:172)
               at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:220)
               at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:147)
               at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:138)
               at TestMultiConnectionMessageProducer.test(TestMultiConnectionMessageProducer.java:140)
               at TestMultiConnectionMessageProducer.main(TestMultiConnectionMessageProducer.java:27)
              Caused by: java.net.ConnectException: Connection refused: connect
               at java.net.PlainSocketImpl.socketConnect(Native Method)
               at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:305)
               at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:171)
               at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:158)
               at java.net.Socket.connect(Socket.java:452)
               at java.net.Socket.connect(Socket.java:402)
               at java.net.Socket.<init>(Socket.java:309)
               at java.net.Socket.<init>(Socket.java:124)
               at org.jboss.remoting.transport.socket.SocketClientInvoker.createSocket(SocketClientInvoker.java:183)
               at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:685)
               at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:319)
               ... 32 more
              


              I've put the session.createQueue calls in for each message dispatch as this is what the Spring JmsTemplate class does (which is what we're using in our application). When I debug our application, the first call to the session.createQueue method after the server has failed takes quite a long time (30 seconds or more). I can't replicate this delay in the test case at the moment. In our application, this causes a delay in the failover as at the point of calling createQueue, we don't know that the server has failed yet.

              You'll notice in the output that the ExceptionListener on the connection can fire quite a while (30 seconds) after the connection has actually been lost. We use the ExceptionListener to shutdown our session pool and initiate reconnection in a separate thread. Until the listener fires, we may still hand out sessions from the pool that are for the server that is no longer there.

              As an aside, when we detect a failed connection what should we do to clean it up? If I put in the code to try closing the Producer, Session and Connection and ignore any exceptions it takes quite a long time to execute (around 60 seconds for session.close() and 60 seconds for connection.close()). With this close code in place, when it tries the second server the error that is generated is:

              2006-11-09 14:41:00,937 ERROR org.jboss.jms.client.container.ExceptionInterceptor - Caught Exception:
              org.jboss.aop.NotFoundInDispatcherException: Object with oid: -2147483641 was not found in the Dispatcher
               at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:85)
               at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127)
               at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:1008)
               at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:857)
               at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:454)
               at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:541)
               at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:261)
               at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:172)
               at org.jboss.remoting.Client.invoke(Client.java:589)
               at org.jboss.remoting.Client.invoke(Client.java:581)
               at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111)
               at org.jboss.jms.client.delegate.ClientSessionDelegate$createQueue_6431069199924553036.invokeNext(ClientSessionDelegate$createQueue_6431069199924553036.java)
               at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
               at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
               at org.jboss.jms.client.delegate.ClientSessionDelegate$createQueue_6431069199924553036.invokeNext(ClientSessionDelegate$createQueue_6431069199924553036.java)
               at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
               at org.jboss.jms.client.delegate.ClientSessionDelegate$createQueue_6431069199924553036.invokeNext(ClientSessionDelegate$createQueue_6431069199924553036.java)
               at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
               at org.jboss.jms.client.delegate.ClientSessionDelegate$createQueue_6431069199924553036.invokeNext(ClientSessionDelegate$createQueue_6431069199924553036.java)
               at org.jboss.jms.client.delegate.ClientSessionDelegate.createQueue(ClientSessionDelegate.java)
               at org.jboss.jms.client.JBossSession.createQueue(JBossSession.java:272)
               at TestMultiConnectionMessageProducer.test(TestMultiConnectionMessageProducer.java:135)
               at TestMultiConnectionMessageProducer.main(TestMultiConnectionMessageProducer.java:27)
              org.jboss.jms.util.MessagingJMSException: Caught exception
              Connection 2 failed:
               at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:99)
               at org.jboss.jms.client.delegate.ClientSessionDelegate$createQueue_6431069199924553036.invokeNext(ClientSessionDelegate$createQueue_6431069199924553036.java)
               at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
               at org.jboss.jms.client.delegate.ClientSessionDelegate$createQueue_6431069199924553036.invokeNext(ClientSessionDelegate$createQueue_6431069199924553036.java)
               at org.jboss.jms.client.delegate.ClientSessionDelegate.createQueue(ClientSessionDelegate.java)
               at org.jboss.jms.client.JBossSession.createQueue(JBossSession.java:272)
               at TestMultiConnectionMessageProducer.test(TestMultiConnectionMessageProducer.java:135)
               at TestMultiConnectionMessageProducer.main(TestMultiConnectionMessageProducer.java:27)
              


              Thanks for your assistance,

              David

              Test Case:

              import java.util.Hashtable;
              
              import javax.jms.Connection;
              import javax.jms.ConnectionFactory;
              import javax.jms.ExceptionListener;
              import javax.jms.JMSException;
              import javax.jms.Message;
              import javax.jms.MessageConsumer;
              import javax.jms.MessageListener;
              import javax.jms.MessageProducer;
              import javax.jms.Queue;
              import javax.jms.Session;
              import javax.jms.TextMessage;
              import javax.naming.Context;
              import javax.naming.InitialContext;
              import javax.naming.NamingException;
              
              public class TestMultiConnectionMessageProducer {
              
               /**
               * @param args
               */
               public static void main(String[] args) {
               TestMultiConnectionMessageProducer ml = new TestMultiConnectionMessageProducer();
              
               try {
               ml.test();
               } catch (JMSException e) {
               // TODO Auto-generated catch block
               e.printStackTrace();
               }
               }
              
               private void test() throws JMSException {
               // Setup connection 1
               Hashtable properties1 = new Hashtable();
               properties1.put(Context.INITIAL_CONTEXT_FACTORY,
               "org.jnp.interfaces.NamingContextFactory");
               properties1.put(Context.URL_PKG_PREFIXES,
               "org.jboss.naming:org.jnp.interfaces");
               properties1.put(Context.PROVIDER_URL, "jnp://localhost:1099");
               properties1.put(Context.SECURITY_PRINCIPAL, "admin");
               properties1.put(Context.SECURITY_CREDENTIALS, "admin");
              
               ConnectionFactory connectionFactory1 = null;
              
               try {
               Context context1 = new InitialContext(properties1);
               connectionFactory1 = (ConnectionFactory) context1
               .lookup("ConnectionFactory");
               } catch (NamingException ne) {
               throw new RuntimeException(ne);
               }
              
               final Connection connection1 = connectionFactory1.createConnection();
               ExceptionListener listener1 = new ExceptionListener() {
               public void onException(JMSException jmse) {
               System.out.println("Fired exception listener on connection 1");
               }
               };
               connection1.setExceptionListener(listener1);
              
               Session session1 = connection1.createSession(false,
               Session.AUTO_ACKNOWLEDGE);
              
               boolean connection1OK = true;
              
               // Setup connection 2
               Hashtable properties2 = new Hashtable();
               properties2.put(Context.INITIAL_CONTEXT_FACTORY,
               "org.jnp.interfaces.NamingContextFactory");
               properties2.put(Context.URL_PKG_PREFIXES,
               "org.jboss.naming:org.jnp.interfaces");
               properties2.put(Context.PROVIDER_URL, "jnp://server2:1099");
               properties2.put(Context.SECURITY_PRINCIPAL, "admin");
               properties2.put(Context.SECURITY_CREDENTIALS, "admin");
              
               ConnectionFactory connectionFactory2 = null;
              
               try {
               Context context2 = new InitialContext(properties2);
               connectionFactory2 = (ConnectionFactory) context2
               .lookup("ConnectionFactory");
               } catch (NamingException ne) {
               throw new RuntimeException(ne);
               }
              
               final Connection connection2 = connectionFactory2.createConnection();
               ExceptionListener listener2 = new ExceptionListener() {
               public void onException(JMSException jmse) {
               System.out.println("Fired exception listener on connection 2");
               }
               };
               connection2.setExceptionListener(listener2);
              
               Session session2 = connection2.createSession(false,
               Session.AUTO_ACKNOWLEDGE);
              
               boolean connection2OK = true;
               boolean finished = false;
               while (!finished && (connection1OK || connection2OK)) {
               if (connection1OK) {
               try {
               TextMessage msg1 = session1.createTextMessage("Message");
               System.out.println("Creating queue 1");
               Queue queue1 = session1.createQueue("publish.request");
               System.out.println("Creating producer 1");
               MessageProducer producer1 = session1.createProducer(queue1);
               System.out.println("Sending message to queue 1");
               producer1.send(msg1);
               producer1.close();
               System.out.println("Connection 1 OK");
               System.out.println("==========================");
               } catch (Exception e) {
               connection1OK = false;
               System.out.println("Connection 1 failed:");
               e.printStackTrace();
              // try {
              // System.out.println("Closing session 1");
              // session1.close();
              // } catch (Exception e1) {
              // }
              // try {
              // System.out.println("Closing connection 1");
              // connection1.close();
              // } catch (Exception e1) {
              // }
               }
               }
               if (!connection1OK && connection2OK) {
               try {
               System.out.println("Trying server 2");
               TextMessage msg2 = session2.createTextMessage("Message");
               System.out.println("Creating queue 2");
               Queue queue2 = session2.createQueue("publish.request");
               System.out.println("Creating producer 2");
               MessageProducer producer2 = session2.createProducer(queue2);
               System.out.println("Sending message to queue 2");
               producer2.send(msg2);
               producer2.close();
               System.out.println("Connection 2 OK");
               System.out.println("==========================");
               } catch (Exception e2) {
               connection2OK = false;
               System.out.println("Connection 2 failed:");
               e2.printStackTrace();
               }
               }
              
               try {
               Thread.currentThread().sleep(1000);
               } catch (InterruptedException e) {
               finished = true;
               }
               }
               }
              }
              


              • 4. Re: Connecting to two JBoss messaging servers causes interfe
                davidrh

                I have tried Clebert's suggestion mentioned in the other post of changing the server id in messaging-server.xml:

                <constructor>
                 <!-- ServerPeerID -->
                 <arg type="java.lang.String" value="server.0" />
                 <!-- DefaultQueueJNDIContext -->
                 <arg type="java.lang.String" value="/queue" />
                 <!-- DefaultTopicJNDIContext -->
                 <arg type="java.lang.String" value="/topic" />
                </constructor>
                

                This seems to make things work if you don't try and close the failed session and connection. If you uncomment the code to try and close the session and connection, then the sending of messages to the second server doesn't work at all, with the same "Object with oid was not found in the Dispatcher" error mentioned previously. On the server, the following entry is logged:

                2006-11-09 16:45:20,457 WARN [org.jboss.jms.server.connectionmanager.SimpleConnectionManager] A problem has been detected with the connection to remote client 4h39k9-33iqxv-euaqkont-1-euaqkq0f-9. It is possible the client has exited without closing its connection(s) or there is a network problem. All connection resources corresponding to that client process will now be removed.


                • 5. Re: Connecting to two JBoss messaging servers causes interfe
                  clebert.suconic

                   

                  If you uncomment the code to try and close the session and connection, then the sending of messages to the second server doesn't work at all, with the same "Object with oid was not found in the Dispatcher" error mentioned previously. On the server, the following entry is logged:



                  You lost me... I tried to find what piece of code you were talking about and didn't figure out.

                  • 6. Re: Connecting to two JBoss messaging servers causes interfe
                    clebert.suconic

                    Anyway... your example here is kind of confusing IMHO... you are opening the connection out of the loop, and closing it within the loop.

                    Are you sure you don't have a coding problem within your example?
                    I have tried two conenctions from the same client (as I'm coding HA support) and never had any problems with it.


                    Clebert Suconic

                    • 7. Re: Connecting to two JBoss messaging servers causes interfe
                      timfox

                      I have a feeling I might know what the problem is (not 100%)

                      In 1.0.x the callbackservers are keyed only on the consumer id, which is created on the server from a sequence that starts from zero on startup.

                      So, if you have more than one server you can have more than one consumer with the same id, which can cause problems.

                      I fixed this some time ago in TRUNK and now the server id + the consumer id used as the key, although this fix is not available currently in 1.0.x

                      • 8. Re: Connecting to two JBoss messaging servers causes interfe
                        clebert.suconic

                        Ah... ok... I"m using trunk. That's why I didn't see any problems

                        • 9. Re: Connecting to two JBoss messaging servers causes interfe
                          davidrh

                          The uncommenting that I was talking about referred to the code in my example that is currently commented out for when any exception is caught on sending a message on connection1. I was attempting to see if cleaning up (i.e. trying to close) the defunct connection on an exception affected the operation of the second connection (it does seem to). I think that the example never uses the first connection again after it has detected any exception.

                          I know this code is not how you would do it in practice, but I was trying to keep the sample as tight as possible to demonstrate the problem. Our actual application is more complex, as it initiates reconnect in a background thread and obtains its sessions from a session pool but I didn't think this was the cause of the problem that we are observing.

                          Is there any chance of getting the fix that Tim is talking about into a 1.0.x release? We are hoping to offer JBoss Messaging support as part of our application, but with this problem we cannot as we need some form of high availability. Was it a relatively confined change that we could patch in ourselves? Would you be able to give us a start and tell us a class and revision to look at?

                          As an aside, could the problem that Tim mentioned cause incorrect messages to get placed on a queue? We have sporadic reports from our testers of incorrect messages being placed on the queue. We only get these errors when running JBoss Messaging (i.e. not other providers that we support) and only when running two instances. The error that they were reporting was that callback message objects were being placed onto the queues which then caused class cast exceptions in our application as weren't expecting that type of message. I don't know if that makes any sense to you, but if it doesn't I can get more information.

                          Thanks for the assistance,

                          David

                          • 10. Re: Connecting to two JBoss messaging servers causes interfe
                            clebert.suconic

                             

                            Is there any chance of getting the fix that Tim is talking about into a 1.0.x release? We are hoping to offer JBoss Messaging support as part of our application, but with this problem we cannot as we need some form of high availability. Was it a relatively confined change that we could patch in ourselves? Would you be able to give us a start and tell us a class and revision to look at?


                            I will talk to Tim about what are the technical implications. I guess we could make it to 1.0.2 if there are no technical implications.

                            You know... HA is being worked at 1.2.0, right?

                            to offer JBoss Messaging support


                            It was dubious to me if you meant support from JBoss Messaging/JBoss. If you are considering paying for support, then we would have contractual obligations about fixing this. I'm not trying to push you anything.. but just in case you meant paying support.

                            • 11. Re: Connecting to two JBoss messaging servers causes interfe
                              davidrh

                              Hi Clebert,

                              We know HA is on the way, but we can't wait. We have an application already and we need to offer a viable open source JMS implementation (we currently use Oracle AQ). Sorry for the confusion around support - I meant our application supporting JBoss Messaging as a JMS provider. We will also consider the paid support option as well though.

                              David

                              • 12. Re: Connecting to two JBoss messaging servers causes interfe
                                davidrh

                                I just ran another test in which our application connected to a single JBoss Messaging server 10 times, before then introducing the second server. I thought this would get the sequence number that Tim mentioned out of step between the two servers. Sure enough, if I stop the first server our application now fails over to the second server correctly and can send messages.

                                It still takes a little while to failover, due to the exception listener not firing for a while after the connection goes down. Is there any way to make the connection verifier run more frequently?

                                • 13. Re: Connecting to two JBoss messaging servers causes interfe
                                  timfox

                                  Assuming the problem is related to what I suspect then it's a fairly simple fix.

                                  Have a look at the differences in the class org.jboss.jms.client.remoting.CallbackManager between the 1.0 branch and TRUNK.

                                  Basically the version in trunk also uses the server id in the key - this is what you want to do.

                                  I can't see any real reason why we shouldn't include this in 1.0.2.

                                  BTW the reason it's not in 1.0 already is I always thought of 1.0 being used in a strictly single server situation. Although I can now see that your use case is a valid one.

                                  • 14. Re: Connecting to two JBoss messaging servers causes interfe
                                    timfox

                                    Also would need to change the ClientDelivery class to pass the server id

                                    1 2 Previous Next