-
1. Re: Connecting to two JBoss messaging servers causes interfe
davidrh Nov 8, 2006 1:43 AM (in response to davidrh)Also, we are running 1.0.1.GA on JBoss 4.0.4.GA deployed as standalone-messaging.
-
2. Re: Connecting to two JBoss messaging servers causes interfe
timfox Nov 8, 2006 7:09 AM (in response to davidrh)Can you replicate this in a simple test case and send it to us?
-
3. Re: Connecting to two JBoss messaging servers causes interfe
davidrh Nov 8, 2006 10:50 PM (in response to davidrh)Simple test class is attached below. It basically sends messages to one server until it gets an exception, at which point it starts sending them to a second server.
I am running this test case on the server1 (localhost) machine. I run it and leave it for a few minutes sending messages to the server1 JBoss Messaging instance on localhost. I then shutdown the server1 instance and the messages begin to be sent to server2 (as expected). If you leave it run for a few minutes more, eventually sending to server2 will fail as well with the exception:2006-11-09 14:21:36,859 ERROR org.jboss.jms.client.container.ExceptionInterceptor - Caught RuntimeException org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for locator - InvokerLocator [socket://161.117.20.9:4457/?dataType=jms&marshaller=org.jboss.jms.server.remoting.JMSWireFormat& serializationtype=jboss&socket.check_connection=false&unmarshaller=org.jboss.jms.server.remoting.JMSWireFormat] at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:323) at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:125) at org.jboss.remoting.Client.invoke(Client.java:589) at org.jboss.remoting.Client.invoke(Client.java:581) at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.invoke(ClientConnectionFactoryDelegate.java:199) at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$getIdBlock_2799534920426283165.invokeNext(ClientConnectionFactoryDelegate$getIdBlock_2799534920426283165.java) at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69) at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$getIdBlock_2799534920426283165.invokeNext(ClientConnectionFactoryDelegate$getIdBlock_2799534920426283165.java) at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107) at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$getIdBlock_2799534920426283165.invokeNext(ClientConnectionFactoryDelegate$getIdBlock_2799534920426283165.java) at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.getIdBlock(ClientConnectionFactoryDelegate.java) at org.jboss.jms.message.MessageIdGenerator.getNextBlock(MessageIdGenerator.java:72) at org.jboss.jms.message.MessageIdGenerator.getId(MessageIdGenerator.java:86) at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:230) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:130) at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java) at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117) at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java) at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69) at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java) at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107) at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java) at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java) at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:172) at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:220) at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:147) at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:138) at TestMultiConnectionMessageProducer.test(TestMultiConnectionMessageProducer.java:140) at TestMultiConnectionMessageProducer.main(TestMultiConnectionMessageProducer.java:27) Caused by: java.net.ConnectException: Connection refused: connect at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:305) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:171) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:158) at java.net.Socket.connect(Socket.java:452) at java.net.Socket.connect(Socket.java:402) at java.net.Socket.<init>(Socket.java:309) at java.net.Socket.<init>(Socket.java:124) at org.jboss.remoting.transport.socket.SocketClientInvoker.createSocket(SocketClientInvoker.java:183) at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:685) at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:319) ... 32 more
I've put the session.createQueue calls in for each message dispatch as this is what the Spring JmsTemplate class does (which is what we're using in our application). When I debug our application, the first call to the session.createQueue method after the server has failed takes quite a long time (30 seconds or more). I can't replicate this delay in the test case at the moment. In our application, this causes a delay in the failover as at the point of calling createQueue, we don't know that the server has failed yet.
You'll notice in the output that the ExceptionListener on the connection can fire quite a while (30 seconds) after the connection has actually been lost. We use the ExceptionListener to shutdown our session pool and initiate reconnection in a separate thread. Until the listener fires, we may still hand out sessions from the pool that are for the server that is no longer there.
As an aside, when we detect a failed connection what should we do to clean it up? If I put in the code to try closing the Producer, Session and Connection and ignore any exceptions it takes quite a long time to execute (around 60 seconds for session.close() and 60 seconds for connection.close()). With this close code in place, when it tries the second server the error that is generated is:2006-11-09 14:41:00,937 ERROR org.jboss.jms.client.container.ExceptionInterceptor - Caught Exception: org.jboss.aop.NotFoundInDispatcherException: Object with oid: -2147483641 was not found in the Dispatcher at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:85) at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:1008) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:857) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:454) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:541) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:261) at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:172) at org.jboss.remoting.Client.invoke(Client.java:589) at org.jboss.remoting.Client.invoke(Client.java:581) at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111) at org.jboss.jms.client.delegate.ClientSessionDelegate$createQueue_6431069199924553036.invokeNext(ClientSessionDelegate$createQueue_6431069199924553036.java) at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117) at org.jboss.jms.client.delegate.ClientSessionDelegate$createQueue_6431069199924553036.invokeNext(ClientSessionDelegate$createQueue_6431069199924553036.java) at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69) at org.jboss.jms.client.delegate.ClientSessionDelegate$createQueue_6431069199924553036.invokeNext(ClientSessionDelegate$createQueue_6431069199924553036.java) at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107) at org.jboss.jms.client.delegate.ClientSessionDelegate$createQueue_6431069199924553036.invokeNext(ClientSessionDelegate$createQueue_6431069199924553036.java) at org.jboss.jms.client.delegate.ClientSessionDelegate.createQueue(ClientSessionDelegate.java) at org.jboss.jms.client.JBossSession.createQueue(JBossSession.java:272) at TestMultiConnectionMessageProducer.test(TestMultiConnectionMessageProducer.java:135) at TestMultiConnectionMessageProducer.main(TestMultiConnectionMessageProducer.java:27) org.jboss.jms.util.MessagingJMSException: Caught exception Connection 2 failed: at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:99) at org.jboss.jms.client.delegate.ClientSessionDelegate$createQueue_6431069199924553036.invokeNext(ClientSessionDelegate$createQueue_6431069199924553036.java) at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107) at org.jboss.jms.client.delegate.ClientSessionDelegate$createQueue_6431069199924553036.invokeNext(ClientSessionDelegate$createQueue_6431069199924553036.java) at org.jboss.jms.client.delegate.ClientSessionDelegate.createQueue(ClientSessionDelegate.java) at org.jboss.jms.client.JBossSession.createQueue(JBossSession.java:272) at TestMultiConnectionMessageProducer.test(TestMultiConnectionMessageProducer.java:135) at TestMultiConnectionMessageProducer.main(TestMultiConnectionMessageProducer.java:27)
Thanks for your assistance,
David
Test Case:import java.util.Hashtable; import javax.jms.Connection; import javax.jms.ConnectionFactory; import javax.jms.ExceptionListener; import javax.jms.JMSException; import javax.jms.Message; import javax.jms.MessageConsumer; import javax.jms.MessageListener; import javax.jms.MessageProducer; import javax.jms.Queue; import javax.jms.Session; import javax.jms.TextMessage; import javax.naming.Context; import javax.naming.InitialContext; import javax.naming.NamingException; public class TestMultiConnectionMessageProducer { /** * @param args */ public static void main(String[] args) { TestMultiConnectionMessageProducer ml = new TestMultiConnectionMessageProducer(); try { ml.test(); } catch (JMSException e) { // TODO Auto-generated catch block e.printStackTrace(); } } private void test() throws JMSException { // Setup connection 1 Hashtable properties1 = new Hashtable(); properties1.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory"); properties1.put(Context.URL_PKG_PREFIXES, "org.jboss.naming:org.jnp.interfaces"); properties1.put(Context.PROVIDER_URL, "jnp://localhost:1099"); properties1.put(Context.SECURITY_PRINCIPAL, "admin"); properties1.put(Context.SECURITY_CREDENTIALS, "admin"); ConnectionFactory connectionFactory1 = null; try { Context context1 = new InitialContext(properties1); connectionFactory1 = (ConnectionFactory) context1 .lookup("ConnectionFactory"); } catch (NamingException ne) { throw new RuntimeException(ne); } final Connection connection1 = connectionFactory1.createConnection(); ExceptionListener listener1 = new ExceptionListener() { public void onException(JMSException jmse) { System.out.println("Fired exception listener on connection 1"); } }; connection1.setExceptionListener(listener1); Session session1 = connection1.createSession(false, Session.AUTO_ACKNOWLEDGE); boolean connection1OK = true; // Setup connection 2 Hashtable properties2 = new Hashtable(); properties2.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory"); properties2.put(Context.URL_PKG_PREFIXES, "org.jboss.naming:org.jnp.interfaces"); properties2.put(Context.PROVIDER_URL, "jnp://server2:1099"); properties2.put(Context.SECURITY_PRINCIPAL, "admin"); properties2.put(Context.SECURITY_CREDENTIALS, "admin"); ConnectionFactory connectionFactory2 = null; try { Context context2 = new InitialContext(properties2); connectionFactory2 = (ConnectionFactory) context2 .lookup("ConnectionFactory"); } catch (NamingException ne) { throw new RuntimeException(ne); } final Connection connection2 = connectionFactory2.createConnection(); ExceptionListener listener2 = new ExceptionListener() { public void onException(JMSException jmse) { System.out.println("Fired exception listener on connection 2"); } }; connection2.setExceptionListener(listener2); Session session2 = connection2.createSession(false, Session.AUTO_ACKNOWLEDGE); boolean connection2OK = true; boolean finished = false; while (!finished && (connection1OK || connection2OK)) { if (connection1OK) { try { TextMessage msg1 = session1.createTextMessage("Message"); System.out.println("Creating queue 1"); Queue queue1 = session1.createQueue("publish.request"); System.out.println("Creating producer 1"); MessageProducer producer1 = session1.createProducer(queue1); System.out.println("Sending message to queue 1"); producer1.send(msg1); producer1.close(); System.out.println("Connection 1 OK"); System.out.println("=========================="); } catch (Exception e) { connection1OK = false; System.out.println("Connection 1 failed:"); e.printStackTrace(); // try { // System.out.println("Closing session 1"); // session1.close(); // } catch (Exception e1) { // } // try { // System.out.println("Closing connection 1"); // connection1.close(); // } catch (Exception e1) { // } } } if (!connection1OK && connection2OK) { try { System.out.println("Trying server 2"); TextMessage msg2 = session2.createTextMessage("Message"); System.out.println("Creating queue 2"); Queue queue2 = session2.createQueue("publish.request"); System.out.println("Creating producer 2"); MessageProducer producer2 = session2.createProducer(queue2); System.out.println("Sending message to queue 2"); producer2.send(msg2); producer2.close(); System.out.println("Connection 2 OK"); System.out.println("=========================="); } catch (Exception e2) { connection2OK = false; System.out.println("Connection 2 failed:"); e2.printStackTrace(); } } try { Thread.currentThread().sleep(1000); } catch (InterruptedException e) { finished = true; } } } }
-
4. Re: Connecting to two JBoss messaging servers causes interfe
davidrh Nov 9, 2006 12:50 AM (in response to davidrh)I have tried Clebert's suggestion mentioned in the other post of changing the server id in messaging-server.xml:
<constructor> <!-- ServerPeerID --> <arg type="java.lang.String" value="server.0" /> <!-- DefaultQueueJNDIContext --> <arg type="java.lang.String" value="/queue" /> <!-- DefaultTopicJNDIContext --> <arg type="java.lang.String" value="/topic" /> </constructor>
This seems to make things work if you don't try and close the failed session and connection. If you uncomment the code to try and close the session and connection, then the sending of messages to the second server doesn't work at all, with the same "Object with oid was not found in the Dispatcher" error mentioned previously. On the server, the following entry is logged:2006-11-09 16:45:20,457 WARN [org.jboss.jms.server.connectionmanager.SimpleConnectionManager] A problem has been detected with the connection to remote client 4h39k9-33iqxv-euaqkont-1-euaqkq0f-9. It is possible the client has exited without closing its connection(s) or there is a network problem. All connection resources corresponding to that client process will now be removed.
-
5. Re: Connecting to two JBoss messaging servers causes interfe
clebert.suconic Nov 9, 2006 11:29 AM (in response to davidrh)If you uncomment the code to try and close the session and connection, then the sending of messages to the second server doesn't work at all, with the same "Object with oid was not found in the Dispatcher" error mentioned previously. On the server, the following entry is logged:
You lost me... I tried to find what piece of code you were talking about and didn't figure out. -
6. Re: Connecting to two JBoss messaging servers causes interfe
clebert.suconic Nov 9, 2006 11:33 AM (in response to davidrh)Anyway... your example here is kind of confusing IMHO... you are opening the connection out of the loop, and closing it within the loop.
Are you sure you don't have a coding problem within your example?
I have tried two conenctions from the same client (as I'm coding HA support) and never had any problems with it.
Clebert Suconic -
7. Re: Connecting to two JBoss messaging servers causes interfe
timfox Nov 9, 2006 11:36 AM (in response to davidrh)I have a feeling I might know what the problem is (not 100%)
In 1.0.x the callbackservers are keyed only on the consumer id, which is created on the server from a sequence that starts from zero on startup.
So, if you have more than one server you can have more than one consumer with the same id, which can cause problems.
I fixed this some time ago in TRUNK and now the server id + the consumer id used as the key, although this fix is not available currently in 1.0.x -
8. Re: Connecting to two JBoss messaging servers causes interfe
clebert.suconic Nov 9, 2006 11:40 AM (in response to davidrh)Ah... ok... I"m using trunk. That's why I didn't see any problems
-
9. Re: Connecting to two JBoss messaging servers causes interfe
davidrh Nov 9, 2006 5:54 PM (in response to davidrh)The uncommenting that I was talking about referred to the code in my example that is currently commented out for when any exception is caught on sending a message on connection1. I was attempting to see if cleaning up (i.e. trying to close) the defunct connection on an exception affected the operation of the second connection (it does seem to). I think that the example never uses the first connection again after it has detected any exception.
I know this code is not how you would do it in practice, but I was trying to keep the sample as tight as possible to demonstrate the problem. Our actual application is more complex, as it initiates reconnect in a background thread and obtains its sessions from a session pool but I didn't think this was the cause of the problem that we are observing.
Is there any chance of getting the fix that Tim is talking about into a 1.0.x release? We are hoping to offer JBoss Messaging support as part of our application, but with this problem we cannot as we need some form of high availability. Was it a relatively confined change that we could patch in ourselves? Would you be able to give us a start and tell us a class and revision to look at?
As an aside, could the problem that Tim mentioned cause incorrect messages to get placed on a queue? We have sporadic reports from our testers of incorrect messages being placed on the queue. We only get these errors when running JBoss Messaging (i.e. not other providers that we support) and only when running two instances. The error that they were reporting was that callback message objects were being placed onto the queues which then caused class cast exceptions in our application as weren't expecting that type of message. I don't know if that makes any sense to you, but if it doesn't I can get more information.
Thanks for the assistance,
David -
10. Re: Connecting to two JBoss messaging servers causes interfe
clebert.suconic Nov 9, 2006 6:15 PM (in response to davidrh)Is there any chance of getting the fix that Tim is talking about into a 1.0.x release? We are hoping to offer JBoss Messaging support as part of our application, but with this problem we cannot as we need some form of high availability. Was it a relatively confined change that we could patch in ourselves? Would you be able to give us a start and tell us a class and revision to look at?
I will talk to Tim about what are the technical implications. I guess we could make it to 1.0.2 if there are no technical implications.
You know... HA is being worked at 1.2.0, right?to offer JBoss Messaging support
It was dubious to me if you meant support from JBoss Messaging/JBoss. If you are considering paying for support, then we would have contractual obligations about fixing this. I'm not trying to push you anything.. but just in case you meant paying support. -
11. Re: Connecting to two JBoss messaging servers causes interfe
davidrh Nov 9, 2006 6:42 PM (in response to davidrh)Hi Clebert,
We know HA is on the way, but we can't wait. We have an application already and we need to offer a viable open source JMS implementation (we currently use Oracle AQ). Sorry for the confusion around support - I meant our application supporting JBoss Messaging as a JMS provider. We will also consider the paid support option as well though.
David -
12. Re: Connecting to two JBoss messaging servers causes interfe
davidrh Nov 9, 2006 6:56 PM (in response to davidrh)I just ran another test in which our application connected to a single JBoss Messaging server 10 times, before then introducing the second server. I thought this would get the sequence number that Tim mentioned out of step between the two servers. Sure enough, if I stop the first server our application now fails over to the second server correctly and can send messages.
It still takes a little while to failover, due to the exception listener not firing for a while after the connection goes down. Is there any way to make the connection verifier run more frequently? -
13. Re: Connecting to two JBoss messaging servers causes interfe
timfox Nov 10, 2006 4:54 AM (in response to davidrh)Assuming the problem is related to what I suspect then it's a fairly simple fix.
Have a look at the differences in the class org.jboss.jms.client.remoting.CallbackManager between the 1.0 branch and TRUNK.
Basically the version in trunk also uses the server id in the key - this is what you want to do.
I can't see any real reason why we shouldn't include this in 1.0.2.
BTW the reason it's not in 1.0 already is I always thought of 1.0 being used in a strictly single server situation. Although I can now see that your use case is a valid one. -
14. Re: Connecting to two JBoss messaging servers causes interfe
timfox Nov 10, 2006 4:55 AM (in response to davidrh)Also would need to change the ClientDelivery class to pass the server id