1 2 Previous Next 19 Replies Latest reply on Oct 16, 2008 9:45 AM by bfach

    JMS Reconnect Issue

    bfach

      Hi,

      I am using JBoss 4.2.3.GA with 1.4.0.SP3. I upgraded both in hopes to fix the following issue.

      In the event that a standalone jms client is connected to multiple JBoss AS via the ClusterConnectionFactory and a server dies, the client will failover to the remaining servers.

      The problem I am trying to solve is if you have 1 server in the cluster at the time. I have found that using an exception listener works reliably if you are using ConnectionFactory but not for ClusteredConnectionFactory.

      If ConnectionFactory is looked up, the connection is reestablished if the server goes down and comes back. If ClusteredConnectionFactory is used, the connection is reestablished the first time the single server dies but fails on the second attempt.

      Here is the code:

      public class JMSConnection implements Runnable {
       protected static Logger log = Logger.getLogger(JMSConnection.class);
       protected static AtomicInteger msgcount = new AtomicInteger();
       private Session session;
       private Queue queue;
       private String dest;
       private BlockingQueue<Serializable> bq;
       private final static ScheduledExecutorService scheduler = Executors
       .newScheduledThreadPool(1);
       private Connection connection = null;
       protected static String sourceName; // the name of the client
       final static int NUM_RETRIES = 1000000000;
       final static int RETRY_INTERVAL = 1000;
       private long id = System.nanoTime();
       volatile boolean doSend = true;
      
       ConnectionFactory connectionFactory;
       MessageProducer producer;
      
       static {
      
       /* The delay between */
       final long delay = 1;
       /* The period to send */
       final long period = 1;
      
       /*
       * Set logging level to debug to have the scheduler report the number of
       * messages being sent by each thread
       */
       scheduler.scheduleAtFixedRate(new Runnable() {
       public void run() {
       if (log.isInfoEnabled()) {
       // if (connected) {
       log.info(sourceName + ": sent " + msgcount + " per "
       + period + " seconds.");
       msgcount.set(0);
       // } else {
       // log.info("Problem with JMS Connection from "
       // + sourceName
       // + "! Please check jboss application server");
       // }
       }
       }
       }, delay, period, TimeUnit.SECONDS);
       }
      
       /**
       * // * // * @param dest // * The name of the destination // * @param bq //
       * * The blocking queue // * @param m_sourceName // * The source name // * @throws
       * Exception //
       */
       public JMSConnection(String dest, BlockingQueue<Serializable> bq,
       String m_sourceName) throws Exception {
       this.dest = dest;
       this.bq = bq;
       sourceName = m_sourceName;
       init();
       log.info("JMS Connection");
       }
      
       /**
       * // * // * @throws Exception //
       */
       private void init() throws Exception {
      
       setUpJMS();
      
       }
      
       public boolean setUpJMS() {
       InitialContext ic;
       try {
       ic = new InitialContext();
       connectionFactory = (ConnectionFactory) ic
       .lookup("ClusteredConnectionFactory");
       queue = (Queue) ic.lookup("queue/RouteManagerQueue");
       connection = connectionFactory.createConnection();
       try {
       log.debug("Connection created ...");
      
       // KEY - register for exception callbacks
       connection.setExceptionListener(new ExceptionListenerImpl());
      
       session = connection.createSession(false,
       Session.AUTO_ACKNOWLEDGE);
       log.debug("Session created ...");
       producer = session.createProducer(queue);
      
       producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT);
       log.debug("Producer created ...");
      
       return true;
       } catch (Exception e) {
       // We failed so close the connection
       try {
       connection.close();
       } catch (JMSException ignored) {
       // Pointless
       }
       // Rethrow the initial problem to where we will log it
       throw e;
       } finally {
       // And close the initial context
       // We don't want to wait for the garbage collector to close it
       // otherwise we'll have useless hanging network connections
       ic.close();
       }
       } catch (Exception e) {
       log.error("Error setting up JMS", e);
       return false;
       }
       }
      
       public void run() {
       int cnt = 0;
       while (doSend) {
       try {
       Serializable s = bq.take();
       ObjectMessage message = session.createObjectMessage();
       message.setObject(s);
       producer.send(message);
       msgcount.incrementAndGet();
       log.info("Sending a attribute from " + id);
       } catch (Exception e) {
       log.error("Problem sending message " + e.getMessage(), e);
       }
       }
       }
      
       private class ExceptionListenerImpl implements ExceptionListener {
       public void onException(JMSException e) {
      
       for (int i = 0; i < NUM_RETRIES; i++) {
       log
       .warn("Connection has problems, trying to re-create it, attempt "
       + (i + 1) + " ...");
      
       try {
       connection.close(); // unregisters the ExceptionListener
       } catch (Exception e2) {
       // I will get an Exception anyway, since the connection to
       // the server is
       // broken, but close() frees up resources associated with
       // the connection
       }
      
       boolean setupOK = setUpJMS();
      
       if (setupOK) {
       log.info("Connection re-established");
       return;
       } else {
       log.warn("Re-creating connection failed, retrying ...");
       }
       }
      
       log.error("Cannot re-establish connection, giving up ...");
       doSend = false;
       }
       }
      

      I am using the remoting jar that comes with 4.2.3.GA, and the messaging-cleint jar that comes with 1.4.0.SP3

      Thanks in advance,

        • 1. Re: JMS Reconnect Issue
          clebert.suconic
          • 2. Re: JMS Reconnect Issue
            bfach

            I have run a test with 1.4.1.CR1 with 4.2.3.GA. The result is the same. The first time the server dies the Client receives an Exception however the second time this does not occur.

            • 3. Re: JMS Reconnect Issue
              bfach

              Thanks for the help. I see the issue in the link is in fact the problem I am having however this problem only occurs the second time the server is killed.

              • 4. Re: JMS Reconnect Issue
                clebert.suconic

                 

                I am using the remoting jar that comes with 4.2.3.GA, and the messaging-cleint jar that comes with 1.4.0.SP3


                I didn't realize this part until I edited your original post adding the code tags to proper format your code.

                You must use the remoting jar we ask you to use on the document according to your release.

                So, when you update to 1.4.1.CR, please use the correct version as stated on the 1.4.1.CR docs.


                Also, are you sure you updated the messaging-client also?


                • 5. Re: JMS Reconnect Issue
                  bfach

                  I think so. Just to be clear. 1.4.1.beta1 is equivalent to 1.4.1.CR1 or is this different?

                  • 6. Re: JMS Reconnect Issue
                    clebert.suconic

                    1.4.1.beta1 is only for JBoss5.
                    You would be better doing this test on JBoss5 (if you want to try 1.4.1.Beta1).

                    For Jboss4, we are working on a release now.

                    • 7. Re: JMS Reconnect Issue
                      bfach

                      Thanks for clearing that up.

                      I should be using 1.4.1.CR1 for jboss 4.2.3.GA then. I will need to download the source and build this as there is no binary release for this yet correct?

                      I will double check that my version of remoting is the correct version.

                      • 8. Re: JMS Reconnect Issue
                        clebert.suconic

                        for 4.2.3 you should use http://anonsvn.jboss.org/repos/messaging/tags/JBossMessaging_1_4_0_SP3_CP04/

                        svn co http://anonsvn.jboss.org/repos/messaging/tags/JBossMessaging_1_4_0_SP3_CP04/ jbm1.4
                        cd jbm1.4
                        ant release-bundle
                        


                        Make sure you update the messaging-client.jar from the release, and that you use remoting-(forgot-the-numbers-now)-SP10, as you will find on the docs for the CP04 (on the generated dir)

                        • 9. Re: JMS Reconnect Issue
                          bfach

                          Clebert,

                          Thanks for the information! The versions are changing all the time so knowing the correct version is difficult.

                          If I were to use 4.2.2.GA, I assume the messaging version would be 1_4_0_SP3_CP02 correct?

                          Thanks again!

                          • 10. Re: JMS Reconnect Issue
                            clebert.suconic

                             

                            If I were to use 4.2.2.GA, I assume the messaging version would be 1_4_0_SP3_CP02 correct?


                            The latest release available is 1.4.0.SP3.

                            The cumulative patches releases are built for JBoss EAP Customers. If you want to use one of the CP releases, use the version I told you before (CP04).

                            Or wait till we release the next community release.

                            • 11. Re: JMS Reconnect Issue
                              bfach

                              Clebert,

                              I will use the CP04 release. Thanks for all the help! I will let you know how it turns out.

                              • 12. Re: JMS Reconnect Issue
                                bfach

                                Clebert,

                                I installed 1.4.0.SP3.CP04 with 4.2.3.GA. The same problem exists. The first reconnect is successful (the exception listener receives the callback). On the first/second attempt the following message is displayed in the logs

                                [JMXMON:JMSConnection] [INFO ] [2008-10-14 13:50:42,337] : simulator: sent 0 per 1 seconds.
                                [JMXMON:SimulatorProducer] [INFO ] [2008-10-14 13:50:42,400] : #Messages sent 1 x Count = 237 of -1
                                [JMXMON:FailoverCommandCenter] [ERROR] [2008-10-14 13:50:42,822] : Failover failed
                                javax.jms.JMSException: Maximum number of failover attempts exceeded. Cannot find a server to failover onto.
                                at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:234)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                at java.lang.reflect.Method.invoke(Method.java:585)
                                at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
                                at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
                                at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
                                at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:129)
                                at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:124)
                                at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                                at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                                at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                                at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                                at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                                at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
                                at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:269)
                                at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect39.invoke(ProducerAspect39.java)
                                at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
                                at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                                at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                                at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
                                at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
                                at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
                                at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)
                                at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
                                at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136)
                                at com.rim.jmxmon.common.jms.JMSConnection.run(JMSConnection.java:159)
                                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
                                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
                                at java.lang.Thread.run(Thread.java:595)


                                The first reconnect you see the exception listener being called however not on the second attempt....

                                Thanks in advance,

                                • 13. Re: JMS Reconnect Issue
                                  bfach

                                  Clebert,

                                  I am following the following example to allow for the JMS Client class to reconnect to JBoss in the event there are no servers in the cluster.

                                  http://www.jboss.org/community/docs/DOC-10931

                                  This works everytime if I am using a non clustered connection factory but not when it is clustered (only reconnects the first time)

                                  • 14. Re: JMS Reconnect Issue
                                    clebert.suconic

                                    If you're using a ClusteredConnectionFactory, why do you want to do manual failover?

                                    That would be done transparently for you.

                                    If you want to control Failover yourself, I would say to not use the ClusteredCFs.

                                    1 2 Previous Next