8 Replies Latest reply on Dec 9, 2013 4:25 PM by jbertram

    Connection Time Out; after sending a number of objects to queue (HQ119013)

    vahid001

      Hello,

       

      We are currently facing some issues, when sending objects (ObjectMessage) from a local machine to a JBoss instance through JMS. The first (approximately) 10 objects are sent and consumed by the MessageDrivenbean, but after the approximate amount of 10 messages an exception starts occuring.

       

      The exception that occurs, only happens on the client side. I have separated the various information (Stacktrace + standalone.xml + client code)

      I have put the log levels to TRACE on the server side and there is no exception, warning or error.

       

      The JBoss instance is running on an Ubuntu server with no firewall (iptables) rules and all ports are open.

       

      The exception that we are getting (on the client side):

      javax.jms.JMSException: Failed to create session factory

        at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:587)

        at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:107)

        at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:102)

        at org.archive.modules.jms.JMSArticleWriter.sendMessage(JMSArticleWriter.java:105)

        at org.archive.modules.writer.HeliosDBProcessor.write(HeliosDBProcessor.java:345)

        at org.archive.modules.writer.HeliosDBProcessor.innerProcessResult(HeliosDBProcessor.java:205)

        at org.archive.modules.Processor.process(Processor.java:142)

        at org.archive.modules.ProcessorChain.process(ProcessorChain.java:131)

        at org.archive.crawler.framework.ToeThread.run(ToeThread.java:151)

      Caused by: HornetQException[errorType=CONNECTION_TIMEDOUT message=HQ119013: Timed out waiting to receive cluster topology. Group:null]

        at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:893)

        at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:583)

        ... 8 more

       

      This is what we have configured in standalone.xml:

              <subsystem xmlns="urn:jboss:domain:messaging:1.3">

                  <hornetq-server>

                      <persistence-enabled>true</persistence-enabled>

                      <security-enabled>false</security-enabled>

                      <journal-type>NIO</journal-type>

                      <journal-min-files>2</journal-min-files>

       

                      <connectors>

                          <netty-connector name="netty" socket-binding="messager"/>

                          <netty-connector name="netty-throughput" socket-binding="messaging-throughput">

                              <param key="batch-delay" value="50"/>

                          </netty-connector>

                          <in-vm-connector name="in-vm" server-id="0"/>

                      </connectors>

       

                      <acceptors>

                          <netty-acceptor name="netty" socket-binding="messager"/>

                          <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">

                              <param key="batch-delay" value="50"/>

                              <param key="direct-deliver" value="false"/>

                          </netty-acceptor>

                          <in-vm-acceptor name="in-vm" server-id="0"/>

                      </acceptors>

       

                      <security-settings>

                          <security-setting match="#">

                              <permission type="send" roles="guest"/>

                              <permission type="consume" roles="guest"/>

                              <permission type="createNonDurableQueue" roles="guest"/>

                              <permission type="deleteNonDurableQueue" roles="guest"/>

                          </security-setting>

                      </security-settings>

       

                      <address-settings>

                          <address-setting match="#">

                              <dead-letter-address>jms.queue.DLQ</dead-letter-address>

                              <expiry-address>jms.queue.ExpiryQueue</expiry-address>

                              <redelivery-delay>0</redelivery-delay>

                              <max-size-bytes>10485760</max-size-bytes>

                              <address-full-policy>BLOCK</address-full-policy>

                              <message-counter-history-day-limit>10</message-counter-history-day-limit>

                          </address-setting>

                      </address-settings>

       

                      <jms-connection-factories>

                          <connection-factory name="InVmConnectionFactory">

                              <connectors>

                                  <connector-ref connector-name="in-vm"/>

                              </connectors>

                              <entries>

                                  <entry name="java:/ConnectionFactory"/>

                              </entries>

                          </connection-factory>

                          <connection-factory name="RemoteConnectionFactory">

                              <connectors>

                                  <connector-ref connector-name="netty"/>

                              </connectors>

                              <entries>

                                  <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>

                              </entries>

                              <client-failure-check-period>60000</client-failure-check-period>

                              <connection-ttl>150000</connection-ttl>

                              <call-timeout>30000</call-timeout>

                              <consumer-window-size>1048576</consumer-window-size>

                          </connection-factory>

                          <pooled-connection-factory name="hornetq-ra">

                              <transaction mode="xa"/>

                              <connectors>

                                  <connector-ref connector-name="in-vm"/>

                              </connectors>

                              <entries>

                                  <entry name="java:/JmsXA"/>

                              </entries>

                          </pooled-connection-factory>

                          <pooled-connection-factory name="hornetq-ra-nonxa">

                              <transaction mode="local"/>

                              <connectors>

                                  <connector-ref connector-name="in-vm"/>

                              </connectors>

                              <entries>

                                  <entry name="java:/JmsNonXA"/>

                              </entries>

                          </pooled-connection-factory>

                      </jms-connection-factories>

       

                      <jms-destinations>

                          <jms-queue name="testerQueue">

                              <entry name="queue/Testers"/>

                              <entry name="java:jboss/exported/jms/queue/Testers"/>

                          </jms-queue>

                      </jms-destinations>

                  </hornetq-server>

              </subsystem>

       

       

      Client code for connection and sendMessage (the client code is a thread):

      private Context getInitialContext() throws javax.naming.NamingException {

           Context initialContext = null;

           Properties properties = new Properties();

       

           properties.put(Context.INITIAL_CONTEXT_FACTORY, "org.jboss.naming.remote.client.InitialContextFactory");

           properties.put(Context.PROVIDER_URL, "remote://ipHERE:4447");

       

           properties.put(Context.SECURITY_AUTHENTICATION, "simple");

           properties.put(Context.SECURITY_PRINCIPAL, "username");

           properties.put(Context.SECURITY_CREDENTIALS, "password");

           properties.put("jboss.naming.client.ejb.context", true);

       

           initialContext = new InitialContext(properties);

           return initialContext;

        }


      private void closeConnection(Connection con) throws JMSArticleWriterException {

      try {

           if (con != null) {

           con.close();

        }

        } catch (JMSException jmse) {

             jmse.printStackTrace();

             throw new JMSArticleWriterException("Could not close connection " + con + " exception was " + jmse, jmse);

             }

      }


      public synchronized void sendMessage(RawArticle article) throws JMSArticleWriterException{

           Context context = null;

           Connection connection = null;

           try {

                context = getInitialContext();

               connectionFactory = (ConnectionFactory) context.lookup("jms/RemoteConnectionFactory");

                Queue queue = (Queue) context.lookup("jms/queue/Testers");

                connection = connectionFactory.createConnection();

                Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

                MessageProducer publisher = session.createProducer(queue);

              

                connection.start();

       

                ObjectMessage objectMessage = session.createObjectMessage(article);

                publisher.send(objectMessage);

       

        } catch (Exception e) {

                e.printStackTrace();

                throw new JMSArticleWriterException("Exception with producing JMS message " + context + " exception was " + e ,e);

        }

        finally {

                if (context != null) {

                try {

        context.close();

        } catch (Exception e) {

             e.printStackTrace();

        throw new JMSArticleWriterException("Could not close context " + context + " exception was " + e ,e);

             }

        }

        closeConnection(connection);

             }

        }

       

       

      Has anyone experienced this issue before or able to advice. Any help would be much appreciated. Thanks in advance.

        • 1. Re: Connection Time Out; after sending a number of objects to queue (HQ119013)
          jbertram

          The first thing I would do is refactor your code so that you don't perform 2 JNDI lookups and create/close a connection for every message you send.

           

          Second, I would avoid javax.jms.ObjectMessage as it requires serialization overhead.

           

          Third, you probably want to switch from using BLOCK to PAGE <address-full-policy>.

           

          Last, please indicate what version of HornetQ you are using.

          • 2. Re: Connection Time Out; after sending a number of objects to queue (HQ119013)
            vahid001

            Hello Justin,

             

            Thanks for the fast response.

            Here some feedback and questions:

            - HornetQ version is 2.3.1-Final.

            - <address-full-policy> is now set to PAGE, which is the more logical option.

            - Is there an alternative to ObjectMessage that we can use? Our preference is to send a custom built object to the queue.

            - Regarding the 2 JNDI lookups; we do require both the queue and the RemoteConnectionFactory for connection I assume? What would be the most efficient way to go forward with this part?

            • 3. Re: Connection Time Out; after sending a number of objects to queue (HQ119013)
              jbertram

              Is there an alternative to ObjectMessage that we can use? Our preference is to send a custom built object to the queue.

              In software development there's almost always an alternative approach to solving a particular problem.  I'm not designing your application so I can't say what you should use instead of ObjectMessage.  I'm just saying that ObjectMessage might have some drawbacks that you're not aware of and you may wish to avoid.

               

              Regarding the 2 JNDI lookups; we do require both the queue and the RemoteConnectionFactory for connection I assume? What would be the most efficient way to go forward with this part?

              Look them up once and cache the result, and do the same for the JMS connection as well.  It is an anti-pattern to do all this every time you send a message.

              • 4. Re: Connection Time Out; after sending a number of objects to queue (HQ119013)
                vahid001

                Thanks for the advice Justin.

                For the example (and for my learning) I have kept using the ObjectMessage (for now). We are doing the lookup just once now.

                 

                The way I have written the client side; we are using threads now. After sending the first 50 objects and starting connection for the next (second thread), I still keep getting the following error:


                HornetQException[errorType=CONNECTION_TIMEDOUT message=HQ119013: Timed out waiting to receive cluster topology. Group:null]

                 

                We collect the output of different threads in a queue. Once the queue reaches a certain treshold we spool the queue with 1 single JMS connection. For example 50 objects are send in one batch. After that we close all JMS related instances. Processing a queue of 50 objects takes approximately 15 minutes so we create the JMS connection et all with this frequency.

                 

                I am getting really stuck here, hit a wall at this point. Any advice/idea on where we might need to look further?

                Any kind of help would be much appreciated. Thanks in advance.

                • 5. Re: Connection Time Out; after sending a number of objects to queue (HQ119013)
                  jbertram

                  Can you provide a reproducible test-case?

                  • 6. Re: Connection Time Out; after sending a number of objects to queue (HQ119013)
                    vahid001

                    Just to make sure; which information besides; client class (producer) + messagedrivenbean (consumer) + standalone.xml would you preferably require within the test case?

                    • 7. Re: Connection Time Out; after sending a number of objects to queue (HQ119013)
                      jbertram

                      A script to run it all automatically would be ideal.  The less manual set-up required the better.

                      • 8. Re: Connection Time Out; after sending a number of objects to queue (HQ119013)
                        jbertram

                        One more thing...The test-case should only include exactly what is required to trigger the error.  Based on your description of the problem I would be surprised if the MDB was required to reproduce the problem as it appears to be between the broker and the producer.