7 Replies Latest reply on Mar 9, 2011 4:12 AM by ataylor

    After publishing 22068 msgs to a queue msg producer does not publishes any more msgs

    rdgupt

      Environment: Windows XP, Java 1.6.0_20

       

      I am evaluating HornetQ performance and msg size+volume capacity to consider it as a enterprise msgng system for our org.

      Using out of the box configuration of HornetQ 2.1.1 Final, I tried to send 600,000 small text msgs to it. But after sending 22068 msgs, producer didn't publish any more msgs and kind of hung. It is consistently getting stuck at 22068. There is a possibility that some configuration parameter that I don't know of. Please help. Let me know if I am doing anything wrong anywhere.

       

      Below is the relevant java code snippet which was used (referenced from hornetq jms client examples).

                  // Step 1. Create an initial context to perform the JNDI lookup.

                  Hashtable<String, String> env = new Hashtable<String, String>();

                  env.put(Context.PROVIDER_URL, "jnp://localhost:1099");

                  env.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory");

                  env.put(Context.URL_PKG_PREFIXES, "org.jboss.naming:org.jnp.interfaces  ");

                  Context ctx = new InitialContext(env);

       

                  // Step 2. Lookup the connection factory

                  ConnectionFactory cf = (ConnectionFactory)ctx.lookup("/ConnectionFactory");

       

                  // Step 3. Lookup the JMS queue

                  Queue queue = (Queue)ctx.lookup("/queue/ExampleQueue");

       

                  // Step 4. Create the JMS objects to connect to the server and manage a session

                  Connection connection = cf.createConnection();

                  Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

       

                  // Step 5. Create a JMS Message Producer to send a message on the queue

                  MessageProducer producer = session.createProducer(queue);

                  TextMessage message = null;

                  // Step 6. Create a Text Message and send it using the producer

                  for(int i=0;i<600000;i++){

                      message = session.createTextMessage("Hello, HornetQ!" + i);

                      producer.send(message);

                      System.out.println(" - " + i);

                  }

                  // Finally, we clean up all the JMS resources

                  connection.close();

        • 1. After publishing 22068 msgs to a queue msg producer does not publishes any more msgs
          clebert.suconic

          Look at paging on the user manual.

          1 of 1 people found this helpful
          • 2. After publishing 22068 msgs to a queue msg producer does not publishes any more msgs
            rdgupt

            If I comment max-size-bytes in hornetq-configuration.xml file, as shown in below snippet then the problem is resolved.

            Producer continously keeps on sending msgs even beyond 22068. Though in a good configuration one should never comment this setting and rather configure as appropriate to their needs. This was just part of testing/evaluation and learning.

             

               <address-settings>

                  <!--default for catch all-->

                  <address-setting match="#">

                     <dead-letter-address>jms.queue.DLQ</dead-letter-address>

                     <expiry-address>jms.queue.ExpiryQueue</expiry-address>

                     <redelivery-delay>0</redelivery-delay>

                     <!--max-size-bytes>10485760</max-size-bytes-->

                     <message-counter-history-day-limit>10</message-counter-history-day-limit>

                     <address-full-policy>BLOCK</address-full-policy>

                  </address-setting>

               </address-settings>

             

            Thanks to Clebert for his helpful clue.

            • 3. After publishing 22068 msgs to a queue msg producer does not publishes any more msgs
              clebert.suconic

              We advice people to consume their messages through the message broker.

               

              Sometimes people tend to use a message system like a cache or db system, and we always advice against that. (The same advice goes to any messaging system.. use the right tool to the right problem)

               

               

              On the case, people should always have a consumer ready, unless in cases of failure (in which case page will kick in).

               

               

              You shouldn't see this issue if you had a consumer ready, hence that's why we keep the max-size configured by default, and we also document the reason so people can make it custom accordingly to exceptions that may be suitable to someone's architecture. (Maybe yours)

              • 4. After publishing 22068 msgs to a queue msg producer does not publishes any more msgs
                rdgupt

                In real life scenarios definitely a consumer will be ready for a queue, but as I mentioned in the beginning of my question that I am evaluating hornetq and doing it's capacity testing. That is why not keeping any consumer and testing how hornetq performance degrades with continous pumping of msgs to it.

                This initial test was with small size msgs. Next we are going to test with each msg more than 3 MB and pump in about 100000 msgs.

                 

                Why are doing this kind of test is, because in real life scenarios when disparate systems send msgs to each other sometimes consumers are slow because of doing various business processing as well a batch process triggered on another system wants to push all msgs (say 10000) to queue and finish the batch job. With various other bottlenecks in enterprise and system constraints we needs to figure out upto what level our messaging system can help decouple systems from each other.

                • 5. After publishing 22068 msgs to a queue msg producer does not publishes any more msgs
                  clebert.suconic
                  Why are doing this kind of test is, because in real life scenarios when disparate systems send msgs to each other sometimes consumers are slow because of doing various business processing as well a batch process triggered on another system wants to push all msgs (say 10000) to queue and finish the batch job. With various other bottlenecks in enterprise and system constraints we needs to figure out upto what level our messaging system can help decouple systems from each other.

                   

                   

                  That's the reason we offer paging.

                  • 6. After publishing 22068 msgs to a queue msg producer does not publishes any more msgs
                    257980

                    Hi Clebert,

                     

                    It is regarding the configuration Rishi used.

                    Address Full Policy = Blocked (When the memory allocated for the queue becomes full block the sender).

                    Max Memory Size(max-size-bytes) allocated for the queue is commented out does this means the queue can store messages untill the memory becomes full.

                    Also could you tell us which memory is used by hornetQ for maintaing queue's in memory(is it heap memory?? or a different memory).

                     

                    Thanks,

                    Ravisankar C

                    • 7. After publishing 22068 msgs to a queue msg producer does not publishes any more msgs
                      ataylor

                      having a Block policy and no max-size-bytes makes no sense. basically the server will keep keep receiving messages until the JVM runs out of memory and the server will hang and thro OotOfMemoryException, this is not good. We offer Block and paging to stop this happening.

                       

                      As to your question about which memory is used, messages are just java objects and memory is controlled by the JVM as always.

                       

                      Maybe if you tell us what you are trying to acheive we could be of more help.