10 Replies Latest reply on Feb 17, 2010 10:36 PM by clebert.suconic

    OutOfMemory error with HornetQ 2.0

    hgomez

      Hi to all,

       

      I'm conducting a JMS bench for JMS brokers, including IBM Websphere MQ 6.x, ASF ActiveMQ 5.3.0 and HornetQ 2.0.

       

      The bench itself is really simple, one or more 'producers' send MapMessages to the brokers and one or more 'consumers' get them.

      The tests are conducted in persistant/non persistant mode (producers) and durable/non durable for consumers.

       

      Clients applications are on Linux machine and the broker on another, thru a 1Gb/s LAN.

       

      I tried to do the same with HornetQ 2.0 GA but get OOM errors :

       

      ie :

       

      [New I/O server worker #1-2] 14:32:38,326 WARNING [org.jboss.netty.channel.socket.nio.NioWorker]  Unexpected exception in the selector loop.
      java.lang.OutOfMemoryError: GC overhead limit exceeded
              at org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:49)
              at org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChannelBuffer.java:39)
              at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:131)
              at org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferFactory.java:72)
              at org.jboss.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChannelBufferFactory.java:53)
              at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:298)
              at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)
              at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)
              at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
              at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:179)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
              at java.lang.Thread.run(Thread.java:619)

       

      I tried IBM JVM 1.6.0-sr6, Sun JDK 1.6.0-18 and still got the same problem.

       

      HornetQ is launched with -Xms512m -Xmx1024m.

       

      export JVM_ARGS="-Xms512M -Xmx1024M -Djava.util.logging.config.file=$CONFIG_DIR/logging.properties -Dhornetq.remoting.netty.host=0.0.0.0 -Dhornetq.config.dir=$CONFIG_DIR -Djava.library.path=."

       

      I tried with and without libAIO, same problem.

       

      I disable JNDI and even JMX console, no way.

       

      The producer detect a problem after between 600/700 messages sent, each being a MapMessage with 20 keys of Strings of 40 chars.

       

      To avoid the OOM, I updated the address like this :

       

         <address-settings>
            <!--default for catch all-->
            <address-setting match="#">
               <dead-letter-address>jms.queue.DLQ</dead-letter-address>
               <expiry-address>jms.queue.ExpiryQueue</expiry-address>
               <redelivery-delay>0</redelivery-delay>
               <max-size-bytes>20485760</max-size-bytes>
               <page-size-bytes>10485760</page-size-bytes>
               <message-counter-history-day-limit>10</message-counter-history-day-limit>
            </address-setting>
         </address-settings>

       

       

      Now I see producer sending 1000, 2000, 3000 messages but consumer still at 100 messages received

      Could it be because the producer use the Session.AUTO_ACKNOWLEDGE ?

       

      Regards

        • 1. Re: OutOfMemory error with HornetQ 2.0
          clebert.suconic
          Do you have some code replicating this?
          • 2. Re: OutOfMemory error with HornetQ 2.0
            hgomez
            Yes and I could send it to you if needed (it's a Maven project).
            • 3. Re: OutOfMemory error with HornetQ 2.0
              clebert.suconic

              The best would be If that's something you could share with everybody, then you could just attach here on a message.

              • 4. Re: OutOfMemory error with HornetQ 2.0
                timfox

                You probably didn't set a max size in memory for the topic, you need something like this:

                 

                      <address-setting match="jms.topic.#">
                         <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                         <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                         <redelivery-delay>0</redelivery-delay>
                         <max-size-bytes>104857600</max-size-bytes>
                         <address-full-policy>BLOCK</address-full-policy>
                         <message-counter-history-day-limit>10</message-counter-history-day-limit>
                      </address-setting>

                 

                See the user manual for more info.

                • 5. Re: OutOfMemory error with HornetQ 2.0
                  hgomez

                  I also tried :

                   

                     <address-settings>
                        <!--default for catch all-->
                        <address-setting match="#">
                           <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                           <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                           <redelivery-delay>0</redelivery-delay>
                           <max-size-bytes>20485760</max-size-bytes>
                           <page-size-bytes>10485760</page-size-bytes>
                           <message-counter-history-day-limit>10</message-counter-history-day-limit>
                        </address-setting>
                     </address-settings>

                  • 6. Re: OutOfMemory error with HornetQ 2.0
                    timfox

                    Benchmarks have been discussed several times recently on this forum in different threads, here is an example:

                     

                    http://community.jboss.org/thread/147533?tstart=0

                     

                    It is very easy to make mistakes with benchmarks. The vast majority of benchmarks I've seen are flawed in one way or another. E.g. not leaving enough time for warmup, testing with too few consumers/producers, not turning off disk cache etc.

                     

                    We are ourselves producing benchmark comparisons pretty soon. We'll also publish the clients so you can run the them yourselves to validate the results, so you can verify everything is above board and kosher.

                    • 7. Re: OutOfMemory error with HornetQ 2.0
                      timfox
                      Like I say, use BLOCK don't page
                      • 8. Re: OutOfMemory error with HornetQ 2.0
                        hgomez

                        BLOCK mode used.

                         

                        benchs are running

                        • 9. Re: OutOfMemory error with HornetQ 2.0
                          clebert.suconic

                          Henri sent me his stuff in a pvt mail.

                           

                          I've run it once and it was taking forever for no absolute reason. I was too curious then and decided to investigate:

                           

                           

                          As a result I've opened a new JIRA:

                           

                           

                          https://jira.jboss.org/jira/browse/HORNETQ-304

                           

                           

                          But I don't think that's a bug.

                           

                          You will be changing a message that is being asynchronously send by another thread. So.. this is causing synchronization issues.

                           

                           

                          Each message holds a buffer on it.. that's part of our optimization. If you keep changing while another async thread is sending it.. it may cause issues.

                           

                           

                          So, @Henri you need to create a new MapMessage on every message you send. I think that basically this test is invalid (not just for HornetQ) until you change how you send the message.

                          • 10. Re: OutOfMemory error with HornetQ 2.0
                            clebert.suconic

                            Actually, yeah.. it is a bug..

                             

                            I'm fixing it.

                             

                             

                            Henry, if you could please workaround it by creating the MapMessage on each send.