7 Replies Latest reply on Mar 10, 2011 8:46 AM by S ks

    Problem with hornetq and relatively larger message

    S ks Newbie

      Hi ,

       

      Iam using Hornetq server as a transport iam using following paging setting in hornetq-configuration.xml .

         <address-settings>

            <!--default for catch all-->

            <address-setting match="#">

               <!--<dead-letter-address>jms.queue.DLQ</dead-letter-address>-->

               <!--<expiry-address>jms.queue.ExpiryQueue</expiry-address>-->

               <redelivery-delay>0</redelivery-delay>

               <max-size-bytes>154857600</max-size-bytes>

               <page-size-bytes>10485760</page-size-bytes>

               <message-counter-history-day-limit>10</message-counter-history-day-limit>

               <address-full-policy>PAGE</address-full-policy>

            </address-setting>

         </address-settings>

       

      My typical messages are minimum 2.5 MB. But i do not want to let them be treated as large message (As i want them to be stored in the memory )so i have following setting in hornetq-jms.xml

          <connection-factory name="NettyConnectionFactory">

             <connectors>

               <connector-ref connector-name="netty"/>

             </connectors>

              <entries>

                  <entry name="/ConnectionFactory"/>

                  <entry name="/XAConnectionFactory"/>

              </entries>

              <min-large-message-size>650000</min-large-message-size> <!-- This is to make sure that large mesage is anything over 6.5 MB -->

          </connection-factory>

       

      But i still see files getting created under the large-message directory . also i see directories under paging.  Though from JConsole i can see that only 400 mb is used .My -xMX setting is 2048 though. My understanding was that Paging would only kick in when we reach almost the to xMX but not so soon.

       

      Also when i reach around 3000 to 4000 messages , hornetq just simply refuses to take any more messages. System doesnt hangs but lots of we do not recieve any more messages from the producer. I do not see any exception too in the logs.

       

      We do not send acknowledgement soon . We send it after 8-9 hours of processing so these messages remain in the memory or on disk for long time.  We have 8Gb ram and 250GB disk system , and we havent yet reached the memory limit. But i still see the hornetq refusing any further messages

       

      Any idea why this is happening?

        • 1. Problem with hornetq and relatively larger message
          Clebert Suconic Master

          Your min large message size is 650 K (less then that of course.. 650,000 bytes) as opposed to 6M (you're missing a zero, right?)

           

          Second.. the old page model would only depage as you ack messages. look at the docs for more information. The new model won't keep you locked but that's the upcoming version.

          • 2. Problem with hornetq and relatively larger message
            S ks Newbie

            Thank you Clebert.

            But one thing iam still not clear is to why producer stops sending messages to the Hornetq JMS topic ? Or does Hornetq blocks him but i have enabled the paging, so things should be fine. I do understand that depage would happen once the ack is sent but suddenly i stop seeing any further addition to my queues

            • 3. Problem with hornetq and relatively larger message
              Clebert Suconic Master

              It's not clear if the system blocks sends...  or if you just stop receiving messages... If it's blocking you may have some issue with flow control. Try looking at the CF settings for that. (I can give you more information depending on what you tell me about it).

               

               

              Paging happens at the address level at 2.1.2. It protects the memory from OME by not letting more messages to be added to the queues.

               

              It will then read from the Page to the queue once you ACK messages. (i.e. once you free memory).

              • 4. Problem with hornetq and relatively larger message
                S ks Newbie

                WHat i have observed is typically after 3000 unacknowledged messages or so Hornetq server stops taking any more messages and producers go real slow.

                These are my settings

                <journal-buffer-size>7000000</journal-buffer-size>

                 

                  <address-settings>

                      <!--default for catch all-->

                      <address-setting match="#">

                         <!--<dead-letter-address>jms.queue.DLQ</dead-letter-address>-->

                         <!--<expiry-address>jms.queue.ExpiryQueue</expiry-address>-->

                         <redelivery-delay>0</redelivery-delay>

                         <max-size-bytes>154857600</max-size-bytes>

                         <page-size-bytes>10485760</page-size-bytes>

                         <message-counter-history-day-limit>10</message-counter-history-day-limit>

                         <address-full-policy>PAGE</address-full-policy>

                      </address-setting>

                   </address-settings>

                 

                My topic consumers are slow , but i do not want my Topic producers to get slow because of this. i thought its hornetq server's responsibility to take this load via Paging , making sure that Producers still do not get slow.

                 

                Is there any limit on no of UNACKED messages in Hornetq server

                • 5. Problem with hornetq and relatively larger message
                  Clebert Suconic Master

                  when you send over to paging.. you will be writing to page files. You may get a performance hit on syncs at the 2.1 while we have optimized syncs on paging at 2.2 a.so.

                  • 6. Problem with hornetq and relatively larger message
                    S ks Newbie

                    OK, thank you for the input.

                     

                    I saw one more behaviour , i assigned -xmX512M for hornetq  so using jconsole when i see the mem usaage things look fine , as in it is always around 200 mb or less. But when i do free -m , the process memory keeps increasing. As data is coming in .

                     

                    Also incase of Paging , my DurableMessageCounter metric doesnt increase. As in it stops after reaching around 3K , but producer keeps sending the data and also my paging directory keeps increasing .

                    • 7. Problem with hornetq and relatively larger message
                      S ks Newbie

                      My Hornetq server is crashing with the following dump:

                       

                       

                      Exception in thread "hornetq-buffer-timeout" java.lang.OutOfMemoryError: Java heap space

                          at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)

                          at java.nio.ByteBuffer.allocate(ByteBuffer.java:329)

                          at org.hornetq.core.journal.impl.NIOSequentialFileFactory.newBuffer(NIOSequentialFileFactory.java:80)

                          at org.hornetq.core.journal.impl.AbstractSequentialFile.newBuffer(AbstractSequentialFile.java:309)

                          at org.hornetq.core.journal.impl.NIOSequentialFile.newBuffer(NIOSequentialFile.java:278)

                          at org.hornetq.core.journal.impl.AbstractSequentialFile$LocalBufferObserver.newBuffer(AbstractSequentialFile.java:332)

                          at org.hornetq.core.journal.impl.TimedBuffer.flush(TimedBuffer.java:319)

                          at org.hornetq.core.journal.impl.TimedBuffer.flush(TimedBuffer.java:294)

                          at org.hornetq.core.journal.impl.TimedBuffer$CheckTimer.run(TimedBuffer.java:434)

                          at java.lang.Thread.run(Thread.java:636)

                      Mar 10, 2011 1:29:07 PM org.hornetq.core.logging.impl.JULLogDelegate error

                      SEVERE: java.lang.OutOfMemoryError: Java heap space

                      java.lang.OutOfMemoryError: Java heap space

                          at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)

                          at java.nio.ByteBuffer.allocate(ByteBuffer.java:329)

                          at org.hornetq.core.journal.impl.NIOSequentialFileFactory.newBuffer(NIOSequentialFileFactory.java:80)

                          at org.hornetq.core.journal.impl.AbstractSequentialFile.newBuffer(AbstractSequentialFile.java:309)

                          at org.hornetq.core.journal.impl.NIOSequentialFile.newBuffer(NIOSequentialFile.java:278)

                          at org.hornetq.core.journal.impl.AbstractSequentialFile$LocalBufferObserver.newBuffer(AbstractSequentialFile.java:332)

                          at org.hornetq.core.journal.impl.TimedBuffer.flush(TimedBuffer.java:319)

                          at org.hornetq.core.journal.impl.TimedBuffer.flush(TimedBuffer.java:294)

                          at org.hornetq.core.journal.impl.TimedBuffer.checkSize(TimedBuffer.java:222)

                          at org.hornetq.core.journal.impl.AbstractSequentialFile.fits(AbstractSequentialFile.java:162)

                          at org.hornetq.core.journal.impl.JournalImpl.appendRecord(JournalImpl.java:3031)

                          at org.hornetq.core.journal.impl.JournalImpl.appendAddRecordTransactional(JournalImpl.java:1041)

                          at org.hornetq.core.persistence.impl.journal.JournalStorageManager.storeMessageTransactional(JournalStorageManager.java:568)

                          at org.hornetq.core.postoffice.impl.PostOfficeImpl.processRoute(PostOfficeImpl.java:904)

                          at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:668)

                          at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:536)

                          at org.hornetq.core.paging.impl.PagingStoreImpl.onDepage(PagingStoreImpl.java:1027)

                      at org.hornetq.core.paging.impl.PagingStoreImpl.readPage(PagingStoreImpl.java:649)

                          at org.hornetq.core.paging.impl.PagingStoreImpl.access$400(PagingStoreImpl.java:61)

                          at org.hornetq.core.paging.impl.PagingStoreImpl$DepageRunnable.run(PagingStoreImpl.java:1205)

                          at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)

                          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

                          at java.lang.Thread.run(Thread.java:636)

                       

                       

                       

                      These are my settings in hornetq-configuration.xml:

                       

                      <journal-min-files>10</journal-min-files>

                      <security-enabled>false</security-enabled>

                         <journal-buffer-size>7000000</journal-buffer-size>

                       

                       

                         <address-settings>

                            <!--default for catch all-->

                            <address-setting match="#">

                               <!--<dead-letter-address>jms.queue.DLQ</dead-letter-address>-->

                               <!--<expiry-address>jms.queue.ExpiryQueue</expiry-address>-->

                               <redelivery-delay>0</redelivery-delay>

                               <max-size-bytes>154857600</max-size-bytes>

                               <page-size-bytes>10485760</page-size-bytes>        

                               <message-counter-history-day-limit>10</message-counter-history-day-limit>

                               <address-full-policy>PAGE</address-full-policy>

                            </address-setting>

                         </address-settings>

                       

                      My settings in hornetq-jms.xml

                       

                          <connection-factory name="NettyConnectionFactory">

                             <connectors>

                               <connector-ref connector-name="netty"/>

                             </connectors>

                              <entries>

                                  <entry name="/ConnectionFactory"/>

                                  <entry name="/XAConnectionFactory"/>

                              </entries>

                              <min-large-message-size>6500000</min-large-message-size>       

                          </connection-factory>

                       

                       

                      Any thoughts ? , Any kind of help is deeply appreciated.