7 Replies Latest reply on Mar 10, 2011 8:46 AM by rahul_516

Problem with hornetq and relatively larger message

rahul_516 Mar 4, 2011 11:38 AM

Hi ,

Iam using Hornetq server as a transport iam using following paging setting in hornetq-configuration.xml .

<address-settings>

<address-setting match="#">

<redelivery-delay>0</redelivery-delay>

<max-size-bytes>154857600</max-size-bytes>

<page-size-bytes>10485760</page-size-bytes>

<message-counter-history-day-limit>10</message-counter-history-day-limit>

<address-full-policy>PAGE</address-full-policy>

</address-setting>

</address-settings>

My typical messages are minimum 2.5 MB. But i do not want to let them be treated as large message (As i want them to be stored in the memory )so i have following setting in hornetq-jms.xml

<connection-factory name="NettyConnectionFactory">

<connector-ref connector-name="netty"/>

</connectors>

</entries>

<min-large-message-size>650000</min-large-message-size>

</connection-factory>

But i still see files getting created under the large-message directory . also i see directories under paging. Though from JConsole i can see that only 400 mb is used .My -xMX setting is 2048 though. My understanding was that Paging would only kick in when we reach almost the to xMX but not so soon.

Also when i reach around 3000 to 4000 messages , hornetq just simply refuses to take any more messages. System doesnt hangs but lots of we do not recieve any more messages from the producer. I do not see any exception too in the logs.

We do not send acknowledgement soon . We send it after 8-9 hours of processing so these messages remain in the memory or on disk for long time. We have 8Gb ram and 250GB disk system , and we havent yet reached the memory limit. But i still see the hornetq refusing any further messages

Any idea why this is happening?

1. Problem with hornetq and relatively larger message

clebert.suconic Mar 4, 2011 12:26 PM (in response to rahul_516)

Your min large message size is 650 K (less then that of course.. 650,000 bytes) as opposed to 6M (you're missing a zero, right?)

Second.. the old page model would only depage as you ack messages. look at the docs for more information. The new model won't keep you locked but that's the upcoming version.
Actions
2. Problem with hornetq and relatively larger message

rahul_516 Mar 4, 2011 1:41 PM (in response to clebert.suconic)

Thank you Clebert.
But one thing iam still not clear is to why producer stops sending messages to the Hornetq JMS topic ? Or does Hornetq blocks him but i have enabled the paging, so things should be fine. I do understand that depage would happen once the ack is sent but suddenly i stop seeing any further addition to my queues
Actions
3. Problem with hornetq and relatively larger message

clebert.suconic Mar 4, 2011 2:37 PM (in response to rahul_516)

It's not clear if the system blocks sends... or if you just stop receiving messages... If it's blocking you may have some issue with flow control. Try looking at the CF settings for that. (I can give you more information depending on what you tell me about it).

Paging happens at the address level at 2.1.2. It protects the memory from OME by not letting more messages to be added to the queues.

It will then read from the Page to the queue once you ACK messages. (i.e. once you free memory).
Actions
4. Problem with hornetq and relatively larger message

rahul_516 Mar 7, 2011 9:22 AM (in response to clebert.suconic)

WHat i have observed is typically after 3000 unacknowledged messages or so Hornetq server stops taking any more messages and producers go real slow.
These are my settings
<journal-buffer-size>7000000</journal-buffer-size>

<address-settings>
      
      <address-setting match="#">
         
         
         <redelivery-delay>0</redelivery-delay>
         <max-size-bytes>154857600</max-size-bytes>
         <page-size-bytes>10485760</page-size-bytes>
         <message-counter-history-day-limit>10</message-counter-history-day-limit>
         <address-full-policy>PAGE</address-full-policy>
      </address-setting>
   </address-settings>

My topic consumers are slow , but i do not want my Topic producers to get slow because of this. i thought its hornetq server's responsibility to take this load via Paging , making sure that Producers still do not get slow.

Is there any limit on no of UNACKED messages in Hornetq server
Actions
5. Problem with hornetq and relatively larger message

clebert.suconic Mar 7, 2011 10:34 AM (in response to rahul_516)

when you send over to paging.. you will be writing to page files. You may get a performance hit on syncs at the 2.1 while we have optimized syncs on paging at 2.2 a.so.
Actions
6. Problem with hornetq and relatively larger message

rahul_516 Mar 10, 2011 8:20 AM (in response to clebert.suconic)

OK, thank you for the input.

I saw one more behaviour , i assigned -xmX512M for hornetq so using jconsole when i see the mem usaage things look fine , as in it is always around 200 mb or less. But when i do free -m , the process memory keeps increasing. As data is coming in .

Also incase of Paging , my DurableMessageCounter metric doesnt increase. As in it stops after reaching around 3K , but producer keeps sending the data and also my paging directory keeps increasing .
Actions
7. Problem with hornetq and relatively larger message

rahul_516 Mar 10, 2011 8:46 AM (in response to rahul_516)

My Hornetq server is crashing with the following dump:

Exception in thread "hornetq-buffer-timeout" java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:329)
    at org.hornetq.core.journal.impl.NIOSequentialFileFactory.newBuffer(NIOSequentialFileFactory.java:80)
    at org.hornetq.core.journal.impl.AbstractSequentialFile.newBuffer(AbstractSequentialFile.java:309)
    at org.hornetq.core.journal.impl.NIOSequentialFile.newBuffer(NIOSequentialFile.java:278)
    at org.hornetq.core.journal.impl.AbstractSequentialFile$LocalBufferObserver.newBuffer(AbstractSequentialFile.java:332)
    at org.hornetq.core.journal.impl.TimedBuffer.flush(TimedBuffer.java:319)
    at org.hornetq.core.journal.impl.TimedBuffer.flush(TimedBuffer.java:294)
    at org.hornetq.core.journal.impl.TimedBuffer$CheckTimer.run(TimedBuffer.java:434)
    at java.lang.Thread.run(Thread.java:636)
Mar 10, 2011 1:29:07 PM org.hornetq.core.logging.impl.JULLogDelegate error
SEVERE: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:329)
    at org.hornetq.core.journal.impl.NIOSequentialFileFactory.newBuffer(NIOSequentialFileFactory.java:80)
    at org.hornetq.core.journal.impl.AbstractSequentialFile.newBuffer(AbstractSequentialFile.java:309)
    at org.hornetq.core.journal.impl.NIOSequentialFile.newBuffer(NIOSequentialFile.java:278)
    at org.hornetq.core.journal.impl.AbstractSequentialFile$LocalBufferObserver.newBuffer(AbstractSequentialFile.java:332)
    at org.hornetq.core.journal.impl.TimedBuffer.flush(TimedBuffer.java:319)
    at org.hornetq.core.journal.impl.TimedBuffer.flush(TimedBuffer.java:294)
    at org.hornetq.core.journal.impl.TimedBuffer.checkSize(TimedBuffer.java:222)
    at org.hornetq.core.journal.impl.AbstractSequentialFile.fits(AbstractSequentialFile.java:162)
    at org.hornetq.core.journal.impl.JournalImpl.appendRecord(JournalImpl.java:3031)
    at org.hornetq.core.journal.impl.JournalImpl.appendAddRecordTransactional(JournalImpl.java:1041)
    at org.hornetq.core.persistence.impl.journal.JournalStorageManager.storeMessageTransactional(JournalStorageManager.java:568)
    at org.hornetq.core.postoffice.impl.PostOfficeImpl.processRoute(PostOfficeImpl.java:904)
    at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:668)
    at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:536)
    at org.hornetq.core.paging.impl.PagingStoreImpl.onDepage(PagingStoreImpl.java:1027)
at org.hornetq.core.paging.impl.PagingStoreImpl.readPage(PagingStoreImpl.java:649)
    at org.hornetq.core.paging.impl.PagingStoreImpl.access$400(PagingStoreImpl.java:61)
    at org.hornetq.core.paging.impl.PagingStoreImpl$DepageRunnable.run(PagingStoreImpl.java:1205)
    at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:636)

These are my settings in hornetq-configuration.xml:

<journal-min-files>10</journal-min-files>
<security-enabled>false</security-enabled>
   <journal-buffer-size>7000000</journal-buffer-size>

   <address-settings>
      
      <address-setting match="#">
         
         
         <redelivery-delay>0</redelivery-delay>
         <max-size-bytes>154857600</max-size-bytes>
         <page-size-bytes>10485760</page-size-bytes>
         <message-counter-history-day-limit>10</message-counter-history-day-limit>
         <address-full-policy>PAGE</address-full-policy>
      </address-setting>
   </address-settings>

My settings in hornetq-jms.xml

    <connection-factory name="NettyConnectionFactory">
       <connectors>
         <connector-ref connector-name="netty"/>
       </connectors>
        <entries>
            <entry name="/ConnectionFactory"/>
            <entry name="/XAConnectionFactory"/>
        </entries>
        <min-large-message-size>6500000</min-large-message-size>
    </connection-factory>

Any thoughts ? , Any kind of help is deeply appreciated.
Actions

Go to original post