Why does Hornetq need to support it's own paging implementation?

[jyellin]

This might be a naive question, but I'm wondering what purpose Hornetq's paging capability serves when one could just harness OS backed virtual memory to achieve the same purpose of growing queues beyond physical memory. What am I missing?

[gaohoward]

I don't know if the virtual memory can be controlled via Java. HornetQ is just a process inside a JVM. The heap size for a JVM cannot be growing forever. If you don't want the JVM to be OOM, or just want the HornetQ JVM to use a portion of your memory, paging is the way to go.

[ataylor]

Howard is correct, its a completely different thing. we need to control the amount of memory used within the process/JVM whilst also assuring consistency of messages if a crash happens by reloading the pages. Like Howard I dont know of a Java API to use OS backed paging.

[jyellin]

Thanks for that. Are you saying that all messages are paged even the non persistent once and even before 'max-size-bytes' is reached?

[jbertram]

Once max-size-bytes is reached (but not before) then every message (whether persistent or non-persistent) is paged.

[jyellin]

Looking at all the responses I'm concluding that if I want to use the Paging mechanism as a tool to assure consistency through a crash for non persisted messages then I have to set max-size-bytes to zero, otherwise all messages up to max-size-byes will be lost? That seems a bit odd - wouldn't I just use persisted messages if I wanted to survive a crash?

If I'm just looking for a memory extension mechanism, I think I would just go for OS virtual memory controlled at the JVM level (-Xmx), especially as my host is dedicated entirely to hornetq and isn't running any other apps.

[jbertram]

If you want your messages to survive a crash then use persistent messages.  It's as simple as that.  Paging is not meant as a tool to make non-persistent messages survive a crash.

I suppose you can use set your JVM heap size to accommodate every single message you might possibly have in memory at any given time, but that seems really inefficient and provides no fine-grained control.  It also might kill performance during garbage collection.

[ataylor]

If I'm just looking for a memory extension mechanism, I think I would just go for OS virtual memory controlled at the JVM level (-Xmx), especially as my host is dedicated entirely to hornetq and isn't running any other apps.

would that even work, the OS pages memory between processes not within a process

[jyellin]

yeh - but all I'm concerned about is that my hornetq get's its 2g of heap.  If there isn't room on physical ram for that, i'm happy letting OS page for me.

for my use-case i actually prefer blocking/fail mode, so that consumer processing speed feeds back to producer speed in reasonably rapid fashion; I don't want producers and consumers to go to far out of sync with millions of messages waiting in hornetq (in ram or in page files)

[jbertram]

yeh - but all I'm concerned about is that my hornetq get's its 2g of heap.  If there isn't room on physical ram for that, i'm happy letting OS page for me.

You indicated previously that your "host is dedicated entirely to hornetq and isn't running any other apps."  That being the case I wouldn't expect HornetQ's JVM to have any trouble acquiring 2G of physical RAM from the OS for its heap.  Even my little laptop has 8G of RAM. 

However, it's worth noting that no matter what kind of RAM paging was happening at the OS level the JVM would only ever have that 2G of heap, and if the size of the unconsumed messages in the queues ever exceeded that limit you would get an OutOfMemoryError.  OS paging doesn't solve that problem, but HornetQ paging does.  That's the main reason I find it odd for you to compare OS paging to HornetQ paging.  They solve different problems.

for my use-case i actually prefer blocking/fail mode

Even if you're using FAIL or BLOCK you'll still need to set the max-size-bytes according to your heap because you can still get an OutOfMemoryError if you retain too many messages in memory.

[clebert.suconic]

Feel free to use either of these options...

Letting the OS page won't scale well for most cases as the system would eventually hung and get very slow. If you have options (like a very fast Solid state disk) that would let you perform well with the memory itself being paged you would probably be fine with that.. of course you would have to test it.

you should probably use block or error based on your description.