New Paging Proposal

[clebert.suconic]

These are the initial requirements/thought I have regarding the new Paging at queue level:

I will reuse the current model that is in place, while adding these requirements:

- I will initially add this as a new option for paging. (Paging at Queue Level versus Paging at Address Level). We may drop address level later depending on the outcome after testing. But that will be decided later.

- Each *queue* will have a pointer to what page/message is being used.

- When a queue reads the Page, it will read an entire Page in memory, but it will keep just a configured portion needed to fullfill a request.

- I will be using SoftCache here, so I wont't replicate the references unless memory is critical, and we would avoid reading a page multiple times.

- Also, I won't read a page again when a new address is depaging, it will get it from the SoftCache. We will read the file again case the SoftCache is gone. (SoftCache is only cleared when memory level is low)

Changes to support these requirements:

- Differentiate slow consumers and fast consumers on routing.

- We will page when there's any slow consumer.

- The page method will be probably the same (no changes)

- Depage will be done per queue, where we keep a reference on the journal (increasing the pointer after ACK)

- Depage will play with SoftCache to *avoid* as much as possible message duplication.

These are the initial thoughts only that I believe could be done. Any extra comments are welcome!

[clebert.suconic]

Also: I will also make Browser to inspect the paged files in the case of browsing a queue.

[aengineer]

In order to better gude the HQ team as they consider these enahancements, I have updated post:

http://community.jboss.org/thread/154471

with what we (Putnam Investments) believe how paging should work. This of this as our requirements.

[clebert.suconic]

The only thing I'm not agreeing with all you wrote is transparency of settings and page at address level:

I -

    In my experience this doesn't work. Maybe it would work for Putnan, but not on every scenario.

    The best would be to have a max size at the queue level and this would fire paging.

    Using max-size doesn't work well because of the GC behaviour you usually have on VMs. Sometimes the memory is just waiting a GC.

    Anyway, I'm at an early stage of development to make decisions on this. I will have a better picture in a week.

II - You will have page at queue level the way you want, but there will be  users that will prefer paging at address level. It will depend on the architecture used and on the use case. (which won't certainly be Putnan's use case).

[timfox]

Clebert Suconic wrote:

These are the initial requirements/thought I have regarding the new Paging at queue level:

I will reuse the current model that is in place, while adding these requirements:

- I will initially add this as a new option for paging. (Paging at Queue Level versus Paging at Address Level). We may drop address level later depending on the outcome after testing. But that will be decided later.

- Each *queue* will have a pointer to what page/message is being used.

- When a queue reads the Page, it will read an entire Page in memory, but it will keep just a configured portion needed to fullfill a request.

- I will be using SoftCache here, so I wont't replicate the references unless memory is critical, and we would avoid reading a page multiple times.

- Also, I won't read a page again when a new address is depaging, it will get it from the SoftCache. We will read the file again case the SoftCache is gone. (SoftCache is only cleared when memory level is low)

Changes to support these requirements:

- Differentiate slow consumers and fast consumers on routing.

- We will page when there's any slow consumer.

- The page method will be probably the same (no changes)

- Depage will be done per queue, where we keep a reference on the journal (increasing the pointer after ACK)

- Depage will play with SoftCache to *avoid* as much as possible message duplication.

These are the initial thoughts only that I believe could be done. Any extra comments are welcome!

I have to admit that I am having difficulty in parsing your proposal.

A few questions:

Pagining will occur when the size of references and messages in a queue reaches a certain max size. (Question what if a message is shared between multiple queues, e.g. multiple subscriptions? How do you calculate how much of the message is counted per queue? Don't want to double count here.)

When paging, are we going to write multiple page files? I.e one per queue? In this case the same message will get written more than once to different files? Will this cause performance problems?

If you could perhaps summarise it again, a bit more clearly, it would be a great help.

[clebert.suconic]

When doing it at Queue level, I wouldn't make global counts. What would determine page level would be a slow consumer on a Queue.

That means, when a queue reaches the maxConfigured Size, we start to page.

I'm going to implement PageCursors, so we only page once, and have each queue browsing through the files. I won't always read the message again as we will have a page cache that will be evicted on soft-cache.

PageCursor will have the following signature:

public interface PageCursor

{

   Pair<PagePosition, PagedMessage> moveNext();

   void confirm(PagePosition position);

}

public class PagePosition

{

   private long pageNr;

   private long messageNr;

}

When confirm is called, I will have a control on what files to delete.

Another piece that will have to be implemented are Counters for the queue. I have a few ideas on how to implement the counters transactionally without adding a lot of contention... but that will come later on a separate post.

[clebert.suconic]

Another update on the cursors model:

I - QueueImpl will poll messages from the Cursor also. (It's currently only polling from the LInkedList)

II - When the message is acked, I will also store it on the Journal (the pageID + messageID on the page), preserving the latest movements and deliveries from paging.

III - I will do reference counts. I will remove the page and the stored acks, When a page is totally consumed (on all of its cursors, as you can have more than one)

IV - Depage is gone on that case, since the Queue will poll messages during delivery.

VI - The resume position of the cursor will be the latest acked position + 1

VII - The cursor will also have the hability to redelivery messages (return redelivering messages).

VIII - There will be a PageReference in addition to the MessageReference. (This is something that will be more detailed later).

So far it seems that the old depage model can be completely gone. As I said earlier I will decide it later.

[timfox]

Clebert Suconic wrote:

Another update on the cursors model:

I - QueueImpl will poll messages from the Cursor also. (It's currently only polling from the LInkedList)

Please don't put the code in QueueImpl. The delivery in the queue is quite complex enough already.

You should aim to keep the new paging code cleanly separated in it's own classes.

I.e. it shouldn't "sprawl" across many classes

[clebert.suconic]

I'm not sure I can get around on not changing QueueImpl.

The new model is about cursors, and it won't have a depage thread. It would be the same as delivering a message that was stored on memory but it was delivered on the disk (cursor) instead.

I plan to make the changes as minimal as possible though.

I already have all the new classes on its own package (org.hornetq.core.paging.cursor), and having just a few additions to org.hornetq.core.paging, even if I decide to move a few methods to org.hornetq.core.paging itself).

[timfox]

subclass it then

[clebert.suconic]

My plan was to do some basic refactoring on QueueImpl and then add extensions. That will be done next week though as I'm done with all the scenarios with the cursors... I will keep this thread posted with changes if necessary.

[timfox]

Separation of concerns. There should be no need to add anything to QueueImpl.

If you can be more specific as to why you think it's necessary to change QueueImpl we can discuss it.

[clebert.suconic]

QueueImpl is only sending messages that are in memory, and routed.

With the new paging model, messages are stored on the disk. The Cursor will get a message from the Cache, and it shouldn't store its Message buffer after being delivered. All that will need to be changed on QueueImpl.

As I said I will come up with more specifics next week.

[timfox]

There's too little information here for me to comment further.

[clebert.suconic]

A Cursor has the hability to navigate through the pages...

The PageCursor has basically the following interface:

   Pair<PagePosition, ServerMessage> moveNext() throws Exception;

   void ack(PagePosition position) throws Exception;

   void ackTx(Transaction tx, PagePosition position) throws Exception;

   // Plus oher methods to recover state from the journal. I don't need to describe them here on this scope now.

The idea is to have one cursor per queue.

When a message is paged, routing will have to inform the queues that another message is available on disk.

Currently, QueueImpl is always getting a message from the Collections that are on QueueImpl. I will need to change it to pull messages out of the Cursor.

That could be done either by an extension on QueueImpl, or having some class encapsulating the Collections. I plan to change it as little as possible here, just adding the extra behaviour needed to pull the cursors.