1 2 Previous Next 15 Replies Latest reply on Oct 12, 2010 6:33 PM by Clebert Suconic

    New Paging Proposal

    Clebert Suconic Master

      These are the initial requirements/thought I have regarding the new Paging at queue level:

       

      I will reuse the current model that is in place, while adding these requirements:

       

       

      - I will initially add this as a new option for paging. (Paging at Queue Level versus Paging at Address Level). We may drop address level later depending on the outcome after testing. But that will be decided later.

       

      - Each *queue* will have a pointer to what page/message is being used.

       

      - When a queue reads the Page, it will read an entire Page in memory, but it will keep just a configured portion needed to fullfill a request.

       

      - I will be using SoftCache here, so I wont't replicate the references unless memory is critical, and we would avoid reading a page multiple times.

       

      - Also, I won't read a page again when a new address is depaging, it will get it from the SoftCache. We will read the file again case the SoftCache is gone. (SoftCache is only cleared when memory level is low)

       

       

      Changes to support these requirements:

       

      - Differentiate slow consumers and fast consumers on routing.

       

      - We will page when there's any slow consumer.

       

      - The page method will be probably the same (no changes)

       

      - Depage will be done per queue, where we keep a reference on the journal (increasing the pointer after ACK)

       

      - Depage will play with SoftCache to *avoid* as much as possible message duplication.

       

       

       

      These are the initial thoughts only that I believe could be done. Any extra comments are welcome!

        • 1. Re: New Paging Proposal
          Clebert Suconic Master

          Also: I will also make Browser to inspect the paged files in the case of browsing a queue.

          • 2. Re: New Paging Proposal
            Aspi Engineer Newbie

            In order to better gude the HQ team as they consider these enahancements, I have updated post:

            http://community.jboss.org/thread/154471

            with what we (Putnam Investments) believe how paging should work. This of this as our requirements.

            • 3. Re: New Paging Proposal
              Clebert Suconic Master

              The only thing I'm not agreeing with all you wrote is transparency of settings and page at address level:

               

               

              I -

               

               

                  In my experience this doesn't work. Maybe it would work for Putnan, but not on every scenario.

               

                  The best would be to have a max size at the queue level and this would fire paging.

               

                  Using max-size doesn't work well because of the GC behaviour you usually have on VMs. Sometimes the memory is just waiting a GC.

               

                  Anyway, I'm at an early stage of development to make decisions on this. I will have a better picture in a week.

               

               

               

              II - You will have page at queue level the way you want, but there will be  users that will prefer paging at address level. It will depend on the architecture used and on the use case. (which won't certainly be Putnan's use case).

              • 4. Re: New Paging Proposal
                Tim Fox Master

                Clebert Suconic wrote:

                 

                These are the initial requirements/thought I have regarding the new Paging at queue level:

                 

                I will reuse the current model that is in place, while adding these requirements:

                 

                 

                - I will initially add this as a new option for paging. (Paging at Queue Level versus Paging at Address Level). We may drop address level later depending on the outcome after testing. But that will be decided later.

                 

                - Each *queue* will have a pointer to what page/message is being used.

                 

                - When a queue reads the Page, it will read an entire Page in memory, but it will keep just a configured portion needed to fullfill a request.

                 

                - I will be using SoftCache here, so I wont't replicate the references unless memory is critical, and we would avoid reading a page multiple times.

                 

                - Also, I won't read a page again when a new address is depaging, it will get it from the SoftCache. We will read the file again case the SoftCache is gone. (SoftCache is only cleared when memory level is low)

                 

                 

                Changes to support these requirements:

                 

                - Differentiate slow consumers and fast consumers on routing.

                 

                - We will page when there's any slow consumer.

                 

                - The page method will be probably the same (no changes)

                 

                - Depage will be done per queue, where we keep a reference on the journal (increasing the pointer after ACK)

                 

                - Depage will play with SoftCache to *avoid* as much as possible message duplication.

                 

                 

                 

                These are the initial thoughts only that I believe could be done. Any extra comments are welcome!

                I have to admit that I am having difficulty in parsing your proposal.

                 

                A few questions:

                 

                Pagining will occur when the size of references and messages in a queue reaches a certain max size. (Question what if a message is shared between multiple queues, e.g. multiple subscriptions? How do you calculate how much of the message is counted per queue? Don't want to double count here.)

                 

                When paging, are we going to write multiple page files? I.e one per queue? In this case the same message will get written more than once to different files? Will this cause performance problems?

                 

                If you could perhaps summarise it again, a bit more clearly, it would be a great help.

                • 5. Re: New Paging Proposal
                  Clebert Suconic Master

                  When doing it at Queue level, I wouldn't make global counts. What would determine page level would be a slow consumer on a Queue.

                   

                  That means, when a queue reaches the maxConfigured Size, we start to page.

                   

                  I'm going to implement PageCursors, so we only page once, and have each queue browsing through the files. I won't always read the message again as we will have a page cache that will be evicted on soft-cache.

                   

                   

                  PageCursor will have the following signature:

                   

                   

                   

                  public interface PageCursor

                  {

                     Pair<PagePosition, PagedMessage> moveNext();

                   

                     void confirm(PagePosition position);

                  }

                   

                  public class PagePosition

                  {

                     private long pageNr;

                   

                     private long messageNr;

                  }

                   

                   

                  When confirm is called, I will have a control on what files to delete.

                   

                   

                   

                  Another piece that will have to be implemented are Counters for the queue. I have a few ideas on how to implement the counters transactionally without adding a lot of contention... but that will come later on a separate post.

                  • 6. Re: New Paging Proposal
                    Clebert Suconic Master

                    Another update on the cursors model:

                     

                    I - QueueImpl will poll messages from the Cursor also. (It's currently only polling from the LInkedList)

                    II - When the message is acked, I will also store it on the Journal (the pageID + messageID on the page), preserving the latest movements and deliveries from paging.

                    III - I will do reference counts. I will remove the page and the stored acks, When a page is totally consumed (on all of its cursors, as you can have more than one)

                    IV - Depage is gone on that case, since the Queue will poll messages during delivery.

                     

                    VI - The resume position of the cursor will be the latest acked position + 1

                     

                    VII - The cursor will also have the hability to redelivery messages (return redelivering messages).

                     

                    VIII - There will be a PageReference in addition to the MessageReference. (This is something that will be more detailed later).

                     

                    So far it seems that the old depage model can be completely gone. As I said earlier I will decide it later.

                    • 7. Re: New Paging Proposal
                      Tim Fox Master

                      Clebert Suconic wrote:

                       

                      Another update on the cursors model:

                       

                      I - QueueImpl will poll messages from the Cursor also. (It's currently only polling from the LInkedList)

                      Please don't put the code in QueueImpl. The delivery in the queue is quite complex enough already.

                       

                      You should aim to keep the new paging code cleanly separated in it's own classes.

                       

                      I.e. it shouldn't "sprawl" across many classes

                      • 8. Re: New Paging Proposal
                        Clebert Suconic Master

                        I'm not sure I can get around on not changing QueueImpl.

                         

                        The new model is about cursors, and it won't have a depage thread. It would be the same as delivering a message that was stored on memory but it was delivered on the disk (cursor) instead.

                         

                        I plan to make the changes as minimal as possible though.

                         

                         

                        I already have all the new classes on its own package (org.hornetq.core.paging.cursor), and having just a few additions to org.hornetq.core.paging, even if I decide to move a few methods to org.hornetq.core.paging itself).

                        • 9. Re: New Paging Proposal
                          Tim Fox Master

                          subclass it then

                          • 10. Re: New Paging Proposal
                            Clebert Suconic Master

                            My plan was to do some basic refactoring on QueueImpl and then add extensions. That will be done next week though as I'm done with all the scenarios with the cursors... I will keep this thread posted with changes if necessary.

                            • 11. Re: New Paging Proposal
                              Tim Fox Master

                              Separation of concerns. There should be no need to add anything to QueueImpl.

                               

                              If you can be more specific as to why you think it's necessary to change QueueImpl we can discuss it.

                              • 12. Re: New Paging Proposal
                                Clebert Suconic Master

                                QueueImpl is only sending messages that are in memory, and routed.

                                 

                                With the new paging model, messages are stored on the disk. The Cursor will get a message from the Cache, and it shouldn't store its Message buffer after being delivered. All that will need to be changed on QueueImpl.

                                 

                                As I said I will come up with more specifics next week.

                                • 13. Re: New Paging Proposal
                                  Tim Fox Master

                                  There's too little information here for me to comment further.

                                  • 14. Re: New Paging Proposal
                                    Clebert Suconic Master

                                    A Cursor has the hability to navigate through the pages...

                                     

                                     

                                    The PageCursor has basically the following interface:

                                     

                                       Pair<PagePosition, ServerMessage> moveNext() throws Exception;

                                     

                                       void ack(PagePosition position) throws Exception;

                                     

                                       void ackTx(Transaction tx, PagePosition position) throws Exception;

                                     

                                       // Plus oher methods to recover state from the journal. I don't need to describe them here on this scope now.

                                     

                                     

                                    The idea is to have one cursor per queue.

                                     

                                     

                                    When a message is paged, routing will have to inform the queues that another message is available on disk.

                                     

                                    Currently, QueueImpl is always getting a message from the Collections that are on QueueImpl. I will need to change it to pull messages out of the Cursor.

                                     

                                    That could be done either by an extension on QueueImpl, or having some class encapsulating the Collections. I plan to change it as little as possible here, just adding the extra behaviour needed to pull the cursors.

                                     

                                     

                                     

                                    1 2 Previous Next