12 Replies Latest reply on Jun 18, 2012 9:19 AM by stefan_fink

    Why hornetq paging very slow?

    bjrenyong74

      Hi,

           I'm using hornetq 2.2.5, linux aio. I wrote some codes to test hornetq. The program continue sent messages to hornetq, and don't cosume them. Before paging, the speed is fast. But when began paging, the speed became very slow. Search the log, I found the sender sent messages about 10ms, then block about 3s. Why?    

           The log and config is in the attachments.

           Thanks a lot.

        • 1. Re: Why hornetq paging very slow?
          ataylor

          it takes longer because the messages are persisted to disk

          • 2. Re: Why hornetq paging very slow?
            bjrenyong74

            But I should say it's too slow. I think it's not normal. I use persistent queue, why write journal so fast, but so slow to write paging?

            • 3. Re: Why hornetq paging very slow?
              ataylor

              there are extra overheads with paging

              • 4. Re: Why hornetq paging very slow?
                clebert.suconic

                There were some improvements on the latest versions. Why don't you try the version that comes with AS7? you could just download the latest AS7 and try it out as it was a standalone or you could download the branch and build it yourself.

                • 5. Re: Why hornetq paging very slow?
                  bjrenyong74

                  Hi, I used iostat to check:

                  Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
                  cciss/c0d0        0.00     0.00  0.00 61.00     0.00    30.50     1.00     0.22    3.54   2.92  17.80
                  cciss/c0d0p1      0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
                  cciss/c0d0p2      0.00     0.00  0.00 61.00     0.00    30.50     1.00     0.22    3.54   2.92  17.80
                  cciss/c0d0p3      0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
                  cciss/c0d0p4      0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
                  cciss/c0d0p5      0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
                  cciss/c0d0p6      0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
                  cciss/c0d0p7      0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
                  cciss/c0d0p8      0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
                  cciss/c0d1        0.00   123.00  0.00 152.00     0.00  1096.00    14.42     0.99    6.51   6.52  99.10
                  cciss/c0d1p1      0.00   123.00  0.00 152.00     0.00  1096.00    14.42     0.99    6.51   6.52  99.10

                   

                  paging datas on c0d1.

                  Is that means my disk IO performance bottleneck? And I have no choice to tuning?

                  • 6. Re: Why hornetq paging very slow?
                    clebert.suconic

                    There were some improvements as when we sync and making it more asynchronous. Did you get a chance to test the latest version?

                     

                    How are you commiting your data? How are you producing? How are you consuming? (General observations that will help on any message system)

                    • 7. Re: Why hornetq paging very slow?
                      luismcosta

                      Hi,

                      I have a question that relates with this one. Besides the documentation (http://docs.jboss.org/hornetq/2.2.5.Final/user-manual/en/html_single/index.html#paging) I didn’t find any other source of information where I could find an answer.

                      When in Paging Mode, which means that “max-size-bytes” has been reached, what happens to the new messages that are delivered in that destination? Are they paged one by one? I would expect that newest “page-size-bytes” messages would be paged in blocks, and the old ones (who are the first to be delivered) to be kept in memory. This implies that “page-size-bytes” should be less or equal to “max-size-bytes”.

                       

                      Thanks for the help,

                         Luís M. Costa

                      • 8. Re: Why hornetq paging very slow?
                        clebert.suconic

                        That is correct...

                         

                         

                        if you actually size page-size-bytes > max-size that would be a configuration error (you would immediately starve the memory if the page-size > max-page-size whenever you added stuff to the memory).

                         

                         

                        Paging is a overflow of the memory, so the first messages are kept in memory.

                        • 9. Re: Why hornetq paging very slow?
                          luismcosta

                          Hi Clebert!

                           

                          Thank you for your response, but... what is correct?

                          My assumption that 1) the newest "page-size-bytes" messages are stored in blocks or, instead, 2) they are paged one by one on receive?

                           

                          Thanks,

                             Luís M. Costa

                          • 10. Re: Why hornetq paging very slow?
                            stefan_fink

                            I have exactly the same timeout of about 3 seconds when messages are paged out.

                            I tested with 2.2.14 Final, using the core API to generate the messages and also the core API to consume the messages. Consuming isn't a issue.

                            With my testing I found that the buffers which is paged out ist quite small (385 messages which look like "Messsage: 1234").

                            Every time this Buffer is wirtten to disk the delay occurs. I checked it with wireshark and the delay is also there visible.

                             

                            Regards

                             

                            Stefan

                            • 11. Re: Why hornetq paging very slow?
                              clebert.suconic

                              There's a timed buffer writing to the pages. If you have a single producer, you need to wait the conclusion of that write.. it should scale when you have more than one producer... and you can configure the timed buffer.

                              • 12. Re: Why hornetq paging very slow?
                                stefan_fink

                                Ok I have to wait until the write has finished. But why does this take 3 seconds. It can't be a write (IO) bound problem. Something else ist waiting here.

                                The only place I find timed buffer in the docs ist the journal-buffer-timeout and journal-buffer-size and here a max of 490k are written many hundred times a second!

                                Any idea where the paging buffer is configured?