1 2 Previous Next 23 Replies Latest reply on Mar 24, 2010 3:23 PM by kreide

    Journal compaction

      This might be a silly question, but I am having some trouble with getting journal compaction to work. I have downloaded HornetQ 2.0 GA and I am using the core API to test. I made two changes to the configuration file, I set max-size-bytes to 300MB (to avoid running out of memory) and address-full-policy to BLOCK.

       

      In my test I have a loop which adds 1000 "Hello" messages to a durable queue and then reads those 1000 messages back; I repeat this loop forever and I would expect this to keep working since the consumed messages would be deleted. However, what happens is that the journal directory fills up with 15 hornetq-data files, each 10MB, and then message sending just blocks. Since all the messages have been consumed I would expect the journal to be compacted and/or old files deleted, but that is not what is happening.

       

      So my question is: what am I doing wrong? I am simply using ClientConsumer.receive() to obtain the messages and I am calling the acknowledge() method (the session has auto-commit on acknowledge). Is there any way I can find out why the files are not being compacted?

       

      I would appreciate any help or pointers.

        • 1. Re: Journal compaction
          timfox

          No-one is going to be able to help you unless you actually post your client code. We are not psychic

           

          Probably you are not acknowledging your messages.

          • 2. Re: Journal compaction

            Sure, thanks for responding. My code is very similar to an example I found in the docs, "8.2. A simple example of using Core", but I did add the acknowledge() call which (surprisingly) was not part of that example. I also send the messages in a transaction to improve performance (I found it was very slow sending the messages one-by-one). Here is my code:

             

            ClientSessionFactory sessionFactory = ...               

            while (true) {

                 ClientSession session = sessionFactory.createTransactedSession();

                 ClientProducer producer = session.createProducer("bar");

             

                 for (int i = 0; i < 1000; i++) {

                      ClientMessage message = session.createMessage(true);

                      message.getBodyBuffer().writeString("Hello");

                      producer.send(message);

                 }

                 session.commit();

                 session.close();

             

                 session = sessionFactory.createSession();

                 session.start();

                 ClientConsumer consumer = session.createConsumer("bar");

             

                 for (int i = 0; i < 1000; i++) {

                      ClientMessage msgReceived = consumer.receive();

                      msgReceived.acknowledge();

                 }

                 session.close();

            }

             

            My queue is defined like this:

             

              <queues>    

                <queue name="bar">
                  <address>bar</address>
                  <durable>true</durable>
                </queue>
              </queues>

             

            I hope something obvious jumps out. Thanks!

             

            • 3. Re: Journal compaction
              timfox

              What happens if you use a non transacted session?

              • 4. Re: Journal compaction

                The same thing happens actually, but sending the messages is on the order of 100 times slower. It is somewhat interesting that despite not using transactions for the receiving part that is still very fast. Without transactions my send loop (1000 messages) takes 3-5 seconds, while the receive loop takes only 10 milliseconds. With transactions in the send loop it takes on the order of 20 milliseconds. Using transactions in the receive loop does not seem to impact performance significantly. Perhaps this is a clue that the acknowledge is not working properly?

                • 5. Re: Journal compaction
                  timfox

                  Kristian Eide wrote:

                   

                  The same thing happens actually, but sending the messages is on the order of 100 times slower. It is somewhat interesting that despite not using transactions for the receiving part that is still very fast. Without transactions my send loop (1000 messages) takes 3-5 seconds, while the receive loop takes only 10 milliseconds. With transactions in the send loop it takes on the order of 20 milliseconds. Using transactions in the receive loop does not seem to impact performance significantly. Perhaps this is a clue that the acknowledge is not working properly?

                  That's to be expected.

                   

                  When sending non transacted, you're probably sending persistent messages, and you're probably sending them blocking (see user manual), so you'll have a) a full network round trip + b) each message will be persisted individually and the response won't be sent back until the message has been synced to disk.

                   

                  An average disk can do around 100-200 syncs per sec. Network round trip time is dependent on your network, but latencies are often around the order of 0.1 ms. Both of these are limits of the hardware.

                   

                  When you send transacted, it doesn't need a round trip or a sync per message, only one for the commit.

                  • 6. Re: Journal compaction
                    timfox

                    BTW this has been discussed many times on this forum. It might be worth a search.

                    • 7. Re: Journal compaction
                      timfox

                      Now.. looking at your code.

                       

                      I don't understand what you're doing here.

                       

                      You send all your messages before you consume any of them, so clearly if you set policy to BLOCK, without consuming any, the producer is going to block.

                      • 8. Re: Journal compaction

                        I send a batch of 1000 messages, then I receive (consume) a batch of 1000 and acknowledge them as well. I would expect that at this point the server would not retain those 1000 messages since they have already been acknowledged. Is this incorrect? Is there anything more I need to do to get the server to "forget" about these messages and reclaim the disk space the next time it does compaction? I guess this is essentially what I am asking.

                        • 9. Re: Journal compaction

                          I should add that I am able to send and receive over 300.000 messages before the server blocks and I can see 15 journal files on the server.

                          • 10. Re: Journal compaction
                            timfox

                            If the code you pasted is not the actual code you are using to demonstrate the issue, you need to post an example that does demonstrate the issue, otherwise we can't tell what you're doing.

                            • 11. Re: Journal compaction

                              The code I posted is in fact exactly the code I am using. Are you saying the same code does not produce a similar issue for you? Could there be something in my server configuration that needs to be changed? I am basically using the configuration file that came with HornetQ 2.0 GA; the only changes I made in addition to defining the queue are:

                               

                              <max-size-bytes>302428800</max-size-bytes>

                              <address-full-policy>BLOCK</address-full-policy>

                               

                              Basically I want to limit the resources used by the server to make sure consumed messages are being cleaned up properly, which is what it not happening. Can you tell me how I can get the server to compact the journal files? What needs to happen for that to occur?

                               

                              Thanks!

                              • 12. Re: Journal compaction
                                timfox

                                Ok sorry.

                                 

                                I'll try your test program tomorrow and try and figure out what is going on.

                                • 13. Re: Journal compaction

                                  Thanks. I also read in the doc that async acknowledgement of messages is the default, so that explains why this part is fast even without transactions. I am also using the NIO journal if that makes a difference.

                                  • 14. Re: Journal compaction
                                    mcleanl

                                    Sorry to butt-in however I noticed a difference between the example and your actual code. In the example I didn't see any session.commit(); or session.close(); on the producer nor a second session being created. I don't know if that was a typo in the docs or intended.

                                     

                                    i.e. instead of....

                                     

                                    ClientSessionFactory sessionFactory = ...              

                                    while (true) {

                                         ClientSession session = sessionFactory.createTransactedSession();

                                         ClientProducer producer = session.createProducer("bar");

                                     

                                         for (int i = 0; i < 1000; i++) {

                                              ClientMessage message = session.createMessage(true);

                                              message.getBodyBuffer().writeString("Hello");

                                              producer.send(message);

                                         }

                                        session.commit();

                                         session.close();

                                     

                                         session = sessionFactory.createSession();

                                         session.start();

                                         ClientConsumer consumer = session.createConsumer("bar");

                                     

                                         for (int i = 0; i < 1000; i++) {

                                              ClientMessage msgReceived = consumer.receive();

                                              msgReceived.acknowledge();

                                         }

                                         session.close();

                                    }

                                     

                                    have you tried with the ones session like...

                                     

                                    ClientSessionFactory sessionFactory = ...              

                                    while (true) {

                                         ClientSession session = sessionFactory.createTransactedSession();

                                         ClientProducer producer = session.createProducer("bar");

                                     

                                         for (int i = 0; i < 1000; i++) {

                                              ClientMessage message = session.createMessage(true);

                                              message.getBodyBuffer().writeString("Hello");

                                              producer.send(message);

                                         }

                                        //session.commit();

                                         //session.close();

                                         //session = sessionFactory.createSession();


                                         session.start();

                                         ClientConsumer consumer = session.createConsumer("bar");

                                     

                                         for (int i = 0; i < 1000; i++) {

                                              ClientMessage msgReceived = consumer.receive();

                                              msgReceived.acknowledge();

                                         }

                                         session.close();

                                    }

                                     

                                    or even changing

                                     

                                    ClientSession session = sessionFactory.createTransactedSession();

                                     

                                    to

                                     

                                    ClientSession session = sessionFactory.createSession();

                                     

                                    and using the one session for the entire transaction.

                                    1 2 Previous Next