5 Replies Latest reply on Mar 4, 2014 6:41 PM by lifeonatrip

    Deleting the messagingjournal as soon as possible

    lifeonatrip
    lifeonatrip Feb 19, 2014 12:07 AM

      Hi all,

       

      I am using a hornetq on Jboss As 7.2 in a cluster active/passive.

      Because of the fact that one of my queues contains sensitive data, I would like to delete the .hq journal files as soon as the message is moved to another queue or consumed.

      I've been playing around with these parameters:

       

      <journal-file-size>51200</journal-file-size>
<journal-min-files>5</journal-min-files>
<journal-compact-min-files>6</journal-compact-min-files>
<journal-compact-percentage>80</journal-compact-percentage>

       

      I have around 800/day of small (few kilobytes) object messages and I would also like to create an empty .hq file in advance in case of spikes.

      I thought that journal-compact-percentage was the answer to my questions since the manual states:

       

      The threshold to start compacting. When less than this percentage is considered live data, we start compacting. Note also that compacting won't kick in until you have at least journal-compact-min-files data files on the journal

      The default for this parameter is 30

      So if less than 80% of data is live, the collection starts (am I right?).

      But unfortunately the files are persisted there for days, I need something that tells to hornetq to delete quickly (a 5 minutes wait time is reasonable).

       

      Thanks for any help with this.

        • 1. Re: Deleting the messagingjournal as soon as possible
          clebert.suconic
          clebert.suconic Feb 19, 2014 4:41 PM (in response to lifeonatrip)

          you could force compactation through a method... it's not currently exposed through management.. but maybe we could think about that.

           

          We don't have such feature on our task list (to cleanup messages from the journal).

           

           

          BTW: Non Persistent Message would be a solution for you? I don't know your use case and if you could maybe request the data again in case it didn't achieve your destination?

            • Actions
          • 2. Re: Deleting the messagingjournal as soon as possible
            lifeonatrip
            lifeonatrip Feb 19, 2014 9:16 PM (in response to clebert.suconic)

            Hi Clebert,

            Thanks for your answer.

             

            Non persistent is not an option because I want the message to survive to crash and restarts and unfortunately there is no way I can retrieve the data again for security reasons.

             

            So, can I get what I need with only manipulating the values of the configuration that I mentioned early in the post?

            If not, in which way I can force the compaction process?

              • Actions
            • 3. Re: Deleting the messagingjournal as soon as possible
              lifeonatrip
              lifeonatrip Feb 28, 2014 1:00 AM (in response to clebert.suconic)

              Hi,

               

              Any Idea?

                • Actions
              • 4. Re: Deleting the messagingjournal as soon as possible
                clebert.suconic
                clebert.suconic Mar 3, 2014 9:24 AM (in response to lifeonatrip)

                You could set min-journal-files as 2 (that's the minimum we can play) and set the file-size to something small, and make sure large-message-size < journal-size.

                 

                That way the messages files will be removed as fast as they are consumed.

                 

                 

                I think you're being over cautious on this. you will affect performance but maybe that's an option for you.

                 

                 

                maybe you could set min-fies = 2 and a file-size of 1M?

                  • Actions
                • 5. Re: Deleting the messagingjournal as soon as possible
                  lifeonatrip
                  lifeonatrip Mar 4, 2014 6:41 PM (in response to lifeonatrip)

                  I think we found the problem.

                  The issue is that the compaction does not work in active/passive cluster mode. When 2 queues are in active/passive the active one, does not compact the journal while the passive one does.

                  This bug has been fixed in this commit , the commit is not just renaming the variable, it's also deleting enableCompact() and disableCompact() from JournalImpl.java:

                   

                  public void replicationSyncPreserveOldFiles()
{
setAutoReclaim(false);
 disableCompact();
}

@Override
public void replicationSyncFinished()
{
 enableCompact();
setAutoReclaim(true);
}

                   

                  These lines basically are always disabling the compaction if a sync if performed, even if the sync is finished. Looking at the code deeply, enableCompact() is actually disabling the compaction, due to the way is called in the other classes (negated). So basically inverting the enableCompact with the disableCompact:

                   

                  public void replicationSyncPreserveOldFiles()
{
setAutoReclaim(false);
 enableCompact();
}

@Override
public void replicationSyncFinished()
{
 disableCompact();
setAutoReclaim(true);
}

                   

                  Will make everything works (at least in 2.3.0.CR1). So now active and passive node are replicating and the active node is also compacting.

                  However in the version 2.4, as per above commit, the disable/enable has been deleted completely.

                   

                  Now the journal-compaction parameters that I am setting up on the <hornetq-server> element are working as expected.

                   

                  Another question would be: Is this change in 2.3.0.CR1 safe? Can we invert the enable/disable here and just replace the existing jboss provided JAR?

                    • Actions