[bmmc]

I plan on working on it this afternoon. I will try recreating the problem with and without compacting enabled.

[bmmc]

I have made no configuration changes (other than deleting the journal files to make the error go away) but I have yet been able to re-create the problem.

Therefore, I have not changed to using compact-percentage/compact-min-files = 0 yet. If the problem happens again I will do so.

After starting 1 thread to put 500,000 messages while another thread reads from (so 2 connections/sesions in total) the test completed in ~2 hours with only 1,014 message remaining on the queue (according to jconsole MessageCount attribute). However, it was not until bouncing HQ and restarting the receiver were the last 1,014 messages consumed. For some reason, HQ appeared to have stalled out delivering the final 1,014 messages to the JMSReceiver.

We have seen this behaviour on several tests so it's likely I can reproduced it if you want me to (and post to a different dicussion thread)?

[clebert.suconic]

The only way I could replicate this so far was with compacting enabled.


I would be interested in your tests without compacting.. then we are sure it is the same issue.


However, even on the compacting scenario I'm dealing with, I had a few exceptions due to memory constraints that made the journal unstable. I'm also interested if you had seen other exceptions during your tests.


The next step after disabling compacting in your test, will be to configure paging attribute to protect possible OME in your tests.

[bmmc]

I can easily run through test test scenario this afternoon again with compacting disabled, but since I could not re-create the issue with whatever the default policy is for compacting (what I would have been using on the first occurence of the problem) I don't understand how that's going to help.

I did not have any other reported exceptions to the clients or the hq log previous to the occurence of the problem during a bounce.

I do not think I had any memory issues with the previous occurence of the problem. I was monitoring heap with jconsole and it was only actively using 300MB of a total 2GB I had set for min and max.

Paging was not enabled either other than whatever the default would be (I think that means no paging '-1').

Thanks,

BMMc.

PS: Could you please explain compacting further? If compacting is disabled, would the journal files still be deleted eventually? What I was thinking is that if I had huge file system where journal files are written, I could just permenately disable compacting to increase performance since there is less work to be done? Please let me know your thoughts. We want the absolute best performance possible. We have SAN and server resources available to accomodate whatever you suggest.

[clebert.suconic]

The journal is a circular file system. When a file has every record deleted, that file is then reused.

However, if a file only has just a few records, that file is eligible for compacting. Say... you have a destination that doesn't have any consumers on it... messages will just be added to the journal... while other destinations are sending and consuming fast.

On that case you could have one message per file. When there are too many files on that scenario we kick in compacting to merge those records in a single file.

If you only have a queue in your system, you shouldn't have compacting being used, and on that case it wouldn't make any difference for you. But since I don't know what happened at your system, that's the only possibility.

So far, the only scenario I could damage the journal like the way you have was when compacting failed with OME.

It would help me a lot if you still had those failing files.. so I would be able to determine any extra scenarios.

[bmmc]

Thanks for the information.

I forgot to tell you that I did something dumb and failed to make a copy of the ./data/ dir before cleaning it to start a new test. I really wish I hadn't done that because now I can't seem to re-create the error. We still have time allocated to test further so if I can get it to fail then I won't forget again to save the files first :)

With our current JMS provider we have hundreds of queues and thousand of concurrent connections/sessions from dozens of JVMs on one node/physical server (we have dozens of these nodes joined together using various type of network clustering). Each node sends/receives about ~100 million messages a day of varying sizes with a total of ~8 billion a month.

If we were to have 1 HQ JVM per node with the same type of load and configuration what kind of compacting policy (and paging policy) would you suggest (ignoring the error we have been discussing for a momment).

Your thoughts would be appreciated.

Thanks,

BMMc.

[clebert.suconic]

I would keep compacting the way it is.. I asked you disable just for a debug purpose.

As for paging.. I would set a max size on each queue. Look at the page documentation, and on the page example.

[timfox]

Is there are a JIRA for this issue?

[bmmc]

I did not create one because I could not recreate it so my definition of the 'problem' was pretty loose. I think Clebert was able to recreate it some extent so maybe he can make a JIRA that's more intelligent/accurate.

[clebert.suconic]

"timfox" wrote:
Is there are a JIRA for this issue?

I just created a JIRA,

https://jira.jboss.org/jira/browse/HORNETQ-124

I could replicate and fix one issue with compacting

(It's not committed yet.. I'm finishing some tests before doing it).



however I' m nore sure this is the same issue reported by bmmc, since we don't have much information.