Journal Compacting committed...

[clebert.suconic]

I just committed the journal compacting into trunk.


I have tested it on the performance labs, and compacting happens totally in parallel.

I have added a test that was adding one message per second in a destination (without consuming it):

num-messages=1000
num-warmup-messages=0
message-size=1024
durable=true
transacted=true
batch-size=100
drain-queue=true
throttle-rate=10
address=perfAddress2
queue-name=perfQueue2
host=172.16.8.10
port=5445
tcp-buffer=1048576
tcp-no-delay=false
send-window=1048576
pre-ack=true
block-ack=false
block-persistent=false

Then I started runSender and runListener as usual. Running both transactional (as the next code example) and non transactional.

num-messages=500000
num-warmup-messages=20000
message-size=1024
durable=true
transacted=true
batch-size=1000
drain-queue=true
queue-name=perfQueue
throttle-rate=-1
address=perfAddress
queue-name=perfQueue
host=172.16.8.10
port=5445
tcp-buffer=1048576
tcp-no-delay=false
send-window=1048576
pre-ack=true
block-ack=false
block-persistent=false

and I could get 28K messages / second if either having the first sender or or not.

On my desktop, with a more limited disk.. I had 20K messages / second without the first sender.. and 15 messages / second with the first sender.


I also compared the performance on the regular scenario on with and without those changes, and I could get the same performance as expected.



Now... as I said on the other thread, we need to implement clean up next.

[clebert.suconic]

Having the compacting running too frequently will make compacting to compete for resources.

Messaging-01 has a pretty fast disk, because of that the numbers were good on both scenarios.


We have to minimize the need for compacting. Having living records on the journal will cause compacting to be executed too frequently. We need to cleanup the links between those files.