what hardware you have? it seems your hardware has 4096 alignment... I could get you some code to try it out.
I'm highly interested on this.. but I will need some of your help... you can reach me on the IRC #hornetq @ Freenode on US hours if you like.
Where to change the alignment for the system? is it a config or that's up to the OS version?
the code is written with 512, I couldn't find an API back then and got blurred with other tasks...we can revisit / fix this. let me know when you're available to help fixing this... thanks
this is a ready answer we give to our users on support cases hitting this (we didn't have many I believe, but it became a Knowledge Base Article)
HornetQ is using AIO to write a partial block to the filesystem, which means there is free space that needs to be accounted for in the block. In the past there was a problem in EXT4 where filesystem corruption could occur if another thread using AIO also began writing to the remaining free space in the block at the same time. EXT4 was patched to protect from corruption, but it will cause a performance impact. Part of the patch was to implement this warning message which aims to inform you of the potential slowdown. Whether a performance impact is actually seen or not will depend entirely on the workload and system, but if testing shows that performance with AIO is unacceptable, we recommend switching to the NIO <journal-type> or changing the filesystem to EXT3 to avoid this ext4 filesystem "feature". You can read more about this issue on upstream patch http://www.spinics.net/lists/linux-ext4/msg22690.html.
I'm not sure yet.. but the conclusion from support was that this was something wrong with ext4 at some version? I'm still verifying it.. emailed some internal guys...
and I will be waiting for more info from you now.. but you really caught my attention here. waiting for it.
Sorry, for the delayed response.
I am working with a colleague to get specific details about the hardware setup.
Its been quite the rabbit hole.
Servers are HP DL 360 G6
RAID Controller is HP P410i (using RAID 1)
The disks themselves are setup for 512 block sizes, but the filesystem is set to 4096 (which is the default).
I believe it is possible to reduce the blocksize of the filesystem, but I hesitate to do so because there should be a performance loss.
I've been testing today and NIO is still slower than AIO, even with the added synchronization.
So, NIO is not really an option.
I would gladly test anything you can send me that may help.
can you change your filesystem to 512?
I will ping one of the guys on the native world...
NIO shouldn't be much slower than AIO. Are you sure about it?
in some bechmarks I did some time ago it was about 15% slower... but that depends on the use case.
Yeah, there is definitely a drop for me when using NIO, about 25% slower than when configured for AIO.
I will look into changing the block size to 512 tomorrow and testing.
I am curious to see the difference as well.
Do you know any way to discover the alignment of the dispositive being used? I'm googling for it but I'm being not very successful on it.
From the research I did that message is being removed on RHEL 6.5, so you can safely ignore it.
However I'm changing the code to cope with the block size now, it seems lstat would give me the correct block size. So I will make the code changes on next week.