-
1. Re: Unaligned AIO/DIO warnings
clebert.suconic Feb 12, 2015 2:11 PM (in response to rich.vibeskowski)what hardware you have? it seems your hardware has 4096 alignment... I could get you some code to try it out.
-
2. Re: Unaligned AIO/DIO warnings
clebert.suconic Feb 12, 2015 2:39 PM (in response to rich.vibeskowski)I'm highly interested on this.. but I will need some of your help... you can reach me on the IRC #hornetq @ Freenode on US hours if you like.
-
3. Re: Unaligned AIO/DIO warnings
clebert.suconic Feb 12, 2015 2:51 PM (in response to rich.vibeskowski)Where to change the alignment for the system? is it a config or that's up to the OS version?
the code is written with 512, I couldn't find an API back then and got blurred with other tasks...we can revisit / fix this. let me know when you're available to help fixing this... thanks
-
4. Re: Unaligned AIO/DIO warnings
clebert.suconic Feb 12, 2015 3:10 PM (in response to rich.vibeskowski)this is a ready answer we give to our users on support cases hitting this (we didn't have many I believe, but it became a Knowledge Base Article)
HornetQ is using AIO to write a partial block to the filesystem, which means there is free space that needs to be accounted for in the block. In the past there was a problem in EXT4 where filesystem corruption could occur if another thread using AIO also began writing to the remaining free space in the block at the same time. EXT4 was patched to protect from corruption, but it will cause a performance impact. Part of the patch was to implement this warning message which aims to inform you of the potential slowdown. Whether a performance impact is actually seen or not will depend entirely on the workload and system, but if testing shows that performance with AIO is unacceptable, we recommend switching to the NIO <journal-type> or changing the filesystem to EXT3 to avoid this ext4 filesystem "feature". You can read more about this issue on upstream patch http://www.spinics.net/lists/linux-ext4/msg22690.html.
I'm not sure yet.. but the conclusion from support was that this was something wrong with ext4 at some version? I'm still verifying it.. emailed some internal guys...
and I will be waiting for more info from you now.. but you really caught my attention here. waiting for it.
-
5. Re: Unaligned AIO/DIO warnings
rich.vibeskowski Feb 12, 2015 4:01 PM (in response to clebert.suconic)Sorry, for the delayed response.
I am working with a colleague to get specific details about the hardware setup.
Its been quite the rabbit hole.
Servers are HP DL 360 G6
RAID Controller is HP P410i (using RAID 1)
The disks themselves are setup for 512 block sizes, but the filesystem is set to 4096 (which is the default).
I believe it is possible to reduce the blocksize of the filesystem, but I hesitate to do so because there should be a performance loss.
I've been testing today and NIO is still slower than AIO, even with the added synchronization.
So, NIO is not really an option.
I would gladly test anything you can send me that may help.
-
6. Re: Unaligned AIO/DIO warnings
clebert.suconic Feb 12, 2015 4:57 PM (in response to rich.vibeskowski)can you change your filesystem to 512?
-
7. Re: Unaligned AIO/DIO warnings
clebert.suconic Feb 12, 2015 4:58 PM (in response to clebert.suconic)I will ping one of the guys on the native world...
NIO shouldn't be much slower than AIO. Are you sure about it?
-
8. Re: Unaligned AIO/DIO warnings
clebert.suconic Feb 12, 2015 4:58 PM (in response to clebert.suconic)in some bechmarks I did some time ago it was about 15% slower... but that depends on the use case.
-
9. Re: Unaligned AIO/DIO warnings
rich.vibeskowski Feb 12, 2015 5:56 PM (in response to clebert.suconic)Yeah, there is definitely a drop for me when using NIO, about 25% slower than when configured for AIO.
I will look into changing the block size to 512 tomorrow and testing.
I am curious to see the difference as well.
-
10. Re: Unaligned AIO/DIO warnings
clebert.suconic Feb 13, 2015 10:57 AM (in response to rich.vibeskowski)Do you know any way to discover the alignment of the dispositive being used? I'm googling for it but I'm being not very successful on it.
-
11. Re: Unaligned AIO/DIO warnings
clebert.suconic Feb 13, 2015 11:28 AM (in response to clebert.suconic)this is subscription based content, but useful if you have an account:
-
12. Re: Unaligned AIO/DIO warnings
clebert.suconic Feb 13, 2015 11:24 AM (in response to clebert.suconic)Interesting links:
http://www.spinics.net/lists/linux-ext4/msg22690.html
It seems this message is being removed upstream though.
-
13. Re: Unaligned AIO/DIO warnings
clebert.suconic Feb 13, 2015 2:01 PM (in response to clebert.suconic)From the research I did that message is being removed on RHEL 6.5, so you can safely ignore it.
However I'm changing the code to cope with the block size now, it seems lstat would give me the correct block size. So I will make the code changes on next week.