It will be easier to solve on your side locally using for example FireBug to check which styles applied causes such effect.
I found the problem which was causing the checkbox to move up (using FireBug). By default, <rich:extendedDataTable> header was putting blank image. I overrode the .extdt-header-sort-img in css.
This fix is working fine with only firefox and not with other browsers(IE and safari).
Please suggest some solutions to work the same in other browsers also.