My guess is, that hibernate first validates that the value falls into the range of the datatype byte. Since a value above 255 cannot be represented by one byte, hibernate stops the validation process at this point (returning the message that the user should please enter a value that is in the byte range) and your specific more granular validation is never reached.
This is, however, just a guess.
Yes. That's my best guess too. I was just wondering if there is a way around this. I want to override the standard message which is triggered by the 'byte validation' and just use my custom message. Does anyone know a way how to achieve this?