Off the top of my head I think you could implement this with an in-memory data structure that holds a checksum of the data you want to compare as well as the corresponding ID for each message. When a message is sent to the broker you could perform a checksum on the incoming message and then ensure that checksum doesn't exist in the data structure. Also, when messages are acknowledge you can remove the corresponding entry from the data structure (which is why you would need to track the message ID and not just the checksum).
Keep in mind that this could have quite a significant negative impact on performance. As the queue grows in size the checksum comparison will take increasingly long and the data structure will occupy more & more memory.
It's also worth noting that the HornetQ code-base has not been under active development in several years since it was donated to the Apache ActiveMQ community. The code-base is continuing now as the ActiveMQ Artemis broker. If you're going to fork something I recommend you fork Artemis. Most of the internals are the same.