I think you could create an MBean to keep a list of each user and how many tries they've done.
Then increment the amount on a login attempt. Check the MBean each time the user tries to log in. Keep the timestamp of the initial and/or final try and compare.
Set the MBean on a timer and purge when time difference is > 30 minutes.
This would be a customization of a jboss login module or your own login module that tracked the attempts.