When a message process failed, i just rollback the transaction, and it will be processed next time, but when i delete all records in jbm_msg and jbm_msg_ref table, the message is still there and be processed.
I guess there is some cache there, how to delete the message completely
without restart jboss server?
No you can't delete the table data while the server is still up and running. If you rollback the tx, the messages in the tx will be cancelled back to queue and redelivered.
However you can try to do it from jmx-console. But first you need to close the receiving sessions.