What are your feelings about Derby? When I get my hands on an M5 version, I'll try Derby and give some feedback.
Any database that support Locators for its Blobs, therefore providing efficient partial writes supported by the JDBC 3 Blob API, will work well with JBoss Mail. Any database that doesn't, can be used with the PagedStore, which emulates locators by splitting large objects over multiple rows, however it won't perform anywhere near as well as a database with good locator support.
One test is use the JDBC3 API and set your -Xmx setting quite low, then try sending emails with very large attachments (e.g. 20MB+) and seeing if it blows the heap.
Note that Mike refers strictly to POP/SMTP stack.
The IMAP and WebMail stacks (HEAD only) are still pretty hairbrained.. They are the kind of code a crazy person would write at 2am. Eventually the'll work in a similar manner.