Make that set ANT_OPTS=-Xmx500m or more.
I watched memory use today, and while Ant makes the seam_reference.pdf the memory use is way up.
400 meg wasn't enough after I wrote the previous post.
What's heinous about it is that if you don't notice it there isn't a complaint at the end of the build, it says "build successful" and yet you have a bad .pdf file.
Hope this helps you some trouble in releases and such.
The problem hasn't gone away yet.
Seems that in target "refdoc" that ant task to build the subproject doesn't use ANT_OPTS, or at least not -Xmx or -Xms.
Still getting apparently random success or failure at getting a good seam_reference.pdf, almost but not exactly tied to an out of memory error in the Ant output.
Problem solved much better with in