The Sun JVM attempts to allocate a contiguous section of memory from its available address space. Considering that on a 32-bit box you have only 2GB of available address per process, and that typical application stuff (code and whatnot) takes up low memory space and that the majority of the Windows DLLs take up high memory, you will end up with a significantly smaller amount of memory avaiulable for the heap. So you will have to decrease your heap size. I can't recall the maximum heap we have ever gotten out of Windows 2003, though 1.4 GB comes to mind.
Also, setting the /3GB boot option will not help because even with that option there is not enough contiguous memory available for the size of heap you want.
However, you could use a JVM that doesn't need contiguous memory space to allocate the heap, and such a JVM might be able to allocate a big heap even without the /3GB heap option. The Unisys JVM (available at http://ecommunity.unisys.com/ecommunity/templates/longdescription.aspx?DocID=62563, requires free registration) does not require contiguous memory. If I recall correctly, we have gotten heaps of about 1.7GB without the /3GB boot option and around 2.7GB with the /3GB boot option. Of course, your mileage may vary.
Thanks very much, Peter.