On a 32-bit system each process is limited to 4GB of address space. 2GB of that is used by the OS (handles and other data used by the OS to service your app), Some of the low memory is occupied by stuff that dates back to MS-DOS. Much of the memory area just below the 2GB mark is occupied by system DLLs (even the JVM DLLs are compiled to load just below that area). The java.exe code loads just above the MS-DOS area. This leaves the memory between the java.exe code and the JVM DLLs available to run the Java app. The JVM reserves some room for the thread stacks.

Given that, the largest heap space I have been able to use is around 1.7GB (the exact number changes with the exact level of the OS and service packs and at time with patches).

You set the permgen to 512MB - this reduced the available space for your heap by that amount, so a 1.3GB max for you sounds about right.