I'm going to respond to myself. By adding -server to the java command launching JBoss, we seem to have overcome the problem.
I heard that the problem lies in the compiler itself.
everytime you compile a jsp (thus with evry deployment) the compiler leaks a chunk of memory that doens't get GC'ed. hence the OutofMemory-error.
looks like you have it under control, but just fyi, you really should get off of windows, change to jikes, and run at least 1.5gig memory if you plan to use this server in production.