I had already used the tuning doc. I used the WEB configuration, but still removed all services other than tomcat and jndi based (and left jcs and transaction since I get error starting JBoss otherwise). I also commented out the annotation mentioned in the doc.
The impact is still not very much. I see a reduction of at the most 100mb out of 4-500mb.
What else could I possibly disabled or remove? is disabling VFS an option?
AS5 is known to consume more memory than AS4. Although some of it can be tuned (like you already did), there isn't much you can do other than that. Disabling VFS isn't an option. I can't recollect any other changes that you could do to optimize the memory usage.