128 megs of heap isn't huge, you could still be running out of memory if Cocoon is loading large XML/XSL parsers into JVM (not sure what it uses).
There are several other reasons why you might get OutOfMemory errors, some are bugs, some are configuration issues, see the FAQ for some common causes.
Note that you probably have to do some fiddling with Xerces as well. Cocoon is very sensitive to using the latest version of Xerces. The stuff which is included in JBoss seems to be at least differently organized.