I don't understand well but I'll try gto give some hints.
It seems that you do not want to run whole JBoss AS just to use Cache. You do not have to, the Cache can be used standalone - it's a shame that there is almost no documentation for this, but there is some example config in the demo application.
I do not see any advantage in moving it to separate JVM on the same machine - there won't be more memory. Cache can be set to overflow to disk. If you want to move it to separate machine, you have to consider additional the overhead (at least increased latence) - for this I do not know what to use, I've heard about project Cajo, mayby you can give it a try.