Mladen Turk from JBoss is working on a port of the apache APR library that lets you do things like fork/exec a JVM. This maybe relevant to what you want, although you probably don't want to fork the jboss jvm because of all the open ports, log file writes, etc.
You'd be better of exec a jvm running your jni code and have a way to connect to it. Maybe you could create a minimal jboss configuration as a container for this. This configuration would be trimmed down to only run your service and log stuff to a different place.
Just some ideas :)
I do this exact same thing. I'll describe how I do this, although it's a bit complicated.
I have a RemoteJvmService MBean that I deploy in JBoss. The MBean is configured with a ServiceFactory class name, a ServiceInterface class name, and a classpath. For the classpath, I use a list of URLs, since I want it to access artifacts in the EAR.
The RemoteJvmService creates a new JVM with the main class RemoteJvmDriver. I use a simple JvmLauncher class to launch the new JVM - it simply builds up a command string and executes it using Runtime.exec(..) It also uses a custom URL classloader I wrote that determines what URLs to use based on the system property "url.class.loader".
When the RemoteJvmDriver runs, it instantiates an instance of the service factory class, generates an instance of the service interface, and then binds the service interface (which contains the JNI calls) using an RMI proxy.
Clients who need to make the JNI calls look up the service proxy via RMI and make calls. Actually, I use Spring to inject the service into objects - all they know about is the service interface.
Most of what I described is infrastructure code - the actual use of this service is fairly trivial. I hope this helps. Let me know if you have any questions.
A deployer approach could be a really elegant idea as well. You could perhaps deploy a special archive type, have it create the remote JVM, etc. similar to how I described in the previous post. Of course - I don't know if a deployer is necessary - I simply use a special MBean.