I don't know of a method to unload a native library from Java. In general, it is not recommended to access JNI from EJBs. It may be for this reason that it is not recommended - perhaps due to the interlocking.
I suppose you can get around this problem by building an MBean service that provides an interface by which you can access the DLL. You can look up and access the MBean worker which will in turn perform the actual work. This would allow you to deploy you application normally - assuming your DLL itself does not often need redeployment.
Your only other alternative is to reboot JBoss.
maybe could be usefull ask to Sun a solution for this problem: it seems that there is a hole in the philosophy of classloader ;-)