I was able to get an asynchronous doc/literal web service using JWSDP 1.4 to deploy on JBoss 3.2.5. When I tried to drop the JWSDP-style cooked WAR into the deploy directory, I encountered similar class loading issues to the ones you described above. Here are a series of steps that worked for me:
1) Copy the default server (let's call the new directory 'myserver').
2) Delete myserver/deploy/jboss-ws4ee.sar and myserver/deploy/ebxmlrr-service.sar
3) Delete myserver/lib/jboss-jaxrpc.jar and myserver/lib/jboss-saaj.jar
4) Copy all of the JWSDP JAR files, except saaj-*.jar, activation.jar, and mail.jar to myserver/lib/
5) Copy JWSDP's saaj-*.jar to the web service's WEB-INF/lib/ directory.
6) Copy the cooked WAR to myserver/deploy/
Packing the JWSDP JARs within the WAR file seems important if you wish to avoid the deployment error. But placing all of the other JARs in the lib directory seems important for avoiding other class loading problems. It still feels like a hack to me, and I don't think it will work for every scenario. For example, I can convert from a SOAPElement to a JAXB object, but I get an exception if I try to go from a JAXB object to a SOAPElement. That happened to be OK for my situation, since my service was async.
I also changed some of the system properties, as you mentioned. Instead of creating files in META-INF/services, I used System.setProperty(). You should also be able to use -D, if preferred. I went a little crazy and set all of the following:
An exhaustive investigation of what's happening would involve inspecting the class loader behavior, as tdiesler mentioned. If you go that route, don't forget that Jboss uses a shared loader by default, and if you can't get that to work, you could try forcing it into using the hierarchical loader.
Good luck, and I would enjoy hearing the outcome. I agree, a page in the Wiki would be useful.