I think I may have it, please let me know if I am barking up the wrong tree. Turns out the JAXM spec was the best resource, should have read that earlier.
In a nutshell, a JAXM provider is DIFFERENT from say, a JMS Provider. While conceptually similar, they are very different in reality.
The JAXM Provider acts as an intermediary between the container and your application, which consists of your application specific code for sending and consuming messages. The JAXM Provider is ultimately an endpoint for you to stick your message. The provider listens for messages to come in on the endpoint through a normal SOAP message call that we all know and love. It then sticks the message in some kind of queue / persistence mechanism. A listener then listens on that queue and attempts to deliver the message. The message stays on the queue until the message is deliver.
How the message is queued, and how that is all setup is JAXM provider implementation specific. Here is the sample scenario:
JAXM Client gets a Provider connection.
JAXM Client uses provider connection to send a message.
Provider connection calls a SOAP service to deliver your message to an internal JAXM Provider (which is just a JAXM Servlet).
The JAXM Provider onMessage will stick your message in a queue.
A separate program will try and deliver your message to the original endpoint. The separate program will make sure your message stays on that queue until it is delivered.
You can have the JAXM Provider be a remote web service call, but what the heck is the point of doing that? You can be pretty sure that your JAXM Provider running in the same container as your client code will work as long as your client code is working. If you use a remote JAXM Provider, your call could get dropped, and then what?
I think this is it. If I am misguided please do let me know. I am trying to get my arms around this poorly documented JAXM thingamabob right now, and could use all the help I can get.