No as far as I know there isnt a jgroups impl of JMS (although the jgroups forum would know better), in fact since a lot of the JMS spec is centered around durability and acknowledgment of messages it would probably be quite difficult to do. saying that you could partially implement the spec, say just implement simple jms producers and consumers.
If you wanted to use jgroups with MDB's you would have to implement your own Resource Adapter to do this, you could then plug this into Wildfly quite easily, you could take a look at the HornetQ resource adapter or the generic one. One question tho, you say you want a brokerless system but are going to be using Wildfly, cant you just deploy HornetQ in one of the Wildfly instances?
lastly, there is a HornetQ JIRA to implement a peer to peer mode [HORNETQ-87] Peer to peer mode / serverless mode / disconnect mode - JBoss Issue Tracker. This would probably be simpler to implement than a jgroups JMS/new resource adapter implementation.
Thank you for your detailed answer.
The reason I'm not using HornetQ on the servers is quite simple. I don't want my servers to be aware of each other. I'd like the producer to "publish" a message to a "virtual" group of servers which is dynamic by nature - servers can join and leave the group any time so the JGroups membership concept suits me well.
Once server from the target group processed the message, it can be publish again to another group and so on. The last processing unit in the chain will send the response back to the caller server (initiator) - by looking at the message payload which specifies the caller name (address).
This way no central broker is required and message producers are not aware of queue/topic location.