only explanation which come to my mind is that RemoteConnectionFactory which is looked up from remote Wildfly is configured with connector which points to local Wildfly. Could you share the config of remote server, please?
I am using the RemoteConnectionFactory that is defined in standalone-full.xml :
<connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnetionFactory" connectors = "http-connector"/>
Do I need to create my own pooled connection factory ? I hope not because I am trying to do this without adding to standalone-full.xml because that is hard to maintain/CM.
Could you check if http-connector has correct socket binding? It should point to "http" socket binding which has configured "public" interface. Public interface then should have IP address of remote Wildfly intance.