I have got to the bottom of this and found a workaround.
It's not clear in my mind if this is a bug, or just an oddity, but this is why the problem happened.
Please forgive me if there are some inaccuracies since I haven't had time to investigate in real depth.
I have a client machine with name client.acme.com and a server machine with name server.acme.com.
I look up the queue connection factory via JNDI, which gets serialized back to the client.
The server object (OilServerIl instance) attempts to open a socket connection back to the server (createConnection method), it uses the addr member (InetAddress instance) as the address for this.
This is where the problem occurs: This instance was created on the SERVER before the serialization, hence the ip lookup was done on the SERVER.
In my /etc/hosts file on the server, I map server.acme.com to 127.0.0.1 - a very standard thing to do.
The client then attempts to open a socket to 127.0.0.1 instead of the server's real IP address and fails!
So... I removed the server.acme.com -> 127.0.0.1 mapping from /etc/hosts on the server.
But there's still a problem... The ip address address
a) Has to go to a name server to resolve (Very slow)
b) Returns the external ip address, which would be routed OUTSIDE our firewall. Naturally we block those ports on our firewall, so the connection is refused again.
Ok, third attempt: I add a mapping in /etc/hosts on the server mapping server.acme.com to the internal 192.168.x.y address on our network.
So moral of the story:
Never map your own hostname to 127.0.0.1 in /etc/hosts on a machine running Jboss, and add an entry with the 192.168.x.y address if you have one.
My feeling is that the look-up should always be done on the machine that is attempting the connection. It's currently seems to be done on the machine that creates the mbean.
I'm uneasy about getting rid of my hostname->127.0.0.1 mapping from my /etc/hosts file for performance reasons, since it's my understanding that a 127.0.0.1 will never hit the network card, but a 192.168.x.y address will.
I'm using jboss 2.4.3