Ok I seem to have the test working now.
In summary the failures were occurring because when creating a new JMS connection, the call to Client.connect() was failing since the server we were trying against was dead.
That is fine, and normally we would deal with this by catching the org.jboss.remoting.CannotConnectionException and retry on the next server.
The problem as that the current 2_2_x branch of remoting does not throw a org.jboss.remoting.CannotConnectionException but instead it throws a RuntimeException with a CannotConnectionException nested two deep inside it's initCause() :)
I have changed our catch code to take this case into consideration, the code is ugly but it works:
if (t instanceof JMSException) { return (JMSException)t; } else if ((t instanceof CannotConnectException) || (t instanceof IOException) || (t instanceof ConnectionFailedException)) { log.warn("Captured Exception:" + t, t); return new MessagingNetworkFailureException((Exception)t); } else if (t instanceof RuntimeException) { RuntimeException re = (RuntimeException)t; Throwable initCause = re.getCause(); if (initCause != null) { do { if ((initCause instanceof CannotConnectException) || (initCause instanceof IOException) || (initCause instanceof ConnectionFailedException)) { log.warn("Captured Exception:" + initCause, initCause); return new MessagingNetworkFailureException((Exception)initCause); } initCause = initCause.getCause(); } while (initCause != null); } }
It's odd that this problem should suddenly surface. I haven't changed anything related to it. The RuntimeException was in 2.2.0.GA.