Weird port number on connection fail

[chipschoch]

We are running JBossAS 4.2.2 with JBM 1.4.4. Recently we experience a problem and I am trying to track it down. It appeared to be a network issue in that we got the following in the log:

2009-08-25 10:45:22,902 ERROR | Thread-24 | org.jboss.remoting.callback.ServerInvokerCa
llbackHandler | Error handling callback
org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing so
cket connection for InvokerLocator [bisocket://10.33.32.22
:1810191292/callback?callbackServerHost=10.33.32.22&callbackServerPort=1810191292&callbackServerProtocol=bisocket&clientMaxPoolSize=1&clientSocketClass=org.jb
oss.jms.client.remoting.ClientSocketWrapper&datatype=jms&guid=axwm-58z961-fy0jth8v-1-fy0jumsb-4y&isCallbackServer=true&onewayThreadPool=org.jboss.jms.server.r
emoting.DirectThreadPool&serverSocketClass=org.jboss.jms.server.remoting.ServerSocketWrapper]
 at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientI
nvoker.java:548)
 at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvok
er.java:422)
 at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141)
 at org.jboss.remoting.Client.invoke(Client.java:1858)
 at org.jboss.remoting.Client.invoke(Client.java:718)
 at org.jboss.remoting.Client.invokeOneway(Client.java:769)
 at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCall
backHandler.java:829)
 at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvok
erCallbackHandler.java:700)
 at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint
.java:1467)
 at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.
java:1379)
 at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:3
27)
 at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119
)
 at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.jav
a:583)
 at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistri
butor.java:79)
 at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:670)
 at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:506)
 at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:251)
 at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostO
ffice.java:2228)
 at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.ja
va:515)
 at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoi
nt.java:769)
 at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:399)
 at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advise
d$SessionAdvised$send$aop(SessionAdvised.java:87)
 at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(
SessionAdvised$send_7280680627620114891.java)
 at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
 at sun.reflect.GeneratedMethodAccessor114.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
 at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(
SessionAdvised$send_7280680627620114891.java)
 at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
 at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
 at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandle
r.java:143)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:862)
 at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)

 at org.jboss.remoting.Client.invoke(Client.java:1858)
 at org.jboss.remoting.Client.invoke(Client.java:718)
 at org.jboss.remoting.Client.invoke(Client.java:706)
 at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
 at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
 at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientS
essionDelegate$send$aop(ClientSessionDelegate.java:499)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(C
lientSessionDelegate$send_6145266547759487588.java)
 at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:661)
 at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect2
8.java)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(C
lientSessionDelegate$send_6145266547759487588.java)
 at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.j
ava:92)
 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(C
lientSessionDelegate$send_6145266547759487588.java)
 at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(C
lientSessionDelegate$send_6145266547759487588.java)
 at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
 at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:276)
 at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect39.invoke(ProducerAspec
t39.java)
 at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(
ClientProducerDelegate$send_3961598017717988886.java)
 at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)

I am particularly curious about the line

callbackServerPort=1810191292

Does this port number look to be correct? If not, could this be the cause of the failure to get a connection?

Thanks alot.

[ron_sigal]

Actually, that number, strange as it seems, is fine. Normally, that number would be the port where the bisocket://10.33.32.22:1810191292/callback? ...
callback server (on the client side) would be listening, but the bisocket transport doesn't open a port on the client, so this number is just a unique value to distinguish this InvokerLocator from any other InvokerLocators.

When a callback client (on the server side), which JBossMessaging uses to sends messages to a consumer, wants a new socket, it writes a message to the client, and the exception you're seeing means that that process failed. It could be a problem with the connection over which that message is written, but it's more likely that there's a problem on the client side. Take a look at your client side logs. If you see a complaint from ConnectionValidator about a failed connection, try increasing the values of

 <attribute name="validatorPingPeriod" isParam="true">10000</attribute>
 <attribute name="validatorPingTimeout" isParam="true">5000</attribute>

in your $JBOSS_HOME/server/$CONFIG/jboss-messaging.sar/remoting-bisocket-service.xml. That will reduce the likelihood of spurious connection failures on a busy network or between busy hosts.