javax.jms.JMSException: Failure on underlying remoting conne

[krisfra]

JBoss EAP 4.3.0.GA_CP02
JBoss Messaging 1.4.0.SP3-CP03
JBoss Remoting 2.2.2.SP9

My client code is setup to consume messages from the testQueue in the out-of-the-box installation of the aforementioned EAP. Also, I have a javax.jms.ExceptionListener callback setup to monitor the connection. Another client begins sending messages to testQueue and my client consumes them ... all is working well. However, the exception listener's onException callback appears to be spuriously called for no apparent reason. If I ignore the spurious callbacks, I can just continue consuming messages as if no problem ever occurred. However, I would like to use the callback to indicate that I really do have a lost connection which cannot be ignored! I have logged these callbacks and they are coming randomly every 30 seconds to 10 minutes apart. remoting-bisocket-service.xml contains ...



<!-- Standard bisocket connector - the bisocket transport only opens connection from client->server
so can be used with firewalls where only outgoing connections are allowed.
For examples of HTTP and SSL transports see docs/examples -->
<mbean code="org.jboss.remoting.transport.Connector"
name="jboss.messaging:service=Connector,transport=bisocket"
display-name="Bisocket Transport Connector">




<!-- There should be no reason to change these parameters - warning!
Changing them may stop JBoss Messaging working correctly -->
org.jboss.jms.wireformat.JMSWireFormat
org.jboss.jms.wireformat.JMSWireFormat
jms
false
${jboss.bind.address}
4457
org.jboss.jms.client.remoting.ClientSocketWrapper
org.jboss.jms.server.remoting.ServerSocketWrapper

1
214748364
10
org.jboss.jms.server.remoting.DirectThreadPool
<!-- End immutable parameters -->

true


<!-- Periodicity of client pings. Server window by default is twice this figure -->
<!-- 10000 -->
3600000


<!-- added by KMF -->
10000
5000
false


0

<!-- Number of seconds to wait for a connection in the client pool to become free -->
10

<!-- Max Number of connections in client pool. This should be significantly higher than
the max number of sessions/consumers you expect -->
200

<!-- The maximum time to wait before timing out on trying to write a message to socket for delivery -->
10000

<!-- Use these parameters to specify values for binding and connecting control connections to
work with your firewall/NAT configuration
xyz
abc
-->



org.jboss.jms.server.remoting.JMSServerInvocationHandler







Being a JBoss novice, I wouldn't be surprised if this is just a configuration issue because this seems like basic funtionality. As you might observe in the configuration above, I have tried the workaround given here ... https://jira.jboss.org/jira/browse/SOA-835

Is there anyone that can give me some advice on how to make these exceptions go away? Thanks for your help!
Kris

[krisfra]

Oops, how about this ...

<server>

 <!-- Standard bisocket connector - the bisocket transport only opens connection from client->server
 so can be used with firewalls where only outgoing connections are allowed.
 For examples of HTTP and SSL transports see docs/examples -->
 <mbean code="org.jboss.remoting.transport.Connector"
 name="jboss.messaging:service=Connector,transport=bisocket"
 display-name="Bisocket Transport Connector">
 <attribute name="Configuration">
 <config>
 <invoker transport="bisocket">

 <!-- There should be no reason to change these parameters - warning!
 Changing them may stop JBoss Messaging working correctly -->
 <attribute name="marshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
 <attribute name="unmarshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
 <attribute name="dataType" isParam="true">jms</attribute>
 <attribute name="socket.check_connection" isParam="true">false</attribute>
 <attribute name="serverBindAddress">${jboss.bind.address}</attribute>
 <attribute name="serverBindPort">4457</attribute>
 <attribute name="clientSocketClass" isParam="true">org.jboss.jms.client.remoting.ClientSocketWrapper</attribute>
 <attribute name="serverSocketClass">org.jboss.jms.server.remoting.ServerSocketWrapper</attribute>

 <attribute name="numberOfCallRetries" isParam="true">1</attribute>
 <attribute name="pingFrequency" isParam="true">214748364</attribute>
 <attribute name="pingWindowFactor" isParam="true">10</attribute>
 <attribute name="onewayThreadPool">org.jboss.jms.server.remoting.DirectThreadPool</attribute>
 <!-- End immutable parameters -->

 <attribute name="stopLeaseOnFailure" isParam="true">true</attribute>

 <!-- Periodicity of client pings. Server window by default is twice this figure -->
 <!-- <attribute name="clientLeasePeriod" isParam="true">10000</attribute> -->
 <attribute name="clientLeasePeriod" isParam="true">3600000</attribute>


 <!-- added by KMF -->
 <attribute name="validatorPingPeriod" isParam="true">10000</attribute>
 <attribute name="validatorPingTimeout" isParam="true">5000</attribute>
 <attribute name="registerCallbackListener">false</attribute>


 <attribute name="timeout" isParam="true">0</attribute>

 <!-- Number of seconds to wait for a connection in the client pool to become free -->
 <attribute name="numberOfRetries" isParam="true">10</attribute>

 <!-- Max Number of connections in client pool. This should be significantly higher than
 the max number of sessions/consumers you expect -->
 <attribute name="JBM_clientMaxPoolSize" isParam="true">200</attribute>

 <!-- The maximum time to wait before timing out on trying to write a message to socket for delivery -->
 <attribute name="callbackTimeout">10000</attribute>

 <!-- Use these parameters to specify values for binding and connecting control connections to
 work with your firewall/NAT configuration
 <attribute name="secondaryBindPort">xyz</attribute>
 <attribute name="secondaryConnectPort">abc</attribute>
 -->

 </invoker>
 <handlers>
 <handler subsystem="JMS">org.jboss.jms.server.remoting.JMSServerInvocationHandler</handler>
 </handlers>
 </config>
 </attribute>
 </mbean>

</server>

[ron_sigal]

Hi Kris,

I don't know enough about JBossMessaging to comment on these ExceptionListener callbacks. Could you tell me what they are saying?

Thanks,
Ron

[krisfra]

Yes, when exception listener callback occurs, the exception being reported is essentially the subject (title) of this post, that is, javax.jms.JMSException: Failure on underlying remoting connection. So, I concluded that I have a remoting problem ... made sense to me.

I do have somewhat of an update. Further investigation into the spurious exceptions related to the underlying remoting connection ultimately led me to the topic Unexplained ConnectionValidator Timeout (see http://www.jboss.org/index.html?module=bb&op=viewtopic&t=151168 for details). After reading the description and noting that the topic poster was using Red Hat (RH) JBoss EAP 4.3.0.GA_CP03, I did the obvious … I updated from RH JBoss EAP 4.3.0.GA_CP02 to 4.3.0.GA_CP04 (latest/greatest non-alpha install). Then, I re-ran my test. The test ran overnight with over 150K message transfers to the JMS queue and over 150K from JMS queue without any random exceptions "heard" on the client connections. The spurious exceptions appeared to have completely disappeared!

Unfortunately, I ran my test again several days later, executed roughly the same number of message transfers, but observed one failure.

So, it appears the problem may still be there, but has been greatly reduced.

[ron_sigal]

Hi Kris,

Well, that just shows that every day, in every way, we get better and better. :)

I have two suggestions.

1. On a busy network, or busy hosts, a spurious failure is always possible. If you lengthen the ping cycle, e.g.,

 <attribute name="validatorPingPeriod" isParam="true">20000</attribute>
 <attribute name="validatorPingTimeout" isParam="true">10000</attribute>

you might reduce the number of occurrences.

2. We just added some major fixes on the JBossRemoting / JBossMessaging boundary in EAP 4.3.CP05 (Remoting 2.2.3 / JBM 1.4.0.SP3_CP08), which seems to be available now. I'm not sure if they will affect your situation, but I think that anyone using JBossMessaging should probably upgrade to these new jars.

Of course, if you have a support contract, you would have to do the whole upgrade.

-Ron