More issues with Pinger

[timfox]

Sorry to be a pain, but I'm going through our remaining remoting issues we're having in messaging. :(

We've been having connections mysteriously disappear from beneath us due even though the pinger was going on them so the lease shouldn't have expired.

I have tracked it down to two issues.

1) ServerInvoker.updateClientLease - this attempts to update the client lease whether or not the client session id is null, ending up with new leases with a null session id.

clientSessionId can be null if we've used a custom marshaller for the invocation that does not propagate client session id (which is true in our case)

2) The pinger interval.
The pinger on the client side is started with the same interval as the lease on the client side.

This means there is a fairly good chance that a ping *won't* arrive during the lease period, due to timing differences.

In practice this happens after 4 or 5 pings causing the connection cleanup code to kick in when the connection is still alive.

The client side ping interval needs to be *less* than the lease interval to be pretty damn sure this doesn't occur.

So far I have fixed this in my local copy of remoting so we can moving.

My fix to the lease pinger is temporar and kludgy, I just start the client side pinger with an interval of lease_period - 3000.

Clearly this is only a temp solution. I would suggest that the client and server side periods need to be configured separately.

I have also fixed the case for when clientSessionId is null by only creating a new Lease when the clientSessionId is not null.

[timfox]

Created JIRA task:

http://jira.jboss.com/jira/browse/JBREM-341

[abhatia]

We have migrated our application from JBoss-4.0.3SP1 to JBoss-5.1.0.GA. We are using ejb2 & messaging. The messaging is highly available part of the application. In our application, we have not touched/modified/overridden any functionality/implementation related to jboss messaging & remoting and rely on default implemntation/functionality provided by JBoss AS. We have only created multiple topics using org.jboss.jms.server.destination.TopicService.

Following exception appears on server side:

Application error detected : WorkerThread#2[127.0.0.1:2440] exception occurred during first invocation

java.io.EOFException

        at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:693)

        at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:524)

        at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232)

On further analysis it has been found that

(here we are talking about <jboss_home>\server\<config>\deploy-hasingleton\messaging\remoting-bisocket-service.xml)

1) if clientLeasePeriod is set to 0 then exception disappears

2) if clientLeasePeriod is non-zero then

     a) if clientLeasePeriod > (validatorPingPeriod + 2000) and both are not divisible by 10

          then exception disappears

     b) if clientLeasePeriod > validatorPingPeriod and either divisible by 10 or less than (validatorPingPeriod + 2000)

          then exception appears

     c) if clientLeasePeriod < validatorPingPeriod then exception appears (this situation is justifiable as explained above in this forum/discussion).

Please help me to understand the reason for the issue & the feasible solutions for the same. Right now we are able to temporarily fix it using scenario 2)a).

Here is our "messaging\remoting-bisocket-service.xml"

<?xml version="1.0" encoding="UTF-8"?>

<!--

     Standard bisocket-based Remoting service deployment descriptor.

     $Id: remoting-bisocket-service.xml,v 1.1.2.2 2011/02/23 03:16:14 abhatia Exp $

-->

<server>

   <!-- Standard bisocket connector - the bisocket transport only opens connection from client->server

        so can be used with firewalls where only outgoing connections are allowed.

        For examples of HTTP and SSL transports see docs/examples -->

   <mbean code="org.jboss.remoting.transport.Connector"

          name="jboss.messaging:service=Connector,transport=bisocket"

          display-name="Bisocket Transport Connector">

      <attribute name="Configuration">

         <config>

            <invoker transport="bisocket">

               <!-- There should be no reason to change these parameters - warning!

                    Changing them may stop JBoss Messaging working correctly -->           

               <attribute name="marshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>

               <attribute name="unmarshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>

               <attribute name="dataType" isParam="true">jms</attribute>

               <attribute name="socket.check_connection" isParam="true">false</attribute>              

               <attribute name="serverBindAddress">192.168.5.206</attribute>

               <attribute name="serverBindPort">${jboss.messaging.connector.bisocket.port:4457}</attribute>

               <attribute name="clientSocketClass" isParam="true">org.jboss.jms.client.remoting.ClientSocketWrapper</attribute>

               <attribute name="serverSocketClass">org.jboss.jms.server.remoting.ServerSocketWrapper</attribute>

               <attribute name="numberOfCallRetries" isParam="true">1</attribute>

               <attribute name="pingFrequency" isParam="true">214748364</attribute>

               <attribute name="pingWindowFactor" isParam="true">10</attribute>

               <attribute name="onewayThreadPool">org.jboss.jms.server.remoting.DirectThreadPool</attribute>              

               <!-- End immutable parameters -->

               <attribute name="stopLeaseOnFailure" isParam="true">true</attribute>

               <!-- Periodicity of client pings. Server window by default is twice this figure -->                              

               <attribute name="clientLeasePeriod" isParam="true">123456</attribute>

               <attribute name="validatorPingPeriod" isParam="true">98765</attribute>

               <attribute name="validatorPingTimeout" isParam="true">5000</attribute>

               <attribute name="registerCallbackListener">false</attribute>

                <attribute name="timeout" isParam="true">0</attribute>

               <!-- Number of seconds to wait for a connection in the client pool to become free -->

                <attribute name="connectionWait" isParam="true">10</attribute>

               <!-- Max Number of connections in client pool. This should be significantly higher than

                    the max number of sessions/consumers you expect -->

               <attribute name="JBM_clientMaxPoolSize" isParam="true">200</attribute>

               <!-- The maximum time to wait before timing out on trying to write a message to socket for delivery -->

               <attribute name="callbackTimeout">10000</attribute>

               <!-- Use these parameters to specify values for binding and connecting control connections to

                    work with your firewall/NAT configuration

               <attribute name="secondaryBindPort">xyz</attribute>                          

               <attribute name="secondaryConnectPort">abc</attribute>              

               -->

            </invoker>

            <handlers>

               <handler subsystem="JMS">org.jboss.jms.server.remoting.JMSServerInvocationHandler</handler>

            </handlers>

         </config>

      </attribute>

   </mbean>

</server>