1 Reply Latest reply on Jan 14, 2013 7:32 PM by ron_sigal

    detected failure on control connection

    ajnagaraj

      We are seeing a number of 'detected failure on control connection' errors in our logs which eventually makes our system unresponsive.

       

       

      2012-11-27 19:04:07,242 WARN  [org.jboss.remoting.transport.bisocket.BisocketServerInvoker] org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@bbfa5e7: detected failure on control connection Thread[control: Socket[addr=oxford/192.168.30.75,port=4460,localport=52236],5,jboss] (5c4o146e-4limzg-ha0fuq7m-1-ha14020p-q3w: requesting new control connection

       

       

      The environment we use -

      Jboss : 4.2.1-GA

      Jboss Messaging : 1.4.5

      Jboss Remoting : 2.2.3-SP3

       

       

      Any suggestions as to what causes this issue? Similar discussions suggest a overload issue on the system, if so will the upgrade to JBoss Remoting version 2.2.4 help?

       

       

      -Ajay

        • 1. Re: detected failure on control connection
          ron_sigal

          Hi Ajay,

           

          The bisocket control connection is used by the server to send requests to the client to create a connection to the bisocket secondary ServerSocket on the server.  It sounds convoluted, but the goal was to create a connection from the server to the client without having to run a ServerSocket on the client.  Depending on the configuration, the server will send PINGs to the client, and the client will emit a "detected failure on control connection" message, and try to recreate the control connection, if a PING is late.  One thing you could try is to expand the window in which a PING is expected.  If the window is too small and 1) the network is congested or 2) the server is busy, you could get spurious failures.  The two relevant configuration parameters are:

           

          * pingFrequency: determines how often a ping is sent

          * pingWindowFactor: the value is multipled times the pingFrequency to determine the window during which the PING is expected

           

          These parameters are configured in $JBOSS_HOME/server/$CONFIG/deploy/jboss-messaging.sar/remoting-bisocket-service.xml.  Well, that's true for AS 4.2.3.GA.  I don't have a copy of 4.2.1.GA at hand.  In any case, look for remoting-bisocket-service.xml.

           

          Note that in some versions of messaging, PINGing is, in effect, turned off:

           

            <attribute name="pingFrequency" isParam="true">214748364</attribute>

            <attribute name="pingWindowFactor" isParam="true">10</attribute>

           

          In others, PINGS are sent frequently, but pingWindowFactor is set to "infinity".  That strategy is used to keep connections open through a firewall.

           

          You can read more about configuration of the bisocket transport in the Remoting Guide: http://docs.jboss.org/jbossremoting/2.2.3.SP3/html/chapter-configuration.html#d0e3235.

           

          As for upgrading to Remoting 2.2.4, I would always recomment upgrading if you can.  Here are the release notes for 2.2.4:

           

          Release Notes - JBoss Remoting - Version 2.2.4

           

          Bug

           

          * [JBREM-1242] - Fix deadlock between Client and MicroRemoteClientInvoker

          * [JBREM-1261] - Prevent DOS attack on BisocketServerInvoker$SecondaryServerSocketThread

          * [JBREM-1268] - RemoteClientInvoker can change configuration map and prevent InvokerRegistry from reusing client invokers

          * [JBREM-1269] - Fix deadlock between Client and MicroRemoteClientInvoker, part 2

          * [JBREM-1276] - SecondaryServerSocketThread should catch java.lang.Error

           

          Enhancement

           

          * [JBREM-1245] - Consider javax.net.ssl.SSLException("Connection has been shutdown") to be retriable

          * [JBREM-1248] - Avoid connecting to a server in catch clause in case of HttpClientInvoker connection failure

          * [JBREM-1262] - Consider java.io.IOException("Software caused connection abort: socket write error") retriable

          * [JBREM-1263] - ClientSocketWrapper.checkConnection() should check returned value

          * [JBREM-1267] - Allow HTTPClientInvoker to call disconnect on HttpURLConnection after use

          * [JBREM-1275] - Make maxthreads, timeout configurable in BisocketServerInvoker.SecondaryServerSocketThread

          * [JBREM-1277] - Allow configuration of socket and bisocket accept thread priority

           

          Feature Request

           

          * [JBREM-1144] - Extend connection identity to server side

           

          There was enough there to bump the version to 2.2.4.

           

          -Ron