1 Reply Latest reply on Sep 11, 2011 2:31 PM by fharms

    Failover for JMS client takes too long and fail

    fharms

      Hi

       

      I have problem I don't know how to solve. Sometimes the JMS client take to long to failover and throw an exception on the client, this is not consistent but something we can reproduce 6-10 times.
      The servers is not stressed at all and the same goes for the client.

      - 2 JBoss 5.1 servers running in a cluster

       

      - 2 clients connect to a topic queue
      - Messaging client 1.4.7.GA
      - remoting 2.5.1

       

      1) if I use the default values "remote-sslbisocket-service.xml" our client thinks it's should failover on startup. This fail with an JMSException and In this case we reconnect to the topic and after this every thing works fine. if I bring down one of the servers, it fail over as expected. But because it fail on startup there is possibility we miss messages because it use a minute to discover the failover didn't work 

       

       

      2) if I change the values validatorPingTimeout to 10000 and validatorPingPeriod = 15000 we are able to start our client with out the JMS client thinks it should failover and messages are received. But if I bring down one of the two servers it fail in the failover procedure with an JMSExeception

       

       

      2011-09-10 07:37:11,801 [Thread-213] WARN  com.apc.dsmf.client.webservices.event.QueueRemoteEventServiceImpl - Failover event FAILURE_DETECTED

       

      2011-09-10 07:37:11,807 [Thread-213] WARN  com.apc.dsmf.client.webservices.event.QueueRemoteEventServiceImpl - Failover event FAILOVER_STARTED

      2011-09-10 07:38:19,319 [Thread-213] WARN  com.apc.dsmf.client.webservices.event.QueueRemoteEventServiceImpl - Failover event FAILOVER_FAILED

      2011-09-10 07:38:19,321 [Thread-213] ERROR com.apc.dsmf.client.webservices.event.QueueRemoteEventServiceImpl - Got JMS Exception

      javax.jms.JMSException: Failure on underlying remoting connection

                at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:99)

                at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:498)

      client code :

       

      TopicConnection connection = ((TopicConnectionFactory) lookup).createTopicConnection();

       

      session = connection.createTopicSession(true,Session.AUTO_ACKNOWLEDGE);

      Topic topic = (Topic) context.lookup(JNDI_NAME_REMOTE_EVENT_TOPIC);

      subscriber = session.createSubscriber(topic);

      subscriber.setMessageListener(this);

      connection.setExceptionListener(this);

      ((JBossConnection) connection).registerFailoverListener(this);

      connection.start();

      ---------------------------------------------------------------

      remote-sslbisocket-service.xml :

       

      <mbean code="org.jboss.remoting.transport.Connector"

       

                name="jboss.messaging:service=Connector,transport=sslbisocket"

                display-name="SSL Bisocket Transport Connector">

            <attribute name="Configuration">

               <config>

                  <invoker transport="sslbisocket">  

                 

                     <!-- There should be no reason to change these parameters - warning!

                          Changing them may stop JBoss Messaging working correctly -->         

                     <attribute name="marshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>

                     <attribute name="unmarshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>

                     <attribute name="dataType" isParam="true">jms</attribute>

                     <attribute name="socket.check_connection" isParam="true">false</attribute>             

                       <attribute name="serverBindAddress">${jboss.bind.address}</attribute>

                     <attribute name="serverBindPort">4459</attribute>              

                     <attribute name="clientSocketClass" isParam="true">org.jboss.jms.client.remoting.ClientSocketWrapper</attribute>

                     <attribute name="serverSocketClass">org.jboss.jms.server.remoting.ServerSocketWrapper</attribute>

                     <attribute name="serverSocketFactory">jboss.messaging:service=ServerSocketFactory,type=SSL</attribute>

                       <attribute name="clientConnectAddress">10.245.256.10</attribute>

                     <attribute name="onewayThreadPool">org.jboss.jms.server.remoting.DirectThreadPool</attribute>

                     <!-- the following parameters are useful when there is a firewall between client and server. Uncomment them if so.-->

                                     

                     <attribute name="numberOfCallRetries" isParam="true">5</attribute>

                     <attribute name="pingFrequency" isParam="true">30000</attribute>

                     <attribute name="pingWindowFactor" isParam="true">71582</attribute>

                     <attribute name="generalizeSocketException" isParam="true">true</attribute>

                    

                     <!-- Now remoting supports socket write timeout configuration. Uncomment this if you need it. -->

                     <!--              

                     <attribute name="writeTimeout" isParam="true">30000</attribute>

                     -->

                     <!-- End immutable parameters -->

                    

                     <attribute name="stopLeaseOnFailure" isParam="true">true</attribute>

                                   

                     <!-- Periodicity of client pings. Server window by default is twice this figure -->                              

                     <attribute name="clientLeasePeriod" isParam="true">10000</attribute>

                     <attribute name="validatorPingPeriod" isParam="true">10000</attribute>

                     <attribute name="validatorPingTimeout" isParam="true">5000</attribute>

                     <attribute name="failureDisconnectTimeout" isParam="true">0</attribute>

                     <attribute name="callbackErrorsAllowed">1</attribute>

                     <attribute name="registerCallbackListener">false</attribute>

                     <attribute name="useClientConnectionIdentity" isParam="true">true</attribute>

                      

                       <attribute name="timeout" isParam="true">0</attribute>

                     <!-- Max Number of connections in client pool. This should be significantly higher than

                          the max number of sessions/consumers you expect -->

                     <attribute name="JBM_clientMaxPoolSize" isParam="true">200</attribute>

                    

                       <!-- The maximum time to wait before timing out on trying to write a message to socket for delivery -->

                     <attribute name="callbackTimeout">10000</attribute>

                                                         

                     <!-- Use these parameters to specify values for binding and connecting control connections to

                          work with your firewall/NAT configuration -->

                     <attribute name="secondaryBindPort">4462</attribute>                          

                     <attribute name="secondaryConnectPort">4462</attribute>                             

                    

                  </invoker>

                  <handlers>

                     <handler subsystem="JMS">org.jboss.jms.server.remoting.JMSServerInvocationHandler</handler>

                  </handlers>

               </config>

            </attribute>

            <depends>jboss.messaging:service=ServerSocketFactory,type=SSL</depends>

         </mbean>

       

      Catch 22, please help!!!

       

      br

      Flemming

        • 1. Re: Failover for JMS client takes too long and fail
          fharms

          I setup a TRACE on the server and I have attached a snip it of the server log from the server is started and to the client connect. There is also client log where you can see it detect a "FAILURE_DETECTED" and the same time the server log print "[org.jboss.remoting.transport.bisocket.BisocketServerInvoker] (Timer-3) checking connections"

           

          So at this time there is no indication on the server about the "FAILURE_DETECTED"

           

          Does any know how to enable logging for remoting and jms on the client, tried to add follow to my log4j configuration with not luck

           

          <category name="org.jboss.remoteing">

                <priority value="TRACE" class="org.jboss.logging.XLevel" />

                <appender-ref ref="FILE" />

          </category>

           

          <category name="org.jboss.messaging">

                <priority value="TRACE" class="org.jboss.logging.XLevel"></priority>

                <appender-ref ref="FILE" />

          </category>

           

          <category name="org.jboss.jms">

                <priority value="TRACE" class="org.jboss.logging.XLevel"></priority>

                <appender-ref ref="FILE" />

            </category>

           

          btw: The time stamp in the 2 log files synchronized because the server and client is running on the same machine for this test

           

          Thanks