6 Replies Latest reply on Nov 5, 2010 5:22 PM by pokerdan

    After network connection failure and reconnect, fails to reestablish connection to a Websphere MQ queue.

    aemcat

      Newbie troubles…

      using JBoss SOA p.5.0.0

       

      We have an ESB configured to read from 2 queues, one using a JBoss Messaging provider, the other using a Websphere MQ provider. The actual WMQ is remote.

       

      The problem: When the network connection goes down, and is brought back up, it cannot reestablish the connection to the Websphere MQ queue.

       

      I see the following messages even after the network connection is restored.

       

      WARN  [JmsGatewayListener] Error reconnecting to Queue, backing off for 32000 milliseconds

       

      I increased the log level for the org.jboss.soa.esb.listeners to Trace and did receive some additional information:

       

      19:15:14,247 DEBUG [JmsGatewayListener]    JMS error on receive.  Attempting JMS Destination reconnect.

      javax.jms.JMSException: MQJMS2002: failed to get message from MQ queue

                      at com.ibm.mq.jms.services.ConfigEnvironment.newException(ConfigEnvironment.java:622)

                      at com.ibm.mq.jms.MQMessageConsumer.getMessage(MQMessageConsumer.java:3778)

                     ……………..

      19:15:14,247 DEBUG [JmsGatewayListener] JMSGateway isTransacted = false

      19:15:14,247 DEBUG [JmsGatewayListener] Reconnecting to Queue

      javax.jms.JMSException: MQJCA1018:Only one session per connection allowed.

                      at com.ibm.mq.connector.services.JCAExceptionBuilder.buildException(JCAExceptionBuilder.java:115)

                      at com.ibm.mq.connector.services.JCAExceptionBuilder.buildException(JCAExceptionBuilder.java:80)

                      at com.ibm.mq.connector.outbound.ConnectionWrapper.createSession(ConnectionWrapper.java:101)

                      at org.jboss.internal.soa.esb.rosetta.pooling.JmsConnectionPool$JmsSessionPool$2.call(JmsConnectionPool.java:830)

                      at org.jboss.internal.soa.esb.rosetta.pooling.JmsConnectionPool$JmsSessionPool$2.call(JmsConnectionPool.java:821)

       

      Currently the only resolution I have is to redeploy the ESB, but seems this connection should be able to be repaired without redeploying.

       

      So far I’ve been unable to determine how to increase the number of sessions per connection, but am unsure that would be the correct solution anyway?

       

      I have a feeling this is something very basic, maybe configuration settings, but after much googling and reading, I’m afraid I’m still clueless.

       

      Please, can anyone provide some advice on this?

       

      In addition, here are relevant parts of the configuration:

       

      wmq-ds.xml

      {code:xml}

      <connection-factories>

       

        <!-- JCA Connection factory definitions -->

        <tx-connection-factory>

          <jndi-name>/WebSphereMQConnectionFactory</jndi-name>

          <xa-transaction />

          <rar-name>wmq.jmsra.rar</rar-name>

          <connection-definition>

            javax.jms.ConnectionFactory

          </connection-definition>

               <!--  remote connection settings -->

          <config-property>SYSTEM.DEF.SVRCONN</config-property>

          <config-property>simba.na.acme.net</config-property>

          <config-property>1414</config-property>

          <config-property>QM_SIMBA</config-property>

          <config-property>CLIENT</config-property>

                

          <min-pool-size>1</min-pool-size>

          <max-pool-size>20</max-pool-size>

        </tx-connection-factory>

      </connection-factories>

      {code}

       

      queue-service.xml

      {code:xml}

        <mbean code="org.jboss.resource.deployment.AdminObject"

            name="jca.wmq:name=MY_INPUT_Q">

          <attribute>

            queue/MY.INPUT.Q

          </attribute>

          <depends optional-attribute-name="RARName">

            jboss.jca:service=RARDeployment,name='wmq.jmsra.rar'

          </depends>

          <attribute>javax.jms.Queue</attribute>

          <attribute>

            baseQueueManagerName=QM_SIMBA

            baseQueueName=MY.INPUT.Q

          </attribute>

        </mbean>

      {code}

       

      jboss-esb.xml (snippets)

      {code:xml}

        <providers>

            <jms-provider connection-factory="OrderedConnectionFactory">

                  <!-- . . . The various jboss messaging queues

            </jms-provider>

        

            <jms-provider connection-factory="java:/WebSphereMQConnectionFactory">

                <jms-bus busid="wmq_input_gw_channel">

                    <jms-message-filter dest-name="queue/MY.INPUT.Q" dest-type="QUEUE" />

                </jms-bus>

            </jms-provider>

        </providers>

        <services>

            <service

                     category="test" >

       

                <listeners>

                    <jms-listener

                                  busidref="jbm_input_gw_channel"

                                  is-gateway="true"/>

                    <jms-listener

                                  busidref=" wmq_input_gw_channel "

                                  is-gateway="true"/>

                    <jms-listener

                                  busidref="esb_input_channel"

                                  is-gateway="false"/>

                </listeners>

      {code}

        • 1. Re: After network connection failure and reconnect, fails to reestablish connection to a Websphere MQ queue.
          jay.howell

          Just hazarding a guess here, but you might fix this by adding

          <track-connection-by-tx> in your outbound adapter specification.

           

          This was fixed in as 5.0.1 and track-connection-by-tx was set to true by default.  https://jira.jboss.org/browse/JBAS-5203

           

          I don't believe you have this fix in your server.  If you do have it and you set track-connection-by-tx to true, it should give you a warn and not hurt anything.

           

          I would add both

           

          <track-connection-by-tx/>

          <application-managed-security/>

           

           


          To your datasource definition and see if this clears up your issue, or you can get a build > 5.0.1 and that should also fix your issue.

           

          Jay:)

          • 2. Re: After network connection failure and reconnect, fails to reestablish connection to a Websphere MQ queue.
            jaswinder.singh

            Rebecca,

            Were you able to resolve the issue with Jay's suggesstion? Anything else that made it work?

            • 3. Re: After network connection failure and reconnect, fails to reestablish connection to a Websphere MQ queue.
              aemcat

              I eventually tried the suggestions by adding the 2 tags recommened, but alas, it still did not solve the problem  (sorry, kept meaning to update this thread!) .

              We have not been able to try upgrading to 5.0.1 yet, though if those 2 settings are the only difference, it seems it wouldn't help.

              We have not had time to pursue the problem otherwise.

               

              So we are still in the same boat right now. If the WebSphere MQ connection goes down, we have no way to recover it except to redeploy the app or restart the app server.

              Sorry I don't have better news!

              • 4. Re: After network connection failure and reconnect, fails to reestablish connection to a Websphere MQ queue.
                pokerdan

                We are experiencing the same issue, and have had a ticket open with JBoss Support since 6/30/10 trying to resolve it.  I would greatly appreciate any information you can provide if you figure anything out.

                 

                We were able to get past the message regarding "Only one session per connection allowed." (Make sure your jboss-esb.xml files have the property <property name="org.jboss.esb.jms.max.sessions.per.connection" value="1"/>), but it did not address the root cause and now we see errors like:

                 

                Caused by: javax.jms.JMSException: MQJMS2005: failed to create MQQueueManager for 'ISOE:MQTG'
                    at com.ibm.mq.jms.services.ConfigEnvironment.newException(ConfigEnvironment.java:614)
                    at com.ibm.mq.jms.MQConnection.createQM(MQConnection.java:2480)
                    at com.ibm.mq.jms.MQConnection.createQMNonXA(MQConnection.java:1830)
                    at com.ibm.mq.jms.MQConnection.<init>(MQConnection.java:811)
                    at com.ibm.mq.jms.MQConnectionFactory.createConnection(MQConnectionFactory.java:4821)
                    at com.ibm.mq.connector.ConnectionBuilder.createConnection(ConnectionBuilder.java:159)
                    ... 23 more
                16:43:40,564 WARN  [JmsGatewayListener] Error reconnecting to Queue, backing off for 32000 milliseconds
                16:44:42,576 WARN  [JmsGatewayListener] Error reconnecting to Queue, backing off for 32000 milliseconds
                16:45:44,593 WARN  [JmsGatewayListener] Error reconnecting to Queue, backing off for 32000 milliseconds
                16:46:46,610 WARN  [JmsGatewayListener] Error reconnecting to Queue, backing off for 32000 milliseconds

                 

                and the connection never comes back.

                • 5. Re: After network connection failure and reconnect, fails to reestablish connection to a Websphere MQ queue.
                  aemcat

                  My co-worker, another member of this forum, opened a ticket last week and came up with the property setting below.
                  I just tried this on (and only on) my laptop environment, but I'm getting reconnected when I reconnect to the network.

                  I didn't try the setting you mentioned, however, this one is similar but not quite the same.

                   

                  Add the following property to your <jms-provider> for WMQ in jboss-esb.xml.

                   

                     <property name="max-sessions-per-connection" value="1" />

                   

                  so my jms-provider section in my jboss-esb.xml looks like 

                  <jms-provider name="WebSphereMQ"
                       connection-factory="java:/Project.CF">

                   

                     <property name="max-sessions-per-connection" value="1" />

                   

                     <jms-bus busid="GWChannel_Project.In.WMQ">
                       <jms-message-filter

                                 dest-name="queue/Project.In.WMQ"
                                 dest-type="QUEUE" />
                     </jms-bus>


                  </jms-provider>

                  • 6. Re: After network connection failure and reconnect, fails to reestablish connection to a Websphere MQ queue.
                    pokerdan

                    Unfortunately, trying that setting did not resolve the problem for us.  However, we were able to setup some custom logging to provide more details, and we now see the following errors with MQ codes in the stack trace.  Still dont know the resolution, however:

                     

                    13:47:08,257 WARN  [JmsGatewayListener] Error reconnecting to Queue, backing off for 1000 milliseconds
                    13:47:09,257 ERROR [LogConnectionExceptionHandler] JMSException: MQJMS2002
                    13:47:09,257 ERROR [LogConnectionExceptionHandler] LinkedException: MQJE001: Completion Code 2, Reason 2019
                    13:47:09,413 ERROR [LogConnectionExceptionHandler] JMSException: MQJMS2002
                    13:47:09,413 ERROR [LogConnectionExceptionHandler] LinkedException: MQJE001: Completion Code 2, Reason 2019
                    13:47:09,429 ERROR [LogConnectionExceptionHandler] JMSException: MQJMS2002
                    13:47:09,429 ERROR [LogConnectionExceptionHandler] LinkedException: MQJE001: Completion Code 2, Reason 2019
                    13:47:38,258 ERROR [LogConnectionExceptionHandler] JMSException: null
                    13:47:38,258  ERROR [LogConnectionExceptionHandler] LinkedException: No  ManagedConnections available within configured blocking timeout ( 30000  [ms] )
                    13:47:38,258 ERROR [LogConnectionExceptionHandler] JMSException: null
                    13:47:38,258  ERROR [LogConnectionExceptionHandler] LinkedException: No  ManagedConnections available within configured blocking timeout ( 30000  [ms] )
                    13:47:38,258 WARN  [JmsGatewayListener] Error reconnecting to Queue, backing off for 1000 milliseconds