2 Replies Latest reply on Jul 9, 2012 6:38 AM by forum.khan

    JBoss auto reconnect issue with Websphere MQ

    forum.khan

      Hi,

         Am facing JBoss auto reconnect issue with websphere MQ. Please let me know if any configuration setting needs to be done. I had raised this same question in messaging: https://community.jboss.org/message/745338 thinking its a messaging issue. My files configuration is similar to this: http://www.ibm.com/developerworks/websphere/library/techarticles/0710_ritchie/0710_ritchie.html

       

         Let me explain the scenario:

         1. I will be sending the messages to the JMS queue via the EJB which is delivered & processed successfully. In the mean time I will only stop the MQ server then it throws the exception marked in Red in the Jboss console.

         2. Next I will restart the MQ server. Exactly after 5 mins it display's the message it got reconnected successfully which is marked in Green.

         3. After it gets connected successfully again am able to send the messages successfully to the queue.

         4. Next I will disconnect/stopping MQ for the whole night without shutting down the JBOSS then next day morning I will restart the MQ  and send few messages. Now my JBOSS will not able to reconnect to MQ and nor be able to process the JMS messages.

       

          Now I like to understand the below points

          1. How this is getting reconnected exactly after 5 mins, where is the configuration setting?

          2. It is observed that if i disconnect/stopping MQ for the whole night without shutting down the JBOSS then next day morning I will restart the MQ  and send few messages then now my JBOSS will not able to reconnect to MQ and nor be able to process the JMS messages. Here I like to understand after shutting down and restarting the MQ it used to connect automatically. But when its closed from the whole night (LONG DURATION), next day JBOSS is not able to identity the MQ. Is there any special configuration setting for this?

       

       

      The below message is from the console.

      Console:

       

      20:06:59,380 WARN  [HttpRouter] Received status code '302' on HTTP org.apache.commons.httpclient.methods.PostMethod@a29bbd request to 'http://localhost:8080/someservice.

      20:09:02,441 INFO  [CNN_Connection_Factory] JMSCC3036:An exception has been delivered to the connections exception listener: '

                             Message : com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ1107: A problem with this connection has occurred. An error has occurred with the WebSphere MQ JMS connection. Use the linked exception to determine the cause of this error.

                               Class : class com.ibm.msg.client.jms.DetailedJMSException

                               Stack : com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:608)

                                     : com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:236)

                                     : com.ibm.msg.client.wmq.internal.WMQConnection.consumer(WMQConnection.java:823)

                                     : com.ibm.mq.jmqi.remote.internal.RemoteAsyncConsume.callEventHandler(RemoteAsyncConsume.java:1018)

                                     : com.ibm.mq.jmqi.remote.internal.RemoteAsyncConsume.driveEventsEH(RemoteAsyncConsume.java:1376)

                                     : com.ibm.mq.jmqi.remote.internal.RemoteDispatchThread.run(RemoteDispatchThread.java:309)

                                     : com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:209)

                                     : com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:100)

                                     : com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:224)

                                     : com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:298)

                                     : com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1220)

           Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').

                               Class : class com.ibm.mq.MQException

                               Stack : com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:223)

                                     : com.ibm.msg.client.wmq.internal.WMQConnection.consumer(WMQConnection.java:823)

                                     : com.ibm.mq.jmqi.remote.internal.RemoteAsyncConsume.callEventHandler(RemoteAsyncConsume.java:1018)

                                     : com.ibm.mq.jmqi.remote.internal.RemoteAsyncConsume.driveEventsEH(RemoteAsyncConsume.java:1376)

                                     : com.ibm.mq.jmqi.remote.internal.RemoteDispatchThread.run(RemoteDispatchThread.java:309)

                                     : com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:209)

                                     : com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:100)

                                     : com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:224)

                                     : com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:298)

                                     : com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1220)

      '.

      20:09:02,441 INFO  [CNN_Connection_Factory] MQJCA4013:A connection to a queue manager failed for activationSpec 'javax.jms.Queue:CNN_Gateway@QM_GIS01234 <3852704>'. Check the queue manager error logs for details.

      20:09:17,441 INFO  [CNN_Connection_Factory] MQJCA4003:A recoverable exception occurred in the JMS layer: 'JMSCMQ0002: The method 'MQCTL' failed.'

      20:09:17,441 INFO  [CNN_Connection_Factory] MQJCA4003:A recoverable exception occurred in the JMS layer: 'JMSWMQ0019: Failed to disconnect from queue manager 'QM_GIS01234' using connection mode '1' and host name 'gis01234.cloud.abc.com(1414)'.'

      20:14:21,687 INFO  [CNN_Connection_Factory] MQJCA4019:ActivationSpec 'javax.jms.Queue:CNN_Gateway@QM_GIS01234<3852704>' reconnected successfully.

       

       

      We are using JMS with EJB 3.0. The queues are being configured in JBOSS 5.0. The message processing is done in ESB, now it looks like i have posted this message in the right window.

       

      The JMS configuration is done in wmq.jmsra-ds.xml

       

      <connection-factories>
        <tx-connection-factory>
          <jndi-name>CNN_Connection_Factory</jndi-name>
          <xa-transaction />
          <rar-name>wmq.jmsra.rar</rar-name>

          <connection-definition>javax.jms.ConnectionFactory</connection-definition>

          <config-property name="channel" type="java.lang.String">someChannel</config-property>
          <config-property name="hostName" type="java.lang.String">GIS01234.cloud.abc.com</config-property> 

          <config-property name="port" type="java.lang.String">5678</config-property>
          <config-property name="queueManager" type="java.lang.String">QM_GIS5678</config-property>
          <config-property name="transportType" type="java.lang.String">CLIENT</config-property>
          <security-domain-and-application>JmsXARealm</security-domain-and-application>
        </tx-connection-factory>

       

        <mbean code="org.jboss.resource.deployment.AdminObject" name="jca.wmq:name=cnngateway">
          <attribute name="JNDIName">CNN_Gateway</attribute>
          <depends optional-attribute-name="RARName">jboss.jca:service=RARDeployment,name='wmq.jmsra.rar'</depends>
          <attribute name="Type">javax.jms.Queue</attribute>
          <attribute name="Properties">
            baseQueueManagerName=QM_GIS01234
            baseQueueName=CNN_Queue

          </attribute>
        </mbean>
      </connection-factories>

         

      Thanks in Advance

      KK

        • 1. Re: JBoss auto reconnect issue with Websphere MQ
          forum.khan

          After analysis I found that in wmq.jmsra.rar\META-INF\ra.xml file there is reconnectionRetryCount = 5 and reconnectionRetryInterval = 300000 that is causing the server to auto reconnect to MQ every 5 mins.

           

          Using RAR software, I extracted the ra.xml file and modified those properties. Again packaged to .rar file.

          After the deployment of .rar file and restarting the server the deployment of .rar fails due to the below errors.

          Please let me know how to resolve this.

           

          12:22:25,231 ERROR [ProfileDeployAction] Failed to add deployment: wmq.jmsra.rar

          org.jboss.deployers.spi.DeploymentException: Error determining structure: wmq.jmsra.rar

          at org.jboss.deployers.spi.DeploymentException.rethrowAsDeploymentException(DeploymentException.java:49)

          at org.jboss.deployers.vfs.plugins.structure.explicit.DeclaredStructure.determineStructure(DeclaredStructure.java:89)

          at org.jboss.deployers.vfs.plugins.structure.StructureDeployerWrapper.determineStructure(StructureDeployerWrapper.java:73)

          at org.jboss.deployers.vfs.plugins.structure.VFSStructuralDeployersImpl.doDetermineStructure(VFSStructuralDeployersImpl.java:196)

          at org.jboss.deployers.vfs.plugins.structure.VFSStructuralDeployersImpl.determineStructure(VFSStructuralDeployersImpl.java:221)

          at org.jboss.deployers.structure.spi.helpers.AbstractStructuralDeployers.determineStructure(AbstractStructuralDeployers.java:77)

          at org.jboss.deployers.plugins.main.MainDeployerImpl.determineStructure(MainDeployerImpl.java:1001)

          at org.jboss.deployers.plugins.main.MainDeployerImpl.determineDeploymentContext(MainDeployerImpl.java:437)

          at org.jboss.deployers.plugins.main.MainDeployerImpl.addDeployment(MainDeployerImpl.java:387)

          at org.jboss.deployers.plugins.main.MainDeployerImpl.addDeployment(MainDeployerImpl.java:297)

          at org.jboss.system.server.profileservice.repository.MainDeployerAdapter.addDeployment(MainDeployerAdapter.java:86)

          at org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:61)

          at org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:53)

          at org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:403)

          at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

          at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)

          at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)

          at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

          at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)

          at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:778)

          at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:543)

          at org.jboss.system.server.profileservice.repository.AbstractProfileService.registerProfile(AbstractProfileService.java:308)

          at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:256)

          at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:461)

          at org.jboss.Main.boot(Main.java:221)

          at org.jboss.Main$1.run(Main.java:556)

          at java.lang.Thread.run(Thread.java:662)

           

          Caused by: java.lang.RuntimeException: java.util.zip.ZipException: error in opening zip file

          at org.jboss.virtual.plugins.context.AbstractExceptionHandler.handleZipEntriesInitException(AbstractExceptionHandler.java:39)

          at org.jboss.virtual.plugins.context.helpers.NamesExceptionHandler.handleZipEntriesInitException(NamesExceptionHandler.java:63)

          at org.jboss.virtual.plugins.context.zip.ZipEntryContext.ensureEntries(ZipEntryContext.java:636)

          at org.jboss.virtual.plugins.context.zip.ZipEntryContext.checkIfModified(ZipEntryContext.java:786)

          at org.jboss.virtual.plugins.context.zip.ZipEntryContext.getChild(ZipEntryContext.java:830)

          at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.createChildHandler(ZipEntryHandler.java:195)

          at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(AbstractVirtualFileHandler.java:690)

          at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getChild(ZipEntryHandler.java:169)

          at org.jboss.virtual.plugins.context.DelegatingHandler.getChild(DelegatingHandler.java:107)

          at org.jboss.virtual.VirtualFile.getChild(VirtualFile.java:492)

          at org.jboss.deployers.vfs.plugins.structure.explicit.DeclaredStructure.determineStructure(DeclaredStructure.java:64)

           

          Caused by: java.util.zip.ZipException: error in opening zip file

          at java.util.zip.ZipFile.open(Native Method)

          at java.util.zip.ZipFile.<init>(ZipFile.java:127)

          at java.util.jar.JarFile.<init>(JarFile.java:135)

          at java.util.jar.JarFile.<init>(JarFile.java:99)

          at org.jboss.virtual.plugins.context.zip.ZipFileWrapper.ensureZipFile(ZipFileWrapper.java:173)

          at org.jboss.virtual.plugins.context.zip.ZipFileWrapper.acquire(ZipFileWrapper.java:250)

          at org.jboss.virtual.plugins.context.zip.ZipEntryContext.initEntries(ZipEntryContext.java:485)

          at org.jboss.virtual.plugins.context.zip.ZipEntryContext.ensureEntries(ZipEntryContext.java:629)

           

           

          • 2. Re: JBoss auto reconnect issue with Websphere MQ
            forum.khan

            The issue was due to improper packaging of .rar file [Caused by: java.util.zip.ZipException: error in opening zip file]

             

            By modifying the properties reconnectionRetryCount = 5 and reconnectionRetryInterval = 300000 in file “wmq.jmsra.rar\META-INF\ra.xml” I was able to resolve the defect.

             

               In order to test the scenario I had set reconnectionRetryInterval = 60000 (1 min) and kept reconnectionRetryCount = 5 attempts only.  I was able to process the messages successfully. On shutting down the MQ and waiting for 5 mins then I found that messages were not processed even after starting MQ. As per the below link it’s clearly mentioned that If all attempts fail to connect MQ, delivery is stopped permanently until the MDBs are restarted manually.

            http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/index.jsp?topic=%2Fcom.ibm.mq.csqzaw.doc%2Fjm40190_.htm

             

            http://www.ibm.com/developerworks/websphere/library/techarticles/0710_ritchie/0710_ritchie.html

             

            Observation:

               Let us assume that MQ will be down only for a period of 1 hour at any point of time, then the values will be reconnectionRetryCount = 12 and reconnectionRetryInterval = 300000 (5 mins) so that the jboss server continues to try connecting to MQ for every 5 mins upto 12 attempts ( Which also means the server will stop pinging MQ after 1 hour having tried to ping for 12 attempts). This is only applicable if the MQ goes down only for 1 hour, else the properties may have to be changed accordingly.