5 Replies Latest reply on Jul 9, 2012 6:39 AM by forum khan

    JBOSS Connection Issue with MQ

    forum khan Newbie

      Hi,

         Let me explain the scenario:

         1. I will be sending the message to the JMS queue which is delivered successfully. In the mean time i will stop the server then it throws the exception marked in Red.

         2. Next i will restart the server. Exactly after 5 mins it display's the message it got reconnected successfully which is marked in Green.

         3. After it gets connected successfully again am able to send the messages successfully to the queue.

       

          Now i like to understand the below points

          1. How this is getting reconnected exactly after 5 mins, where is the configuration setting?

          2. It is observed that if i disconnect MQ for the whole night without shutting down the JBOSS then next day my JBOSS will not able to reconnect to MQ and will not be able to process the JMS messages. Here I like to understand after shutting down and restart the MQ it used to connect automatically. But when its closed from the whole night, next day JBOSS is not able to identity the MQ. Is there any special configuration setting for this?

       

       

      The below message is from the console.

      Console:

       

      20:06:59,380 WARN  [HttpRouter] Received status code '302' on HTTP org.apache.commons.httpclient.methods.PostMethod@a29bbd request to 'http://localhost:8080/someservice.

      20:09:02,441 INFO  [CNN_Connection_Factory] JMSCC3036:An exception has been delivered to the connections exception listener: '

                             Message : com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ1107: A problem with this connection has occurred. An error has occurred with the WebSphere MQ JMS connection. Use the linked exception to determine the cause of this error.

                               Class : class com.ibm.msg.client.jms.DetailedJMSException

                               Stack : com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:608)

                                     : com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:236)

                                     : com.ibm.msg.client.wmq.internal.WMQConnection.consumer(WMQConnection.java:823)

                                     : com.ibm.mq.jmqi.remote.internal.RemoteAsyncConsume.callEventHandler(RemoteAsyncConsume.java:1018)

                                     : com.ibm.mq.jmqi.remote.internal.RemoteAsyncConsume.driveEventsEH(RemoteAsyncConsume.java:1376)

                                     : com.ibm.mq.jmqi.remote.internal.RemoteDispatchThread.run(RemoteDispatchThread.java:309)

                                     : com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:209)

                                     : com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:100)

                                     : com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:224)

                                     : com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:298)

                                     : com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1220)

           Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').

                               Class : class com.ibm.mq.MQException

                               Stack : com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:223)

                                     : com.ibm.msg.client.wmq.internal.WMQConnection.consumer(WMQConnection.java:823)

                                     : com.ibm.mq.jmqi.remote.internal.RemoteAsyncConsume.callEventHandler(RemoteAsyncConsume.java:1018)

                                     : com.ibm.mq.jmqi.remote.internal.RemoteAsyncConsume.driveEventsEH(RemoteAsyncConsume.java:1376)

                                     : com.ibm.mq.jmqi.remote.internal.RemoteDispatchThread.run(RemoteDispatchThread.java:309)

                                     : com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:209)

                                     : com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:100)

                                     : com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:224)

                                     : com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:298)

                                     : com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1220)

      '.

      20:09:02,441 INFO  [CNN_Connection_Factory] MQJCA4013:A connection to a queue manager failed for activationSpec 'javax.jms.Queue:CNN_Gateway@QM_GIS01234 <3852704>'. Check the queue manager error logs for details.

      20:09:17,441 INFO  [CNN_Connection_Factory] MQJCA4003:A recoverable exception occurred in the JMS layer: 'JMSCMQ0002: The method 'MQCTL' failed.'

      20:09:17,441 INFO  [CNN_Connection_Factory] MQJCA4003:A recoverable exception occurred in the JMS layer: 'JMSWMQ0019: Failed to disconnect from queue manager 'QM_GIS01234' using connection mode '1' and host name 'gis01234.cloud.abc.com(1414)'.'

      20:14:21,687 INFO  [CNN_Connection_Factory] MQJCA4019:ActivationSpec 'javax.jms.Queue:CNN_Gateway@QM_GIS01234<3852704>' reconnected successfully.

       

      Thanks in Advance

      KK

        • 1. Re: JBOSS Connection Issue with MQ
          Justin Bertram Master

          How are you integrating with WebsphereMQ?  Have you deployed their JCA RA?  If so, I would investigate the IBM documentation on this subject since their RA maintains the connection to the back-end broker.

           

          Also, how is this related to JBoss Messaging?

          • 2. Re: JBOSS Connection Issue with MQ
            forum khan Newbie

            We are using JMS with EJB 3.0. The queues are being configured in JBOSS 5.0. The message processing is done in ESB, looks like i have posted this message in a wrong window. My file configuration is similar to this: http://www.ibm.com/developerworks/websphere/library/techarticles/0710_ritchie/0710_ritchie.html

             

            The JMS configuration is done in wmq.jmsra-ds.xml

             

            <connection-factories>
              <tx-connection-factory>
                <jndi-name>CNN_Connection_Factory</jndi-name>
                <xa-transaction />
                <rar-name>wmq.jmsra.rar</rar-name>

                <connection-definition>javax.jms.ConnectionFactory</connection-definition>

                <config-property name="channel" type="java.lang.String">someChannel</config-property>
                <config-property name="hostName" type="java.lang.String">GIS01234.cloud.abc.com</config-property> 

                <config-property name="port" type="java.lang.String">5678</config-property>
                <config-property name="queueManager" type="java.lang.String">QM_GIS5678</config-property>
                <config-property name="transportType" type="java.lang.String">CLIENT</config-property>
                <security-domain-and-application>JmsXARealm</security-domain-and-application>
              </tx-connection-factory>

             

              <mbean code="org.jboss.resource.deployment.AdminObject" name="jca.wmq:name=cnngateway">
                <attribute name="JNDIName">CNN_Gateway</attribute>
                <depends optional-attribute-name="RARName">jboss.jca:service=RARDeployment,name='wmq.jmsra.rar'</depends>
                <attribute name="Type">javax.jms.Queue</attribute>
                <attribute name="Properties">
                  baseQueueManagerName=QM_GIS01234
                  baseQueueName=CNN_EDDC
                </attribute>
              </mbean>
            </connection-factories>

            • 3. Re: JBOSS Connection Issue with MQ
              forum khan Newbie

              After analysis I found that in wmq.jmsra.rar\META-INF\ra.xml file there is reconnectionRetryCount = 5 and reconnectionRetryInterval = 300000 that is causing the server to auto reconnect to MQ every 5 mins.

               

              Using RAR software, I extracted the ra.xml file and modified those properties. Again packaged to .rar file.

              After the deployment of .rar file and restarting the server the deployment of .rar fails due to the below errors.

              Please let me know how to resolve this.

               

              12:22:25,231 ERROR [ProfileDeployAction] Failed to add deployment: wmq.jmsra.rar

              org.jboss.deployers.spi.DeploymentException: Error determining structure: wmq.jmsra.rar

              at org.jboss.deployers.spi.DeploymentException.rethrowAsDeploymentException(DeploymentException.java:49)

              at org.jboss.deployers.vfs.plugins.structure.explicit.DeclaredStructure.determineStructure(DeclaredStructure.java:89)

              at org.jboss.deployers.vfs.plugins.structure.StructureDeployerWrapper.determineStructure(StructureDeployerWrapper.java:73)

              at org.jboss.deployers.vfs.plugins.structure.VFSStructuralDeployersImpl.doDetermineStructure(VFSStructuralDeployersImpl.java:196)

              at org.jboss.deployers.vfs.plugins.structure.VFSStructuralDeployersImpl.determineStructure(VFSStructuralDeployersImpl.java:221)

              at org.jboss.deployers.structure.spi.helpers.AbstractStructuralDeployers.determineStructure(AbstractStructuralDeployers.java:77)

              at org.jboss.deployers.plugins.main.MainDeployerImpl.determineStructure(MainDeployerImpl.java:1001)

              at org.jboss.deployers.plugins.main.MainDeployerImpl.determineDeploymentContext(MainDeployerImpl.java:437)

              at org.jboss.deployers.plugins.main.MainDeployerImpl.addDeployment(MainDeployerImpl.java:387)

              at org.jboss.deployers.plugins.main.MainDeployerImpl.addDeployment(MainDeployerImpl.java:297)

              at org.jboss.system.server.profileservice.repository.MainDeployerAdapter.addDeployment(MainDeployerAdapter.java:86)

              at org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:61)

              at org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:53)

              at org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:403)

              at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

              at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)

              at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)

              at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

              at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)

              at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:778)

              at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:543)

              at org.jboss.system.server.profileservice.repository.AbstractProfileService.registerProfile(AbstractProfileService.java:308)

              at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:256)

              at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:461)

              at org.jboss.Main.boot(Main.java:221)

              at org.jboss.Main$1.run(Main.java:556)

              at java.lang.Thread.run(Thread.java:662)

               

              Caused by: java.lang.RuntimeException: java.util.zip.ZipException: error in opening zip file

              at org.jboss.virtual.plugins.context.AbstractExceptionHandler.handleZipEntriesInitException(AbstractExceptionHandler.java:39)

              at org.jboss.virtual.plugins.context.helpers.NamesExceptionHandler.handleZipEntriesInitException(NamesExceptionHandler.java:63)

              at org.jboss.virtual.plugins.context.zip.ZipEntryContext.ensureEntries(ZipEntryContext.java:636)

              at org.jboss.virtual.plugins.context.zip.ZipEntryContext.checkIfModified(ZipEntryContext.java:786)

              at org.jboss.virtual.plugins.context.zip.ZipEntryContext.getChild(ZipEntryContext.java:830)

              at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.createChildHandler(ZipEntryHandler.java:195)

              at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(AbstractVirtualFileHandler.java:690)

              at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getChild(ZipEntryHandler.java:169)

              at org.jboss.virtual.plugins.context.DelegatingHandler.getChild(DelegatingHandler.java:107)

              at org.jboss.virtual.VirtualFile.getChild(VirtualFile.java:492)

              at org.jboss.deployers.vfs.plugins.structure.explicit.DeclaredStructure.determineStructure(DeclaredStructure.java:64)

               

              Caused by: java.util.zip.ZipException: error in opening zip file

              at java.util.zip.ZipFile.open(Native Method)

              at java.util.zip.ZipFile.<init>(ZipFile.java:127)

              at java.util.jar.JarFile.<init>(JarFile.java:135)

              at java.util.jar.JarFile.<init>(JarFile.java:99)

              at org.jboss.virtual.plugins.context.zip.ZipFileWrapper.ensureZipFile(ZipFileWrapper.java:173)

              at org.jboss.virtual.plugins.context.zip.ZipFileWrapper.acquire(ZipFileWrapper.java:250)

              at org.jboss.virtual.plugins.context.zip.ZipEntryContext.initEntries(ZipEntryContext.java:485)

              at org.jboss.virtual.plugins.context.zip.ZipEntryContext.ensureEntries(ZipEntryContext.java:629)

              • 4. Re: JBOSS Connection Issue with MQ
                Justin Bertram Master

                Couple of points:

                1. This has nothing to do with JBoss Messaging.
                2. This has everything to do with the configuration of the IBM WebsphereMQ JCA RA.
                3. The java.util.zip.ZipException you are receiving now is almost certainly caused by your modifications to ra.xml.  My guess is that the file is packaged incorrectly (i.e. it is not in fact a zip file).
                • 5. Re: JBOSS Connection Issue with MQ
                  forum khan Newbie

                  The issue was due to improper packaging of .rar file [Caused by: java.util.zip.ZipException: error in opening zip file]

                   

                  By modifying the properties reconnectionRetryCount = 5 and reconnectionRetryInterval = 300000 in file “wmq.jmsra.rar\META-INF\ra.xml” I was able to resolve the defect.

                   

                     In order to test the scenario I had set reconnectionRetryInterval = 60000 (1 min) and kept reconnectionRetryCount = 5 attempts only.  I was able to process the messages successfully. On shutting down the MQ and waiting for 5 mins then I found that messages were not processed even after starting MQ. As per the below link it’s clearly mentioned that If all attempts fail to connect MQ, delivery is stopped permanently until the MDBs are restarted manually.

                  http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/index.jsp?topic=%2Fcom.ibm.mq.csqzaw.doc%2Fjm40190_.htm

                   

                  http://www.ibm.com/developerworks/websphere/library/techarticles/0710_ritchie/0710_ritchie.html

                   

                  Observation:

                     Let us assume that MQ will be down only for a period of 1 hour at any point of time, then the values will be reconnectionRetryCount = 12 and reconnectionRetryInterval = 300000 (5 mins) so that the jboss server continues to try connecting to MQ for every 5 mins upto 12 attempts ( Which also means the server will stop pinging MQ after 1 hour having tried to ping for 12 attempts). This is only applicable if the MQ goes down only for 1 hour, else the properties may have to be changed accordingly.