6 Replies Latest reply on Oct 18, 2007 12:41 PM by nie feng

    JMS msg get redelivered although setting retry to 0

    nie feng Newbie

      Hi all,

      I have some problem on the JMS msg redelivery after the main node has been down abnormally. I would like to disable such a redelivery due to some business logic requirement.

      My box has the following setup:
      version: JBOSS 4.0.5 GA
      OS: Sun Solaris 10
      JRE: JRE 1.5.10

      Deploy env:
      Two Jboss server running at cluster mode.
      JMS: HA-JMS


      When the main node has been gone down unexpectedly, the JMS service is temporary unavailable to the second node (The handover of the JMS service to the second node will take some time).

      I have observed the case that if the onMessage() of the MD Bean deployed on second node has been called prior the failure of the main node, and that MD Bean finished the processing when the handover process still on going, then a redeliver of that jms msg is kicked in due to the MD Bean hasn't ack that msg. This part is functioning as expected.

      Snippet of the exception after the onMessage of the MD Bean has completed, but at that time the handover process still on going, and it is not able to send back the ACK:



      [09:10:07-16:50:30,815] [ INFO:STDOUT] [JMS SessionPool Worker-5] ReceiveMessageListenerMDBean end onMessage
      [09:10:07-16:50:30,834] [ WARN:org.jboss.tm.TransactionImpl] [JMS SessionPool Worker-5] XAException: tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=apple/168, BranchQual=, localId=168] errorCode=XAER_RMERR
      org.jboss.mq.SpyXAException: Resource manager error during prepare; - nested throwable: (org.jboss.mq.SpyJMSException: Cannot process a transaction; - nested throwable: (java.io.IOException: Client is not connected))
      at org.jboss.mq.SpyXAException.getAsXAException(SpyXAException.java:72)
      at org.jboss.mq.SpyXAResource.prepare(SpyXAResource.java:151)
      at org.jboss.tm.TransactionImpl$Resource.prepare(TransactionImpl.java:2212)
      at org.jboss.tm.TransactionImpl.prepareResources(TransactionImpl.java:1660)
      at org.jboss.tm.TransactionImpl.commit(TransactionImpl.java:347)
      at org.jboss.tm.TxManager.commit(TxManager.java:240)
      at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:351)
      at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:905)
      at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:170)
      at org.jboss.mq.SpySession.run(SpySession.java:323)
      at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194)
      at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
      at java.lang.Thread.run(Unknown Source)
      Caused by: org.jboss.mq.SpyJMSException: Cannot process a transaction; - nested throwable: (java.io.IOException: Client is not connected)
      at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:72)
      at org.jboss.mq.SpyJMSException.rethrowAsJMSException(SpyJMSException.java:57)
      at org.jboss.mq.Connection.send(Connection.java:1114)
      at org.jboss.mq.SpyXAResourceManager.prepare(SpyXAResourceManager.java:255)
      at org.jboss.mq.SpyXAResource.prepare(SpyXAResource.java:147)
      ... 11 more
      Caused by: java.io.IOException: Client is not connected
      at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:264)
      at org.jboss.mq.il.uil2.SocketManager.sendMessage(SocketManager.java:220)
      at org.jboss.mq.il.uil2.UILServerIL.transact(UILServerIL.java:368)
      at org.jboss.mq.Connection.send(Connection.java:1110)
      ... 13 more


      Due to some logic constraint, I need to disable the redelivery of the JMS for the above case. I have tried to disable such a redeliver in the standardjboss.xml as follow. I set the tag "MaxTimesRedelivered" to 0.

      <invoker-proxy-binding>
       <name>message-driven-bean-with-delivered-once</name>
       <invoker-mbean>default</invoker-mbean>
       <proxy-factory>org.jboss.ejb.plugins.jms.JMSContainerInvoker</proxy-factory>
       <proxy-factory-config>
       <JMSProviderAdapterJNDI>DefaultJMSProvider</JMSProviderAdapterJNDI>
       <ServerSessionPoolFactoryJNDI>StdJMSPool</ServerSessionPoolFactoryJNDI>
       <CreateJBossMQDestination>true</CreateJBossMQDestination>
       <!-- WARN: Don't set this to zero until a bug in the pooled executor is
      
      fixed -->
       <MinimumSize>1</MinimumSize>
       <!-- MaximumSize>15</MaximumSize -->
       <MaximumSize>50</MaximumSize>
       <KeepAliveMillis>30000</KeepAliveMillis>
       <MaxMessages>1</MaxMessages>
       <MDBConfig>
       <ReconnectIntervalSec>10</ReconnectIntervalSec>
       <DLQConfig>
       <DestinationQueue>queue/DLQ</DestinationQueue>
       <MaxTimesRedelivered>0</MaxTimesRedelivered>
       <TimeToLive>0</TimeToLive>
       </DLQConfig>
       </MDBConfig>
       </proxy-factory-config>
       </invoker-proxy-binding>
      

      However, I still observed the redelivered from the queue to the MD Bean.

      I also tried to ignore such a redelivery in the code level as follow

      if(message.getJMSRedelivered())
       {
       infoLog("onMessage", "re receive msg from from/queue again. Stop processing");
       return;
       }
      


      However, the flag "jmsRedelivered" is still set to false and the md bean still getting the jms msg that has been processed before.

      Could I know is there anywhere I have mis-configured? Or is there any alternative I can adapt to disable the redelivery in the above case?

      NOTE: The above two methods I mentioned is working fine if I do a messageCtxt.setRollback() in the MD BEAN.

        • 1. Re: JMS msg get redelivered although setting retry to 0
          nie feng Newbie

          Hi all,

          Did I omit something in my description or the information I provided is not sufficient for the case?

          Thanks for your help.

          • 2. Re: JMS msg get redelivered although setting retry to 0
            Adrian Brock Master

            Show System.out.println(message) for both deliveries of the message.

            • 3. Re: JMS msg get redelivered although setting retry to 0
              nie feng Newbie

              Hi Adrian, thanks for your reply.

              Below is the sysout of the jms message.

              1st received msg

              [13:10:07-16:16:22,461][JMS SessionPool Worker-19][ INFO][GN.APP.WORKFLOW][]Msg received: org.jboss.mq.SpyObjectMessage {
              Header {
               jmsDestination : QUEUE.sendSignalQueue
               jmsDeliveryMode : 2
               jmsExpiration : 0
               jmsPriority : 4
               jmsMessageID : ID:240-119226338233616
               jmsTimeStamp : 1192263382336
               jmsCorrelationID: null
               jmsReplyTo : null
               jmsType : null
               jmsRedelivered : false
               jmsProperties : {}
               jmsPropReadWrite: false
               msgReadOnly : true
               producerClientId: ID:240
              }


              The redelivered message
              [13:10:07-16:18:17,913][JMS SessionPool Worker-59][ INFO][GN.APP.WORKFLOW][]Msg received: org.jboss.mq.SpyObjectMessage {
              Header {
               jmsDestination : QUEUE.sendSignalQueue
               jmsDeliveryMode : 2
               jmsExpiration : 0
               jmsPriority : 4
               jmsMessageID : ID:240-119226338233616
               jmsTimeStamp : 1192263382336
               jmsCorrelationID: null
               jmsReplyTo : null
               jmsType : null
               jmsRedelivered : false
               jmsProperties : {}
               jmsPropReadWrite: false
               msgReadOnly : true
               producerClientId: ID:240
              }


              • 4. Re: JMS msg get redelivered although setting retry to 0
                Adrian Brock Master

                Ok, I think I understand what the problem is.

                The issue is that the failover node doesn't know that the message was already
                delivered. The node (the original node) that did know has crashed.

                The problem comes because the failover node uses the transaction log
                to recover the state to the "last known good state" of the message
                (which was in your case before the message was delivered to the MDB).

                The only way I can see to fix the problem, would be to update the
                redelivered status and redelivery count of the message in the database
                before the message is delivered to the client (instead of when the server
                receives the NACK/rollback - which it won't do if the server crashes
                - it will do it if the problem is just the client loosing connectivity).

                i.e. in BasicQueue
                http://viewvc.jboss.org/cgi-bin/viewvc.cgi/jbossas/trunk/jbossmq/src/main/org/jboss/mq/server/BasicQueue.java?revision=61447&view=markup

                This would involve moving the code in "nackMessage()"

                 try
                 {
                ...
                
                 // Set redelivered, vendor-specific flags
                 message.invalidate();
                 // Update the persistent message outside the transaction
                 // We want to know the message might have been delivered regardless
                 if (message.isPersistent())
                 server.getPersistenceManager().update(message, null);
                 }
                 catch (JMSException e)
                 {
                 log.error("Error setting redelivered " + message, e);
                 }
                 }
                


                To the setupMessageAcknowledgement() method instead (which is always
                invoked before delivering the message).

                But, this would obviously have a deterimental effect on performance
                since it would always update persistent messages with the redelivered status
                in the database regardless of whether it is was actually redelivered or not
                (if it doesn't get redelivered, i.e. it is ACKed/committed, it will be deleted so the
                early change in status is irrelevant).

                Do you want to try that patch?

                • 6. Re: JMS msg get redelivered although setting retry to 0
                  nie feng Newbie

                  Hi Adrian,

                  Thanks for the explanation on the problem. I will give it a try for that patch. Could I know where should I get it?

                  Will the solution you suggested (update the redelivery status prior the msg delivery to the client) be applied to the latest Jboss AS also? or it is a temporary solution to resolve my case?

                  In the jira, I saw the fix version will be in JBossAS-5.0.0.Beta3 and JBossAS-4.2.3.GA. Does it mean I need to update my current jboss to the latest version after the official patch make available? Or will you provide a seperate patch for JBoss 4.0.5 GA?