4 Replies Latest reply on Jul 22, 2004 2:21 PM by Adrian Brock

    Handling many NON_PERSISTENT messages and staying alive

    Oliver Henlich Newbie

      Hello.

      This question is about how to configure JBossMQ to cope with a huge flood of messages that are NON_PERSISTENT.

      Assumptions:
      - A NON_PERSISTENT message does not have to be stored persistently, server may drop/lose some. The javadoc says:


      This is the lowest-overhead delivery mode because it does not require
      that the message be logged to stable storage. The level of JMS provider
      failure that causes a NON_PERSISTENT message to be lost is not defined.


      - The MessageCache is good since it stops me hitting a OutOfMemoryException

      - It is fatal (from a performance standpoint) for the cache to begin storing messages to secondary storage.
      This will only make the situation worse if we are being flooded with messages.

      Questions and attempted answers:
      Q1) When does JBoss drop NON_PERSITENT messages?
      A1) Reboot i presume?

      Q2) Can the NullPersistenceManager be used in this scenario?
      A2) I don't think so since it stores all the messages in a Map in memory and will not drop messages.
      So we'll start running out of memory [i tried this, see below].

      Q3) If my requirement is for JBoss to still be responsive when i get a flood of messages and
      i'm happy to lose messages what are my options?
      A3) I imagine i could write my own org.jboss.mq.pm.PersistenceManager/CacheStore whose implementation
      of saveToStorage() just loses the message.

      I would be very grateful for input/feedback on these questions/answer attempts.

      I did try replacing deploy/jms/hsqldb-jdbc-service.xml with examples/jms/null-persistence-service.xml
      and running my stress client with the default jvm heap size. What i observe is that the CPU hits 100%.
      If i enable tracing i very rapidly get GIGA bytes of logs. Looking at them there is an awful lot of the following:
      2004-07-22 16:53:24,187 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
      2004-07-22 16:53:24,187 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
      2004-07-22 16:53:24,187 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
      2004-07-22 16:53:24,187 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
      2004-07-22 16:53:24,187 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
      
      ...
      2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.TracingInterceptor] CALLED : transact
      2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.TracingInterceptor] ARG : org.jboss.mq.TransactionRequest@ac0e87
      2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.BasicQueue] acknowledge AcknowledgementRequest:ACK,TOPIC.testTopic,ID:21-1090507594531644 12614 org.jboss.mq.server.ExclusiveQueue@1b27a72{id=TOPIC.testTopic.ID:1.-2147483648}
      2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.TracingInterceptor] RETURN : transact
      2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
      2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.TracingInterceptor] CALLED : receive
      2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.TracingInterceptor] ARG : -2147483648
      2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.TracingInterceptor] ARG : -1
      2004-07-22 16:53:24,687 TRACE [org.jboss.mq.security.ServerSecurityInterceptor] Checking receive authorize on ConnectionToken:ID:1/0936deaa2c42a73e515828be586c1022 subId=-2147483648
      2004-07-22 16:53:24,687 TRACE [org.jboss.mq.security.SecurityManager] Checking authorize on subjectInfo: SubjectInfo {subject=Betreff:
       Principal: guest
       Principal: Roles(members:j2ee,guest,john)
      ;principal=null;roles=Roles(members:j2ee,guest,john) for rolePrincipals [durpublisher, publisher, guest]
      2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.BasicQueue] receive Subscription[subId=-2147483648connection=ConnectionToken:ID:1/0936deaa2c42a73e515828be586c1022 destination=TOPIC.testTopic messageSelector=null Local Create] wait=false org.jboss.mq.server.ExclusiveQueue@1b27a72{id=TOPIC.testTopic.ID:1.-2147483648}
      2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
      2004-07-22 16:53:24,687 TRACE [org.jboss.mq.SpyXAResource] Start xid=XidImpl [FormatId=257, GlobalId=jboss-laptop//12655, BranchQual=], flags=0 SpyXAResource[session=SpySession@27923397[tx=true txid=-9223372036854763180 XA RUNNING connection=Connection@21513864[token=ConnectionToken:ID:1/0936deaa2c42a73e515828be586c1022 rcvstate=STARTED]]]
      ...
      
      2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
      2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
      2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
      2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
      2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
      


      the saveToStorage appears over 14000 times for msg=12629. I rekon this is where the CPU is spending the time.
      I found the code that does it, but don't really know whats going on here:
       void makeSoft() throws JMSException
       ...
       // HACK: allow the jdbc2 driver to reject saveToStorage for persistent messages
       // it is just about to persist the message when it gets some cpu
       if (stored != STORED)
       {
       if (trace)
       log.trace("saveToStorage rejected by cache " + toString());
       return;
       }
      
      


      cheers

        • 1. Re: Handling many NON_PERSISTENT messages and staying alive
          Adrian Brock Master

          That is a bug, it is not setting the stored status of the message reference:

          org.jboss.mq.pm.none.PersistenceManager should be:

           public void removeFromStorage(MessageReference mh) throws JMSException
           {
           if (delegate == null || mh.inMemory())
           {
           cache.remove(mh);
           mh.setStored(MessageReference.NOT_STORED);
           }
           else
           ((CacheStore) delegate).removeFromStorage(mh);
          
           }
          
           public void saveToStorage(MessageReference mh, SpyMessage message) throws JMSException
           {
           if (delegate == null || mh.inMemory())
           {
           cache.put(mh, message);
           mh.setStored(MessageReference.STORED);
           }
           else
           ((CacheStore) delegate).saveToStorage(mh, message);
           }
          


          • 2. Re: Handling many NON_PERSISTENT messages and staying alive
            Adrian Brock Master

            If you want it to drop NON_PERSISTENT messages rather than putting them
            putting them on disk it would require a modification to the MessageCache
            and BasicQueue to remove the message from the queue
            (assuming it hasn't already been received by a client).

            • 3. Re: Handling many NON_PERSISTENT messages and staying alive
              Oliver Henlich Newbie

              Hi Adrian.
              Thanks for the response.

              So A3) is completely wrong, not the whole story or another valid option.

              I thought that once a message gets passed to saveToStorage() and the
              cache references to it have been softened, then nothing else has got a
              handle on it....i'm not so familiar with the code...sorry.

              I would really appreciate your thoughts on dropping NON_PERSISTENT
              messages though. Is it a requirement you've come accross before? Or
              does it sound like such a requirement shows a broken design?

              Thanks for the pointers to MessageCache and BasicQueue. Hopefully i'll
              get the time to dive in and give it a go. Ah maybe you mean modifiying
              these classes such that they bypass the persistence manager completely. Not sure.

              cheers

              • 4. Re: Handling many NON_PERSISTENT messages and staying alive
                Adrian Brock Master

                I don't see anything wrong with dropping NON_PERSISTENT messages
                rather than putting them on disk.

                They are lost in the event of a reboot, so you need a design that can cope
                with the non guaranteed delivery of the message.

                JBossMQ doesn't currently implement it that way.