1 2 Previous Next 16 Replies Latest reply on Sep 11, 2012 5:44 PM by clebert.suconic

    Paging doesnt seem to be working

    into_java

      Hi All,

       

      The version of HORNETQ is am using is 2.1.2 Final

      I have a simple queue as defined here where i have set the size of the queue to be 20 Mbps and because of the hig rate at which messages are getting inserted i see that the queue gets full and i also see in the paging directory the 000001.page and 00002.page file sgetting created.

       

      Once the producer stops after dumping data i see that the .page files gets deleted but i dont see those messages on the other side of my consumer.

       

      My consumer only get the messages in my queue and not the one which were supposed to be depaged and available. 

       

      Also in the jmx console i observe that the message added property doesnt get incremented for when the depaging starts .. what does that mean  ? Even after depage the messages are some how getting dropped ? I would like to get some pointers of troubleshooting this.

       

       

         <address-setting match="queue.event.outageQueue">

              <dead-letter-address>queue.event.outageQueue.deadLetterQueue</dead-letter-address>

               <max-delivery-attempts>10</max-delivery-attempts>

               <redelivery-delay>5000</redelivery-delay>

               <max-size-bytes>20971520</max-size-bytes>

               <page-size-bytes>10485760</page-size-bytes>

               <send-to-dla-on-no-route>true</send-to-dla-on-no-route>

               <address-full-policy>PAGE</address-full-policy>

        </address-setting>

       

       

      My configuartion for the Queue is

       

       

              try {

                  if (tc == null) {

                      if ( connectionParams.size() != 0 ) {

                          this.factoryOutage = HornetQClient

                                  .createClientSessionFactory(new TransportConfiguration(

                                          NettyConnectorFactory.class.getName(), connectionParams));

                      } else {

                          this.factoryOutage = HornetQClient

                          .createClientSessionFactory(new TransportConfiguration(

                                  NettyConnectorFactory.class.getName()));

                      }

                  } else {

                      this.factoryOutage = HornetQClient.createClientSessionFactory(tc);

       

                  }

              } catch (Exception e) {

                  log.error("Failed in creating factory", e);

              }

              // Create a queue

              try {

                  factoryOutage.setBlockOnDurableSend(false);

                  factoryOutage.setBlockOnNonDurableSend(false);

                  factoryOutage.setPreAcknowledge(true);

                  factoryOutage.setConsumerWindowSize(0); //No Buffering

                  this.coreSessionOutage = this.factoryOutage.createSession(true, true,20000);

       

                  this.producerSessionOutage = this.factoryOutage.createSession(true, true,20000);

                  log.info("Core Session Created " + this.coreSessionOutage.toString());

       

              } catch (HornetQException e) {

                  log.error("Error in Creating a Session from FactoryOutage", e);

              }

       

              // Create a Session for Consumer.

              SimpleString sQueueName = SimpleString.toSimpleString(outageQueueName);

       

              try {

                  ClientSession.QueueQuery qq = coreSessionOutage

                          .queueQuery(sQueueName);

                  if (!qq.isExists()) {

                      this.coreSessionOutage.createQueue(sQueueName, sQueueName, true);

                  }

                  log.info("Outage Queue Created or Exists");

              } catch (HornetQException e) {

                  log.error("Error in Creating Queue", e);

              }

       

              // Create a Consumer

              try {

       

                  this.clientConsumerOutage = coreSessionOutage.createConsumer(sQueueName);

       

                  // log.info("Consumer created "+ clientConsumer.toString());

              } catch (HornetQException e) {

                  log.error("Error in Creating a Outage Consumer", e);

              }

              if (this.clientConsumerOutage != null) {

                  try {

                      this.clientConsumerOutage.setMessageHandler(new OutageEventConsumer());

       

                  } catch (HornetQException e) {

                      log.error("Error in Creating a Message Handler for Outage Consumer", e);

                  }

              }

       

              // start the consumer session

              try {

                  this.coreSessionOutage.start();

              } catch (HornetQException e) {

                  log.error("Error in Starting Outage Core Session", e);

              }

       

       

      And the way i send messages using producer is

       

       

       


      SimpleString uid = SimpleString.toSimpleString(evnObj);

      ClientMessage clientM = clientS.createMessage(false);

      //clientM.putStringProperty(ClientMessage.HDR_DUPLICATE_DETECTION_ID, uid);

      //clientM.putLongProperty(ClientMessage.HDR_SCHEDULED_DELIVERY_TIME, System.currentTimeMillis() + outageSuppressionTime );

      // Convert or Serialize Event Object

      ByteArrayOutputStream out = new ByteArrayOutputStream();

      ObjectOutputStream objOut = null;

      try {

      objOut = new ObjectOutputStream(out);

      objOut.writeObject(evnObj);

      } catch (IOException e1) {

      log.error("Error during Serilizing the Event Object", e1);

      stop();

      }

      byte[] bytes = out.toByteArray();

      log.info("EventObject byte size = " + bytes.length);

      //log.info("Event Object which is send = " + evnObj.toString());

      clientM.putBytesProperty("eventObject", bytes);

      //Send the Message

      try {

      clientP.send(clientM);

      log.info("Outage Message send done");

      } catch (HornetQException e) {

      log.error("Error in sending Message from Producer : MSG = "

      + clientM.toString(), e);

      } finally {

      stop();

      }

       

      Is there any more configuration that i need for paging to work ? Would be great if some one can give me pointers.

        • 1. Re: Paging doesnt seem to be working
          clebert.suconic

          Paging will only kick in after the max size. I'm not sure what you're doing?

          • 2. Re: Paging doesnt seem to be working
            into_java

            Hi Celbert,

            Sorry if i didnt mention details,

             

            I am using my queue for two things to delay each element by 30 seconds(suppression time)

            and Duplicate detection that is why i have these two properties set per message 



            clientM.putLongProperty(ClientMessage.HDR_SCHEDULED_DELIVERY_TIME, System.currentTimeMillis() + outageSuppressionTime );

            clientM.putStringProperty(ClientMessage.HDR_DUPLICATE_DETECTION_ID, uid);

             

            What i am seeing is that when the queue gets full at 20 MB  then a .page file is created each of 10 mb each.

            then i observed that the queue consumer catched up with the queue and consumes the message in queue but on comsumer side i never see all the messages.

             

            what i have seen is everything in .page file doesnt get in to the queue.

             

            the above is the implementation details. Let me know if you get any pointers to what this is about.

            • 3. Re: Paging doesnt seem to be working
              clebert.suconic

              ah... 2.1.2... that's old... Paging was completely changed on the later releases.... It's impossible to me to maintain it...

               

              You should move to a newer version. Paging was totally remodelled.. we page at queue level now.

              • 4. Re: Paging doesnt seem to be working
                clebert.suconic

                But from what you describe it's behaving as expected to 2.1.. that is the message will only enter the queue when all the queues have consumed.

                 

                at 2.2. it's different..

                • 5. Re: Paging doesnt seem to be working
                  into_java

                  Hi Clebert,

                   

                  This is what i have found till now.

                  The property

                  clientM.putStringProperty(ClientMessage.HDR_DUPLICATE_DETECTION_ID, uid);

                  doesnt work correctly on the paged items in the .paged files and rejects them during depaging.

                   

                  Is there anything like this issue which you are aware of was present in 2.1.2..

                   

                  Thanks

                  • 6. Re: Paging doesnt seem to be working
                    clebert.suconic

                    As I said there were so many fixes after 2.1.2... Paging was remodeled... etc... It's inpractical to fix it at 2.1.2... you should move to 2.2.latest ASAP.

                     

                     

                    If I start fixing 2.1.2 to you.. you will get to the point that your 2.1.2 will look like 2.2

                     

                     

                    Why you want to use 2.1.2?

                    • 7. Re: Paging doesnt seem to be working
                      into_java

                      Hey Clebert,

                      Thannks for the reply and motivation to move ...but my manager wont approve of me to change underlying mechanism with the project about to be shipped in 2 weeks.

                      So here is the update i finally have seemed to trace what is happening and why the page files are not getting rejected.

                       

                      when i looked at the trace logs i found this

                       

                      532667: nms-server6: Sep 08 2012 01:22:48.277 +0000: %CGMS-4-UNSPECIFIED: %[ch=PostOfficeImpl][sev=WARN][tid=Thread-24 (group:HornetQ-server-threads1136719061-1415215788)]: Duplicate message detected - transaction will be rejected

                      532893: nms-server6: Sep 08 2012 01:22:48.368 +0000: %CGMS-4-UNSPECIFIED: %[ch=PostOfficeImpl][sev=WARN][tid=Thread-24 (group:HornetQ-server-threads1136719061-1415215788)]: Duplicate message detected - transaction will be rejected

                       

                      I had two page file and both didnt get processed.

                       

                      So what has happened is that the entire page transaction was rejected because in the PostOfficeImpl class it found an duplicate ID.

                       

                      Now the interesting thing is that i am using UUID for generating my uid so there is actually no duplicate which are getting generated time and again in my multiple runs.

                       

                      This is my theory

                       

                      The cache we maintain for duplicate detection is populated with ID before we page a message.

                       

                      After depage we check the cache if there is a message with same ID present and it finds it in that cache since we populated it with that ID before page. This cause the whole transaction to be cancelled and the above log is printed.

                       

                      Do you think this is a right anaylsis .

                       

                      the reason i am asking is i want to be sure that duplicate detection isnt working correctly with paging and its not something wrong that i am putting in config.

                       

                      Once i come to this conculsion and if we agree then i can take off in designing things around this.

                       

                      Let me know

                      Thanks

                      • 8. Re: Paging doesnt seem to be working
                        clebert.suconic

                        You are going to start a new system in 2 weeks with HornetQ 2.1.2?

                         

                        If you're manager is letting you use a version that was done about 2.5 years ago? (2.1.2 was release Aug-18 2010 - post edited the date due to a typo). I have spent since then fixing paging and lot of other things and you're now coming back to me and saying fix everything you've done in the past two years ago because your manager doesn't want you to use the proper version?

                         

                         

                        I'm sorry... I don't want to be rude.. but there's no way that this could be accomplish.. it's not only that I don't want to... it's also:

                         

                        - I don't have time

                        - I don't have money

                        - I don't have resources

                         

                        And mainly.. it makes no sense...

                         

                         

                         

                        If you came to me saying.. I have this system in production now that I need to find a workaround while we move.. that would be something...

                         

                         

                        Now a new system.. in two weeks? (and mainly.. on a friday night?????)

                         

                         

                         

                        You should really stop there and review what you're doing... 

                         

                        If you find issues with 2.2.latest... I will be glad to help you.. but on 2.1.2.. there's no way for me to help you.

                         

                         

                        You could of course download the git repository and look all the changes on paging and move any changes you like... there's no way I can help you there. I'm really sorry!

                        • 9. Re: Paging doesnt seem to be working
                          clebert.suconic

                          This is my theory

                           

                          The cache we maintain for duplicate detection is populated with ID before we page a message.

                          and to answer your question... 2.1.2 we used to page before the message got to the queue. with 2.2. that's different.. but I already said that.. it's a complete refactoring.. for that reason I can't help you on 2.1.2.

                          • 10. Re: Paging doesnt seem to be working
                            into_java

                            I think you higly misunderstood my post.

                             

                            I am not asking you to make any changes, i am not asking you do to do anything for me other than answer my questions..thats it.

                             

                            Yes its friday night and the only reason i am in office is because of this paging issue which i see...and i cant move along as i was looking for validation(but now i will)

                             

                            Also let me explain whats going to happen in 2 weeks. We are going to upgrade a production image of our system in 2 weeks and its less of a time for us to change to 2.1.2 now.

                             

                            Yes for the next release we will move to newer release.

                            • 11. Re: Paging doesnt seem to be working
                              clebert.suconic

                              Ok, apologize me then

                               

                              These are two main things that changed after 2.1.2: paging and duplication detection. On 2.1 the message would just be ignored while on 2.2 it fails a tx.

                               

                              Maybe you could make a little change to where the duplicate cache is called. But any change now is dangerous. We had to make this change for conceptual (and user requests) reasons. But this may be a semantic change for you.

                               

                              If you need to stick with this version you should probably do your own duplicate detection if possible? Or maybe just avoid paging on 2.1.2.

                               

                              Avoiding paging on 2.1.2 may be a good choice? Since paging at 2.1.2 is a producer's cache. But Im not sure the duplicate cache works. If you really need it it's probably an easy change to ignore duplicates as you depage. Take a look on where the messages are read and sent to the producer. (or queque direct... Is old você And i domt relembre it)

                               

                              Also there was a bug in the journal at 2.1.2. Maybe you should disable compacting? Since the bug was associated with compacting and high load.

                               

                              Mainky: move out of 2.1.2 and if I was you u would negotiate more time to go with 2.2.  We are not maintaining 2.1 any longer.

                              • 12. Re: Paging doesnt seem to be working
                                clebert.suconic

                                Typo from my previous post; você=code. I had my iPad into Portuguese and it auto corrected me.

                                 

                                 

                                I'm going to sleep now but I will read this thread tomorrow.

                                • 13. Re: Paging doesnt seem to be working
                                  clebert.suconic

                                  If you promise to move out of 2.1.2 I may tell you where to fix the code for depage on Monday.

                                  • 14. Re: Paging doesnt seem to be working
                                    into_java

                                    Sure the next set of question you would see from me would be on newer versions of hornetq...

                                     

                                    Let me know on monday and i can try it out.

                                     

                                    Thanks

                                    1 2 Previous Next