9 Replies Latest reply on Jun 10, 2011 11:04 PM by clebert.suconic

    I've drunk the Kool Aid, but is there a tool for examining journal files?

    kennardconsulting

      Hi guys,

       

      Okay I've drunk the Kool-Aid. I get it that relational databases are not the best storage medium for JMS, that using the file system directly is a win, that HornetQ is much faster because of it etc etc.

       

      However a really nice feature of the old relational database approach is that I could point an SQL tool at the tables and see what was in them. At the moment I'm trying to debug a problem where messages are not getting processed. Is the problem they are not being produced to the queue? Is the problem they are not being consumed from the queue? To debug this I'd really like to be able to examine the queue itself.

       

      Are there any tools for examining HornetQ journal files? Any plans for one? Such a thing would be really useful to me!

       

      Regards,

       

      Richard.

        • 1. Re: I've drunk the Kool Aid, but is there a tool for examining journal files?
          clebert.suconic

          I have two .sh defined on my $HOME/bin:

           

           

          print-data.sh

          java -cp /work/projects/trunk/build/jars/hornetq-core.jar:/work/projects/trunk/thirdparty/org/jboss/netty/lib/netty.jar org.hornetq.core.persistence.impl.journal.PrintData $1 $2

           

          print-pages.sh

          java -cp /work/projects/trunk/build/jars/hornetq-core.jar:/work/projects/trunk/thirdparty/org/jboss/netty/lib/netty.jar  org.hornetq.core.paging.PrintPages $1 $2

           

           

           

          print-data.sh will show the records for bindings and journal

           

           

          each journal will show the records by file, and at the end you will see the surviving records...

           

           

          Look at JournalStorageManager for the userRecordTypes... and that will give you an idea about what's on the journal.

           

           

          We are also have the export tool exlained on the manual, and we are planning (already writing) another tool that will export in a message centric XML.

          • 2. Re: I've drunk the Kool Aid, but is there a tool for examining journal files?
            clebert.suconic

            print-pages will show the records that you have in case of paging.

             

             

            Let me know if you need help understanding them.

            • 3. Re: I've drunk the Kool Aid, but is there a tool for examining journal files?
              clebert.suconic

              BTW: Those files are not really documented, because they were tools I wrote to help me debug issues.. debug data... and I committed those.

               

              But I plan to replace those by a real export tool. We have already a export tool at journal level, but we are writing one at the message level...

               

               

              Something like

               

              <Queues>

                <queue id>

              <messages>

                  <message .... >

                     <queueBound/>

                  </message>

              </messages>

               

               

              Something like this...

               

               

              We have a branch where this is being written now:

               

              http://anonsvn.jboss.org/repos/hornetq/branches/Branch_2_2_EAP_export_tool/

              • 4. Re: I've drunk the Kool Aid, but is there a tool for examining journal files?
                kennardconsulting

                Clebert,

                 

                Thanks so much for replying!

                 

                Could you please explain a bit what the output of PrintData is? I am trying to understand why my messages are not being processed. I'm pretty sure they're being *sent* to the queue, but I don't know whether they are arriving on the queue and/or being consumed off it.

                 

                Here's what PrintData produces: (attached)

                • 5. Re: I've drunk the Kool Aid, but is there a tool for examining journal files?
                  clebert.suconic

                  You should probably have used attachments on the forums... :-) (Use advanced editor, and you would see it)

                  My browser was really slow on moving around the pages, so I edited it and made the attachment since I'm an admin on this page.

                   

                   

                  For every journal you have, you will see two parts, File by File:

                   

                  #JournalFileImpl: (hornetq-bindings-1.bindings id = 1, recordID = 1)

                  operation@AddRecord;recordID=2;userRecordType=24;isUpdate=false;IDCounterEncoding [id=2147483647]

                  operation@AddRecord;recordID=0;userRecordType=21;isUpdate=false;PersistentQueueBindingEncoding [id=0,

                   

                   

                  And what survived:

                  ### Surviving Records Summary ###

                  recordID=2;userRecordType=24;isUpdate=false;IDCounterEncoding [id=2147483647]

                  recordID=0;userRecordType=21;isUpdate=false;PersistentQueueBindingEncoding [id=0, name=jms.queue.AvantLongRunningProcesses, address=jms.queue.AvantLongRunningProcesses, filterString=null]

                  recordID=3;userRecordType=21;isUpdate=false;PersistentQueueBindingEncoding [id=3, name=jms.queue.AvantEmails, address=jms.queue.AvantEmails, filterString=null]

                   

                   

                   

                  If you had a record being deleted, it would appear on the first report as long as you didn't have the file reclaimed or compacted.

                   

                   

                   

                  So, you will see this both to the bindings and message journal.

                   

                   

                   

                  on the bindings journal, you will have the definition for the core-queues, and for the counters.

                   

                   

                  You see here that the recordID=0 is the queue jms.queue.AvantEmails

                   

                   

                   

                   

                   

                  When you look at the message journal, you will messages being added and routed:

                   

                  #JournalFileImpl: (hornetq-data-1.hq id = 1, recordID = 1)

                   

                  # This means the message was added

                   

                  operation@AddRecordTX;txID=28,recordID=29;userRecordType=31;isUpdate=false;Message(messageID=29;properties=[#properties = 0] - ServerMessage[messageID=29, durable=true, address=jms.queue.AvantEmails,properties=TypedProperties[null]]

                   

                  # This means the message was routed to queueID=3

                  operation@UpdateTX;txID=28,recordID=29;userRecordType=32;isUpdate=true;AddRef;QueueEncoding [queueID=3]

                   

                  #This means a prepare TX, the extraData here is a encoding of the XID:

                  operation@Prepare,txID=28,numberOfRecords=2,extraData=AAIABAAAABwAAAAAAAAAAAAA___AqAEBAAASak3xo6wAAAArAAAAHQAAAAAAAAAAAAD__8CoAQEAABJqTfGjrAAAAB4x

                   

                  #Commit

                  operation@Commit;txID=28,numberOfRecords=2

                   

                  #This means, an ACK inside a commit

                  operation@UpdateTX;txID=30,recordID=29;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=3]

                   

                  # You committed it without XA (probably you didn't need XA here)

                  operation@Commit;txID=30,numberOfRecords=1

                   

                   

                  #The delete happens in background when you ack the message on every queue: The message will still exist until you ACK every route (subscription or core-queue):

                  operation@DeleteRecord;recordID=29

                   

                   

                  If you look at the surviving recods... you will see what didn't get deleted at all.

                   

                  On your case. you have a bunch of messages added to queue 0, and failed transaction (991)  that will be rolled back. You won't be able to receive these messages since you stopped the server without committing it.

                   

                  ### Prepared TX ###

                  ### Failed Transactions (Missing commit/prepare/rollback record) ###

                  Transaction 991 failed with these records:

                  - recordID=992;userRecordType=31;isUpdate=false;Message(messageID=992;properties=[longRunningProcessId=402881813077e7d80130782d7c920044,file=CU90342352011061022213.XML,fileNumber=0,row=1,longRunningProcessor=com.avant.ss.ejb.message.processor.MemberImportProcessor,dataDate=2011-06-09,plan=4028818227fedbfb0127feea87b6013b,memberImportType=snapshot,#properties = 8] - ServerMessage[messageID=992, durable=true, address=jms.queue.AvantLongRunningProcesses,properties=TypedProperties[{longRunningProcessId=402881813077e7d80130782d7c920044, file=CU90342352011061022213.XML, fileNumber=0, row=1, longRunningProcessor=com.avant.ss.ejb.message.processor.MemberImportProcessor, dataDate=2011-06-09, plan=4028818227fedbfb0127feea87b6013b, memberImportType=snapshot}]]

                  - recordID=992;userRecordType=32;isUpdate=true;AddRef;QueueEncoding [queueID=0]

                  • 6. Re: I've drunk the Kool Aid, but is there a tool for examining journal files?
                    clebert.suconic

                    Some extra information, look at the JournalStorageManager.java for the userRecordType ids. (each record type has an userRecordType)

                     

                    (HornetQ is an user of the journal)

                    • 7. Re: I've drunk the Kool Aid, but is there a tool for examining journal files?
                      kennardconsulting

                      Clebert,

                       

                      Thanks for your response. Unfortunately I have a few more questions. Would it be possible to catch you on #hornetq IRC?

                       

                      Richard.

                      • 8. Re: I've drunk the Kool Aid, but is there a tool for examining journal files?
                        kennardconsulting

                        Perhaps we can schedule a time for an interactive IRC conversation?

                        • 9. Re: I've drunk the Kool Aid, but is there a tool for examining journal files?
                          clebert.suconic

                          We are usually available on IRC regular business hours...

                           

                          Andy Taylor is on UK's Time, Francisco Borges Paris Time, Howard (ghoward) on China, and I'm in US.

                           

                           

                          At Friday nights I'm usually drunk or sleepy :-) Although I would be available now for a quick chat.

                           

                           

                          Lets see if we can talk monday US time...     or eventually earlier if you see me online (really online I mean. not just with the computer logged on).