9 Replies Latest reply on Sep 26, 2012 9:26 PM by clebert.suconic

    HornetQ read or export journal files and archive them

    hitjain_83

      I am working on the hornetQ ESB for an Integration project but then unable to find process to export the jms messages recevied onto clear text.

      I would also be interested in knowing the process to delete/archive the journal files without missing any message.

       

      Env:

      Windows 7

      JBoss SOA 5.3 GA

      JBoss ESB 4.11

      HornetQ Server version 2.2.10.GA

       

      When running the export using the process provided in the hornetQ documentation

                                   java -cp hornetq-core.jar org.hornetq.core.journal.impl.ExportJournal <JournalDirectory> <JournalPrefix> <FileExtension> <FileSize> <FileOutput>

       

      I get the response as below

       

      #File,JournalFileImpl: (hornetq-data-1.hq id = 1, recordID = 1)

      operation@AddRecord,id@231,userRecordType@31,length@226,isUpdate@false,compactCount@0,data@AAAAiQAAAHSs7QAFdABtPHVzZXIgdmVyc2lvbj0iMSI-PHVzZXJJZD5oamFpbjE8L3VzZXJJZD48dXNlck5hbWU-SGl0ZXNoPC91c2VyTmFtZT48dXNlclBhc3N3b3JkPjEyMzQ8L3VzZXJQYXNzd29yZD48L3VzZXI-MAAAAO8AAAAAAAAA5wEAAAAwagBtAHMALgBxAHUAZQB1AGUALgBkAGUAcgBlAGcAUgBlAHEAdQBlAHMAdABHAFcAASDgtFX3ihHhi1kAIWqo29oC_wAAAAAAAAAAAAABOZfCFUkEAA==

      operation@Update,id@231,userRecordType@32,length@8,isUpdate@true,compactCount@0,data@AAAAAAAAAAA=

      operation@AddRecord,id@232,userRecordType@31,length@372,isUpdate@false,compactCount@0,data@AAAAigAAAHWs7QAFdABuPHVzZXIgdmVyc2lvbj0iMSI-PHVzZXJJZD5oamFpbjE8L3VzZXJJZD48dXNlck5hbWU-SGl0ZXNoPC91c2VyTmFtZT48dXNlclBhc3N3b3JkPjEyMzQ8L3VzZXJQYXNzd29yZD48L3VzZXI-MDEAAAGBAAAAAAAAAOgBAAAAMGoAbQBzAC4AcQB1AGUAdQBlAC4AZABlAHIAZQBnAFIAZQBxAHUAZQBzAHQARwBXAAEg7E0W94oR4YtZACFqqNvaAv8AAAAAAAAAAAAAATmXwhWVBAEAAAABAAAAfF8ASABRAF8AUgBPAFUAVABFAF8AVABPAHMAZgAuAG0AeQAtAGMAbAB1AHMAdABlAHIALgBkADMAMwBjADkAYgAyAGMALQBmADcAOAA5AC0AMQAxAGUAMQAtAGIAMgA4ADYALQA1ADUAMABjAGUANAA1AGUAOAAxADEANQAEAAAACAAAAAAAAAAA

      operation@Update,id@232,userRecordType@32,length@8,isUpdate@true,compactCount@0,data@AAAAAAAAAEo=

      operation@Update,id@231,userRecordType@33,length@8,isUpdate@true,compactCount@0,data@AAAAAAAAAAA=

      operation@DeleteRecord,id@231

      operation@Update,id@232,userRecordType@33,length@8,isUpdate@true,compactCount@0,data@AAAAAAAAAEo=

      operation@DeleteRecord,id@232

      operation@AddRecord,id@233,userRecordType@31,length@228,isUpdate@false,compactCount@0,data@AAAAiwAAAHas7QAFdABvPHVzZXIgdmVyc2lvbj0iMSI-PHVzZXJJZD5oamFpbjE8L3VzZXJJZD48dXNlck5hbWU-SGl0ZXNoPC91c2VyTmFtZT48dXNlclBhc3N3b3JkPjEyMzQ8L3VzZXJQYXNzd29yZD48L3VzZXI-MDEyAAAA8QAAAAAAAADpAQAAADBqAG0AcwAuAHEAdQBlAHUAZQAuAGQAZQByAGUAZwBSAGUAcQB1AGUAcwB0AEcAVwABIQay5_eKEeGLWQAhaqjb2gL_AAAAAAAAAAAAAAE5l8IWQgQA

      operation@Update,id@233,userRecordType@32,length@8,isUpdate@true,compactCount@0,data@AAAAAAAAAAA=

      operation@Update,id@233,userRecordType@33,length@8,isUpdate@true,compactCount@0,data@AAAAAAAAAAA=

      operation@DeleteRecord,id@233

       

       

      This really doesn't help me understand what has happened to the request and also I am unable to decode the actual message that was transmitted across via the JMS provider.

      As per the HornetQ documentation it says you would be able to get the message in clear text but I don't see the same happening. Has anyone got to the point of getting a clear text message.

       

      My purpose of the above is for monitoring the # of messages that came in via the hornetQ server and really got processed and are pending processing. I know HornetQ takes care of reprocessing the pending messages but I want to have a little more flexibility to drop the request message on the queue for reprocessing, in case the biz decides to reprocess it.

       

      Also I read a lot of issues related to archiving the old files - would we know how this works? Since I see cross-references among the journal files which means we really cannot delete the files AS-IS.

       

      Any inputs would help...

        • 1. Re: HornetQ read or export journal files and archive them
          clebert.suconic

          Take a look on the Print-Data, that prints the journal record by record.. (including adds, deletes).. and at the end it shows a summary of what was left on the journal.

           

           

          I have this sh on my path to help calling it:

           

           

          #java -cp /somewhere/hornetq-logging.jar:/somewhere/hornetq-core.jar:/somewhere/netty.jar org.hornetq.core.persistence.impl.journal.PrintData $1 $2

           

          java -Djava.util.logging.config.file=/home/clebert/logging.properties -Xmx2G -cp /work/hornetq/hornetq-2.2/build/jars/hornetq-logging.jar:/work/hornetq/hornetq-2.2/build/jars/hornetq-core.jar:/work/hornetq/hornetq-2.2/thirdparty/org/jboss/netty/lib/netty.jar org.hornetq.core.persistence.impl.journal.PrintData $1 $2

          • 2. Re: HornetQ read or export journal files and archive them
            clebert.suconic

            There's also the XML Data Exporter and XML Data Importer:

             

             

            org.hornetq.core.persistence.impl.journal.XMLDataExporter

            org.hornetq.core.persistence.impl.journal.XMLDataImporter

             

             

             

            They export the journal in more messaging XML terms instead of low level journal records.

            • 3. Re: HornetQ read or export journal files and archive them
              clebert.suconic

              I'm moving this to the user's forum. I almost missed this message for being on the wrong forum.

              • 4. Re: HornetQ read or export journal files and archive them
                hitjain_83

                Hi Clebert,

                 

                Appreciate your quick response.

                I tried the PrintData, XMLDataExporter but for me it looks similar to whatso ever exportJournal is doing it, don't really find a big difference in it.

                 

                I see the same set of operations like "AddRecord,DeleteRecord,Commit,Queue info and others". I am rather trying to see if I can look at the actual message.

                 

                Just to clarify my main purpose of looking at these messages is to look for any malicious message that isn't desired and later block the originator of the message based on the stats pulled out.

                I know I can print the message at the consumer end but then did not want to add another hop for it to store the msg in a file or a DB instead thought of using the persistence provided by hornetQ.

                 

                Attached the response of the PrintData, XMLDataExporter just for future references.

                • 5. Re: HornetQ read or export journal files and archive them
                  clebert.suconic

                  Look at the XML DataExporter when you have Messages. It's probably what you're looking for.

                   

                  You didn't see any messages because it's empty.

                   

                   

                  A Database wouldn't help you on any message systems, because the DB is so cripted that it's as much as usual as the journal. We try to add tooling around it.

                   

                   

                  A message system is not intended as a db, you need to stop the whole system if you intend to remove data.

                   

                  A message system is not even supposed to persist anything.. as long as it can guarantee the message won't be lost. That's the concept of durable... durable meaning.. it will survive a restart!

                  • 6. Re: HornetQ read or export journal files and archive them
                    jbertram

                    Even if he had messages the body of the message would be Base64 encoded.  He wouldn't be able to just browse through it and read clear text which I tihnk is what he is after.

                    • 7. Re: HornetQ read or export journal files and archive them
                      clebert.suconic

                      why don't you use message properties?

                      • 8. Re: HornetQ read or export journal files and archive them
                        hitjain_83

                        @Justin: Yes this is what I was looking for. When something is encoded I am sure this can obviously be decoded into clear text.

                         

                        @Clebert: What do I do here with the message properties? Is there something you can elaborate.

                        • 9. Re: HornetQ read or export journal files and archive them
                          clebert.suconic

                          You could add properties to your messages as they are exposed better.

                           

                           

                          One thing also could be you change the XML Exporter to generate Text on Text Messages and have some XML differentiation. If you do a nice job on the PR you could submit the change upstream.