9 Replies Latest reply on Sep 25, 2014 5:22 PM by jbertram

    hornetq2.5.5 messages lost after restart

    bwitherell

      Today we had an issue where connections to our hornetq instance was timing out.

      We restarted hornetq and ended up losing all the messages currently in the queues.

       

      I thought this bug was fixed in this release or prior so could someone please help me figure out how this could have happened?

      The data files in the journal directory are all still there so why were the queues empty right after we restarted?

        • 1. Re: hornetq2.5.5 messages lost after restart
          jbertram

          Can you clarify a few things?

          1. What release are you using?  HornetQ 2.5.5 hasn't been released yet.  Perhaps you meant 2.2.5?
          2. Were the messages persistent?
          • 2. Re: hornetq2.5.5 messages lost after restart
            bwitherell

            haha yes sorry about that. its 2.2.5. didnt realize i did that.

             

            yes the messages were persistent.

             

            Thanks for the quick response.

            • 3. Re: hornetq2.5.5 messages lost after restart
              jbertram

              I'm not familiar with any bugs that match the description of your symptoms.  Was there a particular JIRA that was supposed to be fixed in 2.2.5.Final related to this?

               

              Have you considered upgrading to a newer release?  HornetQ 2.2.5.Final was released over 3 years ago now.

              • 4. Re: hornetq2.5.5 messages lost after restart
                bwitherell

                it was well known that in earlier releases messeges didnt persist when hornetq was restarted. not sure on any specific jira ticket. but why would 2.2.5 do this? I didint see anything in the logs. the journal dir was huge 20gb though. is it possible it could have OOMed and somhow that caused the persistance issue i saw?

                • 5. Re: hornetq2.5.5 messages lost after restart
                  jbertram

                  it was well known that in earlier releases messeges didnt persist when hornetq was restarted. not sure on any specific jira ticket.

                  I have been either supporting or developing HornetQ for the last few years and I was not aware of this issue.  If you could be more specific that would be helpful.

                   

                  but why would 2.2.5 do this?

                  I have no idea.

                   

                  I didint see anything in the logs. the journal dir was huge 20gb though.

                  Is it still huge?  Perhaps you could run the PrintData tool on it and attach the output?

                   

                  is it possible it could have OOMed and somhow that caused the persistance issue i saw?

                  In my experience the journal is robust and wouldn't fail in the case of an OOME such that messages wouldn't be read upon a server restart.

                  • 6. Re: hornetq2.5.5 messages lost after restart
                    bwitherell

                    So I ran the print tool as described by the discussion you linked to.

                    It returned 8,000 or so lines. Unfortunately I am prohibited from sending you the data.

                    If you could just give me a few helpful hints on how to interpret this and what I should look for I would appreciate it.

                     

                    Thanks a lot for your continued help on this!

                    • 7. Re: hornetq2.5.5 messages lost after restart
                      jbertram

                      I didn't write the PrintData tool so I'm not familiar enough with its output to direct you blindly.  The main thing I'd recommend is simply looking at the results from the messaging journal, specifically the surviving records.  At the end of the day, however, I don't think there's much that can be done since you can't submit the data for analysis and possible correction.  Even if you could analyze the output from the tool it's unlikely you'd be able to resolve any issues you found.


                      Like I said, I'm not familiar with any bugs that would cause this kind of problem.  I certainly haven't heard of any enterprise customers running into this, but then again we started enterprise support with a build of 2.2.10.

                      • 8. Re: hornetq2.5.5 messages lost after restart
                        bwitherell

                        Thanks! Is surviving data the stuff that should be put back on the queue after restart?

                        • 9. Re: hornetq2.5.5 messages lost after restart
                          jbertram

                          I believe so, but again I'm not terribly familiar with the tool.