4 Replies Latest reply on Mar 18, 2009 2:29 PM by dave.stubbs

    Fuse MQ ceases processing when 256th Journal created

    dave.stubbs

      I've been running soak tests for our application over the weekend, putting ~4 million messages through but about two thirds of the way through all processing from the client and server sides of our application stopped.

       

      Stopping and restarting all applications including Fuse MQ did not fix the situation.

       

      The client had read every message from the queue, so the problem was only with the writes.

       

      When I checked the data folder I found that the last created journal was data-256. Only around a dozen journal files remained though.

       

      As the client had processed all the messages I decided to take the step to delete all the contents of the data folder and restart MQ.

       

      This worked, and processing started normally again.

       

      The application we are using is very high volume and mission critical, so we need an aswer to this quite quickly as it may prevent the planned roll out.

       

      I was using the current download from the website (5.3)

       

      Edited by: dave.stubbs on Mar 9, 2009 10:55 AM

        • 1. Re: Fuse MQ ceases processing when 256th Journal created
          stlewis

          Hey Dave,

           

          Just doing a test here where I've set the journal size to 8192 bytes to force the condition to happen much sooner for me.  The test is still running, however I've got 1550 journal files, the highest named file is data-7512.  So it doesn't look like it's the number of journal files.  Is there any chance you're running low on disk space?  Maybe it's worth trying a test where you increase the journal file size to see if that helps things?

          • 2. Re: Fuse MQ ceases processing when 256th Journal created
            dave.stubbs

            I'm running a second test now, different data but this time it doesn't appear to stop at the 256th file.

             

            Trouble is over the last year of building and testing aps with Fuse I've had a number of these weird and unexplained and often unrepeatable scenarios.

             

            Earlier this year I had issues with out of memory errors from the JVM, then I had issues with messages going missing, now this. I'm concerned about its stability.

            • 3. Re: Fuse MQ ceases processing when 256th Journal created
              stlewis

              I know there were some deadlock issues in the broker related to the persistent store that were resolved awhile ago, but it's possible there's still one edge condition that's yet to be discovered.  If you do run into this again the best thing to do to help diagnose the problem is to get a thread dump, if you're on UNIX do a kill -3 on the JVM or in Windows you'd do Ctrl-Break and capture the output.  A thread dump would help us determine if it's indeed a deadlock and give an idea of where it's happening.

              • 4. Re: Fuse MQ ceases processing when 256th Journal created
                dave.stubbs

                Unfortunately despite many attempts I've not been able to create this again. Figures.

                 

                I've had plenty of other one off scenarios I can only create once. I must start backing up the logs.