1 2 3 4 Previous Next 57 Replies Latest reply on Jul 22, 2010 2:55 AM by azserve.luca Go to original post
      • 30. Re: IllegalStateException when loading journal on startup
        clebert.suconic

        @dfisher and BJ Chipanddale:

         

        Let me see if I understand:

         

        Where's the network issue you guys are seeing? Between live and backup.. or between the client and server?

         

        The error happens at the live node after the backup has been activated.. but you guys are actually using the live node after a restart?

        • 31. Re: IllegalStateException when loading journal on startup
          dfisher

          Our configuration does not use live-backup.

          We do have one core netty bridge configured.

          • 32. Re: IllegalStateException when loading journal on startup
            dfisher

            Can you share the journal data with me for this?

            No, it's contains private data.

            We are working on a test that we can share.

            • 33. Re: IllegalStateException when loading journal on startup
              dfisher

              According to the logs, scheduled messages are being used.

              I believe that is a result of the following config:

              <redelivery-delay>5000</redelivery-delay>

              <max-delivery-attempts>-1</max-delivery-attempts>

              We don't explicitly use scheduled messages, however we do set priority on each message.

               

              Can you provide a test program that replicates the case?

              Not yet, but we're working on it.

              • 34. Re: IllegalStateException when loading journal on startup
                johnnysoccer

                I would also like to chime in. I also see this error on a regular basis, though I haven't noticed a pattern as to what causes it. What I have noticed is that the files in my journal directory look something like this.

                 

                There were the journal files at the time of error:
                -rw-rw-r-- 1 jboss0 tomax  2354176 May 27 12:16 hornetq-data-493.hq
                -rw-rw-r-- 1 jboss0 tomax    80896 May 27 15:53 hornetq-data-504.hq
                -rw-rw-r-- 1 jboss0 tomax   148992 May 27 16:01 hornetq-data-509.hq
                -rw-rw-r-- 1 jboss0 tomax 10485760 May 27 16:42 hornetq-data-522.hq
                -rw-rw-r-- 1 jboss0 tomax 10485760 May 27 16:42 hornetq-data-520.hq
                -rw-rw-r-- 1 jboss0 tomax 10485760 May 27 17:00 hornetq-data-521.hq

                 

                Notice that the three files with odd sizing are also out of sequence with the last 3 files. I've assumed it has something to do with hornet not completely cleaning up after itself when being shut down, or at some point while it is up and running.  If I remove the 3 out-of-sync files, HornetQ is able to start back up as expected.

                 

                My configuration has HornetQ running standalone, non-clustered using libaio on local disk.

                 

                In case you want the error stack again, here it is.

                 

                [main] 20:13:36,173 SEVERE [org.hornetq.integration.bootstrap.HornetQBootstrapServer]  Failed to start server
                java.lang.IllegalStateException: Incompletely deployed:
                DEPLOYMENTS IN ERROR:
                  Deployment "JMSServerManager" is in error due to: java.lang.IllegalStateException: Cannot find queue messages 3465
                        at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.internalValidate(AbstractKernelDeployer.java:278)
                        at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.validate(AbstractKernelDeployer.java:174)
                        at org.hornetq.integration.bootstrap.HornetQBootstrapServer.bootstrap(HornetQBootstrapServer.java:159)
                        at org.jboss.kernel.plugins.bootstrap.AbstractBootstrap.run(AbstractBootstrap.java:83)
                        at org.hornetq.integration.bootstrap.HornetQBootstrapServer.run(HornetQBootstrapServer.java:117)
                        at org.hornetq.integration.bootstrap.HornetQBootstrapServer.main(HornetQBootstrapServer.java:73)
                Exception in thread "main" java.lang.IllegalStateException: Incompletely deployed:

                • 35. Re: IllegalStateException when loading journal on startup
                  clebert.suconic

                  What version are you using?

                  • 36. Re: IllegalStateException when loading journal on startup
                    clebert.suconic

                    I need to know if this would happen with 2.1.0.CR1

                    • 37. Re: IllegalStateException when loading journal on startup
                      johnnysoccer

                      I'm using 2.0.0.GA

                       

                      I can try and get 2.1.0.CR1 running in one of our environments, but since I haven't been able to find a pattern of behavior that will consistently cause the problem, I'm not sure how to test for it.

                       

                      Does 2.1.0.CR1 have changes that you would expect to stop this behavior?

                      • 38. Re: IllegalStateException when loading journal on startup
                        clebert.suconic

                        I was thinking about https://jira.jboss.org/browse/HORNETQ-237, but it was done before GA...

                         

                        On the journal itself, the only change made was:

                         

                         

                        https://jira.jboss.org/browse/HORNETQ-318

                        and

                        https://jira.jboss.org/browse/HORNETQ-387

                         

                         

                        But we also tweaked some of the dependencies on shutdown for the AS integration. it would worth a try if you can.

                        • 39. Re: IllegalStateException when loading journal on startup
                          clebert.suconic

                          John Muhlestein wrote:

                           

                          I would also like to chime in. I also see this error on a regular basis, though I haven't noticed a pattern as to what causes it. What I have noticed is that the files in my journal directory look something like this.

                           

                          There were the journal files at the time of error:
                          -rw-rw-r-- 1 jboss0 tomax  2354176 May 27 12:16 hornetq-data-493.hq
                          -rw-rw-r-- 1 jboss0 tomax    80896 May 27 15:53 hornetq-data-504.hq
                          -rw-rw-r-- 1 jboss0 tomax   148992 May 27 16:01 hornetq-data-509.hq
                          -rw-rw-r-- 1 jboss0 tomax 10485760 May 27 16:42 hornetq-data-522.hq
                          -rw-rw-r-- 1 jboss0 tomax 10485760 May 27 16:42 hornetq-data-520.hq
                          -rw-rw-r-- 1 jboss0 tomax 10485760 May 27 17:00 hornetq-data-521.hq

                           

                          this helped do some investigation now...

                           

                          https://jira.jboss.org/browse/HORNETQ-399

                           

                          there are two routines on the journal to cleanup dead records:

                           

                          - Compacting: It will read the entire journal, create new records.

                          - Cleanup: Say you have a journal file that has only a few records alive. That file will create a linked-list effect what would avoid files being reused. (you would have lots of files for just a few records on the journal).

                           

                           

                          the journal will choose between the two algorithms depending on min-compact settings.

                           

                           

                          and there was a bug on the cleanup. The files wasn't supposed for being reused. The file should have a smaller size than the 10MiB configured.. but it shouldn't be reused in case of being freed.

                           

                           

                          I have committed a fix on trunk. I will be committing a testcase over the weekend.

                          • 40. Re: IllegalStateException when loading journal on startup
                            dhawes

                            I work with dfsher and am helping him reproduce the IllegalStateException(s).

                             

                            I do notice 1 journal file with an odd size when the last IllegalStateException was encountered, but removing it results in the same exception on startup.

                             

                            Do you have reason to believe that the fix in trunk will fix this issue?

                             

                            We are still working on getting journal data that does not contain private data.

                            • 41. Re: IllegalStateException when loading journal on startup
                              clebert.suconic

                              Yes.. I have a strong reason to believe the fix on trunk will fix it.

                               

                              I could reproduce a similar issue but not quite the same yet.

                               

                               

                              BTW: The fix on trunk is valid, but I will change the fix. (Instead of ignoring files with a different size.. I will just avoid files with different sizes).

                              • 42. Re: IllegalStateException when loading journal on startup
                                dhawes

                                Clebert, I sent you a link to a corrupted journal that does not contain private/confidential data.

                                 

                                I hope it's useful with this issue.

                                • 43. Re: IllegalStateException when loading journal on startup
                                  clebert.suconic

                                  I'm pretty sure at this point that the issue is fixed.

                                   

                                  If you guys could confirm it by trying trunk in your tests.

                                  • 44. Re: IllegalStateException when loading journal on startup
                                    dhawes

                                    After testing for several days I have not seen the IllegalStateException(s).  If that changes, I'll be sure to follow up on this thread.