-
30. Re: IllegalStateException when loading journal on startup
clebert.suconic May 27, 2010 5:47 PM (in response to bjchip)@dfisher and BJ Chipanddale:
Let me see if I understand:
Where's the network issue you guys are seeing? Between live and backup.. or between the client and server?
The error happens at the live node after the backup has been activated.. but you guys are actually using the live node after a restart?
-
31. Re: IllegalStateException when loading journal on startup
dfisher May 27, 2010 5:55 PM (in response to clebert.suconic)Our configuration does not use live-backup.
We do have one core netty bridge configured.
-
32. Re: IllegalStateException when loading journal on startup
dfisher May 27, 2010 6:05 PM (in response to clebert.suconic)Can you share the journal data with me for this?
No, it's contains private data.
We are working on a test that we can share.
-
33. Re: IllegalStateException when loading journal on startup
dfisher May 27, 2010 6:03 PM (in response to timfox)According to the logs, scheduled messages are being used.
I believe that is a result of the following config:
<redelivery-delay>5000</redelivery-delay>
<max-delivery-attempts>-1</max-delivery-attempts>
We don't explicitly use scheduled messages, however we do set priority on each message.
Can you provide a test program that replicates the case?
Not yet, but we're working on it.
-
34. Re: IllegalStateException when loading journal on startup
johnnysoccer May 28, 2010 1:21 PM (in response to dfisher)I would also like to chime in. I also see this error on a regular basis, though I haven't noticed a pattern as to what causes it. What I have noticed is that the files in my journal directory look something like this.
There were the journal files at the time of error:
-rw-rw-r-- 1 jboss0 tomax 2354176 May 27 12:16 hornetq-data-493.hq
-rw-rw-r-- 1 jboss0 tomax 80896 May 27 15:53 hornetq-data-504.hq
-rw-rw-r-- 1 jboss0 tomax 148992 May 27 16:01 hornetq-data-509.hq
-rw-rw-r-- 1 jboss0 tomax 10485760 May 27 16:42 hornetq-data-522.hq
-rw-rw-r-- 1 jboss0 tomax 10485760 May 27 16:42 hornetq-data-520.hq
-rw-rw-r-- 1 jboss0 tomax 10485760 May 27 17:00 hornetq-data-521.hqNotice that the three files with odd sizing are also out of sequence with the last 3 files. I've assumed it has something to do with hornet not completely cleaning up after itself when being shut down, or at some point while it is up and running. If I remove the 3 out-of-sync files, HornetQ is able to start back up as expected.
My configuration has HornetQ running standalone, non-clustered using libaio on local disk.
In case you want the error stack again, here it is.
[main] 20:13:36,173 SEVERE [org.hornetq.integration.bootstrap.HornetQBootstrapServer] Failed to start server
java.lang.IllegalStateException: Incompletely deployed:
DEPLOYMENTS IN ERROR:
Deployment "JMSServerManager" is in error due to: java.lang.IllegalStateException: Cannot find queue messages 3465
at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.internalValidate(AbstractKernelDeployer.java:278)
at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.validate(AbstractKernelDeployer.java:174)
at org.hornetq.integration.bootstrap.HornetQBootstrapServer.bootstrap(HornetQBootstrapServer.java:159)
at org.jboss.kernel.plugins.bootstrap.AbstractBootstrap.run(AbstractBootstrap.java:83)
at org.hornetq.integration.bootstrap.HornetQBootstrapServer.run(HornetQBootstrapServer.java:117)
at org.hornetq.integration.bootstrap.HornetQBootstrapServer.main(HornetQBootstrapServer.java:73)
Exception in thread "main" java.lang.IllegalStateException: Incompletely deployed: -
35. Re: IllegalStateException when loading journal on startup
clebert.suconic May 28, 2010 3:39 PM (in response to johnnysoccer)What version are you using?
-
36. Re: IllegalStateException when loading journal on startup
clebert.suconic May 28, 2010 3:40 PM (in response to clebert.suconic)I need to know if this would happen with 2.1.0.CR1
-
37. Re: IllegalStateException when loading journal on startup
johnnysoccer May 28, 2010 3:55 PM (in response to clebert.suconic)I'm using 2.0.0.GA
I can try and get 2.1.0.CR1 running in one of our environments, but since I haven't been able to find a pattern of behavior that will consistently cause the problem, I'm not sure how to test for it.
Does 2.1.0.CR1 have changes that you would expect to stop this behavior?
-
38. Re: IllegalStateException when loading journal on startup
clebert.suconic May 28, 2010 4:55 PM (in response to johnnysoccer)I was thinking about https://jira.jboss.org/browse/HORNETQ-237, but it was done before GA...
On the journal itself, the only change made was:
https://jira.jboss.org/browse/HORNETQ-318
and
https://jira.jboss.org/browse/HORNETQ-387
But we also tweaked some of the dependencies on shutdown for the AS integration. it would worth a try if you can.
-
39. Re: IllegalStateException when loading journal on startup
clebert.suconic May 28, 2010 8:17 PM (in response to johnnysoccer)John Muhlestein wrote:
I would also like to chime in. I also see this error on a regular basis, though I haven't noticed a pattern as to what causes it. What I have noticed is that the files in my journal directory look something like this.
There were the journal files at the time of error:
-rw-rw-r-- 1 jboss0 tomax 2354176 May 27 12:16 hornetq-data-493.hq
-rw-rw-r-- 1 jboss0 tomax 80896 May 27 15:53 hornetq-data-504.hq
-rw-rw-r-- 1 jboss0 tomax 148992 May 27 16:01 hornetq-data-509.hq
-rw-rw-r-- 1 jboss0 tomax 10485760 May 27 16:42 hornetq-data-522.hq
-rw-rw-r-- 1 jboss0 tomax 10485760 May 27 16:42 hornetq-data-520.hq
-rw-rw-r-- 1 jboss0 tomax 10485760 May 27 17:00 hornetq-data-521.hqthis helped do some investigation now...
https://jira.jboss.org/browse/HORNETQ-399
there are two routines on the journal to cleanup dead records:
- Compacting: It will read the entire journal, create new records.
- Cleanup: Say you have a journal file that has only a few records alive. That file will create a linked-list effect what would avoid files being reused. (you would have lots of files for just a few records on the journal).
the journal will choose between the two algorithms depending on min-compact settings.
and there was a bug on the cleanup. The files wasn't supposed for being reused. The file should have a smaller size than the 10MiB configured.. but it shouldn't be reused in case of being freed.
I have committed a fix on trunk. I will be committing a testcase over the weekend.
-
40. Re: IllegalStateException when loading journal on startup
dhawes Jun 1, 2010 5:32 PM (in response to clebert.suconic)I work with dfsher and am helping him reproduce the IllegalStateException(s).
I do notice 1 journal file with an odd size when the last IllegalStateException was encountered, but removing it results in the same exception on startup.
Do you have reason to believe that the fix in trunk will fix this issue?
We are still working on getting journal data that does not contain private data.
-
41. Re: IllegalStateException when loading journal on startup
clebert.suconic Jun 1, 2010 9:12 PM (in response to dhawes)Yes.. I have a strong reason to believe the fix on trunk will fix it.
I could reproduce a similar issue but not quite the same yet.
BTW: The fix on trunk is valid, but I will change the fix. (Instead of ignoring files with a different size.. I will just avoid files with different sizes).
-
42. Re: IllegalStateException when loading journal on startup
dhawes Jun 2, 2010 11:40 AM (in response to clebert.suconic)Clebert, I sent you a link to a corrupted journal that does not contain private/confidential data.
I hope it's useful with this issue.
-
43. Re: IllegalStateException when loading journal on startup
clebert.suconic Jun 2, 2010 12:04 PM (in response to dhawes)I'm pretty sure at this point that the issue is fixed.
If you guys could confirm it by trying trunk in your tests.
-
44. Re: IllegalStateException when loading journal on startup
dhawes Jun 8, 2010 11:08 AM (in response to clebert.suconic)After testing for several days I have not seen the IllegalStateException(s). If that changes, I'll be sure to follow up on this thread.