With HornetQ 2.4 seen error - HQ224000: Failure in initialisation: java.nio.BufferUnderflowException and HornetQ server not started

[veenaonnet]

Hi,

I have multiple journal directories created along with paging and binding directories.

On starting my JBoss server, I check max_replicated_journal_size and accordingly delete some old journal directories(based on the modification time). This is done for multiple failover/failback to go through. Not deleting old binding/paging directories though.

This time, I saw some journal directories with only server.lock files inside them. While starting HornetQ server, have seen the below exception and HornetQ server never came up.

From the logs, it seems that the error came while loading binding journals.

What could be the reason for these errors?

2014-04-06 21:36:43,483 ERROR [org.hornetq.core.server] (MSC service thread 1-13) HQ224000: Failure in initialisation: java.nio.BufferUnderflowException

at java.nio.Buffer.nextGetIndex(Buffer.java:498) [rt.jar:1.7.0_45]

at java.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:355) [rt.jar:1.7.0_45]

at org.hornetq.core.journal.impl.JournalImpl.readFileHeader(JournalImpl.java:2666) [hornetq-journal-2.4.0-SNAPSHOT.jar:]

at org.hornetq.core.journal.impl.JournalImpl.orderFiles(JournalImpl.java:384) [hornetq-journal-2.4.0-SNAPSHOT.jar:]

at org.hornetq.core.journal.impl.JournalImpl.load(JournalImpl.java:1728) [hornetq-journal-2.4.0-SNAPSHOT.jar:]

at org.hornetq.core.journal.impl.JournalImpl.load(JournalImpl.java:1322) [hornetq-journal-2.4.0-SNAPSHOT.jar:]

at org.hornetq.core.journal.impl.JournalImpl.load(JournalImpl.java:1306) [hornetq-journal-2.4.0-SNAPSHOT.jar:]

at org.hornetq.core.persistence.impl.journal.JournalStorageManager.loadBindingJournal(JournalStorageManager.java:2173) [hornetq-server-2.4.0-SNAPSHOT.jar:]

at org.hornetq.core.server.impl.HornetQServerImpl.loadJournals(HornetQServerImpl.java:1735) [hornetq-server-2.4.0-SNAPSHOT.jar:]

at org.hornetq.core.server.impl.HornetQServerImpl.initialisePart2(HornetQServerImpl.java:1624) [hornetq-server-2.4.0-SNAPSHOT.jar:]

at org.hornetq.core.server.impl.HornetQServerImpl.access$1400(HornetQServerImpl.java:172) [hornetq-server-2.4.0-SNAPSHOT.jar:]

at org.hornetq.core.server.impl.HornetQServerImpl$SharedNothingLiveActivation.run(HornetQServerImpl.java:2680) [hornetq-server-2.4.0-SNAPSHOT.jar:]

at org.hornetq.core.server.impl.HornetQServerImpl.start(HornetQServerImpl.java:432) [hornetq-server-2.4.0-SNAPSHOT.jar:]

at org.hornetq.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:483) [hornetq-jms-server-2.4.0-SNAPSHOT.jar:]

at org.jboss.as.messaging.jms.JMSService.start(JMSService.java:74) [jboss-as-messaging-7.2.0.Final.jar:7.2.0.Final]

at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.4.GA.jar:1.0.4.GA]

at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.4.GA.jar:1.0.4.GA]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45]

at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]

Regards,

Veena

[ataylor]

could you please start using the correct forum, ive moved your last couple of posts. This is the developer forum for all things HQ dev

[jbertram]

The exception indicates that the buffer doesn't have the expected number of bytes so the read operation fails.

Are you able to reproduce the failure?

Have you ever seen this failure when you weren't deleting journal directories?

[veenaonnet]

My apologies for the trouble caused Andy. I will take care of it next time onwards.

Regards,

Veena

[veenaonnet]

Hi Justin,

I did not see this error ever in the past, not even after deleting the journals. This is the only instance.

I will try to reproduce it.

But in general, is it right to delete old journal backup files(based on the modification time)? or should the whole directory be deleted when it reaches max_replicated_journal_size?

Do you think that binding and paging files should also be deleted along with the journal files?

After I started deleting these journal backup files, sometimes I saw  errors on starting HA node where JMS server didn't start and it cached commands to create queues/connectionfactory but never created those. Though not sure if deleting journal files caused it. Any thoughts?

Regards,

Veena

[jbertram]

Deleting old journals that you don't need anymore is fine.  Whether or not you use "modification time" to determine that is up to you.

In general, it's hard to comment on errors that you see only occasionally and which aren't directly connected to a specific use-case or are not accompanied with logs, etc.  A good error report includes all the information required to reproduce the error or at least a detailed description of the environment and use-case when the error occurred.  As far as I can tell, you haven't provided any of that.

[clebert.suconic]

I believe you have damaged the sequence somehow that the server could start.

[veenaonnet]

Yes Clebert. I also believe the same.

Will test it further and try to reproduce the issue.

Regards,

Veena