KahaDB recovery problem after out of free disk space

[ciand7]

Hi,

Activemq (5.3.1-01-00) can`t start after out of free disk space crash.

The following exception throws:

====

2010-07-07 18:19:49,941 | ERROR | Failed to start ActiveMQ JMS Message Broker. Reason: java.io.EOFException | org.apache.activemq.broker.BrokerService | main

java.io.EOFException

        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)

        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)

        at org.apache.kahadb.page.PageFile.readPage(PageFile.java:792)

        at org.apache.kahadb.page.Transaction$2.readPage(Transaction.java:440)

        at org.apache.kahadb.page.Transaction$2.(Transaction.java:431)

        at org.apache.kahadb.page.Transaction.openInputStream(Transaction.java:428)

        at org.apache.kahadb.page.Transaction.load(Transaction.java:404)

        at org.apache.kahadb.page.Transaction.load(Transaction.java:361)

        at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)

        at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:224)

        at org.apache.kahadb.index.BTreeNode.getLast(BTreeNode.java:553)

        at org.apache.kahadb.index.BTreeIndex.getLast(BTreeIndex.java:244)

        at org.apache.activemq.store.kahadb.MessageDatabase.loadStoredDestination(MessageDatabase.java:1224)

        at org.apache.activemq.store.kahadb.MessageDatabase.access$000(MessageDatabase.java:82)

        at org.apache.activemq.store.kahadb.MessageDatabase$2.execute(MessageDatabase.java:227)

        at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)

        at org.apache.activemq.store.kahadb.MessageDatabase.loadPageFile(MessageDatabase.java:223)

        at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:277)

        at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:321)

        at org.apache.activemq.store.kahadb.MessageDatabase.start(MessageDatabase.java:182)

        at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.start(KahaDBPersistenceAdapter.java:183)

        at org.apache.activemq.broker.BrokerService.start(BrokerService.java:469)

        at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:85)

====

The same results with ActiveMQ 5.4-SNAPSOT.

Is there way to restore storage?

Thanks.

[garytully]

Can you try the options: checkForCorruptJournalFiles and ingoreMissingJournalFiles on the KahaDBPersistenceAdapter. Enable both of them, which will sync up the index with any missing journal entries.

[ciand7]

It was starting about 1 minutes with lot of disk activity and then crashed with same exception:

======

2010-07-13 16:45:27,516 | INFO  | Using Persistence Adapter: org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter@4d98defc | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain

2010-07-13 16:46:23,435 | ERROR | Failed to start ActiveMQ JMS Message Broker. Reason: java.io.EOFException | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain

java.io.EOFException

        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)

        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)

        at org.apache.kahadb.page.PageFile.readPage(PageFile.java:792)

        at org.apache.kahadb.page.Transaction$2.readPage(Transaction.java:440)

        at org.apache.kahadb.page.Transaction$2.(Transaction.java:431)

        at org.apache.kahadb.page.Transaction.openInputStream(Transaction.java:428)

        at org.apache.kahadb.page.Transaction.load(Transaction.java:404)

        at org.apache.kahadb.page.Transaction.load(Transaction.java:361)

        at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)

        at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:224)

        at org.apache.kahadb.index.BTreeNode.getLast(BTreeNode.java:553)

        at org.apache.kahadb.index.BTreeIndex.getLast(BTreeIndex.java:244)

        at org.apache.activemq.store.kahadb.MessageDatabase.loadStoredDestination(MessageDatabase.java:1224)

        at org.apache.activemq.store.kahadb.MessageDatabase.access$000(MessageDatabase.java:82)

        at org.apache.activemq.store.kahadb.MessageDatabase$2.execute(MessageDatabase.java:227)

        at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)

        at org.apache.activemq.store.kahadb.MessageDatabase.loadPageFile(MessageDatabase.java:223)

        at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:277)

        at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:321)

        at org.apache.activemq.store.kahadb.MessageDatabase.start(MessageDatabase.java:182)

        at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.start(KahaDBPersistenceAdapter.java:183)

        at org.apache.activemq.broker.BrokerService.start(BrokerService.java:469)

        at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:85)

======

[houndz17]

Does it work?

[ciand7]

No, it does not.

Broker crashed on startup with exception java.io.EOFException. Suggested parameters checkForCorruptJournalFiles and ingoreMissingJournalFiles has no effect.

[garytully]

in this case, it looks like it is necessary to rebuild the index by replaying the journal. Take a copy of the data directory as a backup, then remove the index file, db.data and restart the broker.

The missing index will cause the journal to be replayed but any missing journal files will be ignored.

[albertwilson]

I really like this blog, It's always nice when you can not only be informed, but also get entertain, from these type of blog, that is such a nice award show,  really enjoying to watch this, nice entry. Thanks

Term Papers